Hadoop S3A plugin and Ceph RGW - Files bigger than 5G causing issues during upload and upload is failing.

Solution In Progress - Updated -

Issue

  • Hadoop S3A plugin and Ceph RGW - Files bigger than 5G causing issues during upload and upload is failing.

  • We have following errors in the logs in Hadoop distcp command :

Unable to copy part: Status Code: 404, AWS Service: Amazon S3, AWS Request ID: tx000000000000000000106-0059676cf1-6dbb4-default, AWS Error Code: NoSuchKey, AWS Error Message: null
  • and in the RGW logs 404 and with this patch applied we have following error in RGW logs:
ERROR: client_io->complete_request() returned -5

Environment

  • Hadoop S3A plugin version:
# hadoop version
Hadoop 2.7.3
  • Red Hat Ceph Storage 2.3

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content