Hadoop S3A plugin and Ceph RGW - Files bigger than 5G causing issues during upload and upload is failing.

Solution In Progress - Updated -

Issue

  • Hadoop S3A plugin and Ceph RGW - Files bigger than 5G causing issues during upload and upload is failing.

  • We have following errors in the logs in Hadoop distcp command :

Unable to copy part: Status Code: 404, AWS Service: Amazon S3, AWS Request ID: tx000000000000000000106-0059676cf1-6dbb4-default, AWS Error Code: NoSuchKey, AWS Error Message: null
  • and in the RGW logs 404 and with this patch applied we have following error in RGW logs:
ERROR: client_io->complete_request() returned -5

Environment

  • Hadoop S3A plugin version:
# hadoop version
Hadoop 2.7.3
  • Red Hat Ceph Storage 2.3

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.