Ceph - RGW reports ERROR: STREAM_IO
Environment
- Red Hat Ceph Storage 2.x
- Red Hat Ceph Storage 3.x
Issue
- Rados Gateway (RGW) log contains the following errors:
2016-09-06 09:52:07.036430 7f6cf0088700 0 ERROR: flush_read_list(): d->client_c->handle_data() returned -5
2016-09-06 09:52:07.039092 7f6cf0088700 0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-09-06 09:52:07.039130 7f6cf0088700 0 ERROR: s->cio->send_content_length() returned err=-5
2016-09-06 09:52:07.039133 7f6cf0088700 0 ERROR: s->cio->print() returned err=-5
2016-09-06 09:52:07.039134 7f6cf0088700 0 ERROR: STREAM_IO(s)->print() returned err=-5
2016-09-06 09:52:07.039143 7f6cf0088700 0 ERROR: STREAM_IO(s)->complete_header() returned err=-5
2016-09-06 09:52:07.039156 7f6cf0088700 2 req 386004:85.277926:s3:GET /mycontainers66/myobjects467:get_obj:completing
2016-09-06 09:52:07.039162 7f6cf0088700 2 req 386004:85.277933:s3:GET /mycontainers66/myobjects467:get_obj:op status=-5
2016-09-06 09:52:07.039163 7f6cf0088700 2 req 386004:85.277934:s3:GET /mycontainers66/myobjects467:get_obj:http status=500
2016-09-06 09:52:07.039170 7f6cf0088700 1 ====== req done req=0x7f6cf0082710 op status=-5 http_status=500 ======
Root Cause
- This error is usually observed when the Rados Gateway detects that the client has closed the connection before the requests have finished.
- This issue could be due to a timeout in the application that talks to the RGW node, or an HAproxy issue.
- Please check with your network team as this is typically an indication of client disconnects.
- In addition, investigate the application servers and logs of the nodes that are communicating to the RGW nodes.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
