Ceph - After upgrade to RHCS 3.3z1 RGW daemon experiences segmentation fault

Solution Verified - Updated -

Issue

  • After upgrading the Red Hat Ceph Storage Cluster to 3.3z1 (12.2.12-74.el7cp or 12.2.12-67redhat1xenial) the RGW daemons experiences segmentation faults.
  • Errata release with issue is RHBA-2019:3173
  • Segmentation fault will appear similar to this output:
2019-10-24 10:35:56.813839 7f9a20fd3700  1 civetweb: 0x55a720ef2000: 10.134.32.28 - - [24/Oct/2019:10:35:56 +0200] "GET /cc-droplets/?prefix=buildpack_cache%2Fb7%2F09%2Fb70908c0-5e3e-4fd4-8489-69c3407b996a HTTP/1.1" 200 849 - fog-core/1.43.0
2019-10-24 10:35:56.816960 7f9a207d2700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f9a207d2700 thread_name:civetweb-worker

 ceph version 12.2.12-74.el7cp (6c4a9c2235eb0c7e3d61719cdc1d6b7b2dcbdea9) luminous (stable)
 1: (()+0x2d82a1) [0x55a71e13f2a1]
 2: (()+0xf630) [0x7f9a5719e630]
 3: (ceph_str_hash_rjenkins(char const*, unsigned int)+0x20) [0x7f9a4e9c4500]
 4: (pg_pool_t::hash_key(std::string const&, std::string const&) const+0xc0) [0x7f9a4e95e980]
 5: (OSDMap::map_to_pg(long, std::string const&, std::string const&, std::string const&, pg_t*) const+0x75) [0x7f9a4e917975]
 6: (OSDMap::object_locator_to_pg(object_t const&, object_locator_t const&, pg_t&) const+0xa6) [0x7f9a4e917a66]
 7: (()+0xcc447) [0x7f9a58159447]
 8: (()+0xd9c59) [0x7f9a58166c59]
 9: (()+0xe6df8) [0x7f9a58173df8]
 10: (()+0xe707a) [0x7f9a5817407a]
 11: (librados::IoCtxImpl::aio_operate(object_t const&, ObjectOperation*, librados::AioCompletionImpl*, SnapContext const&, int, blkin_trace_info const*)+0x1a1) [0x7f9a58124191]
 12: (librados::IoCtx::aio_operate(std::string const&, librados::AioCompletion*, librados::ObjectWriteOperation*)+0x53) [0x7f9a580e6d33]
 13: (RGWRados::cls_bucket_list_ordered(RGWBucketInfo&, int, cls_rgw_obj_key const&, std::string const&, unsigned int, bool, std::map<std::string, rgw_bucket_dir_entry, std::less<std::string>, std::allocator<std::pair<std::string const, rgw_bucket_dir_entry> > >&, bool*, cls_rgw_obj_key*, bool (*)(std::string const&))+0x13f1) [0x55a71e2cb501]
 14: (RGWRados::Bucket::List::list_objects_ordered(long, std::vector<rgw_bucket_dir_entry, std::allocator<rgw_bucket_dir_entry> >*, std::map<std::string, bool, std::less<std::string>, std::allocator<std::pair<std::string const, bool> > >*, bool*)+0x3fa) [0x55a71e2cbc4a]
 15: (RGWListBucket::execute()+0x25c) [0x55a71e22481c]
 16: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x188) [0x55a71e25a8b8]
 17: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::string const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0xb88) [0x55a71e25b678]
 18: (RGWCivetWebFrontend::process(mg_connection*)+0x3a2) [0x55a71e0bdaa2]
 19: (()+0x2c8517) [0x55a71e12f517]
 20: (()+0x2c9dd2) [0x55a71e130dd2]
 21: (()+0x2ca5a8) [0x55a71e1315a8]
 22: (()+0x7ea5) [0x7f9a57196ea5]
 23: (clone()+0x6d) [0x7f9a4b6348cd]

Environment

  • Upgrade to Red Hat Ceph Storage 3.3z1
  • Red Hat Ceph Storage 3.3z1
  • Ceph 12.2.12-74.el7cp
  • Ceph 12.2.12-67redhat1xenial
  • Errata RHBA-2019:3173

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content