Ceph - radosgw crashes upon boot on Red Hat Ceph Storage 2.5.z4 version 10.2.10-49.el7cp..x86_64(RHEL)/10.2.10-43redhat1xenial(Ubuntu), Why?

Issue

Ceph - radosgw crashes upon boot on Red Hat Ceph Storage 2.5.z4 version 10.2.10-49.el7cp..x86_64(RHEL)/10.2.10-43redhat1xenial(Ubuntu), Why?

• From radosgw logs:

   2019-05-16 19:11:55.240349 7f3bf02f3700 -1 *** Caught signal (Aborted) **
 in thread 7f3bf02f3700 thread_name:ms_pipe_read

 ceph version 10.2.10-43redhat1xenial (b060638dd829a84de4a4f13a0fa6b374389c52fd)
 1: (()+0x6130be) [0x7f41add6c0be]
 2: (()+0x11390) [0x7f41ad13e390]
 3: (gsignal()+0x38) [0x7f41ac800428]
 4: (abort()+0x16a) [0x7f41ac80202a]
 5: (()+0x777ea) [0x7f41ac8427ea]
 6: (()+0x7e913) [0x7f41ac849913]
 7: (()+0x81cde) [0x7f41ac84ccde]
 8: (__libc_malloc()+0x54) [0x7f41ac84f184]
 9: (operator new(unsigned long)+0x18) [0x7f41ace38e78]
 10: (()+0x850ff) [0x7f41a30de0ff]
 11: (()+0x30fad2) [0x7f41a3368ad2]
 12: (()+0x2d3f59) [0x7f41a332cf59]
 13: (()+0x32b9c9) [0x7f41a33849c9]
 14: (()+0x338790) [0x7f41a3391790]
 15: (()+0x3423dd) [0x7f41a339b3dd]
 16: (()+0x76ba) [0x7f41ad1346ba]
 17: (clone()+0x6d) [0x7f41ac8d241d]

2019-06-10 20:05:55.850004 7f17d880c700 -1 *** Caught signal (Aborted) **
 in thread 7f17d880c700 thread_name:ms_pipe_write

 ceph version 10.2.10-49.el7cp (b060638dd829a84de4a4f13a0fa6b374389c52fd)
 1: (()+0x5a676a) [0x7f18d2e0a76a]
 2: (()+0xf5d0) [0x7f18d244c5d0]
 3: (gsignal()+0x37) [0x7f18d1985207]
 4: (abort()+0x148) [0x7f18d19868f8]
 5: (()+0x78d27) [0x7f18d19c7d27]
 6: (()+0x7f5d4) [0x7f18d19ce5d4]
 7: (()+0x82d10) [0x7f18d19d1d10]
 8: (__libc_malloc()+0x4c) [0x7f18d19d41ac]
 9: (PORT_Alloc_Util()+0x20) [0x7f18c77ff050]
 10: (SECITEM_ArenaDupItem_Util()+0x82) [0x7f18c77fd912]
 11: (()+0x38038) [0x7f18c7a50038]
 12: (PK11_CreateContextBySymKey()+0x3f) [0x7f18c7a500df]
 13: (()+0x39b0cf) [0x7f18c85590cf]
 14: (()+0x39be4a) [0x7f18c8559e4a]
 15: (()+0x381012) [0x7f18c853f012]
 16: (()+0x384e00) [0x7f18c8542e00]
 17: (()+0x333b9a) [0x7f18c84f1b9a]
 18: (()+0x336551) [0x7f18c84f4551]
 19: (()+0x339bcd) [0x7f18c84f7bcd]
 20: (()+0x7dd5) [0x7f18d2444dd5]
 21: (clone()+0x6d) [0x7f18d1a4cead]

• From system logs:

Jun 10 20:05:54 ceph-rgw-01 radosgw: *** Error in `/usr/bin/radosgw': malloc(): smallbin double linked list corrupted: 0x00007ff07c056a10 ***
Jun 10 20:05:54 ceph-rgw-01 radosgw: ======= Backtrace: =========
Jun 10 20:05:54 ceph-rgw-01 radosgw: /lib64/libc.so.6(+0x7f5d4)[0x7ff0d4c885d4]
Jun 10 20:05:54 ceph-rgw-01 radosgw: /lib64/libc.so.6(+0x82d10)[0x7ff0d4c8bd10]
Jun 10 20:05:54 ceph-rgw-01 radosgw: /lib64/libc.so.6(__libc_malloc+0x4c)[0x7ff0d4c8e1ac]
Jun 10 20:05:54 ceph-rgw-01 radosgw: /lib64/libstdc++.so.6(_Znwm+0x1d)[0x7ff0d524aecd]
Jun 10 20:05:54 ceph-rgw-01 radosgw: /lib64/librgw.so.2(_ZN4ceph12XMLFormatter14escape_xml_strEPKc+0x3c)[0x7ff0d618c21c]
Jun 10 20:05:54 ceph-rgw-01 radosgw: /lib64/librgw.so.2(_ZN4ceph12XMLFormatter11dump_stringEPKcRKSs+0xd5)[0x7ff0d618cff5]
Jun 10 20:05:54 ceph-rgw-01 radosgw: /lib64/librgw.so.2(_ZN25RGWListBucket_ObjStore_S313send_responseEv+0x2d7)[0x7ff0d5ff62a7]
Jun 10 20:05:54 ceph-rgw-01 radosgw: /lib64/librgw.so.2(_Z15process_requestP8RGWRadosP7RGWRESTP10RGWRequestP11RGWStreamIOP12OpsLogSocket+0xdb7)[0x7ff0d5f23327]
Jun 10 20:05:54 ceph-rgw-01 radosgw: /usr/bin/radosgw(+0x1c950)[0x5621942f1950]
Jun 10 20:05:54 ceph-rgw-01 radosgw: /usr/bin/radosgw(+0x26a4f)[0x5621942fba4f]
Jun 10 20:05:54 ceph-rgw-01 radosgw: /usr/bin/radosgw(+0x28aa0)[0x5621942fdaa0]
Jun 10 20:05:54 ceph-rgw-01 radosgw: /lib64/libpthread.so.0(+0x7dd5)[0x7ff0d56fedd5]
Jun 10 20:05:54 ceph-rgw-01 radosgw: /lib64/libc.so.6(clone+0x6d)[0x7ff0d4d06ead]
Jun 10 20:05:54 ceph-rgw-01 radosgw: ======= Memory map: ========
Jun 10 20:05:54 ceph-rgw-01 radosgw: 5621942d5000-56219430d000 r-xp 00000000 fc:01 2142599                    /usr/bin/radosgw
Jun 10 20:05:54 ceph-rgw-01 radosgw: 56219450c000-56219450f000 r--p 00037000 fc:01 2142599                    /usr/bin/radosgw
Jun 10 20:05:54 ceph-rgw-01 radosgw: 56219450f000-562194510000 rw-p 0003a000 fc:01 2142599                    /usr/bin/radosgw