Why does "yum update" throws error "*** glibc detected *** /usr/bin/python: malloc(): smallbin double linked list corrupted: 0x0000000xxxxxxxx ***" ?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 6.

Issue

  • what does error "*** glibc detected *** /usr/bin/python: malloc(): smallbin double linked list corrupted: 0x000000000da47120 ***" mean?
# yum update

*** glibc detected *** /usr/bin/python: malloc(): smallbin double linked list corrupted: 0x000000000da47120 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3c23c75dee]
/lib64/libc.so.6[0x3c23c7a4b8]
/lib64/libc.so.6[0x3c23c7bb6a]
/lib64/libc.so.6(realloc+0x158)[0x3c23c7bf08]
/usr/lib64/librpmio.so.1(rrealloc+0x18)[0x3c2e4184c8]
/usr/lib64/librpmio.so.1(rpmlog+0x25c)[0x3c2e41638c]
/usr/lib64/librpm.so.1[0x3c2e0353f3]
/usr/lib64/librpm.so.1[0x3c2e035c9a]
/usr/lib64/librpm.so.1[0x3c2e0368a5]
/usr/lib64/librpm.so.1[0x3c2e0364eb]
/usr/lib64/librpm.so.1(rpmtsRun+0x8c8)[0x3c2e04fbc8]
/usr/lib64/python2.6/site-packages/rpm/_rpmmodule.so(+0xe65c)[0x7f3ee2bc465c]
/usr/lib64/libpython2.6.so.1.0(PyObject_Call+0x53)[0x3c25443c63]
/usr/lib64/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x3c254cfc83]
/usr/lib64/libpython2.6.so.1.0[0x3c2545bb5c]
/usr/lib64/libpython2.6.so.1.0(PyObject_Call+0x53)[0x3c25443c63]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x47d4)[0x3c254d4f64]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x63ef)[0x3c254d6b7f]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x927)[0x3c254d7647]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5304)[0x3c254d5a94]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x63ef)[0x3c254d6b7f]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x927)[0x3c254d7647]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5304)[0x3c254d5a94]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x927)[0x3c254d7647]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5304)[0x3c254d5a94]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x927)[0x3c254d7647]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalCode+0x32)[0x3c254d7722]
/usr/lib64/libpython2.6.so.1.0[0x3c254f1b9c]
/usr/lib64/libpython2.6.so.1.0(PyRun_FileExFlags+0x90)[0x3c254f1c70]
/usr/lib64/libpython2.6.so.1.0(PyRun_SimpleFileExFlags+0xdc)[0x3c254f315c]
/usr/lib64/libpython2.6.so.1.0(Py_Main+0xb62)[0x3c254ff892]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3c23c1ed1d]
/usr/bin/python[0x400649]
.....
======= Memory map: ========
00400000-00401000 r-xp 00000000 fd:00 138212                             /usr/bin/python
00600000-00602000 rw-p 00000000 fd:00 138212                             /usr/bin/python
0150f000-0f855000 rw-p 00000000 00:00 0                                  [heap]
3c23800000-3c23820000 r-xp 00000000 fd:00 394208                         /lib64/ld-2.12.so
3c23a20000-3c23a21000 r--p 00020000 fd:00 394208                         /lib64/ld-2.12.so
3c23a21000-3c23a22000 rw-p 00021000 fd:00 394208                         /lib64/ld-2.12.so
3c23a22000-3c23a23000 rw-p 00000000 00:00 0 
3c23c00000-3c23d8a000 r-xp 00000000 fd:00 394209                         /lib64/libc-2.12.so
3c23d8a000-3c23f8a000 ---p 0018a000 fd:00 394209                         /lib64/libc-2.12.so
3c23f8a000-3c23f8e000 r--p 0018a000 fd:00 394209                         /lib64/libc-2.12.so
3c23f8e000-3c23f90000 rw-p 0018e000 fd:00 394209                         /lib64/libc-2.12.so
3c23f90000-3c23f94000 rw-p 00000000 00:00 0 
3c24000000-3c24017000 r-xp 00000000 fd:00 394210                         /lib64/libpthread-2.12.so
3c24017000-3c24217000 ---p 00017000 fd:00 394210                         /lib64/libpthread-2.12.so
3c24217000-3c24218000 r--p 00017000 fd:00 394210                         /lib64/libpthread-2.12.so
3c24218000-3c24219000 rw-p 00018000 fd:00 394210                         /lib64/libpthread-2.12.so
3c24219000-3c2421d000 rw-p 00000000 00:00 0 
3c24400000-3c24402000 r-xp 00000000 fd:00 394218                         /lib64/libdl-2.12.so
3c24402000-3c24602000 ---p 00002000 fd:00 394218                         /lib64/libdl-2.12.so
3c24602000-3c24603000 r--p 00002000 fd:00 394218                         /lib64/libdl-2.12.so
3c24603000-3c24604000 rw-p 00003000 fd:00 394218                         /lib64/libdl-2.12.so
3c24800000-3c24807000 r-xp 00000000 fd:00 394211                         /lib64/librt-2.12.so
3c24807000-3c24a06000 ---p 00007000 fd:00 394211                         /lib64/librt-2.12.so
3c24a06000-3c24a07000 r--p 00006000 fd:00 394211                         /lib64/librt-2.12.so
3c24a07000-3c24a08000 rw-p 00007000 fd:00 394211                         /lib64/librt-2.12.so
3c24c00000-3c24c15000 r-xp 00000000 fd:00 394227                         /lib64/libz.so.1.2.3
3c24c15000-3c24e14000 ---p 00015000 fd:00 394227                         /lib64/libz.so.1.2.3
3c24e14000-3c24e15000 r--p 00014000 fd:00 394227                         /lib64/libz.so.1.2.3
3c24e15000-3c24e16000 rw-p 00015000 fd:00 394227                         /lib64/libz.so.1.2.3
3c25000000-3c25083000 r-xp 00000000 fd:00 394235                         /lib64/libm-2.12.so
3c25083000-3c25282000 ---p 00083000 fd:00 394235                         /lib64/libm-2.12.so
3c25282000-3c25283000 r--p 00082000 fd:00 394235                         /lib64/libm-2.12.so
3c25283000-3c25284000 rw-p 00083000 fd:00 394235                         /lib64/libm-2.12.so
3c25400000-3c2555d000 r-xp 00000000 fd:00 134393                         /usr/lib64/libpython2.6.so.1.0
3c2555d000-3c2575c000 ---p 0015d000 fd:00 134393                         /usr/lib64/libpython2.6.so.1.0
3c2575c000-3c25798000 rw-p 0015c000 fd:00 134393                         /usr/lib64/libpython2.6.so.1.0
3c25798000-3c257a6000 rw-p 00000000 00:00 0 
3c25800000-3c2581c000 r-xp 00000000 fd:00 137585                         /usr/lib64/libmagic.so.1.0.0
3c2581c000-3c25a1c000 ---p 0001c000 fd:00 137585                         /usr/lib64/libmagic.so.1.0.0
3c25a1c000-3c25a1d000 rw-p 0001c000 fd:00 137585                         /usr/lib64/libmagic.so.1.0.0
3c25a1d000-3c25a1f000 rw-p 00000000 00:00 0 
3c25c00000-3c25c24000 r-xp 00000000 fd:00 137603                         /usr/lib64/librpmbuild.so.1.0.0
3c25c24000-3c25e23000 ---p 00024000 fd:00 137603                         /usr/lib64/librpmbuild.so.1.0.0
3c25e23000-3c25e26000 rw-p 00023000 fd:00 137603                         /usr/lib64/librpmbuild.so.1.0.0
3c25e26000-3c25e2c000 rw-p 00000000 00:00 0 
3c26000000-3c26004000 r-xp 00000000 fd:00 394243                         /lib64/libattr.so.1.1.0
3c26004000-3c26203000 ---p 00004000 fd:00 394243                         /lib64/libattr.so.1.1.0
3c26203000-3c26204000 r--p 00003000 fd:00 394243                         /lib64/libattr.so.1.1.0
3c26204000-3c26205000 rw-p 00004000 fd:00 394243                         /lib64/libattr.so.1.1.0
3c26400000-3c26404000 r-xp 00000000 fd:00 394244                         /lib64/libcap.so.2.16
3c26404000-3c26603000 ---p 00004000 fd:00 394244                         /lib64/libcap.so.2.16
3c26603000-3c26604000 rw-p 00003000 fd:00 394244                         /lib64/libcap.so.2.16
3c26800000-3c26832000 r-xp 00000000 fd:00 394195                         /lib64/libidn.so.11.6.1
3c26832000-3c26a31000 ---p 00032000 fd:00 394195                         /lib64/libidn.so.11.6.1
3c26a31000-3c26a32000 rw-p 00031000 fd:00 394195                         /lib64/libidn.so.11.6.1
3c26c00000-3c26c16000 r-xp 00000000 fd:00 394236                         /lib64/libgcc_s-4.4.7-20120601.so.1
3c26c16000-3c26e15000 ---p 00016000 fd:00 394236                         /lib64/libgcc_s-4.4.7-20120601.so.1
3c26e15000-3c26e16000 rw-p 00015000 fd:00 394236                         /lib64/libgcc_s-4.4.7-20120601.so.1
3c27000000-3c27002000 r-xp 00000000 fd:00 394267                         /lib64/libutil-2.12.so
3c27002000-3c27201000 ---p 00002000 fd:00 394267                         /lib64/libutil-2.12.so
3c27201000-3c27202000 r--p 00001000 fd:00 394267                         /lib64/libutil-2.12.so
3c27202000-3c27203000 rw-p 00002000 fd:00 394267                         /lib64/libutil-2.12.so
3c27400000-3c27415000 r-xp 00000000 fd:00 137484                         /usr/lib64/libelf-0.164.so
3c27415000-3c27614000 ---p 00015000 fd:00 137484                         /usr/lib64/libelf-0.164.so
3c27614000-3c27615000 r--p 00014000 fd:00 137484                         /usr/lib64/libelf-0.164.so
3c27615000-3c27616000 rw-p 00015000 fd:00 137484                         /usr/lib64/libelf-0.164.so
3c27800000-3c27803000 r-xp 00000000 fd:00 394224                         /lib64/libcom_err.so.2.1
3c27803000-3c27a02000 ---p 00003000 fd:00 394224                         /lib64/libcom_err.so.2.1Aborted (co

Resolution

  • Backup the rpmdatabse.
# rm -f /var/lib/rpm/__db*
tar czvf $HOME/rpmdatabase.tar.gz /var/lib/rpm
  • complete the yum transaction.
# yum-complete-transaction
  • Now, clean up only transaction journal files and exit.
# yum-complete-transaction --cleanup-only
# yum update

Note:- Go ahead with the desired yum transaction.

Root Cause

  • The issue occurs due to multiple yum transaction prematurely aborted which were waiting for completion. Now, on completion of yum transaction using yum-complete-transaction the latest yum transaction is completed and the other transaction was unable to be complete and yum fails with dependency issues, leaving missing and bunch of duplicate packages in the system.
# yum-complete-transaction 

Loaded plugins: product-id, rhnplugin, subscription-manager\
This system is receiving updates from RHN Classic or RHN Satellite.\
There are 4 outstanding transactions to complete. Finishing the most recent one\

Diagnostic Steps

  • Check the number of transaction pending for completion, here we have 4 transactions to be completed.
# yum-complete-transaction \
Loaded plugins: product-id, rhnplugin, subscription-manager\
This system is receiving updates from RHN Classic or RHN Satellite.\
There are 4 outstanding transactions to complete. Finishing the most recent one\

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments