Unable to start CRS due to Segfault with processes

Solution Verified - Updated -

Issue

  • Processes are segfaulting:

    Oct 26 22:04:28 LRTP597 kernel: [56258.950355] wc[6493]: segfault at 10358ed269b5 ip 000010358ed269b5 sp 00007ffc01f9aca0 error 14
    Oct 26 22:04:28 LRTP597 kernel: [56258.957757] wc[6505]: segfault at 10358e6220b0 ip 000000358e416729 sp 00007fff7416dff0 error 4 in ld-   2.12.so[358e400000+20000]
    Oct 26 22:04:28 LRTP597 kernel: [56258.971488] wc[6529]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007ffc21d3aa50 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:05:28 LRTP597 kernel: [56319.004396] wc[6818]: segfault at 10358e401130 ip 000010358e401130 sp 00007ffcc0de6ec8 error 14
    Oct 26 22:05:28 LRTP597 kernel: [56319.006922] wc[6822]: segfault at 10358e621cd8 ip 000000358e401467 sp 00007fff0340a550 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:05:28 LRTP597 kernel: [56319.011561] wc[6830]: segfault at 10358e61fdd0 ip 000000358e415d0c sp 00007ffc8769c1a0 error 6 in ld-   2.12.so[358e400000+20000]
    Oct 26 22:05:28 LRTP597 kernel: [56319.016166] wc[6838]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007ffcf63a21f0 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:05:28 LRTP597 kernel: [56319.018566] wc[6842]: segfault at 10358e61fdd0 ip 000000358e415d0c sp 00007ffd1b8f3180 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:07:28 LRTP597 kernel: [56439.098280] wc[7294]: segfault at 10358ec2f09c ip 000010358ec2f09c sp 00007fff97912cf0 error 14 in locale-archive[7fa506a5f000+5e93000]
    Oct 26 22:10:28 LRTP597 kernel: [56619.245747] wc[8042]: segfault at 10358e61fdd0 ip 000000358e415d0c sp 00007ffdb01bf000 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:10:28 LRTP597 kernel: [56619.250331] wc[8050]: segfault at 10358e621f50 ip 000000358e402551 sp 00007ffcd6923550 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:10:39 LRTP597 kernel: [56630.331839] grep[8158]: segfault at 10358ef8de90 ip 000000358ed276a4 sp 00007fff99eee3d0 error 4 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:11:01 LRTP597 kernel: [56652.057770] uname[8313]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007ffe5d397540 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:11:01 LRTP597 kernel: [56652.099986] hostname[8360]: segfault at 358fe1f000 ip 000000358fc130b0 sp 00007fffefd2d7b0 error 4 in libselinux.so.1[358fc00000+1d000]
    Oct 26 22:11:01 LRTP597 kernel: [56652.128895] netstat[8383]: segfault at 10358e41aa40 ip 000000358e406623 sp 00007ffd5dec4520 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:11:28 LRTP597 kernel: [56679.275286] wc[8642]: segfault at 10358e61fba8 ip 000000358e41601f sp 00007ffc753a1b00 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:11:28 LRTP597 kernel: [56679.287316] wc[8662]: segfault at 10358ed512a8 ip 000000358ec7f931 sp 00007ffe3c4dfe58 error 4 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:12:28 LRTP597 kernel: [56739.337880] wc[8970]: segfault at 10358ef93284 ip 000000358ec1f0cb sp 00007fff44ee6ed8 error 6 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:13:28 LRTP597 kernel: [56799.367419] wc[9177]: segfault at 10358e622144 ip 000000358e405447 sp 00007ffc710aa590 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:13:28 LRTP597 kernel: [56799.374760] wc[9189]: segfault at 10358ef8e9a0 ip 000000358ec79903 sp 00007ffd594e69d0 error 6 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:14:28 LRTP597 kernel: [56859.418155] wc[9415]: segfault at 10358e621000 ip 000000358e408260 sp 00007fff69028b20 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:15:28 LRTP597 kernel: [56919.473466] wc[9855]: segfault at 10358ef8fae4 ip 000000358ec2e25d sp 00007fff059e89d0 error 6 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:15:28 LRTP597 kernel: [56919.480201] wc[9867]: segfault at 10358e621998 ip 000000358e40116e sp 00007fffacecb690 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:16:29 LRTP597 kernel: [56979.930429] vmsub.sh[10109]: segfault at 1000006dfb48 ip 000000000046668e sp 00007ffe90ff6750 error 4 in bash[400000+d5000]
    Oct 26 22:19:28 LRTP597 kernel: [57159.651990] expr[10849]: segfault at 10358ecf8346 ip 000010358ecf8346 sp 00007ffc14d79cf8 error 14
    Oct 26 22:20:01 LRTP597 kernel: [57192.686580] awk[11364]: segfault at 10358e417df5 ip 000010358e417df5 sp 00007ffd6dcb2b78 error 14
    Oct 26 22:28:29 LRTP597 kernel: [57700.590183] date[14619]: segfault at 10000060c12c ip 0000000000408f89 sp 00007ffee650ec40 error 4 in date[400000+d000]
    Oct 26 22:30:01 LRTP597 kernel: [57792.516312] date[14951]: segfault at 10358e61fdd0 ip 000000358e415d0c sp 00007ffe5c469800 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:33:04 LRTP597 kernel: [57975.440713] sed[15795]: segfault at 10358e61fdd0 ip 000000358e415d0c sp 00007ffe036eda60 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:36:29 LRTP597 kernel: [58180.443836] wc[16635]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007ffd32c650e0 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:37:09 LRTP597 kernel: [58221.067920] true[16802]: segfault at 10358e621998 ip 000000358e40116e sp 00007ffc104d3860 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:37:49 LRTP597 kernel: [58261.103551] date[16966]: segfault at 10358e400b50 ip 000010358e400b50 sp 00007ffcd0adcfd8 error 14
    Oct 26 22:46:10 LRTP597 kernel: [58761.563534] oswsub.sh[19136]: segfault at 10358e622144 ip 000000358e405447 sp 00007ffe62bf6420 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:47:29 LRTP597 kernel: [58840.962348] wc[19456]: segfault at 10358ec2f09c ip 000010358ec2f09c sp 00007ffeb0572f20 error 14 in locale-archive[7f8a79024000+5e93000]
    Oct 26 22:48:29 LRTP597 kernel: [58900.991972] wc[19670]: segfault at 10358ef93290 ip 000000358ec1f122 sp 00007ffe2032f6c8 error 6 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:48:29 LRTP597 kernel: [58901.001242] wc[19686]: segfault at 10358ef8de90 ip 000000358ed276a4 sp 00007fff08655b00 error 4 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:49:30 LRTP597 kernel: [58961.748431] vmstat[19950]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007fff0d4124b0 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:50:50 LRTP597 kernel: [59041.821050] date[20261]: segfault at 10358f606fc0 ip 000000358f4021a4 sp 00007ffc0fadb9b0 error 4 in librt-2.12.so[358f400000+7000]
    Oct 26 22:51:30 LRTP597 kernel: [59081.859724] true[20498]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007ffccf3facc0 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:51:50 LRTP597 kernel: [59101.877221] vmstat[20555]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007ffe8b520a20 error 4 in ld-2.12.so[358e400000+20000]
    
    • In my Oracle RAC cluster only one node is having this issue\
  • Many RHEL/third-party applications are segfaulting within libc and many other libraries (~47 unique libs/bin; 167 unique applications)

  • Segfaults happen during early dracut stages when using modprobe commands, well before chroot occurs

Environment

  • Red Hat Enterprise Linux 6.8
  • UAT RAC environment

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In