Unable to start CRS due to Segfault with processes

Solution Verified - Updated -

Issue

  • Processes are segfaulting:

    Oct 26 22:04:28 LRTP597 kernel: [56258.950355] wc[6493]: segfault at 10358ed269b5 ip 000010358ed269b5 sp 00007ffc01f9aca0 error 14
    Oct 26 22:04:28 LRTP597 kernel: [56258.957757] wc[6505]: segfault at 10358e6220b0 ip 000000358e416729 sp 00007fff7416dff0 error 4 in ld-   2.12.so[358e400000+20000]
    Oct 26 22:04:28 LRTP597 kernel: [56258.971488] wc[6529]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007ffc21d3aa50 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:05:28 LRTP597 kernel: [56319.004396] wc[6818]: segfault at 10358e401130 ip 000010358e401130 sp 00007ffcc0de6ec8 error 14
    Oct 26 22:05:28 LRTP597 kernel: [56319.006922] wc[6822]: segfault at 10358e621cd8 ip 000000358e401467 sp 00007fff0340a550 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:05:28 LRTP597 kernel: [56319.011561] wc[6830]: segfault at 10358e61fdd0 ip 000000358e415d0c sp 00007ffc8769c1a0 error 6 in ld-   2.12.so[358e400000+20000]
    Oct 26 22:05:28 LRTP597 kernel: [56319.016166] wc[6838]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007ffcf63a21f0 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:05:28 LRTP597 kernel: [56319.018566] wc[6842]: segfault at 10358e61fdd0 ip 000000358e415d0c sp 00007ffd1b8f3180 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:07:28 LRTP597 kernel: [56439.098280] wc[7294]: segfault at 10358ec2f09c ip 000010358ec2f09c sp 00007fff97912cf0 error 14 in locale-archive[7fa506a5f000+5e93000]
    Oct 26 22:10:28 LRTP597 kernel: [56619.245747] wc[8042]: segfault at 10358e61fdd0 ip 000000358e415d0c sp 00007ffdb01bf000 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:10:28 LRTP597 kernel: [56619.250331] wc[8050]: segfault at 10358e621f50 ip 000000358e402551 sp 00007ffcd6923550 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:10:39 LRTP597 kernel: [56630.331839] grep[8158]: segfault at 10358ef8de90 ip 000000358ed276a4 sp 00007fff99eee3d0 error 4 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:11:01 LRTP597 kernel: [56652.057770] uname[8313]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007ffe5d397540 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:11:01 LRTP597 kernel: [56652.099986] hostname[8360]: segfault at 358fe1f000 ip 000000358fc130b0 sp 00007fffefd2d7b0 error 4 in libselinux.so.1[358fc00000+1d000]
    Oct 26 22:11:01 LRTP597 kernel: [56652.128895] netstat[8383]: segfault at 10358e41aa40 ip 000000358e406623 sp 00007ffd5dec4520 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:11:28 LRTP597 kernel: [56679.275286] wc[8642]: segfault at 10358e61fba8 ip 000000358e41601f sp 00007ffc753a1b00 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:11:28 LRTP597 kernel: [56679.287316] wc[8662]: segfault at 10358ed512a8 ip 000000358ec7f931 sp 00007ffe3c4dfe58 error 4 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:12:28 LRTP597 kernel: [56739.337880] wc[8970]: segfault at 10358ef93284 ip 000000358ec1f0cb sp 00007fff44ee6ed8 error 6 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:13:28 LRTP597 kernel: [56799.367419] wc[9177]: segfault at 10358e622144 ip 000000358e405447 sp 00007ffc710aa590 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:13:28 LRTP597 kernel: [56799.374760] wc[9189]: segfault at 10358ef8e9a0 ip 000000358ec79903 sp 00007ffd594e69d0 error 6 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:14:28 LRTP597 kernel: [56859.418155] wc[9415]: segfault at 10358e621000 ip 000000358e408260 sp 00007fff69028b20 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:15:28 LRTP597 kernel: [56919.473466] wc[9855]: segfault at 10358ef8fae4 ip 000000358ec2e25d sp 00007fff059e89d0 error 6 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:15:28 LRTP597 kernel: [56919.480201] wc[9867]: segfault at 10358e621998 ip 000000358e40116e sp 00007fffacecb690 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:16:29 LRTP597 kernel: [56979.930429] vmsub.sh[10109]: segfault at 1000006dfb48 ip 000000000046668e sp 00007ffe90ff6750 error 4 in bash[400000+d5000]
    Oct 26 22:19:28 LRTP597 kernel: [57159.651990] expr[10849]: segfault at 10358ecf8346 ip 000010358ecf8346 sp 00007ffc14d79cf8 error 14
    Oct 26 22:20:01 LRTP597 kernel: [57192.686580] awk[11364]: segfault at 10358e417df5 ip 000010358e417df5 sp 00007ffd6dcb2b78 error 14
    Oct 26 22:28:29 LRTP597 kernel: [57700.590183] date[14619]: segfault at 10000060c12c ip 0000000000408f89 sp 00007ffee650ec40 error 4 in date[400000+d000]
    Oct 26 22:30:01 LRTP597 kernel: [57792.516312] date[14951]: segfault at 10358e61fdd0 ip 000000358e415d0c sp 00007ffe5c469800 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:33:04 LRTP597 kernel: [57975.440713] sed[15795]: segfault at 10358e61fdd0 ip 000000358e415d0c sp 00007ffe036eda60 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:36:29 LRTP597 kernel: [58180.443836] wc[16635]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007ffd32c650e0 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:37:09 LRTP597 kernel: [58221.067920] true[16802]: segfault at 10358e621998 ip 000000358e40116e sp 00007ffc104d3860 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:37:49 LRTP597 kernel: [58261.103551] date[16966]: segfault at 10358e400b50 ip 000010358e400b50 sp 00007ffcd0adcfd8 error 14
    Oct 26 22:46:10 LRTP597 kernel: [58761.563534] oswsub.sh[19136]: segfault at 10358e622144 ip 000000358e405447 sp 00007ffe62bf6420 error 6 in ld-2.12.so[358e400000+20000]
    Oct 26 22:47:29 LRTP597 kernel: [58840.962348] wc[19456]: segfault at 10358ec2f09c ip 000010358ec2f09c sp 00007ffeb0572f20 error 14 in locale-archive[7f8a79024000+5e93000]
    Oct 26 22:48:29 LRTP597 kernel: [58900.991972] wc[19670]: segfault at 10358ef93290 ip 000000358ec1f122 sp 00007ffe2032f6c8 error 6 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:48:29 LRTP597 kernel: [58901.001242] wc[19686]: segfault at 10358ef8de90 ip 000000358ed276a4 sp 00007fff08655b00 error 4 in libc-2.12.so[358ec00000+18a000]
    Oct 26 22:49:30 LRTP597 kernel: [58961.748431] vmstat[19950]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007fff0d4124b0 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:50:50 LRTP597 kernel: [59041.821050] date[20261]: segfault at 10358f606fc0 ip 000000358f4021a4 sp 00007ffc0fadb9b0 error 4 in librt-2.12.so[358f400000+7000]
    Oct 26 22:51:30 LRTP597 kernel: [59081.859724] true[20498]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007ffccf3facc0 error 4 in ld-2.12.so[358e400000+20000]
    Oct 26 22:51:50 LRTP597 kernel: [59101.877221] vmstat[20555]: segfault at 10358e6220b0 ip 000000358e41679e sp 00007ffe8b520a20 error 4 in ld-2.12.so[358e400000+20000]
    
    • In my Oracle RAC cluster only one node is having this issue\
  • Many RHEL/third-party applications are segfaulting within libc and many other libraries (~47 unique libs/bin; 167 unique applications)

  • Segfaults happen during early dracut stages when using modprobe commands, well before chroot occurs

Environment

  • Red Hat Enterprise Linux 6.8
  • UAT RAC environment

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content