System crashes at 'd_path' which is called from '__stp_utrace_task_finder_target_quiesce'

Solution Verified - Updated -

Issue

  • System kept crashing with the below call trace in RHEL 6.
crash> bt
PID: 17377  TASK: ffff8801a6c4aab0  CPU: 0   COMMAND: "java"
 #0 [ffff88012a9a7990] machine_kexec at ffffffff8103d1fb
 #1 [ffff88012a9a79f0] crash_kexec at ffffffff810cc632
 #2 [ffff88012a9a7ac0] oops_end at ffffffff8153d9a0
 #3 [ffff88012a9a7af0] no_context at ffffffff8104e8cb
 #4 [ffff88012a9a7b40] __bad_area_nosemaphore at ffffffff8104eb55
 #5 [ffff88012a9a7b90] bad_area at ffffffff8104ec7e
 #6 [ffff88012a9a7bc0] __do_page_fault at ffffffff8104f483
 #7 [ffff88012a9a7ce0] do_page_fault at ffffffff8153f8ee
 #8 [ffff88012a9a7d10] page_fault at ffffffff8153cc35
    [exception RIP: d_path+0x20]
    RIP: ffffffff811acdc0  RSP: ffff88012a9a7dc8  RFLAGS: 00010282
    RAX: 0000000000000000  RBX: ffff8801a6c4aab0  RCX: 00000000000043e1
    RDX: 0000000000001000  RSI: ffff8803255a1000  RDI: 0000000000000000
    RBP: ffff88012a9a7e18   R8: 0000003099c21000   R9: 0000000000001000
    R10: 65737275636e6269  R11: 372e352e6f732e73  R12: ffff88012084aad0
    R13: ffffffffa096f740  R14: 000000000000014e  R15: 0000000000000026
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0000
 #9 [ffff88012a9a7e20] __stp_utrace_task_finder_target_quiesce at ffffffffa08071cb [stap_98526622d228930268ec8bd2cc1e3b65_18230]
#10 [ffff88012a9a7e90] start_callback at ffffffff810db160
#11 [ffff88012a9a7ee0] utrace_resume at ffffffff810dc044
#12 [ffff88012a9a7f30] do_notify_resume at ffffffff8100ab30
#13 [ffff88012a9a7f50] int_signal at ffffffff8100b3a1
    RIP: 000000308860e76c  RSP: 00007f4f0befce70  RFLAGS: 00000297
    RAX: 0000000000000000  RBX: 00007f4eb40049f8  RCX: ffffffffffffffff
    RDX: 00000000ffffbd7d  RSI: 0000000000004283  RDI: 00007f4f31571541
    RBP: 00007f4f0befce90   R8: 0000000000000000   R9: 0000000000000001
    R10: 000000000000000c  R11: 0000000000000297  R12: 00007f4e98000cf0
    R13: 00007f4e98002918  R14: 00007f4e98000de0  R15: 0000000000000003
    ORIG_RAX: 000000000000003a  CS: 0033  SS: 002b
  • panic with logs in RHEL7
[42784925.643392] Kprobes globally unoptimized
[42784925.643717] stap_pkexec_block: module verification failed: signature and/or required key missing - tainting kernel
[42784925.664997] stap_pkexec_block: systemtap: 2.8/0.163, base: ffffffffa06c8000, memory: 139data/48text/478ctx/2058net/331alloc kb, probes: 1
[42963045.415834] BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
[42963045.415882] IP: [<ffffffff811f67b8>] d_path+0x38/0x170
[42963045.415907] PGD b7b4cc067 PUD 1a0ad6b067 PMD 0 
[42963045.415927] Oops: 0000 [#1] SMP 
[42963045.415942] Modules linked in: stap_pkexec_block(OE) fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat iptable_filter dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc bonding intel_powerclamp coretemp intel_rapl kvm_intel kvm ext4 crc32_pclmul ghash_clmulni_intel aesni_intel lrw iTCO_wdt iTCO_vendor_support gf128mul mbcache jbd2 glue_helper ablk_helper cryptd sg hpilo hpwdt pcspkr ioatdma shpchp lpc_ich i2c_i801 ipmi_devintf mfd_core sb_edac edac_core wmi ipmi_si ipmi_msghandler acpi_power_meter pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 serio_raw syscopyarea sysfillrect
[42963045.416229]  sysimgblt i2c_algo_bit drm_kms_helper ttm ahci libahci drm libata ixgbe tg3 i2c_core mdio dca ptp hpsa pps_core dm_mirror dm_region_hash dm_log dm_mod
[42963045.416284] CPU: 31 PID: 24204 Comm: java Tainted: G           OE  ------------   3.10.0-327.36.3.el7.x86_64 #1
[42963045.416314] Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 10/21/2019
[42963045.416338] task: ffff881ab544dc00 ti: ffff881f2e74c000 task.ti: ffff881f2e74c000
[42963045.416361] RIP: 0010:[<ffffffff811f67b8>]  [<ffffffff811f67b8>] d_path+0x38/0x170
[42963045.416385] RSP: 0018:ffff881f2e74fd90  EFLAGS: 00010246
[42963045.416402] RAX: ffff882f5154e000 RBX: ffff883fb87f4610 RCX: 0000000000000000
[42963045.416423] RDX: 0000000000001000 RSI: ffff882f5154d000 RDI: 0000000000000000
[42963045.416456] RBP: ffff881f2e74fdd0 R08: 00007f16e1d9c000 R09: 0000000000043000
[42963045.416478] R10: 0000000000000098 R11: ffffffffa06cf9b8 R12: ffff882f5154d000
[42963045.416500] R13: ffff883fbcc2aa40 R14: ffff882f5154dfad R15: ffff883fbcc261b0
[42963045.416522] FS:  00007f0f11a77700(0000) GS:ffff883fff2c0000(0000) knlGS:0000000000000000
[42963045.416548] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[42963045.416567] CR2: 0000000000000060 CR3: 0000001fa7579000 CR4: 00000000001407e0
[42963045.416589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[42963045.416611] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[42963045.416633] Stack:
[42963045.416642]  00001000000000d1 ffff882f5154e000 00007f16e1d9c000 ffff881ab544dc00
[42963045.416674]  0000000027e05315 ffff881ab544dc00 ffff882f5154d000 ffff883fbcc2aa40
[42963045.416706]  ffff881f2e74fe30 ffffffffa06d11d9 0000000000000000 00000000000000d1
[42963045.416747] Call Trace:
[42963045.416763]  [<ffffffffa06d11d9>] __stp_call_mmap_callbacks_for_task+0x169/0x240 [stap_pkexec_block]
[42963045.416792]  [<ffffffffa06d140c>] __stp_utrace_task_finder_target_quiesce+0x15c/0x2b0 [stap_pkexec_block]
[42963045.416821]  [<ffffffffa06cb5ce>] start_callback.isra.42+0x7e/0x100 [stap_pkexec_block]
[42963045.416845]  [<ffffffffa06cdf01>] utrace_resume+0xf1/0x410 [stap_pkexec_block]
[42963045.416871]  [<ffffffff810a2377>] task_work_run+0xa7/0xe0
[42963045.416891]  [<ffffffff81014b12>] do_notify_resume+0x92/0xb0
[42963045.416912]  [<ffffffff81646dfd>] int_signal+0x12/0x17
[42963045.416929] Code: 89 e5 41 55 41 54 53 48 89 fb 48 83 ec 28 48 8b 7f 08 48 89 44 24 08 89 54 24 04 65 48 8b 0c 25 28 00 00 00 48 89 4c 24 20 31 c9 <48> 8b 47 60 48 85 c0 74 2f 48 8b 40 40 48 85 c0 74 26 ff d0 48 
[42963045.417030] RIP  [<ffffffff811f67b8>] d_path+0x38/0x170
[42963045.417048]  RSP <ffff881f2e74fd90>
[42963045.417060] CR2: 0000000000000060

Environment

  • Red Hat Enterprise Linux 7.3
    • systemtap-2.8-10.el7
  • Red Hat Enterprise Linux 6.7
    • systemtap-2.7-2.el6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content