System crashes at 'd_path' which is called from '__stp_utrace_task_finder_target_quiesce'
Issue
- System kept crashing with the below call trace in RHEL 6.
crash> bt
PID: 17377 TASK: ffff8801a6c4aab0 CPU: 0 COMMAND: "java"
#0 [ffff88012a9a7990] machine_kexec at ffffffff8103d1fb
#1 [ffff88012a9a79f0] crash_kexec at ffffffff810cc632
#2 [ffff88012a9a7ac0] oops_end at ffffffff8153d9a0
#3 [ffff88012a9a7af0] no_context at ffffffff8104e8cb
#4 [ffff88012a9a7b40] __bad_area_nosemaphore at ffffffff8104eb55
#5 [ffff88012a9a7b90] bad_area at ffffffff8104ec7e
#6 [ffff88012a9a7bc0] __do_page_fault at ffffffff8104f483
#7 [ffff88012a9a7ce0] do_page_fault at ffffffff8153f8ee
#8 [ffff88012a9a7d10] page_fault at ffffffff8153cc35
[exception RIP: d_path+0x20]
RIP: ffffffff811acdc0 RSP: ffff88012a9a7dc8 RFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff8801a6c4aab0 RCX: 00000000000043e1
RDX: 0000000000001000 RSI: ffff8803255a1000 RDI: 0000000000000000
RBP: ffff88012a9a7e18 R8: 0000003099c21000 R9: 0000000000001000
R10: 65737275636e6269 R11: 372e352e6f732e73 R12: ffff88012084aad0
R13: ffffffffa096f740 R14: 000000000000014e R15: 0000000000000026
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
#9 [ffff88012a9a7e20] __stp_utrace_task_finder_target_quiesce at ffffffffa08071cb [stap_98526622d228930268ec8bd2cc1e3b65_18230]
#10 [ffff88012a9a7e90] start_callback at ffffffff810db160
#11 [ffff88012a9a7ee0] utrace_resume at ffffffff810dc044
#12 [ffff88012a9a7f30] do_notify_resume at ffffffff8100ab30
#13 [ffff88012a9a7f50] int_signal at ffffffff8100b3a1
RIP: 000000308860e76c RSP: 00007f4f0befce70 RFLAGS: 00000297
RAX: 0000000000000000 RBX: 00007f4eb40049f8 RCX: ffffffffffffffff
RDX: 00000000ffffbd7d RSI: 0000000000004283 RDI: 00007f4f31571541
RBP: 00007f4f0befce90 R8: 0000000000000000 R9: 0000000000000001
R10: 000000000000000c R11: 0000000000000297 R12: 00007f4e98000cf0
R13: 00007f4e98002918 R14: 00007f4e98000de0 R15: 0000000000000003
ORIG_RAX: 000000000000003a CS: 0033 SS: 002b
- panic with logs in RHEL7
[42784925.643392] Kprobes globally unoptimized
[42784925.643717] stap_pkexec_block: module verification failed: signature and/or required key missing - tainting kernel
[42784925.664997] stap_pkexec_block: systemtap: 2.8/0.163, base: ffffffffa06c8000, memory: 139data/48text/478ctx/2058net/331alloc kb, probes: 1
[42963045.415834] BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
[42963045.415882] IP: [<ffffffff811f67b8>] d_path+0x38/0x170
[42963045.415907] PGD b7b4cc067 PUD 1a0ad6b067 PMD 0
[42963045.415927] Oops: 0000 [#1] SMP
[42963045.415942] Modules linked in: stap_pkexec_block(OE) fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat iptable_filter dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc bonding intel_powerclamp coretemp intel_rapl kvm_intel kvm ext4 crc32_pclmul ghash_clmulni_intel aesni_intel lrw iTCO_wdt iTCO_vendor_support gf128mul mbcache jbd2 glue_helper ablk_helper cryptd sg hpilo hpwdt pcspkr ioatdma shpchp lpc_ich i2c_i801 ipmi_devintf mfd_core sb_edac edac_core wmi ipmi_si ipmi_msghandler acpi_power_meter pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 serio_raw syscopyarea sysfillrect
[42963045.416229] sysimgblt i2c_algo_bit drm_kms_helper ttm ahci libahci drm libata ixgbe tg3 i2c_core mdio dca ptp hpsa pps_core dm_mirror dm_region_hash dm_log dm_mod
[42963045.416284] CPU: 31 PID: 24204 Comm: java Tainted: G OE ------------ 3.10.0-327.36.3.el7.x86_64 #1
[42963045.416314] Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 10/21/2019
[42963045.416338] task: ffff881ab544dc00 ti: ffff881f2e74c000 task.ti: ffff881f2e74c000
[42963045.416361] RIP: 0010:[<ffffffff811f67b8>] [<ffffffff811f67b8>] d_path+0x38/0x170
[42963045.416385] RSP: 0018:ffff881f2e74fd90 EFLAGS: 00010246
[42963045.416402] RAX: ffff882f5154e000 RBX: ffff883fb87f4610 RCX: 0000000000000000
[42963045.416423] RDX: 0000000000001000 RSI: ffff882f5154d000 RDI: 0000000000000000
[42963045.416456] RBP: ffff881f2e74fdd0 R08: 00007f16e1d9c000 R09: 0000000000043000
[42963045.416478] R10: 0000000000000098 R11: ffffffffa06cf9b8 R12: ffff882f5154d000
[42963045.416500] R13: ffff883fbcc2aa40 R14: ffff882f5154dfad R15: ffff883fbcc261b0
[42963045.416522] FS: 00007f0f11a77700(0000) GS:ffff883fff2c0000(0000) knlGS:0000000000000000
[42963045.416548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[42963045.416567] CR2: 0000000000000060 CR3: 0000001fa7579000 CR4: 00000000001407e0
[42963045.416589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[42963045.416611] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[42963045.416633] Stack:
[42963045.416642] 00001000000000d1 ffff882f5154e000 00007f16e1d9c000 ffff881ab544dc00
[42963045.416674] 0000000027e05315 ffff881ab544dc00 ffff882f5154d000 ffff883fbcc2aa40
[42963045.416706] ffff881f2e74fe30 ffffffffa06d11d9 0000000000000000 00000000000000d1
[42963045.416747] Call Trace:
[42963045.416763] [<ffffffffa06d11d9>] __stp_call_mmap_callbacks_for_task+0x169/0x240 [stap_pkexec_block]
[42963045.416792] [<ffffffffa06d140c>] __stp_utrace_task_finder_target_quiesce+0x15c/0x2b0 [stap_pkexec_block]
[42963045.416821] [<ffffffffa06cb5ce>] start_callback.isra.42+0x7e/0x100 [stap_pkexec_block]
[42963045.416845] [<ffffffffa06cdf01>] utrace_resume+0xf1/0x410 [stap_pkexec_block]
[42963045.416871] [<ffffffff810a2377>] task_work_run+0xa7/0xe0
[42963045.416891] [<ffffffff81014b12>] do_notify_resume+0x92/0xb0
[42963045.416912] [<ffffffff81646dfd>] int_signal+0x12/0x17
[42963045.416929] Code: 89 e5 41 55 41 54 53 48 89 fb 48 83 ec 28 48 8b 7f 08 48 89 44 24 08 89 54 24 04 65 48 8b 0c 25 28 00 00 00 48 89 4c 24 20 31 c9 <48> 8b 47 60 48 85 c0 74 2f 48 8b 40 40 48 85 c0 74 26 ff d0 48
[42963045.417030] RIP [<ffffffff811f67b8>] d_path+0x38/0x170
[42963045.417048] RSP <ffff881f2e74fd90>
[42963045.417060] CR2: 0000000000000060
Environment
- Red Hat Enterprise Linux 7.3
- systemtap-2.8-10.el7
- Red Hat Enterprise Linux 6.7
- systemtap-2.7-2.el6
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.