kernel panic at handle_cmd_completion+0x60a

Solution Verified - Updated -

Issue

  • Panic log from RHEL 7:
[1028465.568365] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[1028465.568411] IP: [<ffffffff9d5544fa>] handle_cmd_completion+0x60a/0xef0
[1028465.568445] PGD 0 
[1028465.568459] Oops: 0000 [#1] SMP 
[1028465.568479] Modules linked in: macvlan mpt2sas raid_class scsi_transport_sas mptctl mptbase xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache team_mode_roundrobin ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge 8021q garp mrp stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 team ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu skx_edac intel_powerclamp coretemp
[1028465.568907]  intel_rapl iosf_mbi kvm_intel iTCO_wdt iTCO_vendor_support kvm sunrpc irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dell_smbios dcdbas pcspkr dell_wmi_descriptor vfat fat sg mei_me mei lpc_ich i2c_i801 ipmi_si acpi_power_meter acpi_pad ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) nvidia(POE) sd_mod sr_mod crc_t10dif cdrom crct10dif_generic mgag200 drm_kms_helper ttm syscopyarea sysfillrect sysimgblt fb_sys_fops ahci drm ixgbe crct10dif_pclmul crct10dif_common crc32c_intel libahci igb libata megaraid_sas mdio ptp pps_core i2c_algo_bit dca drm_panel_orientation_quirks wmi nfit ipmi_devintf ipmi_msghandler libnvdimm dm_mirror dm_region_hash dm_log dm_mod
[1028465.569321] CPU: 0 PID: 0 Comm: swapper/0 Kdump: loaded Tainted: P        W  OE  ------------   3.10.0-1127.18.2.el7.x86_64 #1
[1028465.569372] Hardware name: Dell Inc. PowerEdge R640/0RGP26, BIOS 2.3.10 08/15/2019
[1028465.569407] task: ffffffff9dc18480 ti: ffffffff9dc00000 task.ti: ffffffff9dc00000
[1028465.569442] RIP: 0010:[<ffffffff9d5544fa>]  [<ffffffff9d5544fa>] handle_cmd_completion+0x60a/0xef0
[1028465.569487] RSP: 0018:ffff980fee603d90  EFLAGS: 00010046
[1028465.569512] RAX: 0000000000000000 RBX: ffff980fd11707a0 RCX: 0000000000000006
[1028465.570886] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff98107a6ac260
[1028465.572262] RBP: ffff980fee603e20 R08: 000000000000000a R09: 0000000000000000
[1028465.573636] R10: 000000000013e016 R11: ffff980fee603a2e R12: ffff980fec5a3420
[1028465.575013] R13: ffff980fec5a5130 R14: 0000000000000005 R15: ffff98107a6ac260
[1028465.576382] FS:  0000000000000000(0000) GS:ffff980fee600000(0000) knlGS:0000000000000000
[1028465.577754] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1028465.579125] CR2: 0000000000000008 CR3: 000000029fa10000 CR4: 00000000007607f0
[1028465.580505] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1028465.581884] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[1028465.583251] PKRU: 00000000
[1028465.584606] Call Trace:
[1028465.585946]  <IRQ> 
[1028465.585963]  [<ffffffff9d50ef49>] ? usb_hcd_poll_rh_status+0x129/0x180
[1028465.588626]  [<ffffffff9d5564cc>] xhci_irq+0x39c/0xa50
[1028465.589965]  [<ffffffff9d0ae150>] ? __internal_add_timer+0x130/0x130
[1028465.591310]  [<ffffffff9d556b91>] xhci_msi_irq+0x11/0x20
[1028465.592651]  [<ffffffff9d150764>] __handle_irq_event_percpu+0x44/0x1c0
[1028465.593993]  [<ffffffff9d150912>] handle_irq_event_percpu+0x32/0x80
[1028465.595339]  [<ffffffff9d15099c>] handle_irq_event+0x3c/0x60
[1028465.596671]  [<ffffffff9d15378f>] handle_edge_irq+0x7f/0x150
[1028465.597992]  [<ffffffff9d02f5f4>] handle_irq+0xe4/0x1a0
[1028465.599325]  [<ffffffff9d0ded2e>] ? sched_clock_idle_wakeup_event+0x1e/0x20
[1028465.600664]  [<ffffffff9d1118ad>] ? tick_check_idle+0xbd/0xd0
[1028465.602001]  [<ffffffff9d79786d>] do_IRQ+0x4d/0xf0
[1028465.603320]  [<ffffffff9d78936a>] common_interrupt+0x16a/0x16a
[1028465.604628]  <EOI> 
[1028465.604644]  [<ffffffff9d5c61e7>] ? cpuidle_enter_state+0x57/0xd0
[1028465.607241]  [<ffffffff9d5c61dd>] ? cpuidle_enter_state+0x4d/0xd0
[1028465.608525]  [<ffffffff9d5c633e>] cpuidle_idle_call+0xde/0x230
[1028465.609774]  [<ffffffff9d037c8e>] arch_cpu_idle+0xe/0xc0
[1028465.610993]  [<ffffffff9d101c7a>] cpu_startup_entry+0x14a/0x1e0
[1028465.612188]  [<ffffffff9d76e6a7>] rest_init+0x77/0x80
[1028465.613350]  [<ffffffff9dd8b1cf>] start_kernel+0x44b/0x46c
[1028465.614487]  [<ffffffff9dd8ab84>] ? repair_env_string+0x5c/0x5c
[1028465.615594]  [<ffffffff9dd8a120>] ? early_idt_handler_array+0x120/0x120
[1028465.616675]  [<ffffffff9dd8a738>] x86_64_start_reservations+0x24/0x26
[1028465.617726]  [<ffffffff9dd8a88e>] x86_64_start_kernel+0x154/0x177
[1028465.618744]  [<ffffffff9d0000d5>] start_cpu+0x5/0x14
[1028465.619725] Code: 08 00 00 89 c8 25 00 00 1f 00 4c 89 ff c1 e8 10 83 e8 01 89 c2 89 45 90 48 63 45 a8 49 8d 04 c7 48 89 45 98 48 8b 80 c8 01 00 00 <48> 8b 70 08 e8 ad 84 ff ff 49 89 c4 0f 1f 44 00 00 31 c0 44 89 
[1028465.621827] RIP  [<ffffffff9d5544fa>] handle_cmd_completion+0x60a/0xef0
[1028465.622800]  RSP <ffff980fee603d90>
[1028465.623737] CR2: 0000000000000008
  • Panic log from RHEL 8:
[14228151.832142] usb 1-14.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[14228151.832910] hub 1-14.1:1.0: USB hub found
[14228151.833081] hub 1-14.1:1.0: 4 ports detected
[14228151.898008] usb 1-14.4: new high-speed USB device number 101 using xhci_hcd
[14228151.983885] usb 1-14.4: Device not responding to setup address.
[14228151.983915] xhci_hcd 0000:00:14.0: WARN Event TRB for slot 1 ep 0 with no TDs queued?
[14228151.983934] usb 1-14: USB disconnect, device number 99
[14228151.983967] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[14228151.992055] PGD 0 P4D 0 
[14228151.994853] Oops: 0000 [#1] SMP NOPTI
[14228151.998780] CPU: 36 PID: 0 Comm: swapper/36 Kdump: loaded Tainted: G          I      --------- -  - 4.18.0-305.76.1.el8
_4.x86_64 #1
[14228152.010851] Hardware name: Dell Inc. PowerEdge R640/0HG0J8, BIOS 2.15.1 06/15/2022
[14228152.018684] RIP: 0010:handle_cmd_completion+0x8d3/0x1130
[14228152.024259] Code: 24 08 0f 85 46 06 00 00 8b 44 24 08 41 c1 ed 10 48 89 df 41 83 e5 1f 41 8d 55 ff 4c 8d 2c c3 49 8b 85
 00 02 00 00 89 54 24 18 <48> 8b 70 08 e8 44 7f ff ff 48 89 c1 0f 1f 44 00 00 89 e9 48 c7 c2
[14228152.043264] RSP: 0000:ffffb9020d3c8cd8 EFLAGS: 00010002
[14228152.048748] RAX: 0000000000000000 RBX: ffff9450166842c0 RCX: 0000000000000001
[14228152.056143] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9450166842c0
[14228152.063533] RBP: 000000000000000b R08: ffff945020c9b2d0 R09: 0000000000000000
[14228152.070926] R10: 0000000000000000 R11: 00000000158d51f6 R12: ffff94501e2f2a70
[14228152.078320] R13: ffff9450166842c8 R14: ffff9450162fad70 R15: ffff944d8696b420
[14228152.085711] FS:  0000000000000000(0000) GS:ffff945020c80000(0000) knlGS:0000000000000000
[14228152.094057] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[14228152.100063] CR2: 0000000000000008 CR3: 0000002de1410003 CR4: 00000000007726e0
[14228152.107454] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[14228152.114846] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[14228152.122240] PKRU: 55555554
[14228152.125212] Call Trace:
[14228152.127926]  <IRQ>
[14228152.130211]  ? queue_command+0x102/0x220
[14228152.134405]  ? update_load_avg+0x7e/0x610
[14228152.138683]  ? account_entity_enqueue+0x9c/0xd0
[14228152.143476]  xhci_irq+0x1ec/0x430
[14228152.147058]  __handle_irq_event_percpu+0x40/0x180
[14228152.156733]  handle_irq_event+0x36/0x53
[14228152.160833]  handle_edge_irq+0x82/0x190
[14228152.164938]  handle_irq+0x1c/0x30
[14228152.168522]  do_IRQ+0x49/0xd0
[14228152.171752]  common_interrupt+0xf/0xf
[14228152.175678] RIP: 0010:__do_softirq+0x72/0x2d6
[14228152.180298] Code: 67 51 ff 65 81 05 b2 5b 41 79 00 01 00 00 c7 44 24 10 0a 00 00 00 48 c7 c0 00 97 02 00 65 66 c7 00 00 00 fb 66 0f 1f 44 00 00 <b8> ff ff ff ff 48 c7 c3 00 61 60 87 41 0f bc c7 89 c6 83 c6 01 89
[14228152.199300] RSP: 0000:ffffb9020d3c8f68 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff2b
[14228152.207126] RAX: 0000000000029700 RBX: 00000000000000eb RCX: 000000000000001f
[14228152.214518] RDX: 0000000000000044 RSI: 000000003158af9d RDI: ffb96c767e86db3e
[14228152.221910] RBP: ffffb9020ca1fdb8 R08: 0000000000000002 R09: 0000000000029780
[14228152.229302] R10: 013a2fc1227b5cda R11: 0000000000000000 R12: ffff9450159c0600
[14228152.236695] R13: 00000000000000eb R14: 0000000000000000 R15: 0000000000000008
[14228152.244090]  ? common_interrupt+0xa/0xf
[14228152.248188]  ? __do_softirq+0x4b/0x2d6
[14228152.252203]  irq_exit+0xf7/0x100
[14228152.255696]  do_IRQ+0x7f/0xd0
[14228152.258926]  common_interrupt+0xf/0xf
[14228152.262853]  </IRQ>
[14228152.265222] RIP: 0010:cpuidle_enter_state+0xd9/0x3c0
[14228152.270443] Code: e8 6c 5a 9f ff 80 7c 24 07 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 97 02 00 00 31 ff e8 0e c6 a5 ff fb 66 0f 1f 44 00 00 <45> 85 ed 0f 88 23 01 00 00 49 63 d5 48 2b 5c 24 08 48 8d 04 d5 00
[14228152.289447] RSP: 0000:ffffb9020ca1fe68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff14
[14228152.297273] RAX: ffff945020ca9f40 RBX: 00328c6dc8d053b8 RCX: 000000000000001f
[14228152.304666] RDX: 00328c6dc8d053b8 RSI: 000000003158af9d RDI: 0000000000000000
[14228152.312058] RBP: ffffd8ea00c80390 R08: 0000000000000002 R09: 0000000000029780
[14228152.319452] R10: 013a2fc1227b54a2 R11: ffff945020ca8c64 R12: ffffffff87730ba0
[14228152.326842] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001
[14228152.334239]  ? cpuidle_enter_state+0xb4/0x3c0
[14228152.338854]  cpuidle_enter+0x2c/0x40
[14228152.342697]  do_idle+0x234/0x260
[14228152.346188]  cpu_startup_entry+0x6f/0x80
[14228152.350373]  start_secondary+0x198/0x1e0
[14228152.354562]  secondary_startup_64_no_verify+0xc2/0xcb
[14228152.359872] Modules linked in: joydev vhost_net vhost vhost_iotlb tap tun tcp_diag inet_diag xt_CT ipt_rpfilter ip_set_hash_ip ip_set_hash_net veth nf_conntrack_netlink xt_addrtype xt_set ip_set_hash_ipportip ip_set_hash_ipportnet ip_set_hash_ipport ip_set_bitmap_port ip_set dummy ip6t_MASQUERADE ipt_MASQUERADE xt_conntrack xt_comment xt_mark nft_chain_nat nft_counter xt_multiport nft_compat nf_tables overlay 8021q garp mrp bonding intel_rapl_msr iTCO_wdt iTCO_vendor_support wmi_bmof dell_smbios dell_wmi_descriptor dcdbas intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul i40iw ghash_clmulni_intel rapl intel_cstate intel_uncore ib_uverbs snd_pcm snd_timer ib_core snd soundcore pcspkr ipmi_ssif lpc_ich i2c_i801 mei_me mei wmi acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter vxlan ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat vfat fat ext4
[14228152.359900]  mbcache jbd2 auth_rpcgss sunrpc xfs sd_mod t10_pi sg mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops i40e drm ahci libahci megaraid_sas i2c_algo_bit tg3 libata dm_mirror dm_region_hash dm_log dm_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio scsi_transport_fc br_netfilter bridge stp llc ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc32c_intel fuse
[14228152.486061] Features: eBPF/xdp
[14228152.489380] CR2: 0000000000000008

Environment

  • Red Hat Enterprise Linux 8
    • seen on kernel-4.18.0-305.76.1.el8_4
  • Red Hat Enterprise Linux 7
    • seen on kernel-3.10.0-1127.18.2.el7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content