RHEL7: kernel panic at kernel/trace/ftrace.c:1839

Solution In Progress - Updated -

Issue

  • Kernel panic with below logs:
[401273.017886] ------------[ cut here ]------------ [401273.017894] WARNING: CPU: 47 PID: 61359 at kernel/trace/ftrace.c:1839 ftrace_bug+0x223/0x270 [401273.017895] Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill mpt3sas mptctl mptbase udp_diag tcp_diag inet_diag dm_round_robin ib_iser(OE) iscsi_tcp libiscsi_tcp libiscsi dell_rbu nvme_rdma(OE) nvme_fabrics(OE) nvme(OE) nvme_core(OE) scsi_transport_iscsi bonding(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ib_ipoib(OE) ib_cm(OE) ip6t_REJECT nf_reject_ipv6 xt_conntrack ib_umad(OE) mlx5_fpga_tools(OE) ip_set nfnetlink mlx5_ib(OE) ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 mlx5_core(OE) nf_nat_ipv6 mlxfw(OE) ip6table_mangle vfio_mdev(OE) vfio_iommu_type1 vfio ip6table_security mdev(OE) ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack mlx4_en(OE) iptable_mangle [401273.017921]  iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iTCO_wdt iTCO_vendor_support dell_smbios dcdbas sparse_keymap skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr lpc_ich mei_me i2c_i801 mei joydev shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad dm_multipath knem(OE) binfmt_misc ip_tables xfs libcrc32c mlx4_ib(OE) ib_uverbs(OE) ib_core(OE) crc32c_intel mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe mlx4_core(OE) igb drm devlink mdio mlx_compat(OE) ptp i2c_algo_bit pps_core dca i2c_core sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common [401273.017946]  ahci libahci libata mpt2sas raid_class scsi_transport_sas megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[401273.017951] CPU: 47 PID: 61359 Comm: cloudwalker-eng Kdump: loaded Tainted: G           OE  ------------   3.10.0-862.11.6.el7.x86_64 #1
[401273.017953] Hardware name: Hangzhou WOQU Technology WOQUTECH R940/0GCTJ1, BIOS 2.9.4 11/06/2020 [401273.017954] Call Trace:
[401273.017958]  [<ffffffff8ad135d4>] dump_stack+0x19/0x1b [401273.017962]  [<ffffffff8a6945d8>] __warn+0xd8/0x100 [401273.017967]  [<ffffffff8ad29100>] ? __fentry__+0x10/0x10 [401273.017968]  [<ffffffff8a69471d>] warn_slowpath_null+0x1d/0x20 [401273.017970]  [<ffffffff8a7574c3>] ftrace_bug+0x223/0x270 [401273.017973]  [<ffffffff8a661e11>] ftrace_replace_code+0x2b1/0x420 [401273.017975]  [<ffffffff8a757baa>] ftrace_modify_all_code+0x6a/0xb0 [401273.017977]  [<ffffffff8a661f90>] arch_ftrace_update_code+0x10/0x20 [401273.017978]  [<ffffffff8a757c87>] ftrace_run_update_code+0x17/0x70 [401273.017980]  [<ffffffff8a757d19>] ftrace_startup_enable+0x39/0x50 [401273.017981]  [<ffffffff8a7581d4>] ftrace_startup+0x114/0x2e0 [401273.017983]  [<ffffffff8a7583e3>] register_ftrace_function+0x43/0x60
[401273.017985]  [<ffffffff8ad22187>] arm_kprobe+0x87/0xe0 [401273.017987]  [<ffffffff8ad2224b>] enable_kprobe+0x6b/0x80 [401273.017990]  [<ffffffff8a77a3c9>] enable_trace_probe+0xf9/0x150 [401273.017992]  [<ffffffff8ad24708>] kprobe_register+0x48/0x60 [401273.017994]  [<ffffffff8a771d85>] __ftrace_event_enable_disable+0x65/0x1a0
[401273.017995]  [<ffffffff8a77252d>] event_enable_write+0x9d/0xe0 [401273.017998]  [<ffffffff8a81f180>] vfs_write+0xc0/0x1f0 [401273.017999]  [<ffffffff8a81ffaf>] SyS_write+0x7f/0xf0 [401273.018001]  [<ffffffff8ad2579b>] system_call_fastpath+0x22/0x27 [401273.018002] ---[ end trace 522a1a9c731258d7 ]--- [401273.018003] ftrace faulted on modifying [<31ffffffc0772e70>] 0x31ffffffc0772e70 [401273.018007] Failed on adding breakpoints (39185):
[401273.020532] ------------[ cut here ]------------

Environment

  • Red Hat Enterprise Linux 7.5
  • CentOS package installed
    • kernel-debuginfo-3.10.0-862.11.6.el7.x86_64 kernel-debuginfo-common-x86_64-3.10.0-862.11.6.el7.x86_64
  • ftrace enabled by 3rd party application cloudwalker-engine

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In