OSP 16.2: kernel panic when tryinig to use hugepages
Issue
- kernel panic with below logs:
[ 39.723351] vrouter: loading out-of-tree module taints kernel.
[ 39.723638] vrouter: module verification failed: signature and/or required key missing - tainting kernel
[ 39.725789] vrouter version: {"build-info": [{"build-time": "2022-06-20 03:02:45.919367", "build-hostname": "contrail-nightly-tbf5h", "build-user": "contrail-builder", "build-version": "21.4"}]}
[ 39.729090] linux_if_notifier: dev 0000000048dbe25d event REGISTER
[ 39.729095] linux_if_notifier: dev 0000000048dbe25d event UNREGISTER
[ 39.729096] linux_if_notifier: dev 00000000b2294c78 event REGISTER
[ 39.729099] linux_if_notifier: dev 00000000b2294c78 event UNREGISTER
[ 39.729100] linux_if_notifier: dev 00000000f83e4fc1 event REGISTER
[ 39.729103] linux_if_notifier: dev 00000000f83e4fc1 event UNREGISTER
[ 39.729104] linux_if_notifier: dev 00000000a3222b11 event REGISTER
[ 39.729107] linux_if_notifier: dev 00000000a3222b11 event UNREGISTER
[ 39.729108] linux_if_notifier: dev 00000000742ffc82 event REGISTER
[ 39.729111] linux_if_notifier: dev 00000000742ffc82 event UNREGISTER
[ 39.729112] linux_if_notifier: dev 00000000d53b8b31 event REGISTER
[ 39.729115] linux_if_notifier: dev 00000000d53b8b31 event UNREGISTER
[ 39.729115] linux_if_notifier: dev 000000005f264fae event REGISTER
[ 39.729118] linux_if_notifier: dev 000000005f264fae event UNREGISTER
[ 39.729119] linux_if_notifier: dev 0000000038caa1a6 event REGISTER
[ 39.729122] linux_if_notifier: dev 0000000038caa1a6 event UNREGISTER
[ 39.729123] linux_if_notifier: dev 00000000f7a21872 event REGISTER
[ 39.729126] linux_if_notifier: dev 00000000f7a21872 event UNREGISTER
[ 39.729127] linux_if_notifier: dev 00000000f62ec522 event REGISTER
[ 39.729130] linux_if_notifier: dev 00000000f62ec522 event UNREGISTER
[ 39.729130] linux_if_notifier: dev 0000000097410a0d event REGISTER
[ 39.729133] linux_if_notifier: dev 0000000097410a0d event UNREGISTER
[ 39.729134] linux_if_notifier: dev 00000000fccfb6a8 event REGISTER
[ 39.729137] linux_if_notifier: dev 00000000fccfb6a8 event UNREGISTER
[ 39.729138] linux_if_notifier: dev 00000000669bc5e7 event REGISTER
[ 39.729141] linux_if_notifier: dev 00000000669bc5e7 event UNREGISTER
[ 39.729141] linux_if_notifier: dev 00000000d9c171a4 event REGISTER
[ 39.729144] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[ 39.729145] linux_if_notifier: dev 00000000708c867f event REGISTER
[ 39.729148] linux_if_notifier: dev 00000000708c867f event UNREGISTER
[ 39.729148] linux_if_notifier: dev 00000000e62ff6ea event REGISTER
[ 39.729151] linux_if_notifier: dev 00000000e62ff6ea event UNREGISTER
[ 39.729152] linux_if_notifier: dev 0000000082e5b053 event REGISTER
[ 39.729155] linux_if_notifier: dev 0000000034e484db event REGISTER
[ 39.729158] linux_if_notifier: dev 00000000c1fc6e3d event REGISTER
[ 39.977529] linux_if_notifier: dev 0000000082e5b053 event UNREGISTER
[ 39.977588] linux_if_notifier: dev 0000000082e5b053 event UNREGISTER
[ 40.176831] linux_if_notifier: dev 00000000c1fc6e3d event UNREGISTER
[ 40.176876] linux_if_notifier: dev 00000000c1fc6e3d event UNREGISTER
[ 40.375398] linux_if_notifier: dev 0000000034e484db event UNREGISTER
[ 40.375439] linux_if_notifier: dev 0000000034e484db event UNREGISTER
[ 40.378814] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[ 40.378986] linux_if_notifier: dev 00000000de1e73eb event REGISTER
[ 40.383079] Vrouter: vr_interface_get:3548 vif is NULL for vifr_idx: -1
[ 40.383125] Num phy interfaces 1
[ 40.383716] Vrouter: vr_interface_get:3548 vif is NULL for vifr_idx: -1
[ 40.384584] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[ 40.384599] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[ 40.385463] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[ 40.385547] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[ 40.398649] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[ 40.398671] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[ 44.515793] linux_if_notifier: dev 00000000ab576300 event UNREGISTER
[ 44.516085] linux_if_notifier: dev 00000000ab576300 event REGISTER
[ 44.516125] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[ 44.516188] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[ 44.543390] linux_if_notifier: dev 00000000ab576300 event UNREGISTER
[ 44.543456] linux_if_notifier: dev 00000000ab576300 event UNREGISTER
[ 48.657259] linux_if_notifier: dev 00000000b3df36a1 event UNREGISTER
[ 48.657403] linux_if_notifier: dev 00000000b3df36a1 event REGISTER
[ 48.657411] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[ 48.657434] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[ 48.682969] linux_if_notifier: dev 00000000b3df36a1 event UNREGISTER
[ 48.683028] linux_if_notifier: dev 00000000b3df36a1 event UNREGISTER
[ 52.801682] linux_if_notifier: dev 000000006b794c44 event UNREGISTER
[ 52.801956] linux_if_notifier: dev 000000006b794c44 event REGISTER
[ 52.801972] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[ 52.802030] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[ 52.827962] linux_if_notifier: dev 000000006b794c44 event UNREGISTER
[ 52.828018] linux_if_notifier: dev 000000006b794c44 event UNREGISTER
[ 57.974594] cgroup: cgroup: disabling cgroup2 socket matching due to net_prio or net_cls activation
[ 57.999789] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[ 58.003545] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[ 58.161693] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[ 58.618324] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[ 59.179193] tun: Universal TUN/TAP device driver, 1.6
[ 59.179821] linux_if_notifier: dev 000000005a0a064f event UNREGISTER
[ 59.179989] linux_if_notifier: dev 000000005a0a064f event REGISTER
[ 59.194303] linux_if_notifier: dev 000000005a0a064f event UNREGISTER
[ 59.194435] linux_if_notifier: dev 000000005a0a064f event UNREGISTER
[ 59.259446] Config Hugepage vmem 00000000f01948b1 psize 1073741824 mem_sz 20185088 path /dev/hugepages/bridge
[ 59.263128] Pinned huge page uspace_vmem 00000000f01948b1 start_page_addr 0000000005a5d439 num 4k pages 4928 mem_size 20185088 file_path /dev/hugepages/bridge
[ 59.263130] Config Hugepage vmem 000000006ffa70c9 psize 1073741824 mem_sz 161218560 path /dev/hugepages/flow
[ 59.291301] Pinned huge page uspace_vmem 000000006ffa70c9 start_page_addr 000000001b63c890 num 4k pages 39360 mem_size 161218560 file_path /dev/hugepages/flow
[ 59.327011] contrail-nodemg[7742]: segfault at 7fb72c0d9802 ip 00007fb72b2638c0 sp 00007ffc7a6de688 error 7 in libc-2.28.so[7fb72b21c000+1bc000]
[ 59.327018] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 59.327236] nova-compute[7781]: segfault at 80 ip 00007fd23230471d sp 00007ffedde16a70 error 4 in libpython3.6m.so.1.0[7fd232168000+2ab000]
[ 59.327241] Code: 8b 35 87 28 31 00 8b 7e 20 8d 57 01 89 56 20 41 39 16 0f 8c 93 08 00 00 49 39 cb 0f 84 ec 05 00 00 49 39 db 0f 84 13 05 00 00 <49> 8b 8b 80 00 00 00 48 85 c9 0f 84 4d 08 00 00 48 85 ed 0f 88 a3
[ 59.330053] tuned[8456]: segfault at 0 ip 00007f8c02e24b7b sp 00007f8be2eee2e0 error 4 in libpython3.6m.so.1.0[7f8c02d37000+2ab000]
[ 59.330064] Code: 41 8d 5c 24 ff 4c 8d 05 03 84 40 00 c1 eb 03 8d 2c 1b 49 8b 34 e8 48 8b 7e 10 48 39 fe 0f 84 bc 01 00 00 4c 8b 56 08 83 06 01 <49> 8b 02 48 89 46 08 48 85 c0 74 39 45 85 ed 75 64 48 8b 74 24 68
[ 59.330207] traps: yum[9080] general protection fault ip:7efe46339780 sp:7ffda54025b8 error:0 in libc-2.28.so[7efe462eb000+1bc000]
[ 59.336830] contrail-versio[9076]: segfault at 0 ip 00007f44fc5db3f0 sp 00007ffe44d17f18 error 6 in libc-2.28.so[7f44fc58c000+1bc000]
[ 59.336836] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 59.337423] tail[9077]: segfault at 0 ip 00007ff0f47bc350 sp 00007ffca43fe8b8 error 6 in libc-2.28.so[7ff0f476b000+1bc000]
[ 59.337428] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 59.337453] sh[9075]: segfault at 0 ip 0000000000000000 sp 00007ffd4de8c698 error 14 in bash[55df1267c000+108000]
[ 59.337456] Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
[ 59.338664] contrail-vroute[9025]: segfault at 0 ip 000000000093e050 sp 00007fff7b23ae58 error 6 in contrail-vrouter-agent[400000+1a4f000]
[ 59.338668] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 59.338729] entrypoint.sh[7836]: segfault at 0 ip 00007fdb6cbcfd50 sp 00007fffbc949ff8 error 6 in libc-2.28.so[7fdb6cb81000+1bc000]
[ 59.338735] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 59.342797] BUG: unable to handle kernel NULL pointer dereference at 0000000000000050
[ 59.348916] ------------[ cut here ]------------
[ 59.348960] tee[8486]: segfault at 0 ip 00007f276de04350 sp 00007ffd9df3c638 error 6 in libc-2.28.so[7f276ddb3000+1bc000]
[ 59.348965] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 59.350694] PGD 0 P4D 0
[ 59.355356] kernel BUG at include/linux/fs.h:2943!
[ 59.366398] Oops: 0000 [#1] SMP NOPTI
[ 59.396401] CPU: 18 PID: 8456 Comm: tuned Kdump: loaded Tainted: G IOE --------- - - 4.18.0-305.30.1.el8_4.x86_64 #1
[ 59.408138] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 07/16/2020
[ 59.416728] RIP: 0010:filemap_fault+0x47/0xa20
[ 59.421204] Code: 04 25 28 00 00 00 48 89 84 24 a0 00 00 00 31 c0 48 8b 07 48 8b 80 a0 00 00 00 4c 8b a0 f0 00 00 00 48 89 44 24 08 4d 8b 34 24 <49> 8b 46 50 48 05 ff 0f 00 00 48 c1 e8 0c 49 39 c5 0f 83 dd 05 00
[ 59.440118] RSP: 0000:ffffbece6331b838 EFLAGS: 00010246
[ 59.445379] RAX: ffff9d703dfb5100 RBX: ffffbece6331b998 RCX: 0000000000000000
[ 59.452564] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffffbece6331b998
[ 59.459750] RBP: ffff9d8fbdb77b40 R08: 0000000000000000 R09: 0000000001db8093
[ 59.466936] R10: 0000000000000001 R11: 0000000000000100 R12: ffff9d8fbdb77e00
[ 59.474122] R13: 00000000000002b5 R14: 0000000000000000 R15: 0000000000000000
[ 59.481307] FS: 00007f8be2eef700(0000) GS:ffff9d707fa80000(0000) knlGS:0000000000000000
[ 59.489456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 59.495243] CR2: 0000000000000050 CR3: 0000003d7eafe005 CR4: 00000000007706e0
[ 59.502429] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 59.509614] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 59.516800] PKRU: 55555554
[ 59.519524] Call Trace:
[ 59.521991] ? _cond_resched+0x15/0x30
[ 59.526376] __xfs_filemap_fault+0x6d/0x200 [xfs]
[ 59.531652] __do_fault+0x36/0xd0
[ 59.535515] __handle_mm_fault+0x3d7/0xca0
[ 59.540200] ? xfs_iunlock+0xf3/0x100 [xfs]
[ 59.544925] handle_mm_fault+0xc2/0x1d0
[ 59.549296] __get_user_pages+0x2c4/0x840
[ 59.553836] get_dump_page+0x49/0x80
[ 59.557934] elf_core_dump+0x8cf/0xa50
[ 59.562201] do_coredump+0x73b/0xf4e
[ 59.566285] ? sched_clock_cpu+0xc/0xb0
[ 59.570627] ? up+0x12/0x50
[ 59.573912] get_signal+0x14f/0x870
[ 59.577888] ? __send_signal+0x343/0x4a0
[ 59.582295] ? page_fault+0x8/0x30
[ 59.586173] do_signal+0x36/0x660
[ 59.589953] ? force_sig_info+0xc7/0xe0
[ 59.594253] ? force_sig_fault+0x59/0x80
[ 59.598633] ? page_fault+0x8/0x30
[ 59.602485] exit_to_usermode_loop+0x89/0xf0
[ 59.607206] prepare_exit_to_usermode+0x9b/0xa0
[ 59.612178] retint_user+0x8/0x8
[ 59.615825] RIP: 0033:0x7f8c02e24b7b
[ 59.619813] Code: 41 8d 5c 24 ff 4c 8d 05 03 84 40 00 c1 eb 03 8d 2c 1b 49 8b 34 e8 48 8b 7e 10 48 39 fe 0f 84 bc 01 00 00 4c 8b 56 08 83 06 01 <49> 8b 02 48 89 46 08 48 85 c0 74 39 45 85 ed 75 64 48 8b 74 24 68
[ 59.639498] RSP: 002b:00007f8be2eee2e0 EFLAGS: 00010202
[ 59.645151] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 00007f8c02f53aac
[ 59.652730] RDX: 000000000000001c RSI: 00007f8bec5f4000 RDI: 0000000000000000
[ 59.660308] RBP: 0000000000000006 R08: 00007f8c0322cf60 R09: 00007f8beb3ad340
[ 59.667877] R10: 0000000000000000 R11: 00007f8c0326bee8 R12: 000000000000001c
[ 59.675434] R13: 0000000000000000 R14: 000000000000001c R15: 0000000000000001
[ 59.682981] Modules linked in: tun overlay vrouter(OE) 8021q garp mrp bonding nf_log_ipv4 nf_log_ipv6 nf_log_common nft_chain_nat nft_limit ipt_MASQUERADE nft_counter xt_LOG xt_limit xt_multiport xt_comment nf_nat xt_state xt_conntrack xt_addrtype nft_compat nf_tables nfnetlink vfat fat dm_service_time dm_multipath dm_mod intel_rapl_msr intel_rapl_common isst_if_common rpcrdma sunrpc rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm ib_cm nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass ipmi_ssif crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl intel_cstate i40iw mlx5_ib ib_uverbs ib_core intel_uncore pcspkr joydev acpi_ipmi hpwdt hpilo lpc_ich mei_me ipmi_si mei ioatdma ipmi_devintf dca wmi ipmi_msghandler acpi_tad acpi_power_meter ip_tables xfs sd_mod t10_pi sg mlx5_core mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm i40e ahci libahci mlxfw
[ 59.683015] pci_hyperv_intf libata tls tg3 i2c_algo_bit nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc32c_intel fuse br_netfilter bridge stp llc
[ 59.789014] CR2: 0000000000000050
Environment
- Red Hat Openstack Platform 16.2
kernel-4.18.0-305.30.1.el8_4
- 3rd party module
vrouter
21.4
vrouter version: {"build-info": [{"build-time": "2022-06-20 03:02:45.919367", "build-hostname": "contrail-nightly-tbf5h", "build-user": "contrail-builder", "build-version": "21.4"}]}
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.