OSP 16.2: kernel panic when tryinig to use hugepages

Issue

• kernel panic with below logs:

[   39.723351] vrouter: loading out-of-tree module taints kernel.
[   39.723638] vrouter: module verification failed: signature and/or required key missing - tainting kernel
[   39.725789] vrouter version: {"build-info": [{"build-time": "2022-06-20 03:02:45.919367", "build-hostname": "contrail-nightly-tbf5h", "build-user": "contrail-builder", "build-version": "21.4"}]}
[   39.729090] linux_if_notifier: dev 0000000048dbe25d event REGISTER
[   39.729095] linux_if_notifier: dev 0000000048dbe25d event UNREGISTER
[   39.729096] linux_if_notifier: dev 00000000b2294c78 event REGISTER
[   39.729099] linux_if_notifier: dev 00000000b2294c78 event UNREGISTER
[   39.729100] linux_if_notifier: dev 00000000f83e4fc1 event REGISTER
[   39.729103] linux_if_notifier: dev 00000000f83e4fc1 event UNREGISTER
[   39.729104] linux_if_notifier: dev 00000000a3222b11 event REGISTER
[   39.729107] linux_if_notifier: dev 00000000a3222b11 event UNREGISTER
[   39.729108] linux_if_notifier: dev 00000000742ffc82 event REGISTER
[   39.729111] linux_if_notifier: dev 00000000742ffc82 event UNREGISTER
[   39.729112] linux_if_notifier: dev 00000000d53b8b31 event REGISTER
[   39.729115] linux_if_notifier: dev 00000000d53b8b31 event UNREGISTER
[   39.729115] linux_if_notifier: dev 000000005f264fae event REGISTER
[   39.729118] linux_if_notifier: dev 000000005f264fae event UNREGISTER
[   39.729119] linux_if_notifier: dev 0000000038caa1a6 event REGISTER
[   39.729122] linux_if_notifier: dev 0000000038caa1a6 event UNREGISTER
[   39.729123] linux_if_notifier: dev 00000000f7a21872 event REGISTER
[   39.729126] linux_if_notifier: dev 00000000f7a21872 event UNREGISTER
[   39.729127] linux_if_notifier: dev 00000000f62ec522 event REGISTER
[   39.729130] linux_if_notifier: dev 00000000f62ec522 event UNREGISTER
[   39.729130] linux_if_notifier: dev 0000000097410a0d event REGISTER
[   39.729133] linux_if_notifier: dev 0000000097410a0d event UNREGISTER
[   39.729134] linux_if_notifier: dev 00000000fccfb6a8 event REGISTER
[   39.729137] linux_if_notifier: dev 00000000fccfb6a8 event UNREGISTER
[   39.729138] linux_if_notifier: dev 00000000669bc5e7 event REGISTER
[   39.729141] linux_if_notifier: dev 00000000669bc5e7 event UNREGISTER
[   39.729141] linux_if_notifier: dev 00000000d9c171a4 event REGISTER
[   39.729144] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   39.729145] linux_if_notifier: dev 00000000708c867f event REGISTER
[   39.729148] linux_if_notifier: dev 00000000708c867f event UNREGISTER
[   39.729148] linux_if_notifier: dev 00000000e62ff6ea event REGISTER
[   39.729151] linux_if_notifier: dev 00000000e62ff6ea event UNREGISTER
[   39.729152] linux_if_notifier: dev 0000000082e5b053 event REGISTER
[   39.729155] linux_if_notifier: dev 0000000034e484db event REGISTER
[   39.729158] linux_if_notifier: dev 00000000c1fc6e3d event REGISTER
[   39.977529] linux_if_notifier: dev 0000000082e5b053 event UNREGISTER
[   39.977588] linux_if_notifier: dev 0000000082e5b053 event UNREGISTER
[   40.176831] linux_if_notifier: dev 00000000c1fc6e3d event UNREGISTER
[   40.176876] linux_if_notifier: dev 00000000c1fc6e3d event UNREGISTER
[   40.375398] linux_if_notifier: dev 0000000034e484db event UNREGISTER
[   40.375439] linux_if_notifier: dev 0000000034e484db event UNREGISTER
[   40.378814] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   40.378986] linux_if_notifier: dev 00000000de1e73eb event REGISTER
[   40.383079] Vrouter: vr_interface_get:3548 vif is NULL for vifr_idx: -1 
[   40.383125] Num phy interfaces 1
[   40.383716] Vrouter: vr_interface_get:3548 vif is NULL for vifr_idx: -1 
[   40.384584] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   40.384599] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   40.385463] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   40.385547] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   40.398649] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   40.398671] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   44.515793] linux_if_notifier: dev 00000000ab576300 event UNREGISTER
[   44.516085] linux_if_notifier: dev 00000000ab576300 event REGISTER
[   44.516125] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   44.516188] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   44.543390] linux_if_notifier: dev 00000000ab576300 event UNREGISTER
[   44.543456] linux_if_notifier: dev 00000000ab576300 event UNREGISTER
[   48.657259] linux_if_notifier: dev 00000000b3df36a1 event UNREGISTER
[   48.657403] linux_if_notifier: dev 00000000b3df36a1 event REGISTER
[   48.657411] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   48.657434] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   48.682969] linux_if_notifier: dev 00000000b3df36a1 event UNREGISTER
[   48.683028] linux_if_notifier: dev 00000000b3df36a1 event UNREGISTER
[   52.801682] linux_if_notifier: dev 000000006b794c44 event UNREGISTER
[   52.801956] linux_if_notifier: dev 000000006b794c44 event REGISTER
[   52.801972] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   52.802030] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   52.827962] linux_if_notifier: dev 000000006b794c44 event UNREGISTER
[   52.828018] linux_if_notifier: dev 000000006b794c44 event UNREGISTER
[   57.974594] cgroup: cgroup: disabling cgroup2 socket matching due to net_prio or net_cls activation
[   57.999789] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   58.003545] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   58.161693] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   58.618324] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   59.179193] tun: Universal TUN/TAP device driver, 1.6
[   59.179821] linux_if_notifier: dev 000000005a0a064f event UNREGISTER
[   59.179989] linux_if_notifier: dev 000000005a0a064f event REGISTER
[   59.194303] linux_if_notifier: dev 000000005a0a064f event UNREGISTER
[   59.194435] linux_if_notifier: dev 000000005a0a064f event UNREGISTER
[   59.259446] Config Hugepage vmem 00000000f01948b1 psize 1073741824 mem_sz 20185088 path /dev/hugepages/bridge
[   59.263128] Pinned huge page uspace_vmem 00000000f01948b1 start_page_addr 0000000005a5d439 num 4k pages 4928 mem_size 20185088 file_path /dev/hugepages/bridge
[   59.263130] Config Hugepage vmem 000000006ffa70c9 psize 1073741824 mem_sz 161218560 path /dev/hugepages/flow
[   59.291301] Pinned huge page uspace_vmem 000000006ffa70c9 start_page_addr 000000001b63c890 num 4k pages 39360 mem_size 161218560 file_path /dev/hugepages/flow
[   59.327011] contrail-nodemg[7742]: segfault at 7fb72c0d9802 ip 00007fb72b2638c0 sp 00007ffc7a6de688 error 7 in libc-2.28.so[7fb72b21c000+1bc000]
[   59.327018] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[   59.327236] nova-compute[7781]: segfault at 80 ip 00007fd23230471d sp 00007ffedde16a70 error 4 in libpython3.6m.so.1.0[7fd232168000+2ab000]
[   59.327241] Code: 8b 35 87 28 31 00 8b 7e 20 8d 57 01 89 56 20 41 39 16 0f 8c 93 08 00 00 49 39 cb 0f 84 ec 05 00 00 49 39 db 0f 84 13 05 00 00 <49> 8b 8b 80 00 00 00 48 85 c9 0f 84 4d 08 00 00 48 85 ed 0f 88 a3
[   59.330053] tuned[8456]: segfault at 0 ip 00007f8c02e24b7b sp 00007f8be2eee2e0 error 4 in libpython3.6m.so.1.0[7f8c02d37000+2ab000]
[   59.330064] Code: 41 8d 5c 24 ff 4c 8d 05 03 84 40 00 c1 eb 03 8d 2c 1b 49 8b 34 e8 48 8b 7e 10 48 39 fe 0f 84 bc 01 00 00 4c 8b 56 08 83 06 01 <49> 8b 02 48 89 46 08 48 85 c0 74 39 45 85 ed 75 64 48 8b 74 24 68
[   59.330207] traps: yum[9080] general protection fault ip:7efe46339780 sp:7ffda54025b8 error:0 in libc-2.28.so[7efe462eb000+1bc000]
[   59.336830] contrail-versio[9076]: segfault at 0 ip 00007f44fc5db3f0 sp 00007ffe44d17f18 error 6 in libc-2.28.so[7f44fc58c000+1bc000]
[   59.336836] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[   59.337423] tail[9077]: segfault at 0 ip 00007ff0f47bc350 sp 00007ffca43fe8b8 error 6 in libc-2.28.so[7ff0f476b000+1bc000]
[   59.337428] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[   59.337453] sh[9075]: segfault at 0 ip 0000000000000000 sp 00007ffd4de8c698 error 14 in bash[55df1267c000+108000]
[   59.337456] Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
[   59.338664] contrail-vroute[9025]: segfault at 0 ip 000000000093e050 sp 00007fff7b23ae58 error 6 in contrail-vrouter-agent[400000+1a4f000]
[   59.338668] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[   59.338729] entrypoint.sh[7836]: segfault at 0 ip 00007fdb6cbcfd50 sp 00007fffbc949ff8 error 6 in libc-2.28.so[7fdb6cb81000+1bc000]
[   59.338735] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[   59.342797] BUG: unable to handle kernel NULL pointer dereference at 0000000000000050
[   59.348916] ------------[ cut here ]------------
[   59.348960] tee[8486]: segfault at 0 ip 00007f276de04350 sp 00007ffd9df3c638 error 6 in libc-2.28.so[7f276ddb3000+1bc000]
[   59.348965] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[   59.350694] PGD 0 P4D 0 
[   59.355356] kernel BUG at include/linux/fs.h:2943!
[   59.366398] Oops: 0000 [#1] SMP NOPTI
[   59.396401] CPU: 18 PID: 8456 Comm: tuned Kdump: loaded Tainted: G          IOE    --------- -  - 4.18.0-305.30.1.el8_4.x86_64 #1
[   59.408138] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 07/16/2020
[   59.416728] RIP: 0010:filemap_fault+0x47/0xa20
[   59.421204] Code: 04 25 28 00 00 00 48 89 84 24 a0 00 00 00 31 c0 48 8b 07 48 8b 80 a0 00 00 00 4c 8b a0 f0 00 00 00 48 89 44 24 08 4d 8b 34 24 <49> 8b 46 50 48 05 ff 0f 00 00 48 c1 e8 0c 49 39 c5 0f 83 dd 05 00
[   59.440118] RSP: 0000:ffffbece6331b838 EFLAGS: 00010246
[   59.445379] RAX: ffff9d703dfb5100 RBX: ffffbece6331b998 RCX: 0000000000000000
[   59.452564] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffffbece6331b998
[   59.459750] RBP: ffff9d8fbdb77b40 R08: 0000000000000000 R09: 0000000001db8093
[   59.466936] R10: 0000000000000001 R11: 0000000000000100 R12: ffff9d8fbdb77e00
[   59.474122] R13: 00000000000002b5 R14: 0000000000000000 R15: 0000000000000000
[   59.481307] FS:  00007f8be2eef700(0000) GS:ffff9d707fa80000(0000) knlGS:0000000000000000
[   59.489456] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   59.495243] CR2: 0000000000000050 CR3: 0000003d7eafe005 CR4: 00000000007706e0
[   59.502429] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   59.509614] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   59.516800] PKRU: 55555554
[   59.519524] Call Trace:
[   59.521991]  ? _cond_resched+0x15/0x30
[   59.526376]  __xfs_filemap_fault+0x6d/0x200 [xfs]
[   59.531652]  __do_fault+0x36/0xd0
[   59.535515]  __handle_mm_fault+0x3d7/0xca0
[   59.540200]  ? xfs_iunlock+0xf3/0x100 [xfs]
[   59.544925]  handle_mm_fault+0xc2/0x1d0
[   59.549296]  __get_user_pages+0x2c4/0x840
[   59.553836]  get_dump_page+0x49/0x80
[   59.557934]  elf_core_dump+0x8cf/0xa50
[   59.562201]  do_coredump+0x73b/0xf4e
[   59.566285]  ? sched_clock_cpu+0xc/0xb0
[   59.570627]  ? up+0x12/0x50
[   59.573912]  get_signal+0x14f/0x870
[   59.577888]  ? __send_signal+0x343/0x4a0
[   59.582295]  ? page_fault+0x8/0x30
[   59.586173]  do_signal+0x36/0x660
[   59.589953]  ? force_sig_info+0xc7/0xe0
[   59.594253]  ? force_sig_fault+0x59/0x80
[   59.598633]  ? page_fault+0x8/0x30
[   59.602485]  exit_to_usermode_loop+0x89/0xf0
[   59.607206]  prepare_exit_to_usermode+0x9b/0xa0
[   59.612178]  retint_user+0x8/0x8
[   59.615825] RIP: 0033:0x7f8c02e24b7b
[   59.619813] Code: 41 8d 5c 24 ff 4c 8d 05 03 84 40 00 c1 eb 03 8d 2c 1b 49 8b 34 e8 48 8b 7e 10 48 39 fe 0f 84 bc 01 00 00 4c 8b 56 08 83 06 01 <49> 8b 02 48 89 46 08 48 85 c0 74 39 45 85 ed 75 64 48 8b 74 24 68
[   59.639498] RSP: 002b:00007f8be2eee2e0 EFLAGS: 00010202
[   59.645151] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 00007f8c02f53aac
[   59.652730] RDX: 000000000000001c RSI: 00007f8bec5f4000 RDI: 0000000000000000
[   59.660308] RBP: 0000000000000006 R08: 00007f8c0322cf60 R09: 00007f8beb3ad340
[   59.667877] R10: 0000000000000000 R11: 00007f8c0326bee8 R12: 000000000000001c
[   59.675434] R13: 0000000000000000 R14: 000000000000001c R15: 0000000000000001
[   59.682981] Modules linked in: tun overlay vrouter(OE) 8021q garp mrp bonding nf_log_ipv4 nf_log_ipv6 nf_log_common nft_chain_nat nft_limit ipt_MASQUERADE nft_counter xt_LOG xt_limit xt_multiport xt_comment nf_nat xt_state xt_conntrack xt_addrtype nft_compat nf_tables nfnetlink vfat fat dm_service_time dm_multipath dm_mod intel_rapl_msr intel_rapl_common isst_if_common rpcrdma sunrpc rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm ib_cm nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass ipmi_ssif crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl intel_cstate i40iw mlx5_ib ib_uverbs ib_core intel_uncore pcspkr joydev acpi_ipmi hpwdt hpilo lpc_ich mei_me ipmi_si mei ioatdma ipmi_devintf dca wmi ipmi_msghandler acpi_tad acpi_power_meter ip_tables xfs sd_mod t10_pi sg mlx5_core mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm i40e ahci libahci mlxfw
[   59.683015]  pci_hyperv_intf libata tls tg3 i2c_algo_bit nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc32c_intel fuse br_netfilter bridge stp llc
[   59.789014] CR2: 0000000000000050