OSP 16.2: kernel panic when tryinig to use hugepages

Solution Verified - Updated -

Issue

  • kernel panic with below logs:
[   39.723351] vrouter: loading out-of-tree module taints kernel.
[   39.723638] vrouter: module verification failed: signature and/or required key missing - tainting kernel
[   39.725789] vrouter version: {"build-info": [{"build-time": "2022-06-20 03:02:45.919367", "build-hostname": "contrail-nightly-tbf5h", "build-user": "contrail-builder", "build-version": "21.4"}]}
[   39.729090] linux_if_notifier: dev 0000000048dbe25d event REGISTER
[   39.729095] linux_if_notifier: dev 0000000048dbe25d event UNREGISTER
[   39.729096] linux_if_notifier: dev 00000000b2294c78 event REGISTER
[   39.729099] linux_if_notifier: dev 00000000b2294c78 event UNREGISTER
[   39.729100] linux_if_notifier: dev 00000000f83e4fc1 event REGISTER
[   39.729103] linux_if_notifier: dev 00000000f83e4fc1 event UNREGISTER
[   39.729104] linux_if_notifier: dev 00000000a3222b11 event REGISTER
[   39.729107] linux_if_notifier: dev 00000000a3222b11 event UNREGISTER
[   39.729108] linux_if_notifier: dev 00000000742ffc82 event REGISTER
[   39.729111] linux_if_notifier: dev 00000000742ffc82 event UNREGISTER
[   39.729112] linux_if_notifier: dev 00000000d53b8b31 event REGISTER
[   39.729115] linux_if_notifier: dev 00000000d53b8b31 event UNREGISTER
[   39.729115] linux_if_notifier: dev 000000005f264fae event REGISTER
[   39.729118] linux_if_notifier: dev 000000005f264fae event UNREGISTER
[   39.729119] linux_if_notifier: dev 0000000038caa1a6 event REGISTER
[   39.729122] linux_if_notifier: dev 0000000038caa1a6 event UNREGISTER
[   39.729123] linux_if_notifier: dev 00000000f7a21872 event REGISTER
[   39.729126] linux_if_notifier: dev 00000000f7a21872 event UNREGISTER
[   39.729127] linux_if_notifier: dev 00000000f62ec522 event REGISTER
[   39.729130] linux_if_notifier: dev 00000000f62ec522 event UNREGISTER
[   39.729130] linux_if_notifier: dev 0000000097410a0d event REGISTER
[   39.729133] linux_if_notifier: dev 0000000097410a0d event UNREGISTER
[   39.729134] linux_if_notifier: dev 00000000fccfb6a8 event REGISTER
[   39.729137] linux_if_notifier: dev 00000000fccfb6a8 event UNREGISTER
[   39.729138] linux_if_notifier: dev 00000000669bc5e7 event REGISTER
[   39.729141] linux_if_notifier: dev 00000000669bc5e7 event UNREGISTER
[   39.729141] linux_if_notifier: dev 00000000d9c171a4 event REGISTER
[   39.729144] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   39.729145] linux_if_notifier: dev 00000000708c867f event REGISTER
[   39.729148] linux_if_notifier: dev 00000000708c867f event UNREGISTER
[   39.729148] linux_if_notifier: dev 00000000e62ff6ea event REGISTER
[   39.729151] linux_if_notifier: dev 00000000e62ff6ea event UNREGISTER
[   39.729152] linux_if_notifier: dev 0000000082e5b053 event REGISTER
[   39.729155] linux_if_notifier: dev 0000000034e484db event REGISTER
[   39.729158] linux_if_notifier: dev 00000000c1fc6e3d event REGISTER
[   39.977529] linux_if_notifier: dev 0000000082e5b053 event UNREGISTER
[   39.977588] linux_if_notifier: dev 0000000082e5b053 event UNREGISTER
[   40.176831] linux_if_notifier: dev 00000000c1fc6e3d event UNREGISTER
[   40.176876] linux_if_notifier: dev 00000000c1fc6e3d event UNREGISTER
[   40.375398] linux_if_notifier: dev 0000000034e484db event UNREGISTER
[   40.375439] linux_if_notifier: dev 0000000034e484db event UNREGISTER
[   40.378814] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   40.378986] linux_if_notifier: dev 00000000de1e73eb event REGISTER
[   40.383079] Vrouter: vr_interface_get:3548 vif is NULL for vifr_idx: -1 
[   40.383125] Num phy interfaces 1
[   40.383716] Vrouter: vr_interface_get:3548 vif is NULL for vifr_idx: -1 
[   40.384584] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   40.384599] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   40.385463] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   40.385547] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   40.398649] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   40.398671] linux_if_notifier: dev 00000000de1e73eb event UNREGISTER
[   44.515793] linux_if_notifier: dev 00000000ab576300 event UNREGISTER
[   44.516085] linux_if_notifier: dev 00000000ab576300 event REGISTER
[   44.516125] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   44.516188] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   44.543390] linux_if_notifier: dev 00000000ab576300 event UNREGISTER
[   44.543456] linux_if_notifier: dev 00000000ab576300 event UNREGISTER
[   48.657259] linux_if_notifier: dev 00000000b3df36a1 event UNREGISTER
[   48.657403] linux_if_notifier: dev 00000000b3df36a1 event REGISTER
[   48.657411] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   48.657434] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   48.682969] linux_if_notifier: dev 00000000b3df36a1 event UNREGISTER
[   48.683028] linux_if_notifier: dev 00000000b3df36a1 event UNREGISTER
[   52.801682] linux_if_notifier: dev 000000006b794c44 event UNREGISTER
[   52.801956] linux_if_notifier: dev 000000006b794c44 event REGISTER
[   52.801972] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   52.802030] linux_if_notifier: dev 00000000d9c171a4 event UNREGISTER
[   52.827962] linux_if_notifier: dev 000000006b794c44 event UNREGISTER
[   52.828018] linux_if_notifier: dev 000000006b794c44 event UNREGISTER
[   57.974594] cgroup: cgroup: disabling cgroup2 socket matching due to net_prio or net_cls activation
[   57.999789] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   58.003545] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   58.161693] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   58.618324] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   59.179193] tun: Universal TUN/TAP device driver, 1.6
[   59.179821] linux_if_notifier: dev 000000005a0a064f event UNREGISTER
[   59.179989] linux_if_notifier: dev 000000005a0a064f event REGISTER
[   59.194303] linux_if_notifier: dev 000000005a0a064f event UNREGISTER
[   59.194435] linux_if_notifier: dev 000000005a0a064f event UNREGISTER
[   59.259446] Config Hugepage vmem 00000000f01948b1 psize 1073741824 mem_sz 20185088 path /dev/hugepages/bridge
[   59.263128] Pinned huge page uspace_vmem 00000000f01948b1 start_page_addr 0000000005a5d439 num 4k pages 4928 mem_size 20185088 file_path /dev/hugepages/bridge
[   59.263130] Config Hugepage vmem 000000006ffa70c9 psize 1073741824 mem_sz 161218560 path /dev/hugepages/flow
[   59.291301] Pinned huge page uspace_vmem 000000006ffa70c9 start_page_addr 000000001b63c890 num 4k pages 39360 mem_size 161218560 file_path /dev/hugepages/flow
[   59.327011] contrail-nodemg[7742]: segfault at 7fb72c0d9802 ip 00007fb72b2638c0 sp 00007ffc7a6de688 error 7 in libc-2.28.so[7fb72b21c000+1bc000]
[   59.327018] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[   59.327236] nova-compute[7781]: segfault at 80 ip 00007fd23230471d sp 00007ffedde16a70 error 4 in libpython3.6m.so.1.0[7fd232168000+2ab000]
[   59.327241] Code: 8b 35 87 28 31 00 8b 7e 20 8d 57 01 89 56 20 41 39 16 0f 8c 93 08 00 00 49 39 cb 0f 84 ec 05 00 00 49 39 db 0f 84 13 05 00 00 <49> 8b 8b 80 00 00 00 48 85 c9 0f 84 4d 08 00 00 48 85 ed 0f 88 a3
[   59.330053] tuned[8456]: segfault at 0 ip 00007f8c02e24b7b sp 00007f8be2eee2e0 error 4 in libpython3.6m.so.1.0[7f8c02d37000+2ab000]
[   59.330064] Code: 41 8d 5c 24 ff 4c 8d 05 03 84 40 00 c1 eb 03 8d 2c 1b 49 8b 34 e8 48 8b 7e 10 48 39 fe 0f 84 bc 01 00 00 4c 8b 56 08 83 06 01 <49> 8b 02 48 89 46 08 48 85 c0 74 39 45 85 ed 75 64 48 8b 74 24 68
[   59.330207] traps: yum[9080] general protection fault ip:7efe46339780 sp:7ffda54025b8 error:0 in libc-2.28.so[7efe462eb000+1bc000]
[   59.336830] contrail-versio[9076]: segfault at 0 ip 00007f44fc5db3f0 sp 00007ffe44d17f18 error 6 in libc-2.28.so[7f44fc58c000+1bc000]
[   59.336836] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[   59.337423] tail[9077]: segfault at 0 ip 00007ff0f47bc350 sp 00007ffca43fe8b8 error 6 in libc-2.28.so[7ff0f476b000+1bc000]
[   59.337428] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[   59.337453] sh[9075]: segfault at 0 ip 0000000000000000 sp 00007ffd4de8c698 error 14 in bash[55df1267c000+108000]
[   59.337456] Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
[   59.338664] contrail-vroute[9025]: segfault at 0 ip 000000000093e050 sp 00007fff7b23ae58 error 6 in contrail-vrouter-agent[400000+1a4f000]
[   59.338668] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[   59.338729] entrypoint.sh[7836]: segfault at 0 ip 00007fdb6cbcfd50 sp 00007fffbc949ff8 error 6 in libc-2.28.so[7fdb6cb81000+1bc000]
[   59.338735] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[   59.342797] BUG: unable to handle kernel NULL pointer dereference at 0000000000000050
[   59.348916] ------------[ cut here ]------------
[   59.348960] tee[8486]: segfault at 0 ip 00007f276de04350 sp 00007ffd9df3c638 error 6 in libc-2.28.so[7f276ddb3000+1bc000]
[   59.348965] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[   59.350694] PGD 0 P4D 0 
[   59.355356] kernel BUG at include/linux/fs.h:2943!
[   59.366398] Oops: 0000 [#1] SMP NOPTI
[   59.396401] CPU: 18 PID: 8456 Comm: tuned Kdump: loaded Tainted: G          IOE    --------- -  - 4.18.0-305.30.1.el8_4.x86_64 #1
[   59.408138] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 07/16/2020
[   59.416728] RIP: 0010:filemap_fault+0x47/0xa20
[   59.421204] Code: 04 25 28 00 00 00 48 89 84 24 a0 00 00 00 31 c0 48 8b 07 48 8b 80 a0 00 00 00 4c 8b a0 f0 00 00 00 48 89 44 24 08 4d 8b 34 24 <49> 8b 46 50 48 05 ff 0f 00 00 48 c1 e8 0c 49 39 c5 0f 83 dd 05 00
[   59.440118] RSP: 0000:ffffbece6331b838 EFLAGS: 00010246
[   59.445379] RAX: ffff9d703dfb5100 RBX: ffffbece6331b998 RCX: 0000000000000000
[   59.452564] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffffbece6331b998
[   59.459750] RBP: ffff9d8fbdb77b40 R08: 0000000000000000 R09: 0000000001db8093
[   59.466936] R10: 0000000000000001 R11: 0000000000000100 R12: ffff9d8fbdb77e00
[   59.474122] R13: 00000000000002b5 R14: 0000000000000000 R15: 0000000000000000
[   59.481307] FS:  00007f8be2eef700(0000) GS:ffff9d707fa80000(0000) knlGS:0000000000000000
[   59.489456] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   59.495243] CR2: 0000000000000050 CR3: 0000003d7eafe005 CR4: 00000000007706e0
[   59.502429] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   59.509614] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   59.516800] PKRU: 55555554
[   59.519524] Call Trace:
[   59.521991]  ? _cond_resched+0x15/0x30
[   59.526376]  __xfs_filemap_fault+0x6d/0x200 [xfs]
[   59.531652]  __do_fault+0x36/0xd0
[   59.535515]  __handle_mm_fault+0x3d7/0xca0
[   59.540200]  ? xfs_iunlock+0xf3/0x100 [xfs]
[   59.544925]  handle_mm_fault+0xc2/0x1d0
[   59.549296]  __get_user_pages+0x2c4/0x840
[   59.553836]  get_dump_page+0x49/0x80
[   59.557934]  elf_core_dump+0x8cf/0xa50
[   59.562201]  do_coredump+0x73b/0xf4e
[   59.566285]  ? sched_clock_cpu+0xc/0xb0
[   59.570627]  ? up+0x12/0x50
[   59.573912]  get_signal+0x14f/0x870
[   59.577888]  ? __send_signal+0x343/0x4a0
[   59.582295]  ? page_fault+0x8/0x30
[   59.586173]  do_signal+0x36/0x660
[   59.589953]  ? force_sig_info+0xc7/0xe0
[   59.594253]  ? force_sig_fault+0x59/0x80
[   59.598633]  ? page_fault+0x8/0x30
[   59.602485]  exit_to_usermode_loop+0x89/0xf0
[   59.607206]  prepare_exit_to_usermode+0x9b/0xa0
[   59.612178]  retint_user+0x8/0x8
[   59.615825] RIP: 0033:0x7f8c02e24b7b
[   59.619813] Code: 41 8d 5c 24 ff 4c 8d 05 03 84 40 00 c1 eb 03 8d 2c 1b 49 8b 34 e8 48 8b 7e 10 48 39 fe 0f 84 bc 01 00 00 4c 8b 56 08 83 06 01 <49> 8b 02 48 89 46 08 48 85 c0 74 39 45 85 ed 75 64 48 8b 74 24 68
[   59.639498] RSP: 002b:00007f8be2eee2e0 EFLAGS: 00010202
[   59.645151] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 00007f8c02f53aac
[   59.652730] RDX: 000000000000001c RSI: 00007f8bec5f4000 RDI: 0000000000000000
[   59.660308] RBP: 0000000000000006 R08: 00007f8c0322cf60 R09: 00007f8beb3ad340
[   59.667877] R10: 0000000000000000 R11: 00007f8c0326bee8 R12: 000000000000001c
[   59.675434] R13: 0000000000000000 R14: 000000000000001c R15: 0000000000000001
[   59.682981] Modules linked in: tun overlay vrouter(OE) 8021q garp mrp bonding nf_log_ipv4 nf_log_ipv6 nf_log_common nft_chain_nat nft_limit ipt_MASQUERADE nft_counter xt_LOG xt_limit xt_multiport xt_comment nf_nat xt_state xt_conntrack xt_addrtype nft_compat nf_tables nfnetlink vfat fat dm_service_time dm_multipath dm_mod intel_rapl_msr intel_rapl_common isst_if_common rpcrdma sunrpc rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm ib_cm nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass ipmi_ssif crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl intel_cstate i40iw mlx5_ib ib_uverbs ib_core intel_uncore pcspkr joydev acpi_ipmi hpwdt hpilo lpc_ich mei_me ipmi_si mei ioatdma ipmi_devintf dca wmi ipmi_msghandler acpi_tad acpi_power_meter ip_tables xfs sd_mod t10_pi sg mlx5_core mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm i40e ahci libahci mlxfw
[   59.683015]  pci_hyperv_intf libata tls tg3 i2c_algo_bit nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc32c_intel fuse br_netfilter bridge stp llc
[   59.789014] CR2: 0000000000000050

Environment

  • Red Hat Openstack Platform 16.2
  • kernel-4.18.0-305.30.1.el8_4
  • 3rd party module vrouter 21.4
vrouter version: {"build-info": [{"build-time": "2022-06-20 03:02:45.919367", "build-hostname": "contrail-nightly-tbf5h", "build-user": "contrail-builder", "build-version": "21.4"}]}

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content