Vmware guest panics with NULL pointer dereference during cpu hot-add on low memory situation.

Solution Verified - Updated -

Issue

  • Vmware guest panics with NULL pointer dereference during cpu hot-add on low memory situation.
[   67.663790] smpboot: Booting Node 0 Processor 9 APIC 0x12
[   67.664264] Disabled fast string operations
[   67.665215] smpboot: CPU 9 Converting physical 18 to logical package 11
[   67.665321] Skipped synchronization checks as TSC is reliable.
[   67.666202] Will online and init hotplugged CPU: 9
[   67.678517] BUG: unable to handle kernel NULL pointer dereference at           (null)
[   67.678524] IP: [<ffffffffb479efab>] __list_add+0x1b/0xc0
[   67.678530] PGD 0
[   67.678532] Oops: 0000 [#1] SMP
[   67.678535] Modules linked in: binfmt_misc macsec vsock_diag tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter devlink ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac ppdev iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common
[   67.678568]  crc32c_intel ahci nfit drm libnvdimm ata_piix libahci serio_raw libata vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
[   67.678577] CPU: 9 PID: 1522 Comm: in:imjournal Kdump: loaded Tainted: G               ------------ T 3.10.0-1062.el7.x86_64 #1
[   67.678579] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[   67.678581] task: ffff91396a6c62a0 ti: ffff913965af0000 task.ti: ffff913965af0000
[   67.678583] RIP: 0010:[<ffffffffb479efab>]  [<ffffffffb479efab>] __list_add+0x1b/0xc0 
[   67.678586] RSP: 0000:ffff913965af34d8  EFLAGS: 00010246
[   67.678588] RAX: 00000000ffffffff RBX: ffff913965af3500 RCX: 0000000000000000
[   67.678589] RDX: ffff913975858050 RSI: 0000000000000000 RDI: ffff913965af3500
[   67.678591] RBP: ffff913965af34f0 R08: 0000000000000000 R09: 0000000000000002
[   67.678592] R10: ffffffffb515a260 R11: 0000000000000001 R12: ffff913975858050
[   67.678594] R13: 0000000000000000 R14: 00000000ffffffff R15: ffff913975858050
[   67.678596] FS:  00007fc7935f5700(0000) GS:ffff913975840000(0000) knlGS:0000000000000000
[   67.678598] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   67.678600] CR2: 00007f8ea23ce000 CR3: 0000000226b7e000 CR4: 00000000001607e0
[   67.678623] Call Trace:
[   67.678630]  [<ffffffffb4b7ded6>] __mutex_lock_slowpath+0xa6/0x1d0
[   67.678633]  [<ffffffffb4b7d2cf>] mutex_lock+0x1f/0x2f
[   67.678636]  [<ffffffffb45e4c7b>] get_swap_page+0x9b/0x1b0
[   67.678640]  [<ffffffffb4605699>] add_to_swap+0x19/0x80
[   67.678644]  [<ffffffffb45d081b>] shrink_page_list+0x69b/0xc30
[   67.678647]  [<ffffffffb45d13d6>] shrink_inactive_list+0x1c6/0x5d0
[   67.678650]  [<ffffffffb45d1ed5>] shrink_lruvec+0x385/0x740
[   67.678653]  [<ffffffffb45d2306>] shrink_zone+0x76/0x1a0
[   67.678656]  [<ffffffffb45d27f0>] do_try_to_free_pages+0xf0/0x520
[   67.678659]  [<ffffffffb45d2d1c>] try_to_free_pages+0xfc/0x180
[   67.678663]  [<ffffffffb4b7459d>] __alloc_pages_slowpath+0x457/0x724
[   67.678666]  [<ffffffffb45c6b84>] __alloc_pages_nodemask+0x404/0x420
[   67.678670]  [<ffffffffb4618105>] alloc_pages_vma+0xb5/0x200
[   67.678672]  [<ffffffffb4605ac5>] __read_swap_cache_async+0x115/0x190
[   67.678675]  [<ffffffffb4605b66>] read_swap_cache_async+0x26/0x60
[   67.678677]  [<ffffffffb4605d4b>] swapin_readahead+0x1ab/0x210
[   67.678680]  [<ffffffffb4786722>] ? radix_tree_lookup_slot+0x22/0x50
[   67.678683]  [<ffffffffb45bb4ee>] ? __find_get_page+0x1e/0xa0
[   67.678686]  [<ffffffffb45efe36>] handle_pte_fault+0xd66/0xe20
[   67.678689]  [<ffffffffb45f200d>] handle_mm_fault+0x39d/0x9b0
[   67.678693]  [<ffffffffb4b86633>] __do_page_fault+0x213/0x500
[   67.678695]  [<ffffffffb4b86955>] do_page_fault+0x35/0x90
[   67.678698]  [<ffffffffb4b82768>] page_fault+0x28/0x30
[   67.678700] Code: ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 d4 53 4c 8b 42 08 48 89 fb 49 39 f0 75 2a <4d> 8b 45 00 4d 39 c4 75 68 4c 39 e3 74 3e 4c 39 eb 74 39 49 89
[   67.678720] RIP  [<ffffffffb479efab>] __list_add+0x1b/0xc0
[   67.678722]  RSP <ffff913965af34d8>
[   67.678724] CR2: 0000000000000000
  • As per customer's wording who is able to reproduce this behavior, this happens when free memory is around ~110MB to ~120MB and hot-add is triggered during that situation.

Environment

  • Red Hat Enterprise Linux 7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content