rhel7.7 -rhel7.9: panic with "BUG: unable to handle kernel paging request at"

Solution Verified - Updated -

Issue

  • The memcg_params field of kmem_cache struct contains an old slab address that is to small for the current size of memcg_limited_groups_array_size.
  • On affected kernels, we are hitting a panic in mm/memcontrol.c.
       PANIC: "BUG: unable to handle kernel paging request at 000000010000024e"

crash> bt
PID: 30317  TASK: ffff9281db27c1c0  CPU: 3   COMMAND: "kworker/u12:3"
…
 #9 [ffff928193a9fc00] page_fault at ffffffff8dd84778
    [exception RIP: memcg_kmem_destroy_cache+0x1b]
    RIP: ffffffff8d83988b  RSP: ffff928193a9fcb0  RFLAGS: 00010206
    RAX: 000000010000024e  RBX: ffff92841d7ed800  RCX: ffff928193a9ffd8
    RDX: 0000000000000000  RSI: ffff9285bfcdfd18  RDI: ffff92841d7ed800
    RBP: ffff928193a9fcc0   R8: fffff420d3e7b560   R9: 0000000000000000
    R10: ffff9285bfbd5fe0  R11: ffff9284a5b290c0  R12: ffff92859837b600
    R13: 000000000000000f  R14: 0000000000000000  R15: 0000000080000003
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff928193a9fcc8] __kmem_cache_destroy_memcg_children at ffffffff8d83ee05
#11 [ffff928193a9fcf8] kmem_cache_destroy at ffffffff8d7e3aa4
#12 [ffff928193a9fd18] nf_conntrack_cleanup_net_list at ffffffffc0897a3b [nf_conntrack]
#13 [ffff928193a9fd60] nf_conntrack_pernet_exit at ffffffffc089886d [nf_conntrack]
#14 [ffff928193a9fd88] ops_exit_list at ffffffff8dc43779
#15 [ffff928193a9fdb8] cleanup_net at ffffffff8dc44800
#16 [ffff928193a9fe20] process_one_work at ffffffff8d6be21f
…

crash> dis -lr memcg_kmem_destroy_cache+0x1b
…
/usr/src/debug/kernel-3.10.0-1062.9.1.el7/linux-3.10.0-1062.9.1.el7.x86_64/mm/memcontrol.c: 3323
0xffffffff8d83987c <memcg_kmem_destroy_cache+0xc>:      mov    0xb8(%rdi),%rax
…
/usr/src/debug/kernel-3.10.0-1062.9.1.el7/linux-3.10.0-1062.9.1.el7.x86_64/mm/slab.h: 129
…
0xffffffff8d83988b <memcg_kmem_destroy_cache+0x1b>:     cmpb   $0x0,(%rax)

crash> whatis memcg_kmem_destroy_cache
void memcg_kmem_destroy_cache(struct kmem_cache *);

crash> struct -o kmem_cache | grep b8
    [0xb8] struct memcg_cache_params *memcg_params;

crash> struct kmem_cache.memcg_params ffff92841d7ed800
  memcg_params = 0x10000024e

crash> kmem ffff92841d7ed800
CACHE             OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE  NAME
ffff9280ffc01900      128       7925     10272    321     4k  kmalloc-128
  SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
  fffff420d275fb40  ffff92841d7ed000     0     32         22    10
  FREE / [ALLOCATED]
   ffff92841d7ed800
…

Environment

  • Red Hat Enterprise Linux (RHEL) 7.7, 7.8 and 7.9
  • kernel panic

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content