RHEL7: Kernel panic at cgroup_is_descendant+0x1d

Solution Verified - Updated -

Issue

  • After frequent oom, kernel crashed with below logs:
[12581201.985136] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[12581201.985655] [ 9720]     0  9720    44153      389      47      140           985 su
[12581201.985665] [ 9827]     0  9827    28811      343      15       67           985 bash
[12581201.985673] [ 9856]     0  9856 12611759   909872    5224  1074492           985 java
[12581201.985735] Memory cgroup out of memory: Kill process 12796 (java) score 1933 or sacrifice child
[12581201.987743] Killed process 9856 (java), UID 0, total-vm:50447036kB, anon-rss:3628704kB, file-rss:10912kB, shmem-rss:0kB
[12609882.427338] BUG: unable to handle kernel paging request at 0000001d00008040
[12609882.529072] IP: [<ffffffffb7126e1d>] cgroup_is_descendant+0x1d/0x40
[12609882.607703] PGD 0 
[12609882.607803] Oops: 0000 [#1] SMP 
[12609882.620181] Modules linked in: nfsv3 binfmt_misc tcp_diag inet_diag xt_set xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ip_set ipip tunnel4 ip_tunnel veth xt_statistic xt_nat xt_recent xt_comment xt_mark fuse nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables overlay(T) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache iptable_filter snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm snd_timer joydev snd sg virtio_rng soundcore pcspkr i2c_piix4 virtio_balloon parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace
[12609882.621025]  sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common virtio_net net_failover virtio_console virtio_scsi failover ata_generic pata_acpi bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci drm_panel_orientation_quirks virtio_ring serio_raw virtio floppy dm_mirror dm_region_hash dm_log dm_mod
[12609882.621363] CPU: 14 PID: 1 Comm: systemd Kdump: loaded Tainted: G               ------------ T 3.10.0-1160.el7.x86_64 #1
[12609882.621448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
[12609882.621919] task: ffff8a8043b70000 ti: ffff8a804337c000 task.ti: ffff8a804337c000
[12609882.621989] RIP: 0010:[<ffffffffb7126e1d>]  [<ffffffffb7126e1d>] cgroup_is_descendant+0x1d/0x40
[12609882.622073] RSP: 0018:ffff8a804337fcc0  EFLAGS: 00010286
[12609882.622135] RAX: 0000000000000000 RBX: ffffffffb7d5de80 RCX: 0000000000000020
[12609882.622202] RDX: ffff8a925f83c400 RSI: ffff8a803c791600 RDI: 0000001d00008000
[12609882.622269] RBP: ffff8a804337fcc0 R08: ffffffffb7d5de80 R09: 0000000000000010
[12609882.622344] R10: d5848d28d25d7b40 R11: 00000000000003ff R12: 00000000000181a0
[12609882.622432] R13: ffff8a9ebf2181a0 R14: 0000000000000010 R15: 0000000000000010
[12609882.622503] FS:  00007fde6652f940(0000) GS:ffff8a9ebf180000(0000) knlGS:0000000000000000
[12609882.622575] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12609882.622638] CR2: 00000000022a600e CR3: 000000015db54000 CR4: 00000000000006e0
[12609882.631524] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[12609882.631603] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[12609882.631672] Call Trace:
[12609882.631764]  [<ffffffffb723e55c>] __mem_cgroup_same_or_subtree+0x2c/0x40
[12609882.631839]  [<ffffffffb723e60b>] drain_all_stock+0x9b/0x180
[12609882.639844]  [<ffffffffb723e741>] mem_cgroup_reparent_charges+0x51/0x3c0
[12609882.639947]  [<ffffffffb72668b8>] ? select_collect+0x68/0x120
[12609882.640014]  [<ffffffffb72686a3>] ? d_walk+0xf3/0x310
[12609882.640077]  [<ffffffffb723ec64>] mem_cgroup_css_offline+0x84/0x170
[12609882.640150]  [<ffffffffb712ae8a>] cgroup_destroy_locked+0xea/0x370
[12609882.640219]  [<ffffffffb712b132>] cgroup_rmdir+0x22/0x40
[12609882.640286]  [<ffffffffb725b4ec>] vfs_rmdir+0xdc/0x150
[12609882.640349]  [<ffffffffb7260171>] do_rmdir+0x1f1/0x220
[12609882.640433]  [<ffffffffb7793ed5>] ? system_call_after_swapgs+0xa2/0x13a
[12609882.640501]  [<ffffffffb7793ec9>] ? system_call_after_swapgs+0x96/0x13a
[12609882.640570]  [<ffffffffb7793ed5>] ? system_call_after_swapgs+0xa2/0x13a
[12609882.640639]  [<ffffffffb7793ec9>] ? system_call_after_swapgs+0x96/0x13a
[12609882.640709]  [<ffffffffb7793ed5>] ? system_call_after_swapgs+0xa2/0x13a
[12609882.640785]  [<ffffffffb7793ec9>] ? system_call_after_swapgs+0x96/0x13a
[12609882.640857]  [<ffffffffb72613a6>] SyS_rmdir+0x16/0x20
[12609882.650724]  [<ffffffffb7793f92>] system_call_fastpath+0x25/0x2a
[12609882.651794]  [<ffffffffb7793ed5>] ? system_call_after_swapgs+0xa2/0x13a
[12609882.652842] Code: f7 ff 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 55 48 85 ff 48 89 e5 74 18 48 39 f7 75 0a eb 1b 0f 1f 00 48 39 fe 74 13 <48> 8b 7f 40 48 85 ff 75 f2 31 c0 5d c3 66 0f 1f 44 00 00 b8 01 
[12609882.655163] RIP  [<ffffffffb7126e1d>] cgroup_is_descendant+0x1d/0x40
[12609882.656235]  RSP <ffff8a804337fcc0>
[12609882.657318] CR2: 0000001d00008040

Environment

  • Red Hat Enterprise Linux 7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content