Kernel panic in gfs2_inplace_reserve when processing link system call
Environment
- Red Hat Enterprise Linux Server 6.4 (from
kernel-2.6.32-358.el6.x86_64up tokernel-2.6.32-358.13.1.el6.x86_64) - Red Hat Enterprise Linux Server 6.5 (kernel versions preceding
kernel-2.6.32-431.el6) - Global File System 2 (GFS2) filesystem
Issue
- kernel panicking in
gfs2_inplace_reserve()because of NULL pointer dereference - the message buffer messages show the following:
BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
IP: [<ffffffffa05d86ef>] gfs2_inplace_reserve+0x54f/0x7e0 [gfs2]
PGD 0
Oops: 0002 [#1] SMP
last sysfs file: /sys/devices/pci0000:80/0000:80:03.0/0000:90:00.0/host4/rport-4:0-2/target4:0:0/4:0:0:2/state
CPU 8
...
Pid: 13087, comm: sshd Not tainted 2.6.32-358.el6.x86_64 #1 HP ProLiant DL580 G7
RIP: 0010:[<ffffffffa05d86ef>] [<ffffffffa05d86ef>] gfs2_inplace_reserve+0x54f/0x7e0 [gfs2]
RSP: 0018:ffff8810156f3bb8 EFLAGS: 00010287
RAX: ffff88082504ff28 RBX: 0000000000000001 RCX: 0000000000faf667
RDX: 0000000000faf667 RSI: 0000000000fac85c RDI: ffff8808337ea440
RBP: ffff8810156f3c78 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000001000 R11: 0000000000000000 R12: ffff880fee76f200
R13: 0000000000000000 R14: ffff88101b144be0 R15: ffff88101b248e30
FS: 00007fddce0fb7c0(0000) GS:ffff88085c400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000060 CR3: 0000001015415000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process sshd (pid: 13087, threadinfo ffff8810156f2000, task ffff8810313f9540)
Stack:
ffff88101b248e30 ffffffffa05b32c0 ffff8810156f3bf8 ffff88101b144be0
<d> ffff88101b248e30 00000115a05b32c0 ffff8810156f3c70 ffff8808337ea000
<d> ffff8810156f3c58 ffffffffa05b488f ffff88082504ff28 0000000000000000
Call Trace:
[<ffffffffa05b32c0>] ? gfs2_dirent_find_space+0x0/0x50 [gfs2]
[<ffffffffa05b488f>] ? gfs2_dirent_search+0xff/0x1a0 [gfs2]
[<ffffffffa05cef07>] gfs2_link+0x217/0x2f0 [gfs2]
[<ffffffffa05ced72>] ? gfs2_link+0x82/0x2f0 [gfs2]
[<ffffffffa05ced89>] ? gfs2_link+0x99/0x2f0 [gfs2]
[<ffffffff8118f442>] vfs_link+0x102/0x170
[<ffffffff8118e76a>] ? lookup_hash+0x3a/0x50
[<ffffffff81192704>] sys_linkat+0x114/0x150
[<ffffffff8119275e>] sys_link+0x1e/0x20
[<ffffffffa065ad1a>] do_linux_dazuko_sys_link+0x13a/0x360 [dazuko]
[<ffffffff81192740>] ? sys_link+0x0/0x20
[<ffffffffa065af75>] linux_dazuko_sys_link+0x15/0x20 [dazuko]
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Resolution
- If you insist on staying on RHEL 6 Update 4, update to
kernel-2.6.32-358.14.1.el6or later. The fix is part of the Errata RHSA-2013-1051-1 - Otherwise for RHEL 6 Update 5 or later, update to
kernel-2.6.32-431.el6or later. The fix is included in the Errata RHSA-2013-1645-2
Root Cause
- A missing reservation structure caused the link code to reference an invalid pointer.
- The NULL
gfs2_inode->i_resis a result ofgfs2_rs_alloc()failing and the code path continuing on in an attempt to allocate space without that field set
NOTE: Technically kernel-2.6.32-358.2.1.el6 and later should be sufficient to fix this problem, but the kernels from kernel-2.6.32-358.2.1.el6 up until (but not including) kernel-2.6.32-358.14.el6 still contain a similar problem. For those reasons we recommend kernel-2.6.32-358.14.1.el6 and later as the solution.
Diagnostic Steps
- The trigger is running a
link()system call in GFS2 environment - The kernel panic stack trace is
crash> bt
PID: 13087 TASK: ffff8810313f9540 CPU: 8 COMMAND: "sshd"
#0 [ffff8810156f3780] machine_kexec at ffffffff81035b7b
#1 [ffff8810156f37e0] crash_kexec at ffffffff810c0db2
#2 [ffff8810156f38b0] oops_end at ffffffff815111d0
#3 [ffff8810156f38e0] no_context at ffffffff81046bfb
#4 [ffff8810156f3930] __bad_area_nosemaphore at ffffffff81046e85
#5 [ffff8810156f3980] bad_area at ffffffff81046fae
#6 [ffff8810156f39b0] __do_page_fault at ffffffff81047760
#7 [ffff8810156f3ad0] do_page_fault at ffffffff8151311e
#8 [ffff8810156f3b00] page_fault at ffffffff815104d5
[exception RIP: gfs2_inplace_reserve+0x54f]
RIP: ffffffffa05d86ef RSP: ffff8810156f3bb8 RFLAGS: 00010287
RAX: ffff88082504ff28 RBX: 0000000000000001 RCX: 0000000000faf667
RDX: 0000000000faf667 RSI: 0000000000fac85c RDI: ffff8808337ea440
RBP: ffff8810156f3c78 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000001000 R11: 0000000000000000 R12: ffff880fee76f200
R13: 0000000000000000 R14: ffff88101b144be0 R15: ffff88101b248e30
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff8810156f3bb0] gfs2_inplace_reserve at ffffffffa05d86eb [gfs2]
#10 [ffff8810156f3c80] gfs2_link at ffffffffa05cef07 [gfs2]
#11 [ffff8810156f3d60] vfs_link at ffffffff8118f442
#12 [ffff8810156f3db0] sys_linkat at ffffffff81192704
#13 [ffff8810156f3eb0] sys_link at ffffffff8119275e
#14 [ffff8810156f3ec0] do_linux_dazuko_sys_link at ffffffffa065ad1a [dazuko]
#15 [ffff8810156f3f70] linux_dazuko_sys_link at ffffffffa065af75 [dazuko]
#16 [ffff8810156f3f80] system_call_fastpath at ffffffff8100b072
The exception is hit in gfs2_inplace_reserve()
crash> dis gfs2_inplace_reserve
...
0xffffffffa05d86e6 <gfs2_inplace_reserve+0x546>: callq 0xffffffffa05d5d60 <gfs2_blk2rgrpd>
0xffffffffa05d86eb <gfs2_inplace_reserve+0x54b>: mov %rax,-0x70(%rbp)
0xffffffffa05d86ef <gfs2_inplace_reserve+0x54f>: mov %rax,0x60(%r13) <-- exception here
-
Dereferencing a NULL pointer in register %r13 is the reason for the exception.
R13: 0000000000000000 -
The argument
struct gfs2_inode *ipis passed from the caller routinegfs2_link()
0xffffffffa05ceef2 <gfs2_link+0x202>: mov -0xc0(%rbp),%rax
0xffffffffa05ceef9 <gfs2_link+0x209>: mov %r14,%rdi <-- argument initialized here
0xffffffffa05ceefc <gfs2_link+0x20c>: mov 0x14c(%rax),%esi
0xffffffffa05cef02 <gfs2_link+0x212>: callq 0xffffffffa05d81a0 <gfs2_inplace_reserve>
The register r14 is saved on stack by the callee:
0xffffffffa05d81a0 <gfs2_inplace_reserve>: push %rbp
0xffffffffa05d81a1 <gfs2_inplace_reserve+0x1>: mov %rsp,%rbp
0xffffffffa05d81a4 <gfs2_inplace_reserve+0x4>: push %r15
0xffffffffa05d81a6 <gfs2_inplace_reserve+0x6>: push %r14 <-- saved on stack here
0xffffffffa05d81a8 <gfs2_inplace_reserve+0x8>: push %r13
This allows us to figure out the address of the struct gfs2_inode and to inspect its content
crash> bt -f
...
#9 [ffff8810156f3bb0] gfs2_inplace_reserve at ffffffffa05d86eb [gfs2]
ffff8810156f3bb8: ffff88101b248e30 ffffffffa05b32c0
ffff8810156f3bc8: ffff8810156f3bf8 ffff88101b144be0
ffff8810156f3bd8: ffff88101b248e30 00000115a05b32c0
ffff8810156f3be8: ffff8810156f3c70 ffff8808337ea000
ffff8810156f3bf8: ffff8810156f3c58 ffffffffa05b488f
ffff8810156f3c08: ffff88082504ff28 0000000000000000
ffff8810156f3c18: ffff88101b144be0 ffff880fee609130
ffff8810156f3c28: ffff88101b144be0 00000000ffffffe1
ffff8810156f3c38: ffff880fee76f200 ffff8810156f3ca8
ffff8810156f3c48: ffff88101b144be0 0000000000000001
ffff8810156f3c58: ffff880fee76f200 ffff8810156f3ca8
ffff8810156f3c68: ffff88101b144be0 ffff88101b248e30
^^^^^^^^^^^^^^^^
struct gfs2_inode
ffff8810156f3c78: ffff8810156f3d58 ffffffffa05cef07
#10 [ffff8810156f3c80] gfs2_link at ffffffffa05cef07 [gfs2]
...
crash> struct gfs2_inode 0xffff88101b144be0
struct gfs2_inode {
i_inode = {
...
i_ino = 0x35ae7,
i_count = {
counter = 0x1
},
i_nlink = 0x2,
i_uid = 0x1f9,
i_gid = 0x1f9,
i_rdev = 0x0,
i_version = 0x0,
i_size = 0x1000,
...
},
i_gh = {
gh_list = {
next = 0xffff88101b144ea0,
prev = 0xffff88101b144ea0
},
gh_gl = 0xffff88101b144eb0,
gh_owner_pid = 0xffff88101b144eb0,
gh_state = 0x0,
gh_flags = 0x0,
gh_error = 0x81620fc0,
gh_iflags = 0xffff880833dd0c00,
gh_ip = 0x0
},
i_res = 0x0, <--- NULL, reason for the panic
i_rgd = 0x0,
i_goal = 0xfac85c,
i_rw_mutex = {
count = 0x0,
wait_lock = {
raw_lock = {
slock = 0x0
}
},
wait_list = {
next = 0xffff88101b144f08,
prev = 0xffff88101b144f08
}
},
...
}
The reason for panic is dereferencing a NULL i_res pointer here:
int gfs2_inplace_reserve(struct gfs2_inode *ip, const struct gfs2_alloc_parms *a
{
struct gfs2_sbd *sdp = GFS2_SB(&ip->i_inode);
struct gfs2_rgrpd *begin = NULL;
struct gfs2_blkreserv *rs = &ip->i_res;
int error, rg_locked, flags = LM_FLAG_TRY;
u64 last_unlinked = NO_BLOCK;
int loops = 0;
u32 skip = 0;
if (gfs2_rs_active(rs)) {
begin = rs->rs_rbm.rgd;
} else {
if (ip->i_rgd && rgrp_contains_block(ip->i_rgd, ip->i_goal)) {
rs->rs_rbm.rgd = begin = ip->i_rgd;
} else {
check_and_update_goal(ip);
rs->rs_rbm.rgd = begin = gfs2_blk2rgrpd(sdp, ip->i_goal, 1); <--- exception happens HERE
}
if (S_ISDIR(ip->i_inode.i_mode) && (ap->aflags & GFS2_AF_ORLOV))
skip = gfs2_orlov_skip(ip);
}
...
}
The struct gfs2_blkreserv "rs" (taken from ip->i_res) is NULL and is the cause of the panic.
Therefore the NULL pointer dereference is caused by the gfs2_inode->i_res being NULL as a result of an earlier gfs2_rs_alloc() failing and the code path continuing on in an attempt to allocate space without that field being set.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
