Server freezes or panics after executing db2stop command showing freeary() frame in the call trace
Issue
When the db2stop
command is executed, the servers either hangs experiencing soft lockups, or panics.
An example of a soft lockup message is given below:
kernel: BUG: soft lockup - CPU#0 stuck for 67s! [db2stop2:10791]
kernel: Pid: 10791, comm: db2stop2 Not tainted 2.6.32-220.4.2.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Plat
form
kernel: RIP: 0010:[<ffffffff814ef2ae>] [<ffffffff814ef2ae>] _spin_lock+0x1e/0x30
kernel: RSP: 0018:ffff880334299e78 EFLAGS: 00000206
kernel: RAX: 0000000000000000 RBX: ffff880334299e78 RCX: dead000000200200
kernel: RDX: 00000000000006e1 RSI: ffff8800282112c0 RDI: ffff880325045de4
kernel: RBP: ffffffff8100bc0e R08: ffff88031d57f8e8 R09: 0000000000000000
kernel: R10: 00000000000006e1 R11: 0000000000000002 R12: ffffffff81ad8720
kernel: R13: ffff88031b045bd0 R14: 0000000000000000 R15: ffff880334299dd8
kernel: FS: 00007fe7e3e50720(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 00007fe7eb0f2070 CR3: 0000000321662000 CR4: 00000000000006f0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
kernel: Process db2stop2 (pid: 10791, threadinfo ffff880334298000, task ffff8803349b6100)
kernel: Stack:
kernel: ffff880334299eb8 ffffffff811fd295 0000000000000000 0000000000000000
kernel: <0> ffff88031b045bd0 ffffffff81ad8720 ffff880334299ec8 0000000000000000
kernel: <0> ffff880334299f68 ffffffff811fd73f ffff88031a255cc0 ffff880037578380
kernel: Call Trace:
kernel: [<ffffffff811fd295>] ? freeary+0x65/0x150
kernel: [<ffffffff811fd73f>] ? semctl_down.clone.0+0xaf/0x100
Corresponding call trace from the vmcore:
PID: 30421 TASK: ffff8807043bc0c0 CPU: 3 COMMAND: "db2stop2"
[...]
[exception RIP: __bfs+232]
RIP: ffffffff810bcb98 RSP: ffff88073a98fb28 RFLAGS: 00010096
RAX: 0000000001f725fb RBX: ffffffff827f6ce0 RCX: 0000000000000000
RDX: ffff88073a98fb50 RSI: 0000000000000001 RDI: 0000000000000009
RBP: ffff88073a98fb88 R8: 0000000000000000 R9: 0000000000000001
R10: ffffffff818c1ad8 R11: 0000000000000003 R12: 0000000000000000
R13: ffffffff810bc970 R14: ffff88073a98fbf0 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff88073a98fb90] check_usage_backwards at ffffffff810c050e
#10 [ffff88073a98fc20] mark_lock at ffffffff810c0f8b
#11 [ffff88073a98fc70] __lock_acquire at ffffffff810c2367
#12 [ffff88073a98fd50] lock_acquire at ffffffff810c3364
#13 [ffff88073a98fdd0] _spin_lock at ffffffff81570a16
#14 [ffff88073a98fe00] freeary at ffffffff81241a35
#15 [ffff88073a98fe70] semctl_down.clone.0 at ffffffff81241cad
#16 [ffff88073a98ff20] sys_semctl at ffffffff81243a12
#17 [ffff88073a98ff80] system_call_fastpath at ffffffff8100b0d2
RIP: 0000003600aeaf8a RSP: 00007fff86a16f38 RFLAGS: 00010202
RAX: 0000000000000042 RBX: ffffffff8100b0d2 RCX: 00007fff86b70937
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000038004
RBP: 0000000000038004 R8: 0000000000000000 R9: 00007f7b77bdaae4
R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000410568
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: 0000000000000042 CS: 0033 SS: 002b
An example of a panic as it is seen on a PowerPC machine:
crash> bt
PID: 6737 TASK: c00000010e337e20 CPU: 43 COMMAND: "db2stop2"
…
#5 [c000001da9e6fa20] .__spin_yield at c00000000006aea0
[Link Register] [c000001da9e6fa20] ._raw_spin_lock at c0000000008f1ec8
#6 [c000001da9e6faa0] .freeary at c0000000003bd1d4
#7 [c000001da9e6fb80] .semctl_down at c0000000003bd8f8
#8 [c000001da9e6fcd0] .sys_semctl at c0000000003be874
#9 [c000001da9e6fd90] .sys_ipc at c0000000003c2c0c
#10 [c000001da9e6fe30] syscall_exit at c00000000000a17c
…
Corresponding message from the kernel log:
[9048908.066410] kernel BUG at arch/powerpc/lib/locks.c:34!
[9048908.066413] Oops: Exception in kernel mode, sig: 5 [#1]
Environment
- Red Hat Enterprise Linux 6 or 7
- DB2
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.