Server freezes or panics after executing db2stop command showing freeary() frame in the call trace

Solution Verified - Updated -

Issue

When the db2stop command is executed, the servers either hangs experiencing soft lockups, or panics.

An example of a soft lockup message is given below:

kernel: BUG: soft lockup - CPU#0 stuck for 67s! [db2stop2:10791]
kernel: Pid: 10791, comm: db2stop2 Not tainted 2.6.32-220.4.2.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Plat
form
kernel: RIP: 0010:[<ffffffff814ef2ae>]  [<ffffffff814ef2ae>] _spin_lock+0x1e/0x30
kernel: RSP: 0018:ffff880334299e78  EFLAGS: 00000206
kernel: RAX: 0000000000000000 RBX: ffff880334299e78 RCX: dead000000200200
kernel: RDX: 00000000000006e1 RSI: ffff8800282112c0 RDI: ffff880325045de4
kernel: RBP: ffffffff8100bc0e R08: ffff88031d57f8e8 R09: 0000000000000000
kernel: R10: 00000000000006e1 R11: 0000000000000002 R12: ffffffff81ad8720
kernel: R13: ffff88031b045bd0 R14: 0000000000000000 R15: ffff880334299dd8
kernel: FS:  00007fe7e3e50720(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 00007fe7eb0f2070 CR3: 0000000321662000 CR4: 00000000000006f0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
kernel: Process db2stop2 (pid: 10791, threadinfo ffff880334298000, task ffff8803349b6100)
kernel: Stack:
kernel: ffff880334299eb8 ffffffff811fd295 0000000000000000 0000000000000000
kernel: <0> ffff88031b045bd0 ffffffff81ad8720 ffff880334299ec8 0000000000000000
kernel: <0> ffff880334299f68 ffffffff811fd73f ffff88031a255cc0 ffff880037578380
kernel: Call Trace:
kernel: [<ffffffff811fd295>] ? freeary+0x65/0x150
kernel: [<ffffffff811fd73f>] ? semctl_down.clone.0+0xaf/0x100

Corresponding call trace from the vmcore:

PID: 30421  TASK: ffff8807043bc0c0  CPU: 3   COMMAND: "db2stop2"
[...]
    [exception RIP: __bfs+232]
    RIP: ffffffff810bcb98  RSP: ffff88073a98fb28  RFLAGS: 00010096
    RAX: 0000000001f725fb  RBX: ffffffff827f6ce0  RCX: 0000000000000000
    RDX: ffff88073a98fb50  RSI: 0000000000000001  RDI: 0000000000000009
    RBP: ffff88073a98fb88   R8: 0000000000000000   R9: 0000000000000001
    R10: ffffffff818c1ad8  R11: 0000000000000003  R12: 0000000000000000
    R13: ffffffff810bc970  R14: ffff88073a98fbf0  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #9 [ffff88073a98fb90] check_usage_backwards at ffffffff810c050e
#10 [ffff88073a98fc20] mark_lock at ffffffff810c0f8b
#11 [ffff88073a98fc70] __lock_acquire at ffffffff810c2367
#12 [ffff88073a98fd50] lock_acquire at ffffffff810c3364
#13 [ffff88073a98fdd0] _spin_lock at ffffffff81570a16
#14 [ffff88073a98fe00] freeary at ffffffff81241a35
#15 [ffff88073a98fe70] semctl_down.clone.0 at ffffffff81241cad
#16 [ffff88073a98ff20] sys_semctl at ffffffff81243a12
#17 [ffff88073a98ff80] system_call_fastpath at ffffffff8100b0d2
    RIP: 0000003600aeaf8a  RSP: 00007fff86a16f38  RFLAGS: 00010202
    RAX: 0000000000000042  RBX: ffffffff8100b0d2  RCX: 00007fff86b70937
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 0000000000038004
    RBP: 0000000000038004   R8: 0000000000000000   R9: 00007f7b77bdaae4
    R10: 0000000000000000  R11: 0000000000000206  R12: 0000000000410568
    R13: 0000000000000000  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: 0000000000000042  CS: 0033  SS: 002b

An example of a panic as it is seen on a PowerPC machine:

crash> bt
PID: 6737   TASK: c00000010e337e20  CPU: 43  COMMAND: "db2stop2"
…
 #5 [c000001da9e6fa20] .__spin_yield at c00000000006aea0
 [Link Register] [c000001da9e6fa20] ._raw_spin_lock at c0000000008f1ec8
 #6 [c000001da9e6faa0] .freeary at c0000000003bd1d4
 #7 [c000001da9e6fb80] .semctl_down at c0000000003bd8f8
 #8 [c000001da9e6fcd0] .sys_semctl at c0000000003be874
 #9 [c000001da9e6fd90] .sys_ipc at c0000000003c2c0c
#10 [c000001da9e6fe30] syscall_exit at c00000000000a17c
…

Corresponding message from the kernel log:

[9048908.066410] kernel BUG at arch/powerpc/lib/locks.c:34!
[9048908.066413] Oops: Exception in kernel mode, sig: 5 [#1]

Environment

  • Red Hat Enterprise Linux 6 or 7
  • DB2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content