Shared ACFS filesystem was full on a 2-node Oracle Cluster and the nodes became unresponsive.

Solution Unverified - Updated -

Issue

  • Shared ACFS filesystem was full on a 2-node Oracle Cluster and the nodes became unresponsive. Following messages were seen on the console just before the hang:
kernel: INFO: task df:4286 blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: df            D ffffffff80151248     0  4286   4282                     (NOTLB)
kernel:  ffff81037411bb38 0000000000000082 0000000000000000 ffff810880003c10
kernel:  0000000000000292 0000000000000005 ffff810581fa7080 ffff81011df650c0
kernel:  001767e59c11013a 0000000000085412 ffff810581fa7268 0000000a7f388000
kernel: Call Trace:
kernel:  [<ffffffff88663713>] :oracleoks:KsWaitEvent+0x113/0x171
kernel:  [<ffffffff8008e16d>] default_wake_function+0x0/0xe
kernel:  [<ffffffff88264d5c>] :dm_mod:dm_any_congested+0x38/0x3f
kernel:  [<ffffffff886651e8>] :oracleoks:KsCondWaitTimeout+0x59/0x1c8
kernel:  [<ffffffff886fb74a>] :oracleacfs:OfsAcquireDLMLock_int+0x80d/0xfab
kernel:  [<ffffffff886f67de>] :oracleacfs:ofs_dlm_blkrtn+0x0/0x126
kernel:  [<ffffffff886fbf44>] :oracleacfs:OfsOpAcquireDLMLock_int+0x5c/0x445
kernel:  [<ffffffff80019b4b>] __follow_mount+0x26/0x7f
kernel:  [<ffffffff887194c4>] :oracleacfs:OfsGenStatFs+0x1d/0xd3
kernel:  [<ffffffff8875343d>] :oracleacfs:ofs_statfs+0xba/0x14d
kernel:  [<ffffffff8000a64f>] __link_path_walk+0xe14/0xf42
kernel:  [<ffffffff800e0ac2>] vfs_statfs+0x63/0x7f

Environment

  • Red Hat Enterprise Linux 5
  • Oracle RAC

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content