Shared ACFS filesystem was full on a 2-node Oracle Cluster and the nodes became unresponsive.
Issue
- Shared ACFS filesystem was full on a 2-node Oracle Cluster and the nodes became unresponsive. Following messages were seen on the console just before the hang:
kernel: INFO: task df:4286 blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: df D ffffffff80151248 0 4286 4282 (NOTLB)
kernel: ffff81037411bb38 0000000000000082 0000000000000000 ffff810880003c10
kernel: 0000000000000292 0000000000000005 ffff810581fa7080 ffff81011df650c0
kernel: 001767e59c11013a 0000000000085412 ffff810581fa7268 0000000a7f388000
kernel: Call Trace:
kernel: [<ffffffff88663713>] :oracleoks:KsWaitEvent+0x113/0x171
kernel: [<ffffffff8008e16d>] default_wake_function+0x0/0xe
kernel: [<ffffffff88264d5c>] :dm_mod:dm_any_congested+0x38/0x3f
kernel: [<ffffffff886651e8>] :oracleoks:KsCondWaitTimeout+0x59/0x1c8
kernel: [<ffffffff886fb74a>] :oracleacfs:OfsAcquireDLMLock_int+0x80d/0xfab
kernel: [<ffffffff886f67de>] :oracleacfs:ofs_dlm_blkrtn+0x0/0x126
kernel: [<ffffffff886fbf44>] :oracleacfs:OfsOpAcquireDLMLock_int+0x5c/0x445
kernel: [<ffffffff80019b4b>] __follow_mount+0x26/0x7f
kernel: [<ffffffff887194c4>] :oracleacfs:OfsGenStatFs+0x1d/0xd3
kernel: [<ffffffff8875343d>] :oracleacfs:ofs_statfs+0xba/0x14d
kernel: [<ffffffff8000a64f>] __link_path_walk+0xe14/0xf42
kernel: [<ffffffff800e0ac2>] vfs_statfs+0x63/0x7f
Environment
- Red Hat Enterprise Linux 5
- Oracle RAC
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.