Failed to start clvmd service on one node while clustat showing both the nodes are online.

Issue

• Failed to start clvmd service on one node while clustat showing both the nodes are online.

  • Found the following messages logs indicating clvmd in "D" state.

    dlm_controld[2410]: dlm_controld 3.0.12 started
    gfs_controld[2466]: gfs_controld 3.0.12 started
    fence_node[2475]: unfence node1 success
    kernel: dlm: Using TCP for communications
    kernel: dlm: connecting to 2
    kernel: INFO: task clvmd:2586 blocked for more than 120 seconds.
    kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    kernel: clvmd         D ee85fe3c     0  2586      1 0x00000080
    kernel: ee3f5030 00000082 00000002 ee85fe3c d16c3b54 00000000 00000000 ee3f5030
    kernel: f70bc570 ee1f4480 000000c7 5a0cd0db 000000c7 c0ae2120 c0ae2120 ee3f52d8
    kernel: c0ae2120 c0addb54 c0ae2120 ee3f52d8 d16c3b54 00000002 00088099 ee3f5030
    kernel: Call Trace:
    kernel: [<c08231e5>] ? schedule_timeout+0x195/0x250
    kernel: [<c0474081>] ? finish_wait+0x31/0x80
    kernel: [<c0822f49>] ? wait_for_common+0xe9/0x150
    kernel: [<c044bf70>] ? default_wake_function+0x0/0x10
    kernel: [<fccf9057>] ? dlm_new_lockspace+0x807/0x880 [dlm]
    kernel: [<fcd0048d>] ? device_write+0x21d/0x610 [dlm]
    kernel: [<c059d6cc>] ? security_file_permission+0xc/0x10
    kernel: [<c0527fb6>] ? rw_verify_area+0x66/0xe0
    kernel: [<fcd00270>] ? device_write+0x0/0x610 [dlm]
    kernel: [<c05280d0>] ? vfs_write+0xa0/0x190
    kernel: [<c04adecc>] ? audit_syscall_entry+0x21c/0x240
    kernel: [<c0528b51>] ? sys_write+0x41/0x70
    kernel: [<c0409adf>] ? sysenter_do_call+0x12/0x28
    kernel: INFO: task clvmd:2586 blocked for more than 120 seconds.