"Unable to create lockspace for CLVMD: Address already in use" error when starting clvmd on a RHEL5 cluster node
Issue
- While starting clvmd on boot following messages are displayed:
Dec 1 16:37:57 kernel: dlm: Using TCP for communications Dec 1 16:37:57 kernel: dlm: Can't create listening comms socket Dec 1 16:37:57 kernel: dlm: cannot start dlm lowcomms -98 Dec 1 16:37:57 clvmd: Unable to create lockspace for CLVM: Address already in use - After rebooting one cluster node, gfs filesystems aren't mounting
- Receive the following traceback when the cluster is quorate and after running service clvmd start manually on the console and in /var/log/messages:
Mar 5 17:24:02 hostname kernel: dlm: Using TCP for communications Mar 5 17:24:02 hostname kernel: dlm: Can't create listening comms socket Mar 5 17:24:12 hostname kernel: BUG: soft lockup - CPU#3 stuck for 10s! [clvmd:18026] Mar 5 17:24:12 hostname kernel: CPU 3:Mar 5 17:24:12 hostname kernel: Modules linked in: hangcheck_timer bonding autofs4 hidp rfcomm l2cap bluetooth ipv6 xfrm_nalgo crypto_api lock_dlm(U) gfs2(U) dlm configfs sunrpc dm_multipath video sbs backlight i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac parport_pc lp parport shpchp bnx2 serio_raw sg tg3 i5000_edac pcspkr edac_mc dm_snapshot dm_zero dm_mirror dm_mod usb_storage lpfc(U) lpfcmpl(U) scsi_transport_fc ata_piix libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Mar 5 17:24:12 hostname kernel: Pid: 18026, comm: clvmd Tainted: G 2.6.18-92.el5 #1 Mar 5 17:24:12 hostname kernel: RIP: 0010:[<ffffffff80064b54>] [<ffffffff80064b54>] .text.lock.spinlock+0x2/0x30 Mar 5 17:24:12 hostname kernel: RSP: 0018:ffff8101cf8cbc50 EFLAGS: 00000286 Mar 5 17:24:12 hostname kernel: RAX: 0000000000000000 RBX: ffff81022cabae30 RCX: ffffffff8008c2ce Mar 5 17:24:12 hostname kernel: RDX: ffff8101cf8cbc98 RSI: 0000000000000000 RDI: ffff81022cabae34 Mar 5 17:24:12 hostname kernel: RBP: ffff8101cf8cbd58 R08: ffff8101cf8ca000 R09: ffff81022f7b42c8 Mar 5 17:24:12 hostname kernel: R10: 0000000000000092 R11: 0000000000000080 R12: 0000000000000086 Mar 5 17:24:12 hostname kernel: R13: ffff8101cf8cbc88 R14: ffffffff80062efb R15: ffff81022314bee8 Mar 5 17:24:12 hostname kernel: FS: 00002af0ae689300(0000) GS:ffff81022ff22640(0000) knlGS:0000000000000000 Mar 5 17:24:12 hostname kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 5 17:24:12 hostname kernel: CR2: 000000000044c9f9 CR3: 00000001d24b6000 CR4: 00000000000006e0 Mar 5 17:24:12 hostname kernel: Mar 5 17:24:12 hostname kernel: Call Trace: Mar 5 17:24:12 hostname kernel: [<ffffffff80063b84>] __mutex_lock_slowpath+0x1d/0x9b Mar 5 17:24:12 hostname kernel: [<ffffffff80063c11>] .text.lock.mutex+0xf/0x14 Mar 5 17:24:12 hostname kernel: [<ffffffff883bb58c>] :dlm:close_connection+0x15/0x68 Mar 5 17:24:12 hostname kernel: [<ffffffff883bc09a>] :dlm:dlm_lowcomms_start+0x5d4/0x644 Mar 5 17:24:12 hostname kernel: [<ffffffff8008a5d2>] enqueue_task+0x41/0x56 Mar 5 17:24:12 hostname kernel: [<ffffffff8008a60e>] __activate_task+0x27/0x39 Mar 5 17:24:12 hostname kernel: [<ffffffff883b9c40>] :dlm:dlm_new_lockspace+0xab/0x875 Mar 5 17:24:12 hostname kernel: [<ffffffff80021f44>] __up_read+0x19/0x7f Mar 5 17:24:12 hostname kernel: [<ffffffff80066852>] do_page_fault+0x4fe/0x830 Mar 5 17:24:12 hostname kernel: [<ffffffff883bfd9c>] :dlm:device_write+0x42c/0x5e2 Mar 5 17:24:12 hostname kernel: [<ffffffff80016433>] vfs_write+0xce/0x174 Mar 5 17:24:12 hostname kernel: [<ffffffff80016d00>] sys_write+0x45/0x6e Mar 5 17:24:12 hostname kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Mar 5 17:24:12 hostname kernel:
Environment
-
Red Hat Enterprise Linux 5
- update 2 - kernel-2.6.18-92.el5
-
Red Hat Cluster Suite 5
- cman-2.0.84-2.el5-x86_64
- 2 node cluster configuration
- clvmd managed storage
- GFS1 filesystem
- configured in /etc/fstab and mounted via gfs service on boot
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
