"Unable to create lockspace for CLVMD: Address already in use" error when starting clvmd on a RHEL5 cluster node

Solution Verified - Updated -

Issue

  • While starting clvmd on boot following messages are displayed:
    Dec  1 16:37:57 kernel: dlm: Using TCP for communications 
Dec  1 16:37:57 kernel: dlm: Can't create listening comms socket 
Dec  1 16:37:57 kernel: dlm: cannot start dlm lowcomms -98 
Dec  1 16:37:57 clvmd: Unable to create lockspace for CLVM: Address already in use
  • After rebooting one cluster node, gfs filesystems aren't mounting
  • Receive the following traceback when the cluster is quorate and after running service clvmd start manually on the console and in /var/log/messages:
    Mar  5 17:24:02 hostname kernel: dlm: Using TCP for communications
Mar  5 17:24:02 hostname kernel: dlm: Can't create listening comms socket
Mar  5 17:24:12 hostname kernel: BUG: soft lockup - CPU#3 stuck for 10s! [clvmd:18026]
Mar  5 17:24:12 hostname kernel: CPU 3:Mar  5 17:24:12 hostname kernel: Modules linked in: hangcheck_timer bonding autofs4 hidp rfcomm l2cap bluetooth ipv6
xfrm_nalgo crypto_api lock_dlm(U) gfs2(U) dlm configfs sunrpc dm_multipath video sbs backlight i2c_ec i2c_core button
battery asus_acpi acpi_memhotplug ac parport_pc lp parport shpchp bnx2 serio_raw sg tg3 i5000_edac pcspkr edac_mc dm_snapshot
dm_zero dm_mirror dm_mod usb_storage lpfc(U) lpfcmpl(U) scsi_transport_fc ata_piix libata cciss sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd
Mar  5 17:24:12 hostname kernel: Pid: 18026, comm: clvmd Tainted: G      2.6.18-92.el5 #1
Mar  5 17:24:12 hostname kernel: RIP: 0010:[<ffffffff80064b54>]  [<ffffffff80064b54>] .text.lock.spinlock+0x2/0x30
Mar  5 17:24:12 hostname kernel: RSP: 0018:ffff8101cf8cbc50  EFLAGS: 00000286
Mar  5 17:24:12 hostname kernel: RAX: 0000000000000000 RBX: ffff81022cabae30 RCX: ffffffff8008c2ce
Mar  5 17:24:12 hostname kernel: RDX: ffff8101cf8cbc98 RSI: 0000000000000000 RDI: ffff81022cabae34
Mar  5 17:24:12 hostname kernel: RBP: ffff8101cf8cbd58 R08: ffff8101cf8ca000 R09: ffff81022f7b42c8
Mar  5 17:24:12 hostname kernel: R10: 0000000000000092 R11: 0000000000000080 R12: 0000000000000086
Mar  5 17:24:12 hostname kernel: R13: ffff8101cf8cbc88 R14: ffffffff80062efb R15: ffff81022314bee8
Mar  5 17:24:12 hostname kernel: FS:  00002af0ae689300(0000) GS:ffff81022ff22640(0000) knlGS:0000000000000000
Mar  5 17:24:12 hostname kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar  5 17:24:12 hostname kernel: CR2: 000000000044c9f9 CR3: 00000001d24b6000 CR4: 00000000000006e0
Mar  5 17:24:12 hostname kernel:
Mar  5 17:24:12 hostname kernel: Call Trace:
Mar  5 17:24:12 hostname kernel:  [<ffffffff80063b84>] __mutex_lock_slowpath+0x1d/0x9b
Mar  5 17:24:12 hostname kernel:  [<ffffffff80063c11>] .text.lock.mutex+0xf/0x14
Mar  5 17:24:12 hostname kernel:  [<ffffffff883bb58c>] :dlm:close_connection+0x15/0x68
Mar  5 17:24:12 hostname kernel:  [<ffffffff883bc09a>] :dlm:dlm_lowcomms_start+0x5d4/0x644
Mar  5 17:24:12 hostname kernel:  [<ffffffff8008a5d2>] enqueue_task+0x41/0x56
Mar  5 17:24:12 hostname kernel:  [<ffffffff8008a60e>] __activate_task+0x27/0x39
Mar  5 17:24:12 hostname kernel:  [<ffffffff883b9c40>] :dlm:dlm_new_lockspace+0xab/0x875
Mar  5 17:24:12 hostname kernel:  [<ffffffff80021f44>] __up_read+0x19/0x7f
Mar  5 17:24:12 hostname kernel:  [<ffffffff80066852>] do_page_fault+0x4fe/0x830
Mar  5 17:24:12 hostname kernel:  [<ffffffff883bfd9c>] :dlm:device_write+0x42c/0x5e2
Mar  5 17:24:12 hostname kernel:  [<ffffffff80016433>] vfs_write+0xce/0x174
Mar  5 17:24:12 hostname kernel:  [<ffffffff80016d00>] sys_write+0x45/0x6e
Mar  5 17:24:12 hostname kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Mar  5 17:24:12 hostname kernel:

Environment

  • Red Hat Enterprise Linux 5

    • update 2 - kernel-2.6.18-92.el5

  • Red Hat Cluster Suite 5

    • cman-2.0.84-2.el5-x86_64
    • 2 node cluster configuration
    • clvmd managed storage
  • GFS1 filesystem
    • configured in /etc/fstab and mounted via gfs service on boot

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.