glusterd fails to start due to no space

Solution Unverified - Updated -

Issue

Glusterd does not come up on the gluster storage server node. It has following logs

[2016-12-15 11:00:12.786649] E [MSGID: 100017] [glusterfsd.c:1887:glusterfs_pidfile_setup] 0-glusterfsd: pidfile /var/run/glusterd.pid open failed [No space left on device]
[2016-12-15 11:00:38.711243] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.1 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2016-12-15 11:00:38.711969] E [MSGID: 100017] [glusterfsd.c:1887:glusterfs_pidfile_setup] 0-glusterfsd: pidfile /var/run/glusterd.pid open failed [No space left on device]
[2016-12-15 11:09:28.526025] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.1 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2016-12-15 11:09:28.526910] E [MSGID: 100017] [glusterfsd.c:1887:glusterfs_pidfile_setup] 0-glusterfsd: pidfile /var/run/glusterd.pid open failed [No space left on device]
[2016-12-15 11:09:53.554127] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.1 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2016-12-15 11:09:53.554883] E [MSGID: 100017] [glusterfsd.c:1887:glusterfs_pidfile_setup] 0-glusterfsd: pidfile /var/run/glusterd.pid open failed [No space left on device]
[2016-12-15 11:13:58.904807] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.1 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2016-12-15 11:13:58.917045] I [MSGID: 106478] [glusterd.c:1348:init] 0-management: Maximum allowed open file descriptors set to 65536
[2016-12-15 11:13:58.917098] I [MSGID: 106479] [glusterd.c:1397:init] 0-management: Using /var/lib/glusterd as working directory
[2016-12-15 11:13:58.925825] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2016-12-15 11:13:58.925856] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device
[2016-12-15 11:13:58.925867] W [rpc-transport.c:358:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2016-12-15 11:13:58.925960] W [rpcsvc.c:1595:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2016-12-15 11:13:58.925976] E [MSGID: 106243] [glusterd.c:1621:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2016-12-15 11:14:01.870498] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-12-15 11:14:01.870569] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-12-15 11:14:01.870575] I [MSGID: 106514] [glusterd-store.c:2058:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 30703
[2016-12-15 11:14:01.918498] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/vols/RHS-ISO/bricks/ts2rhsn001-stor:-glustervol-ISO. [No such file or directory]
[2016-12-15 11:14:01.918532] E [MSGID: 106201] [glusterd-store.c:3021:glusterd_store_retrieve_volumes] 0-management: Unable to restore volume: RHS-ISO
[2016-12-15 11:14:01.918561] E [MSGID: 101019] [xlator.c:428:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2016-12-15 11:14:01.918574] E [MSGID: 101066] [graph.c:326:glusterfs_graph_init] 0-management: initializing translator failed
[2016-12-15 11:14:01.918584] E [MSGID: 101176] [graph.c:672:glusterfs_graph_activate] 0-graph: init failed
[2016-12-15 11:14:01.919136] W [glusterfsd.c:1219:cleanup_and_exit] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xe5) [0x7f6d460e5ea5] -->/usr/sbin/glusterd(glusterfs_process_volfp+0x126) [0x7f6d460e5d66] -->/usr/sbin/glusterd(cleanup_and_exit+0x71) [0x7f6d460e5ba1] ) 0-: received signum (0), shutting down

Environment

  • RHS-3.0
  • RHGS-3.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.