glusterd fails to start due to no space
Issue
Glusterd does not come up on the gluster storage server node. It has following logs
[2016-12-15 11:00:12.786649] E [MSGID: 100017] [glusterfsd.c:1887:glusterfs_pidfile_setup] 0-glusterfsd: pidfile /var/run/glusterd.pid open failed [No space left on device]
[2016-12-15 11:00:38.711243] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.1 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2016-12-15 11:00:38.711969] E [MSGID: 100017] [glusterfsd.c:1887:glusterfs_pidfile_setup] 0-glusterfsd: pidfile /var/run/glusterd.pid open failed [No space left on device]
[2016-12-15 11:09:28.526025] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.1 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2016-12-15 11:09:28.526910] E [MSGID: 100017] [glusterfsd.c:1887:glusterfs_pidfile_setup] 0-glusterfsd: pidfile /var/run/glusterd.pid open failed [No space left on device]
[2016-12-15 11:09:53.554127] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.1 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2016-12-15 11:09:53.554883] E [MSGID: 100017] [glusterfsd.c:1887:glusterfs_pidfile_setup] 0-glusterfsd: pidfile /var/run/glusterd.pid open failed [No space left on device]
[2016-12-15 11:13:58.904807] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.1 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2016-12-15 11:13:58.917045] I [MSGID: 106478] [glusterd.c:1348:init] 0-management: Maximum allowed open file descriptors set to 65536
[2016-12-15 11:13:58.917098] I [MSGID: 106479] [glusterd.c:1397:init] 0-management: Using /var/lib/glusterd as working directory
[2016-12-15 11:13:58.925825] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2016-12-15 11:13:58.925856] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device
[2016-12-15 11:13:58.925867] W [rpc-transport.c:358:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2016-12-15 11:13:58.925960] W [rpcsvc.c:1595:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2016-12-15 11:13:58.925976] E [MSGID: 106243] [glusterd.c:1621:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2016-12-15 11:14:01.870498] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-12-15 11:14:01.870569] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-12-15 11:14:01.870575] I [MSGID: 106514] [glusterd-store.c:2058:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 30703
[2016-12-15 11:14:01.918498] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/vols/RHS-ISO/bricks/ts2rhsn001-stor:-glustervol-ISO. [No such file or directory]
[2016-12-15 11:14:01.918532] E [MSGID: 106201] [glusterd-store.c:3021:glusterd_store_retrieve_volumes] 0-management: Unable to restore volume: RHS-ISO
[2016-12-15 11:14:01.918561] E [MSGID: 101019] [xlator.c:428:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2016-12-15 11:14:01.918574] E [MSGID: 101066] [graph.c:326:glusterfs_graph_init] 0-management: initializing translator failed
[2016-12-15 11:14:01.918584] E [MSGID: 101176] [graph.c:672:glusterfs_graph_activate] 0-graph: init failed
[2016-12-15 11:14:01.919136] W [glusterfsd.c:1219:cleanup_and_exit] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xe5) [0x7f6d460e5ea5] -->/usr/sbin/glusterd(glusterfs_process_volfp+0x126) [0x7f6d460e5d66] -->/usr/sbin/glusterd(cleanup_and_exit+0x71) [0x7f6d460e5ba1] ) 0-: received signum (0), shutting down
Environment
- RHS-3.0
- RHGS-3.1
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.