Why 'Request timed out' error messages are seen after executing "gluster volume status" command which leads to locking of volume ?

Solution Verified - Updated -

Issue

  • Why 'Request timed out' error messages are seen after executing "gluster volume status" command which leads to locking of volume ?

~ Snippet from gluster logs /var/log/glusterfs/etc-glusterfs-glusterd.vol.log:

[2015-04-06 11:35:53.874604] E [name.c:140:client_fill_address_family] 0-glusterfs: transport.address-family not specified. Could not guess default value from (remote-host:(null) or transport.unix.connect-path:(null)) options
[2015-04-06 11:35:53.874682] T [cli-quotad-client.c:100:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT    <<===============
[2015-04-06 11:35:54.045765] T [rpc-clnt.c:418:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect
[2015-04-06 11:35:54.045831] T [socket.c:2732:socket_connect] (-->/lib64/libpthread.so.0() [0x34e20079d1] (-->/usr/lib64/libgl
usterfs.so.0(gf_timer_proc+0xc8) [0x34e303ca78] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_reconnect+0xd9) [0x34e380e129]))) 0-glus
terfs: connect () called on transport already connected
[2015-04-06 11:35:54.045847] T [rpc-clnt.c:418:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect
[2015-04-06 11:35:54.045853] T [socket.c:2740:socket_connect] 0-glusterfs: connecting 0x17fe110, state=0 gen=0 sock=-1
[2015-04-06 11:35:54.045870] W [dict.c:1059:data_to_str] (-->/usr/lib64/glusterfs/3.6.0.42.1/rpc-transport/socket.so(+0x6bda) 
[0x7fc1a3187bda] (-->/usr/lib64/glusterfs/3.6.0.42.1/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0xad) [0x7fc1a3
18c2bd] (-->/usr/lib64/glusterfs/3.6.0.42.1/rpc-transport/socket.so(client_fill_address_family+0x200) [0x7fc1a318c170]))) 0-di
ct: data is NULL
[2015-04-06 11:35:54.045881] W [dict.c:1059:data_to_str] (-->/usr/lib64/glusterfs/3.6.0.42.1/rpc-transport/socket.so(+0x6bda) 
[0x7fc1a3187bda] (-->/usr/lib64/glusterfs/3.6.0.42.1/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0xad) [0x7fc1a3
18c2bd] (-->/usr/lib64/glusterfs/3.6.0.42.1/rpc-transport/socket.so(client_fill_address_family+0x20b) [0x7fc1a318c17b]))) 0-di
ct: data is NULL
[2015-04-06 11:35:54.045886] E [name.c:140:client_fill_address_family] 0-glusterfs: transport.address-family not specified. Co
uld not guess default value from (remote-host:(null) or transport.unix.connect-path:(null)) options
[2015-04-06 11:35:54.045960] T [cli-quotad-client.c:100:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT
[2015-04-06 11:35:54.109165] D [cli-cmd.c:388:cli_cmd_submit] 0-cli: Returning -1 
[2015-04-06 11:35:54.109183] D [cli-rpc-ops.c:6761:gf_cli_status_volume] 0-cli: Returning: -1  <<================
[2015-04-06 11:35:54.109189] E [cli-rpc-ops.c:6856:gf_cli_status_volume_all] 0-cli: status all failed     <<=======
[2015-04-06 11:35:54.109200] I [input.c:36:cli_batch] 0-: Exiting with: -1    <<====================
[2015-04-06 11:35:55.482207] D [cli.c:588:cli_rpc_init] 0-cli: Connecting to glusterd using default socket
[2015-04-06 11:35:55.482256] D [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins
[2015-04-06 11:35:55.482263] D [rpc-clnt.c:986:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout
[2015-04-06 11:35:55.482274] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.0.42.1/rpc-transport/socket.so
# cat var/log/glusterfs/etc-glusterfs-glusterd.vol.log
.c:160:gd_collate_errors] 0-: Commit failed on 00000000-0000-0000-0000-000000000000. Please check log file for details.
[2015-04-06 11:09:58.993488] E [rpc-clnt.c:201:call_bail] 0-management: bailing out frame type(glusterd mgmt) op(--(4)) xid = 0x489156 sent = 2015-04-06 10:57:16.084496. timeout = 600 for IPADDRESS:24007
[2015-04-06 11:09:58.993499] E [glusterd-syncop.c:160:gd_collate_errors] 0-: Commit failed on 00000000-0000-0000-0000-000000000000. Please check log file for details.       <<==================
[2015-04-06 11:09:58.993512] E [rpc-clnt.c:201:call_bail] 0-management: bailing out frame type(glusterd mgmt) op(--(4)) xid = 0x489155 sent = 2015-04-06 10:57:16.084492. timeout = 600 for IPADDRESS:24007
[2015-04-06 11:09:58.993523] E [glusterd-syncop.c:160:gd_collate_errors] 0-: Commit failed on 00000000-0000-0000-0000-000000000000. Please check log file for details.       <<==================
# tail -5  /var/log/glusterfs/.cmd_log_history
[2015-04-07 11:55:39.597210]  : volume status all detail : SUCCESS
[2015-04-07 11:55:39.835160]  : volume status all detail : SUCCESS
[2015-04-07 11:55:40.018741]  : volume status all detail : SUCCESS
[2015-04-07 11:55:40.181914]  : volume status all detail : SUCCESS
[2015-04-07 11:55:40.342675]  : volume status all detail : SUCCESS
  • Gluster is monitored with zabbix and then "gluster volume status all detail" command is executed every minute and then the output is parsed with scripts where zabbix agent was not responding on a node so tried restarting glusterd service on all nodes but locking failed on one volume.

Environment

  • Red Hat Gluster Storage 3.0
  • Red Hat Gluster Storage 3.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.