Gluster Client Cannot Connect to Brick due to Gluster Reporting Incorrect Port (Multiplex)
Issue
After upgrading Gluster, the heal info count never becomes zero, and logs show client trying to connect to a port that is different from the port Gluster is reporting for the brick:
Client logs say client is connecting to port 49155 (for example):
[2017-06-01 08:56:05.149017] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-data-client-1: changing port to 49155 (from 0)
[2017-06-01 08:56:05.152941] E [socket.c:2318:socket_connect_finish] 0-data-client-1: connection to X.X.X.X:49155 failed (Connection refused); disconnecting socket
[2017-06-01 08:56:09.153591] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-data-client-1: changing port to 49155 (from 0)
[2017-06-01 08:56:09.157781] E [socket.c:2318:socket_connect_finish] 0-data-client-1: connection to X.X.X.X:49155 failed (Connection refused); disconnecting socket
[2017-06-01 08:56:13.158146] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-data-client-1: changing port to 49155 (from 0)
[2017-06-01 08:56:13.162099] E [socket.c:2318:socket_connect_finish] 0-data-client-1: connection to X.X.X.X:49155 failed (Connection refused); disconnecting socket
[2017-06-01 08:56:17.162637] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-data-client-1: changing port to 49155 (from 0)
[2017-06-01 08:56:17.166604] E [socket.c:2318:socket_connect_finish] 0-data-client-1: connection to X.X.X.X:49155 failed (Connection refused); disconnecting socket
[2017-06-01 08:56:21.167186] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-data-client-1: changing port to 49155 (from 0)
[2017-06-01 08:56:21.171153] E [socket.c:2318:socket_connect_finish] 0-data-client-1: connection to X.X.X.X:49155 failed (Connection refused); disconnecting socket
[2017-06-01 08:56:25.172089] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-data-client-1: changing port to 49155 (from 0)
[2017-06-01 08:56:25.176367] E [socket.c:2318:socket_connect_finish] 0-data-client-1: connection to X.X.X.X:49155 failed (Connection refused); disconnecting
However, the brick appears to be listening on port 49152:
# ps ax | grep -i brick | grep data
root 22436 0.5 0.0 1217564 26164 ? Ssl 12:31 0:40 /usr/sbin/glusterfsd -s X.X.X.X --volfile-id data.X.X.X.X.gluster_bricks-data-data -p /var/lib/glusterd/vols/data/run/X.X.X.X-gluster_bricks-data-data.pid -S /var/run/gluster/77a68376a313c246358b3d10487b0e94.socket --brick-name /gluster_bricks/data/data -l /var/log/glusterfs/bricks/gluster_bricks-data-data.log --xlator-option *-posix.glusterd-uuid=f889a05f-648e-4890-b1e6-637e33b69803 --brick-port 49152 --xlator-option data-server.listen-port=49152
Environment
- Red Hat Gluster Storage 3.3
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.