Gluster Client Cannot Connect to Brick due to Gluster Reporting Incorrect Port (Multiplex)

Solution In Progress - Updated -

Issue

After upgrading Gluster, the heal info count never becomes zero, and logs show client trying to connect to a port that is different from the port Gluster is reporting for the brick:

Client logs say client is connecting to port 49155 (for example):

[2017-06-01 08:56:05.149017] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-data-client-1: changing port to 49155 (from 0)
[2017-06-01 08:56:05.152941] E [socket.c:2318:socket_connect_finish] 0-data-client-1: connection to X.X.X.X:49155 failed (Connection refused); disconnecting socket
[2017-06-01 08:56:09.153591] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-data-client-1: changing port to 49155 (from 0)
[2017-06-01 08:56:09.157781] E [socket.c:2318:socket_connect_finish] 0-data-client-1: connection to X.X.X.X:49155 failed (Connection refused); disconnecting socket
[2017-06-01 08:56:13.158146] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-data-client-1: changing port to 49155 (from 0)
[2017-06-01 08:56:13.162099] E [socket.c:2318:socket_connect_finish] 0-data-client-1: connection to X.X.X.X:49155 failed (Connection refused); disconnecting socket
[2017-06-01 08:56:17.162637] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-data-client-1: changing port to 49155 (from 0)
[2017-06-01 08:56:17.166604] E [socket.c:2318:socket_connect_finish] 0-data-client-1: connection to X.X.X.X:49155 failed (Connection refused); disconnecting socket
[2017-06-01 08:56:21.167186] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-data-client-1: changing port to 49155 (from 0)
[2017-06-01 08:56:21.171153] E [socket.c:2318:socket_connect_finish] 0-data-client-1: connection to X.X.X.X:49155 failed (Connection refused); disconnecting socket
[2017-06-01 08:56:25.172089] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-data-client-1: changing port to 49155 (from 0)
[2017-06-01 08:56:25.176367] E [socket.c:2318:socket_connect_finish] 0-data-client-1: connection to X.X.X.X:49155 failed (Connection refused); disconnecting

However, the brick appears to be listening on port 49152:

# ps ax | grep -i brick | grep data
root     22436  0.5  0.0 1217564 26164 ?       Ssl  12:31   0:40 /usr/sbin/glusterfsd -s X.X.X.X --volfile-id data.X.X.X.X.gluster_bricks-data-data -p /var/lib/glusterd/vols/data/run/X.X.X.X-gluster_bricks-data-data.pid -S /var/run/gluster/77a68376a313c246358b3d10487b0e94.socket --brick-name /gluster_bricks/data/data -l /var/log/glusterfs/bricks/gluster_bricks-data-data.log --xlator-option *-posix.glusterd-uuid=f889a05f-648e-4890-b1e6-637e33b69803 --brick-port 49152 --xlator-option data-server.listen-port=49152

Environment

  • Red Hat Gluster Storage 3.3

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content