LVM mirror on GFS2
We have 4 nodes Redhat cluster sharing a GFS2 file system and we use LVM mirror on 2 separate SAN disk for high availibity. With LVM mirror, we noticed a lot of "openais[5695]: [TOTEM] Retransmit List" in the /var/log/messages and application group claimed that I/O writes seem to very slow. Our network engineer captured traffics for non-LVM mirror and LVM mirror and noticed that with LVM mirror there 100 thousand times more packets using multicast address compared to non-LVM mirror.
Responses
Hello Joseph,
First of all using GFS/GFS2 fs in cluster without lvm is not recommended.
As we know, GFS and GFS2 puts additional load on the network in the form of DLM messages that must go between the cluster nodes to manage filesystem locks. This traffic, combined with the other traffic necessary for cluster communication can overload the network or the cluster nodes specially with LVM mirrors with a more overhead than even if the linear LV is used. The service clvmd is a very minimal user actually. It just sends locking notification messages when metadata is altered so there is no serious impact on performance and clvmd should work quite happily in a multicast, broadcast or UDPU installation.
The service cmirror is a different thing altogether. This is required with LVM mirrors. cmirror use the openais or corosync CPG protocol to share information and on a busy system with many mirrors this can be quite a lot of traffic, especially when cluster nodes reboot or resyncs are necessary. Hence the messages "openais[5695]: [TOTEM] Retransmit List" which indicates multicast messages are being delayed for a short amount of time somewhere on the network, but are eventually being delivered.
See https://access.redhat.com/site/solutions/38510 "[TOTEM] Retransmit List" messages repeatedly seen in RHEL 5 or 6 cluster node logs
Also to overcome the performance issue with LVM mirrors, see
https://access.redhat.com/site/solutions/25261 performance or connectivity issues in RHEL when using cmirror.
In case any further assistance is required, please open a case on 'Red Hat Customer Portal' for details analysis as additional details of cluster and the storage would be needed.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
