GFS2 filesystems become hung when running pvmove and cmirror on Red Hat Enterprise Linux

Environment

• Red Hat Enterprise Linux Server 6 (with the High Availability and Resilient Storage Add Ons)

• Red Hat Cluster Suite cluster with 2 or more nodes.

  • Issue observed on corosync-1.4.1-15.el6. It is currently unknown if earlier versions of this package are affected. Later versions are not believed to be affected.
  • In /etc/cluster/cluster.conf, each node has a multicast address explicitly configured:

  # cat /etc/cluster/cluster.conf
  <... some data omitted ...>
      <clusternodes>
      <clusternode name="node1-priv" votes="1" nodeid="1">
          <multicast addr="225.0.0.12" interface="eth3"/>         <----
          <fence>
              <method name="1">
                  <device name="node1_ilo"/>
              </method>
          </fence>
      </clusternode>
      <clusternode name="node2-priv" votes="1" nodeid="2">
          <multicast addr="225.0.0.12" interface="eth3"/>         <----
          <fence>
              <method name="1">
                  <device name="node2_ilo"/>
              </method>
          </fence>
      </clusternode>
  </clusternodes>
  <... some data omitted ...>
  

  • There is no global cman-configured multicast address.

    • If a global cman multicast address was configured, it should look like this:

         <cman>
               <multicast addr="225.0.0.12"/>
         </cman>
    

• Clustered-LVM (CLVM) for shared storage.

  • Clustered-mirror (cmirror) used to facilitate pvmove.
  • pvmove of clustered VG physical volumes:

  pvmove -n /dev/datavg/datavol01 /dev/mapper/mpathc /dev/mapper/mpathdn
  

• GFS2 filesystems

• Kernel semaphore limits have been adjusted from default (unknown if this is related or not):

# sysctl kernel.sem
kernel.sem = 250        32000   32      128