Why are some glusterfsd processes consuming 200% of CPU, when the 'gluster export' tool is running?

Solution Verified - Updated -

Issue

  • Some glusterfsd processes are consuming 200% of CPU:

    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root     26980  203  0.8 2257996 288616 ?      Ssl  Mar13 7179:26 /usr/sbin/glusterfsd -s 10.0.0.1 --volfile-id vol_XXXXXXXXXXXX/10.0.0.1.var-lib-heketi-mounts-vg_XXXXXXXXXXXX-brick_XXXXXXXXXXXX-brick -p /var/run/gluster/vols/vol_XXXXXXXXXXXX/10.0.0.1-var-lib-heketi-mounts-vg_XXXXXXXXXXXX-brick_XXXXXXXXXXXX-brick.pid -S /var/run/gluster/5f46f7aa4de3a23e.socket --brick-name /var/lib/heketi/mounts/vg_XXXXXXXXXXXX/brick_XXXXXXXXXXXX/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_XXXXXXXXXXXX-brick_XXXXXXXXXXXX-brick.log --xlator-option *-posix.glusterd-uuid=695e99df-1a71-4e9c-a3f4-05ca272f4bc2 --process-name brick --brick-port 49214 --xlator-option vol_XXXXXXXXXXXX-server.listen-port=49214
    
  • This occurred shortly after a volume expansion from a replica-2 configuration to replica-3. Because the bricks are consuming so many CPU resources, the healing process that should copy the contents to the new brick is not taking place:

    [root@gfs01 cloud-user]# gluster volume heal vol_XXXXXXXXXXXX statistics heal-count
    Gathering count of entries to be healed on volume vol_XXXXXXXXXXXX has been successful
    
    Brick 10.0.0.1:/var/lib/heketi/mounts/vg_XXXXXXXXXXXX/brick_XXXXXXXXXXXX/brick
    Number of entries: 108948
    
    Brick 10.0.0.2:/var/lib/heketi/mounts/vg_XXXXXXXXXXXX/brick_XXXXXXXXXXXX/brick
    Number of entries: 0
    
    Brick 10.0.0.3:/var/lib/heketi/mounts/vg_XXXXXXXXXXXX/brick_XXXXXXXXXXXX/brick
    Number of entries: 109233
    
  • How to reduce the CPU consumption of the bricks and get the healing ongoing?

Environment

  • Red Hat Openshift Container Storage 3.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content