Why are some glusterfsd processes consuming 200% of CPU, when the 'gluster export' tool is running?
Issue
-
Some
glusterfsd
processes are consuming 200% of CPU:USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 26980 203 0.8 2257996 288616 ? Ssl Mar13 7179:26 /usr/sbin/glusterfsd -s 10.0.0.1 --volfile-id vol_XXXXXXXXXXXX/10.0.0.1.var-lib-heketi-mounts-vg_XXXXXXXXXXXX-brick_XXXXXXXXXXXX-brick -p /var/run/gluster/vols/vol_XXXXXXXXXXXX/10.0.0.1-var-lib-heketi-mounts-vg_XXXXXXXXXXXX-brick_XXXXXXXXXXXX-brick.pid -S /var/run/gluster/5f46f7aa4de3a23e.socket --brick-name /var/lib/heketi/mounts/vg_XXXXXXXXXXXX/brick_XXXXXXXXXXXX/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_XXXXXXXXXXXX-brick_XXXXXXXXXXXX-brick.log --xlator-option *-posix.glusterd-uuid=695e99df-1a71-4e9c-a3f4-05ca272f4bc2 --process-name brick --brick-port 49214 --xlator-option vol_XXXXXXXXXXXX-server.listen-port=49214
-
This occurred shortly after a volume expansion from a replica-2 configuration to replica-3. Because the bricks are consuming so many CPU resources, the healing process that should copy the contents to the new brick is not taking place:
[root@gfs01 cloud-user]# gluster volume heal vol_XXXXXXXXXXXX statistics heal-count Gathering count of entries to be healed on volume vol_XXXXXXXXXXXX has been successful Brick 10.0.0.1:/var/lib/heketi/mounts/vg_XXXXXXXXXXXX/brick_XXXXXXXXXXXX/brick Number of entries: 108948 Brick 10.0.0.2:/var/lib/heketi/mounts/vg_XXXXXXXXXXXX/brick_XXXXXXXXXXXX/brick Number of entries: 0 Brick 10.0.0.3:/var/lib/heketi/mounts/vg_XXXXXXXXXXXX/brick_XXXXXXXXXXXX/brick Number of entries: 109233
-
How to reduce the CPU consumption of the bricks and get the healing ongoing?
Environment
- Red Hat Openshift Container Storage 3.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.