What is vdsm profiling tool and when to use it

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Virtualization (RHEV) 3.5, 3.6
  • Red Hat Virtualization (RHV) 4.0

Issue

What is vdsm profiling tool and when to use it?

Resolution

  • When vdsm process consumes 100% or more of (a single) CPU, it is recommended to run vdsm profiler to see what keeps vdsm busy.

  • Once vdsm profiling is enabled, the issue should be reproduced and the /var/run/vdsm/vdsm.prof file should be collected for further analysis.

Instructions for configuring vdsm profiling:

1. Download and install yappi rpm on the host:
1.1. For RHEL6 hosts with vdsm-4.16, download yappi-0.93-1.x86_64.rpm (provided by Red Hat Support team) to /tmp on the host and install it:

# rpm -ivh /tmp/yappi-0.93-1.x86_64.rpm

1.2. For RHEL7 hosts with vdsm-4.17 and above, download yappi-0.98-1.x86_64.rpm (provided by Red Hat Support team) to /tmp on the host and install it:

# rpm -ivh /tmp/yappi-0.98-1.x86_64.rpm

For RHEV-H hosts, mount the root file system in read-write mode before running the rpm command:

# mount / -o remount,rw

Then remount it as read-only afterwards:

# mount / -o remount,ro

Note: since RHEV-H is stateless, after a reboot all the changes would be gone.

2. Enable vdsm profiling on the host:

2.1.a. For RHEL6 hosts, add the following lines to /etc/vdsm/vdsm.conf under [vars] section:

# Enable whole process profiling (requires yappi profiler).
profile_enable = true

2.1.b. For RHEL7 hosts, add the following lines to /etc/vdsm/vdsm.conf under [devel] section:

# Enable whole process profiling (requires yappi profiler).
[devel]
cpu_profile_enable = true

2.2. Restart vdsmd.

2.2.a. For RHEL6 hosts:

# service vdsmd restart

2.2.b. For RHEL7 hosts:

# systemctl restart vdsmd

Once vdsmd is restarted, it will start collecting the information in the file /var/run/vdsm/vdsm.prof.
Note: this file will not be visible until vdsmd is stopped or restarted again.

3. Disable profiling:

3.1. Modify /etc/vdsm/vdsm.conf.
3.1.a. For RHEL6 hosts:

profile_enable = false

3.1.b. For RHEL7 hosts:

cpu_profile_enable = false

3.2. Stop vdsmd:
3.2.a. For RHEL6 hosts:

# service vdsmd stop

3.2.b. For RHEL7 hosts:

# systemctl stop vdsmd

Note: Power Management should be disabled prior to stopping/restarting vdsmd in order to prevent the host from potentially getting fenced.

4. /var/run/vdsm/vdsm.prof:

  • The /var/run/vdsm/vdsm.prof file will get created when vdsmd is stopped.

  • The vdsmd service should be restarted as soon as possible, otherwise the host could get soft-fenced. This would cause vdsmd to be automatically restarted. If this were to occur before /etc/vdsm/vdsm.conf had been modified, then profiling would still be enabled.

Diagnostic Steps

Output of top while spining the VMs:

# top -b > /tmp/top_with_prof

And then, checking the vdsm process in the output, we can see its excessive CPU use:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU  %MEM    TIME+  COMMAND 
...
18931 vdsm       0 -20 37.2g  31g  10m S   163.7   12.5 234288:45 vdsm                                          
18931 vdsm       0 -20 37.2g  31g  10m S   62.0    12.5 234288:47 vdsm                                           
18931 vdsm       0 -20 37.2g  31g  10m S   72.5    12.5 234288:49 vdsm                                           
18931 vdsm       0 -20 37.2g  31g  10m S   111.5   12.5 234288:53 vdsm

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments