pcc-cpufreq fails to load on some CPUs on HP systems

Solution Verified - Updated -

Red Hat Insights can detect this issue

Proactively detect and remediate issues impacting your systems.
View matching systems and remediation

Environment

  • HP Servers : ProLiant G6, G7, Gen8, or Gen9-series
  • Red Hat Enterprise Linux
  • HP Dynamic Power Savings Mode enabled from BIOS

Issue

  • System is performing more slow than normal
  • Generally setting the CPU governor to performance resolves my issue but am unable to set the CPU governor
  • During server boot, modprobe installs pcc-cpufreq which enables CPU governors on each CPU. However, it sometimes fails to enable on all CPUs.

Resolution

Disclaimer: The article shared above has been provided by Red Hat for reference only. It is outside the scope of the posted Service Level Agreements and support coverage as it involves software not provided by Red Hat. For further information on these issues, you should reach out to the hardware vendor.
Link(s) contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Root Cause

  • With HP Dynamic Power Savings Mode enabled and under certain conditions platform firmware (BIOS) may return a processor frequency of zero to the Linux pcc-cpufreq driver.

  • This may cause the driver to fail to load either for all processors or for a single processor.

Diagnostic Steps

  • The current power management driver can be checked by using cpupower command as follows. If the driver is pcc-cpufreq, it indicates the power control is with the BIOS instead of the OS.
# cpupower frequency-info
analyzing CPU 0:
  driver: pcc-cpufreq        ----------------->>>>>
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 1.20 GHz - 2.40 GHz
  available cpufreq governors: conservative userspace powersave ondemand performance
  current policy: frequency should be within 1.20 GHz and 2.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: 1.66 GHz (asserted by call to hardware)
  boost state support:
    Supported: yes
    Active: yes
  • Once this is set, after some random number of iterations, the driver will fail to load on one CPU. This can be checked as follows :
# for cpu in `ls -1d /sys/devices/system/cpu/cpu[0-9]*`; do ls -ld $cpu/cpufreq; done

drwxr-xr-x. 2 root root 0 Nov  7 11:27 /sys/devices/system/cpu/cpu0/cpufreq
drwxr-xr-x. 2 root root 0 Nov  7 11:27 /sys/devices/system/cpu/cpu1/cpufreq
drwxr-xr-x. 2 root root 0 Nov  7 11:27 /sys/devices/system/cpu/cpu10/cpufreq
drwxr-xr-x. 2 root root 0 Nov  7 11:27 /sys/devices/system/cpu/cpu11/cpufreq
drwxr-xr-x. 2 root root 0 Nov  7 11:27 /sys/devices/system/cpu/cpu2/cpufreq
drwxr-xr-x. 2 root root 0 Nov  7 11:27 /sys/devices/system/cpu/cpu3/cpufreq
ls: cannot access 'sys/devices/system/cpu/cpu4/cpufreq': No such file or directory   ------->>>>>>> HERE
drwxr-xr-x. 2 root root 0 Nov  7 11:27 /sys/devices/system/cpu/cpu5/cpufreq
drwxr-xr-x. 2 root root 0 Nov  7 11:27 /sys/devices/system/cpu/cpu6/cpufreq
drwxr-xr-x. 2 root root 0 Nov  7 11:27 /sys/devices/system/cpu/cpu7/cpufreq
drwxr-xr-x. 2 root root 0 Nov  7 11:27 /sys/devices/system/cpu/cpu8/cpufreq
drwxr-xr-x. 2 root root 0 Nov  7 11:27 /sys/devices/system/cpu/cpu9/cpufreq
  • And at the same time, if the /var/log/messages is checked, following traces can be seen :
Apr 19 19:57:42 hp-bl460cg8-1 kernel: cpufreq: __cpufreq_add_dev: ->get() failed
Apr 19 19:57:42 hp-bl460cg8-1 kernel: 
Apr 19 19:57:42 hp-bl460cg8-1 kernel: ======================================================
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [ INFO: possible circular locking dependency detected ]
Apr 19 19:57:42 hp-bl460cg8-1 kernel: 3.10.0-862.el7.x86_64.debug #1 Not tainted
Apr 19 19:57:42 hp-bl460cg8-1 kernel: -------------------------------------------------------
Apr 19 19:57:42 hp-bl460cg8-1 kernel: modprobe/10822 is trying to acquire lock:
Apr 19 19:57:42 hp-bl460cg8-1 kernel: (s_active#104){++++.+}, at: [<ffffffff827085c3>] kernfs_remove+0x23/0x40
Apr 19 19:57:42 hp-bl460cg8-1 kernel: #012but task is already holding lock:
Apr 19 19:57:42 hp-bl460cg8-1 kernel: (&policy->rwsem){+++++.}, at: [<ffffffff829f5073>] __cpufreq_add_dev.isra.25+0x433/0xb30
Apr 19 19:57:42 hp-bl460cg8-1 kernel: #012which lock already depends on the new lock.
Apr 19 19:57:42 hp-bl460cg8-1 kernel: #012the existing dependency chain (in reverse order) is:
Apr 19 19:57:42 hp-bl460cg8-1 kernel: #012-> #1 (&policy->rwsem){+++++.}:
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff82531699>] lock_acquire+0x99/0x1e0
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff82beda23>] mutex_lock_nested+0x93/0x410
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff8270984e>] kernfs_fop_open+0x17e/0x360
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff8266cc52>] do_dentry_open+0x1c2/0x350
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff8266ce9d>] vfs_open+0x5d/0xb0
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff82680952>] do_last+0x1f2/0x12c0
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff82681af0>] path_openat+0xd0/0x670
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff8268354d>] do_filp_open+0x4d/0xb0
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff8266e407>] do_sys_open+0x137/0x240
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff8266e52e>] SyS_open+0x1e/0x20
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff82bff355>] system_call_fastpath+0x1c/0x21
Apr 19 19:57:42 hp-bl460cg8-1 kernel: #012-> #0 (s_active#104){++++.+}:
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff8253056c>] __lock_acquire+0xdac/0x15f0
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff82531699>] lock_acquire+0x99/0x1e0
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff827074f0>] __kernfs_remove+0x280/0x380
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff827085c3>] kernfs_remove+0x23/0x40
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff8270ad61>] sysfs_remove_dir+0x51/0x80
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff827c4eb8>] kobject_del+0x18/0x50
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff827c4f4e>] kobject_release+0x5e/0x1b0
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff827c4e18>] kobject_put+0x28/0x60
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff829f535c>] __cpufreq_add_dev.isra.25+0x71c/0xb30
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff829f577e>] cpufreq_add_dev+0xe/0x10
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff828f7207>] subsys_interface_register+0xc7/0x120
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff829f39f2>] cpufreq_register_driver+0x112/0x340
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffffc03d062e>] pcc_cpufreq_init+0x4af/0xe81 [pcc_cpufreq]
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff8240210a>] do_one_initcall+0xba/0x240
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff82541e10>] load_module+0x19c0/0x1f00
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff8254242f>] SyS_init_module+0xdf/0x130
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       [<ffffffff82bff355>] system_call_fastpath+0x1c/0x21
Apr 19 19:57:42 hp-bl460cg8-1 kernel: #012other info that might help us debug this:
Apr 19 19:57:42 hp-bl460cg8-1 kernel: Possible unsafe locking scenario:
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       CPU0                    CPU1
Apr 19 19:57:42 hp-bl460cg8-1 kernel:       ----                    ----
Apr 19 19:57:42 hp-bl460cg8-1 kernel:  lock(&policy->rwsem);
Apr 19 19:57:42 hp-bl460cg8-1 kernel:                               lock(s_active#104);
Apr 19 19:57:42 hp-bl460cg8-1 kernel:                               lock(&policy->rwsem);
Apr 19 19:57:42 hp-bl460cg8-1 kernel:  lock(s_active#104);
Apr 19 19:57:42 hp-bl460cg8-1 kernel: #012 *** DEADLOCK ***
Apr 19 19:57:42 hp-bl460cg8-1 kernel: 3 locks held by modprobe/10822:
Apr 19 19:57:42 hp-bl460cg8-1 kernel: #0:  (subsys mutex#5){+.+.+.}, at: [<ffffffff828f71a9>] subsys_interface_register+0x69/0x120
Apr 19 19:57:42 hp-bl460cg8-1 kernel: #1:  (cpufreq_rwsem){++++.+}, at: [<ffffffff829f4cb0>] __cpufreq_add_dev.isra.25+0x70/0xb30
Apr 19 19:57:42 hp-bl460cg8-1 kernel: #2:  (&policy->rwsem){+++++.}, at: [<ffffffff829f5073>] __cpufreq_add_dev.isra.25+0x433/0xb30
Apr 19 19:57:42 hp-bl460cg8-1 kernel: #012stack backtrace:
Apr 19 19:57:42 hp-bl460cg8-1 kernel: CPU: 25 PID: 10822 Comm: modprobe Kdump: loaded Not tainted 3.10.0-862.el7.x86_64.debug #1
Apr 19 19:57:42 hp-bl460cg8-1 kernel: Hardware name: HP ProLiant BL460c Gen8, BIOS I31 08/02/2014
Apr 19 19:57:42 hp-bl460cg8-1 kernel: Call Trace:
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff82be915b>] dump_stack+0x19/0x1b
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff82be2554>] print_circular_bug+0x1f9/0x207
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff8252ed77>] check_prevs_add+0x957/0x960
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff8252b997>] ? add_lock_to_list.isra.28.constprop.45+0x77/0xc0
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff8253056c>] __lock_acquire+0xdac/0x15f0
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff82531699>] lock_acquire+0x99/0x1e0
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff827085c3>] ? kernfs_remove+0x23/0x40
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff827074f0>] __kernfs_remove+0x280/0x380
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff827085c3>] ? kernfs_remove+0x23/0x40
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff8252bbaf>] ? lock_release_holdtime.part.29+0xf/0x1a0
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff827085c3>] kernfs_remove+0x23/0x40
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff8270ad61>] sysfs_remove_dir+0x51/0x80
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff827c4eb8>] kobject_del+0x18/0x50
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff827c4f4e>] kobject_release+0x5e/0x1b0
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff827c4e18>] kobject_put+0x28/0x60
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff829f535c>] __cpufreq_add_dev.isra.25+0x71c/0xb30
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffffc03d017f>] ? pcc_cpufreq_do_osc+0x17f/0x17f [pcc_cpufreq]
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff829f577e>] cpufreq_add_dev+0xe/0x10
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff828f7207>] subsys_interface_register+0xc7/0x120
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff829f39f2>] cpufreq_register_driver+0x112/0x340
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff8264262b>] ? kfree+0xdb/0x2d0
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffffc03d017f>] ? pcc_cpufreq_do_osc+0x17f/0x17f [pcc_cpufreq]
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffffc03d062e>] pcc_cpufreq_init+0x4af/0xe81 [pcc_cpufreq]
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffffc03d017f>] ? pcc_cpufreq_do_osc+0x17f/0x17f [pcc_cpufreq]
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff8240210a>] do_one_initcall+0xba/0x240
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff82541e10>] load_module+0x19c0/0x1f00
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff827f38e0>] ? ddebug_proc_write+0xf0/0xf0
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff8254242f>] SyS_init_module+0xdf/0x130
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff82bff355>] system_call_fastpath+0x1c/0x21
Apr 19 19:57:42 hp-bl460cg8-1 kernel: [<ffffffff82bff2a1>] ? system_call_after_swapgs+0xae/0x146

  • Alternatively, the cpupower service may fail. Checking systemctl status cpupower.service may output logs similar to the following;

    daemon.info systemd:Starting Configure CPU power related settings...
daemon.info cpupower:Setting cpu: 0
daemon.info cpupower:Setting cpu: 1
daemon.info cpupower:Setting cpu: 2
daemon.info cpupower:Setting cpu: 3
daemon.info cpupower:Setting cpu: 4
daemon.info cpupower:Setting cpu: 5
daemon.info cpupower:Setting cpu: 6
daemon.info cpupower:Setting cpu: 7
daemon.info cpupower:Setting cpu: 8
daemon.info cpupower:Error setting new values. Common errors:
daemon.info cpupower:- Do you have proper administration rights? (super-user?)
daemon.info cpupower:- Is the governor you requested available and modprobed?
daemon.info cpupower:- Trying to set an invalid policy?
daemon.info cpupower:- Trying to set a specific frequency, but userspace governor is not available,
daemon.info cpupower:for example because of hardware which cannot be set to a specific frequency
daemon.info cpupower:or because the userspace governor isn't loaded?
daemon.notice systemd:cpupower.service: main process exited, code=exited, status=237/n/a
daemon.err systemd:Failed to start Configure CPU power related settings.
daemon.notice systemd:Unit cpupower.service entered failed state.
daemon.warning systemd:cpupower.service failed.

  • If the power control is given to the OS, intel_pstate driver will get loaded and the issue will not be observed.

# cpupower frequency-info
analyzing CPU 0:
  driver: intel_pstate        ----------------->>>>>
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 1.20 GHz - 2.40 GHz
  available cpufreq governors: conservative userspace powersave ondemand performance
  current policy: frequency should be within 1.20 GHz and 2.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: 1.66 GHz (asserted by call to hardware)
  boost state support:
    Supported: yes
    Active: yes

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments