When new storage is scanned, tuned reloads cpufreq modules and causes a race on Red Hat Enterprise Linux 6
Issue
- After we added a path from the new SAN and went to 5 paths, ktuned went crazy and there were many ktuned and lsblk process running. There are 328 luns allocated to the system. CPU utilization went to 100% for over a half hour until I turned off the tuned service.
- We suffer an I/O delay issue happened when we recover a failed path, and many call traces happening in /var/log/messages:
kernel: ------------[ cut here ]------------
kernel: WARNING: at fs/sysfs/group.c:138 sysfs_remove_group+0xfb/0x100() (Not tainted)
kernel: Hardware name: ProLiant DL580 G7
kernel: sysfs group ffffffffa02f97e0 not found for kobject 'cpufreq'
kernel: Modules linked in: cpufreq_ondemand freq_table sctp libcrc32c fuse mptctl mptbase pcc_cpufreq coretemp bonding 8021q garp stp llc ipv6 dm_round_robin uinput power_meter microcode ses enclosure netxen_nic e1000e serio_raw iTCO_wdt iTCO_vendor_support sg hpilo hpwdt i7core_edac edac_core shpchp ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt pata_acpi ata_generic ata_piix hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: freq_table]
kernel: Pid: 28557, comm: cpuspeed Not tainted 2.6.32-358.6.2.el6.x86_64 #1
kernel: Call Trace:
kernel: [<ffffffff8106e2e7>] ? warn_slowpath_common+0x87/0xc0
kernel: [<ffffffff8106e3d6>] ? warn_slowpath_fmt+0x46/0x50
kernel: [<ffffffff811fd04b>] ? sysfs_remove_group+0xfb/0x100
kernel: [<ffffffffa02f8995>] ? cpufreq_governor_dbs+0xe5/0x470 [cpufreq_ondemand]
kernel: [<ffffffff8112a361>] ? get_page_from_freelist+0x3d1/0x830
kernel: [<ffffffff8150f07e>] ? mutex_lock+0x1e/0x50
kernel: [<ffffffff813082b0>] ? acpi_processor_ppc_notifier+0xb0/0xb6
kernel: [<ffffffff81513685>] ? notifier_call_chain+0x55/0x80
kernel: [<ffffffff81413529>] ? __cpufreq_governor+0xb9/0x180
kernel: [<ffffffff8141379f>] ? __cpufreq_set_policy+0x1af/0x250
kernel: [<ffffffff81413cd4>] ? store_scaling_governor+0xe4/0x210
kernel: [<ffffffff81413970>] ? handle_update+0x0/0x40
kernel: [<ffffffff812793ca>] ? kobject_get+0x1a/0x30
kernel: [<ffffffff81500065>] ? calibrate_delay+0x25/0x319
kernel: [<ffffffff81412da7>] ? store+0x67/0xa0
kernel: [<ffffffff811f9875>] ? sysfs_write_file+0xe5/0x170
kernel: [<ffffffff81181078>] ? vfs_write+0xb8/0x1a0
kernel: [<ffffffff81181971>] ? sys_write+0x51/0x90
kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
kernel: ---[ end trace 5353e93168003015 ]---
cpuspeed: Enabling ondemand cpu frequency scaling governor
cpuspeed: Enabling ondemand cpu frequency scaling governor
cpuspeed: Enabling ondemand cpu frequency scaling governor
cpuspeed: Enabling ondemand cpu frequency scaling governor
cpuspeed: Enabling ondemand cpu frequency scaling governor
kernel: ------------[ cut here ]------------
kernel: WARNING: at fs/sysfs/group.c:138 sysfs_remove_group+0xfb/0x100() (Tainted: G W --------------- )
kernel: Hardware name: ProLiant DL580 G7
kernel: sysfs group ffffffffa02f97e0 not found for kobject 'cpufreq'
kernel: Modules linked in: cpufreq_ondemand freq_table sctp libcrc32c fuse mptctl mptbase pcc_cpufreq coretemp bonding 8021q garp stp llc ipv6 dm_round_robin uinput power_meter microcode ses enclosure netxen_nic e1000e serio_raw iTCO_wdt iTCO_vendor_support sg hpilo hpwdt i7core_edac edac_core shpchp ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt pata_acpi ata_generic ata_piix hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: freq_table]
kernel: Pid: 28560, comm: cpuspeed Tainted: G W --------------- 2.6.32-358.6.2.el6.x86_64 #1
kernel: Call Trace:
kernel: [<ffffffff8106e2e7>] ? warn_slowpath_common+0x87/0xc0
kernel: [<ffffffff8106e3d6>] ? warn_slowpath_fmt+0x46/0x50
kernel: [<ffffffff811fd04b>] ? sysfs_remove_group+0xfb/0x100
kernel: [<ffffffffa02f8995>] ? cpufreq_governor_dbs+0xe5/0x470 [cpufreq_ondemand]
kernel: [<ffffffff8112a361>] ? get_page_from_freelist+0x3d1/0x830
kernel: [<ffffffff8150f07e>] ? mutex_lock+0x1e/0x50
kernel: [<ffffffff813082b0>] ? acpi_processor_ppc_notifier+0xb0/0xb6
kernel: [<ffffffff81513685>] ? notifier_call_chain+0x55/0x80
kernel: [<ffffffff81413529>] ? __cpufreq_governor+0xb9/0x180
kernel: [<ffffffff8141379f>] ? __cpufreq_set_policy+0x1af/0x250
kernel: [<ffffffff81413cd4>] ? store_scaling_governor+0xe4/0x210
kernel: [<ffffffff81413970>] ? handle_update+0x0/0x40
kernel: [<ffffffff812793ca>] ? kobject_get+0x1a/0x30
kernel: [<ffffffff81500065>] ? calibrate_delay+0x25/0x319
kernel: [<ffffffff81412da7>] ? store+0x67/0xa0
kernel: [<ffffffff811f9875>] ? sysfs_write_file+0xe5/0x170
kernel: [<ffffffff81181078>] ? vfs_write+0xb8/0x1a0
kernel: [<ffffffff81181971>] ? sys_write+0x51/0x90
kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
kernel: ---[ end trace 5353e93168003016 ]---
Environment
- Red Hat Enterprise Linux 6 (RHEL6)
-
tuned version prior to tuned-0.2.19-11.el6.1 (RHEL6.4.z)
- tuned is enabled:
$ grep tuned chkconfig tuned 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
