dbus daemon eating 100% of one CPU core

Posted on

On our database servers (Redhat 6.4) after 2-3 day of operation, the dbus daemon eating 100% of one CPU core. After few days this high load still exists, only solved if the server is restarted.
When the dbus is using 100% of the CPU core, the Oracle JOB-s running into a deadlock, also shell sudo command is not responding to any request.
Just for background information: the problematic Oracle JOB-s using some credential mechanism (this is the common of these jobs).

We have tried the same setup on an other oracle DB server, which is running under CentOS and not Redhat, but finally realized the same issue.

DB/Node parameters:

xxxxDB1-root@yyyzzxxxxrac1:~# uname -a
Linux yyyzzxxxxrac1 2.6.32-358.18.1.el6.x86_64 #1 SMP Fri Aug 2 17:04:38 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

xxxxDB1-root@yyyzzxxxxrac1:~# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.4 (Santiago)

SQL> select * from v$instance;
INSTANCE_NUMBER INSTANCE_NAME HOST_NAME VERSION STARTUP_TIME STATUS PAR THREAD# ARCHIVE LOG_SWITCH_WAIT LOGINS SHU
DATABASE_STATUS INSTANCE_ROLE ACTIVE_ST BLO
1 SID1 yyyzzxxxxrac1 11.2.0.3.0 14-JAN. -23 OPEN YES 1 STARTED ALLOWED NO
ACTIVE PRIMARY_INSTANCE NORMAL NO

xxxxDB1-root@yyyzzxxxxrac1:~# dbus-daemon --version
D-Bus Message Bus Daemon 1.2.24

xxxxDB1-root@yyyzzxxxxrac1:~# lspci
00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07)
00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)
00:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07)
00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07)
00:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07)
00:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07)
00:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07)
00:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07)
00:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07)
00:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07)
00:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07)
00:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07)
00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07)
00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07)
00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06)
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06)
00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6)
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06)
00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06)
01:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch [PS]
02:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch [PS]
02:01.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch [PS]
03:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe-PCI Bridge [PPB]
04:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. G200eR2
11:00.0 Fibre Channel: Brocade Communications Systems, Inc. 1867/1860: 16Gbps/10Gbps Fabric Adapter (rev 01)
11:00.1 Fibre Channel: Brocade Communications Systems, Inc. 1867/1860: 16Gbps/10Gbps Fabric Adapter (rev 01)
1b:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
80:00.0 PCI bridge: Intel Corporation Xeon E5/Core i7 DMI2 in PCI Express Mode (rev 07)
80:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07)
80:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b (rev 07)
80:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)
80:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07)
80:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07)
80:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07)
80:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07)
80:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07)
80:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07)
80:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07)
80:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07)
80:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07)
80:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07)
80:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07)
80:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07)

xxxxDB1-root@yyyzzxxxxrac1:~# ifconfig
bond0 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
inet6 addr: fe80::202:c9ff:fee5:6cf0/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:9058519605 errors:0 dropped:0 overruns:0 frame:0
TX packets:28600922879 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3511452359951 (3.1 TiB) TX bytes:38719928355036 (35.2 TiB)

bond0.51 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
inet addr:10.51.XX.XX Bcast:10.51.0.255 Mask:255.255.255.0
inet6 addr: fe80::202:c9ff:fee5:6cf0/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:402143 errors:0 dropped:0 overruns:0 frame:0
TX packets:37684 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:24247361 (23.1 MiB) TX bytes:1872895 (1.7 MiB)

bond0.60 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
inet addr:172.27.XX.XX Bcast:172.27.61.255 Mask:255.255.254.0
inet6 addr: fe80::202:c9ff:fee5:6cf0/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:6262516952 errors:0 dropped:0 overruns:0 frame:0
TX packets:6089735344 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1181302289616 (1.0 TiB) TX bytes:35601212876851 (32.3 TiB)

bond0.60:1 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
inet addr:172.27.XX.XX Bcast:172.27.61.255 Mask:255.255.254.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

bond0.60:2 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
inet addr:172.27.XX.XX Bcast:172.27.61.255 Mask:255.255.254.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

bond0.60:khr Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
inet addr:192.168.XX.XX Bcast:192.168.20.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

bond0.121 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
inet addr:10.121.XX.XX Bcast:10.121.0.255 Mask:255.255.255.0
inet6 addr: fe80::202:c9ff:fee5:6cf0/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:2786184418 errors:0 dropped:0 overruns:0 frame:0
TX packets:2439892520 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2202685530090 (2.0 TiB) TX bytes:1793990838383 (1.6 TiB)

bond0.121:1 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
inet addr:169.254.XX.XX Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

eth0 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:9017550977 errors:0 dropped:0 overruns:0 frame:0
TX packets:28600922879 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3507715042150 (3.1 TiB) TX bytes:38719928355036 (35.2 TiB)

eth1 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:40968628 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3737317801 (3.4 GiB) TX bytes:0 (0.0 b)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:120919318 errors:0 dropped:0 overruns:0 frame:0
TX packets:120919318 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:30168996858 (28.0 GiB) TX bytes:30168996858 (28.0 GiB)

xxxxDB1-root@yyyzzxxxxrac1:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_redhat-lv_root
30G 9.5G 19G 34% /
tmpfs 64G 32G 32G 50% /dev/shm
/dev/mapper/36005076802810d4d900000000000002dp2
485M 93M 367M 21% /boot
/dev/mapper/36005076802810d4d900000000000002dp1
200M 260K 200M 1% /boot/efi
/dev/mapper/vg_redhat-lv_u01
60G 15G 42G 27% /u01
172.27.XX.XX:/volume1/backup
18T 17T 903G 96% /mnt/NAS/backup
172.27.XX.XX:/volume1/admin
18T 17T 903G 96% /mnt/NAS/admin
172.27.XX.XX:/mnt/devops
1.3T 852G 364G 71% /mnt/devops
//yyyxxfile03.yyyhu.net/oracle/xxxxxx
2.5T 2.3T 268G 90% /mnt/oradirs
//172.27.XX.XX/ORACLE
1016G 218G 799G 22% /mnt/oracle_share
//yyyxxfile03.yyyhu.net/yyyxxx
2.5T 2.3T 268G 90% /mnt/yyyxxx

xxxxDB1-root@yyyzzxxxxrac1:~# free -m
total used free shared buffers cached
Mem: 129027 101063 27963 0 397 94404
-/+ buffers/cache: 6261 122765
Swap: 60735 0 60735

AF

Responses