Select Your Language

Infrastructure and Management

Cloud Computing

Storage

Runtimes

Integration and Automation

  • Comments
  • dbus daemon eating 100% of one CPU core

    Posted on

    On our database servers (Redhat 6.4) after 2-3 day of operation, the dbus daemon eating 100% of one CPU core. After few days this high load still exists, only solved if the server is restarted.
    When the dbus is using 100% of the CPU core, the Oracle JOB-s running into a deadlock, also shell sudo command is not responding to any request.
    Just for background information: the problematic Oracle JOB-s using some credential mechanism (this is the common of these jobs).

    We have tried the same setup on an other oracle DB server, which is running under CentOS and not Redhat, but finally realized the same issue.

    DB/Node parameters:

    xxxxDB1-root@yyyzzxxxxrac1:~# uname -a
    Linux yyyzzxxxxrac1 2.6.32-358.18.1.el6.x86_64 #1 SMP Fri Aug 2 17:04:38 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

    xxxxDB1-root@yyyzzxxxxrac1:~# cat /etc/redhat-release
    Red Hat Enterprise Linux Server release 6.4 (Santiago)

    SQL> select * from v$instance;
    INSTANCE_NUMBER INSTANCE_NAME HOST_NAME VERSION STARTUP_TIME STATUS PAR THREAD# ARCHIVE LOG_SWITCH_WAIT LOGINS SHU
    DATABASE_STATUS INSTANCE_ROLE ACTIVE_ST BLO
    1 SID1 yyyzzxxxxrac1 11.2.0.3.0 14-JAN. -23 OPEN YES 1 STARTED ALLOWED NO
    ACTIVE PRIMARY_INSTANCE NORMAL NO

    xxxxDB1-root@yyyzzxxxxrac1:~# dbus-daemon --version
    D-Bus Message Bus Daemon 1.2.24

    xxxxDB1-root@yyyzzxxxxrac1:~# lspci
    00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07)
    00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07)
    00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)
    00:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07)
    00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07)
    00:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07)
    00:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07)
    00:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07)
    00:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07)
    00:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07)
    00:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07)
    00:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07)
    00:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07)
    00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07)
    00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07)
    00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06)
    00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06)
    00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6)
    00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06)
    00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
    00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06)
    00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06)
    01:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch [PS]
    02:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch [PS]
    02:01.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch [PS]
    03:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe-PCI Bridge [PPB]
    04:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. G200eR2
    11:00.0 Fibre Channel: Brocade Communications Systems, Inc. 1867/1860: 16Gbps/10Gbps Fabric Adapter (rev 01)
    11:00.1 Fibre Channel: Brocade Communications Systems, Inc. 1867/1860: 16Gbps/10Gbps Fabric Adapter (rev 01)
    1b:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
    80:00.0 PCI bridge: Intel Corporation Xeon E5/Core i7 DMI2 in PCI Express Mode (rev 07)
    80:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07)
    80:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b (rev 07)
    80:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)
    80:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07)
    80:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07)
    80:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07)
    80:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07)
    80:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07)
    80:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07)
    80:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07)
    80:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07)
    80:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07)
    80:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07)
    80:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07)
    80:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07)

    xxxxDB1-root@yyyzzxxxxrac1:~# ifconfig
    bond0 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
    inet6 addr: fe80::202:c9ff:fee5:6cf0/64 Scope:Link
    UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
    RX packets:9058519605 errors:0 dropped:0 overruns:0 frame:0
    TX packets:28600922879 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:3511452359951 (3.1 TiB) TX bytes:38719928355036 (35.2 TiB)

    bond0.51 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
    inet addr:10.51.XX.XX Bcast:10.51.0.255 Mask:255.255.255.0
    inet6 addr: fe80::202:c9ff:fee5:6cf0/64 Scope:Link
    UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
    RX packets:402143 errors:0 dropped:0 overruns:0 frame:0
    TX packets:37684 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:24247361 (23.1 MiB) TX bytes:1872895 (1.7 MiB)

    bond0.60 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX

    inet addr:172.27.XX.XX Bcast:172.27.61.255 Mask:255.255.254.0
    inet6 addr: fe80::202:c9ff:fee5:6cf0/64 Scope:Link
    UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
    RX packets:6262516952 errors:0 dropped:0 overruns:0 frame:0
    TX packets:6089735344 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:1181302289616 (1.0 TiB) TX bytes:35601212876851 (32.3 TiB)

    bond0.60:1 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
    inet addr:172.27.XX.XX Bcast:172.27.61.255 Mask:255.255.254.0
    UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

    bond0.60:2 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
    inet addr:172.27.XX.XX Bcast:172.27.61.255 Mask:255.255.254.0
    UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

    bond0.60:khr Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
    inet addr:192.168.XX.XX Bcast:192.168.20.255 Mask:255.255.255.0
    UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

    bond0.121 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX

    inet addr:10.121.XX.XX Bcast:10.121.0.255 Mask:255.255.255.0
    inet6 addr: fe80::202:c9ff:fee5:6cf0/64 Scope:Link
    UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
    RX packets:2786184418 errors:0 dropped:0 overruns:0 frame:0
    TX packets:2439892520 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:2202685530090 (2.0 TiB) TX bytes:1793990838383 (1.6 TiB)

    bond0.121:1 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
    inet addr:169.254.XX.XX Bcast:169.254.255.255 Mask:255.255.0.0
    UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

    eth0 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
    UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
    RX packets:9017550977 errors:0 dropped:0 overruns:0 frame:0
    TX packets:28600922879 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:3507715042150 (3.1 TiB) TX bytes:38719928355036 (35.2 TiB)

    eth1 Link encap:Ethernet HWaddr 00:02:C9:XX:XX:XX
    UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
    RX packets:40968628 errors:0 dropped:0 overruns:0 frame:0
    TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:3737317801 (3.4 GiB) TX bytes:0 (0.0 b)

    lo Link encap:Local Loopback

    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:120919318 errors:0 dropped:0 overruns:0 frame:0
    TX packets:120919318 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:30168996858 (28.0 GiB) TX bytes:30168996858 (28.0 GiB)

    xxxxDB1-root@yyyzzxxxxrac1:~# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/vg_redhat-lv_root
    30G 9.5G 19G 34% /
    tmpfs 64G 32G 32G 50% /dev/shm
    /dev/mapper/36005076802810d4d900000000000002dp2
    485M 93M 367M 21% /boot
    /dev/mapper/36005076802810d4d900000000000002dp1
    200M 260K 200M 1% /boot/efi
    /dev/mapper/vg_redhat-lv_u01
    60G 15G 42G 27% /u01
    172.27.XX.XX:/volume1/backup
    18T 17T 903G 96% /mnt/NAS/backup
    172.27.XX.XX:/volume1/admin
    18T 17T 903G 96% /mnt/NAS/admin
    172.27.XX.XX:/mnt/devops
    1.3T 852G 364G 71% /mnt/devops
    //yyyxxfile03.yyyhu.net/oracle/xxxxxx
    2.5T 2.3T 268G 90% /mnt/oradirs
    //172.27.XX.XX/ORACLE
    1016G 218G 799G 22% /mnt/oracle_share
    //yyyxxfile03.yyyhu.net/yyyxxx
    2.5T 2.3T 268G 90% /mnt/yyyxxx

    xxxxDB1-root@yyyzzxxxxrac1:~# free -m
    total used free shared buffers cached
    Mem: 129027 101063 27963 0 397 94404
    -/+ buffers/cache: 6261 122765
    Swap: 60735 0 60735

    by

    points

    Responses

    Red Hat

    Quick Links

    Help

    Site Info

    Related Sites

    © 2025 Red Hat, Inc.