Multiple udevd processes causing high load average

Solution In Progress - Updated -

Issue

  • High CPU load while running udevadm trigger, storage stack is impacted.
  • While udevadm trigger is running, pvs command has a long delay to return, and database is affected (/etc/udev/rules.d/99-udev-oracle.rules content):

    SUBSYSTEM=="block",KERNEL=="emcpower*",PROGRAM=="/lib/udev/scsi_id -gu /dev/%P",RESULT=="360000960000597000004512030521234",OWNER="grid",GROUP="dba",MODE="0660"
    SUBSYSTEM=="block",KERNEL=="emcpower*",PROGRAM=="/lib/udev/scsi_id -gu /dev/%P",RESULT=="36000096000059700000452103062543",OWNER="grid",GROUP="dba",MODE="0660"
    [...]
    
  • When running udevadm trigger, system logs are filled with the output:

    Jan 14 11:12:13 localhost udevd[5913]: worker [34311] unexpectedly returned with status 0x0100
    Jan 14 11:12:13 localhost udevd[5913]: worker [34311] failed while handling '/devices/virtual/block/emcpowerx/emcpowerx1'
    Jan 14 11:12:13 localhost udevd[5913]: worker [34318] unexpectedly returned with status 0x0100
    Jan 14 11:12:13 localhost udevd[5913]: worker [34318] failed while handling '/devices/virtual/block/emcpowery/emcpowery1'
    Jan 14 11:12:13 localhost udevd[5913]: worker [34322] unexpectedly returned with status 0x0100
    Jan 14 11:12:13 localhost udevd[5913]: worker [34322] failed while handling '/devices/virtual/block/emcpowere/emcpowere1'
    Jan 14 11:12:14 localhost udevd[5913]: worker [34328] unexpectedly returned with status 0x0100
    Jan 14 11:12:14 localhost udevd[5913]: worker [34328] failed while handling '/devices/virtual/block/emcpowerf/emcpowerf1'
    
  • There are large number of /sbin/udev -d processes running.

  • I spot checked a few other oracle servers and saw between 1 and 3 udev processes running.

Environment

  • Red Hat Enterprise Linux (RHEL) 6,7
  • EMC PowerPath
  • Oracle Database

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content