I am seeing high usage of the "init" process over 30 min periods

Latest response

This is weird, I am seeing high CPU and IO utilisation for the init process (kernel 2.6.32-358.18.1.el6.x86_64 on RHEL 6). It continues for approx 30 mins then drops off. CPU and memory are fine.

11:10 init = 75%, 756GB read disk
11:20 init = 84%, 1.2TB read disk
11:30 init = 67%, 530G read disk

Anyone seen this before? I don't have a bug report in yet, have yet to capture the system in the act. Share and Enjoy!

Responses

Ian,

There's obviously many variables involved in this. Here is just one of many variables to consider:
-- by any chance have you omitted large storage from being indexed via the /etc/updatedb.conf so that your large storage is not indexed by mlocate by the /etc/cron.daily/mlocate.cron?

Again, there are certainly many more factors not yet addressed in this sum of the current discussion to consider.

Also consider checking the file system to determine what is hitting it with lsof; thanks PixelDrift for the improved use of lsof

  • The next time it happens, examine what is using the disk with lsof against the applicable filesystems

added
See the very useful examples of lsof in the man page

man lsof | less +/examples

One example to a mount point.

lsof /path/to/mount
  • If you know the user (like the user named "jack"), the below would be for a specific user, if you knew the user.
lsof -nPFp -u jack | cut -c2- | xargs ps -o cmd= -p1 | grep -v ^.*/init$

lsof against init

lsof -c init | grep -v ^.*/init$

Here's some more lsof examples at this link, and more at this IBM website.

Let us know how this goes.

Kind Regards...

Thanks Remmele, interesting points, will setup a crib sheet for capturing information the next time it happens.

Ian,

Is there any chance these disks are used via nfs/sftp/ftp/samba or some other means where the files are served to others? If so, tcpdump (and log access files) might help to see who is using it.

(just in case, and this may not be the entire fix at all)
The bit I mentioned above on /etc/updatedb.conf - see this link, the man page and update the PRUNEFS, PRUNENAMES, PRUNEPATHS directives to omit file indexing of your large file systems.

I've seen where even certain vulnerability scans (perhaps ran by your security folks, IA, etc) will attempt to iterate through a file system, such as sanctioned vulnerability scans.

Let us know how it goes...

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.