looking for a summary of what are different kernel threads

Latest response

Hi everyone,

has anyone come upon a good summary of what different kernel threads are? aoi, events, fc_dl_1, ib_addr, ib_cm, iw_cm_wc, kblockd, khelper, kthread, local_sa, migration, kjournald, watchdog, etc. It would be handy to have them described in one document - at least the most common kernel threads.

 

Kernel threads are marked in the ps output with square brackets. An example of output from a RHEL5 system:

 

[root@coke ~]# ps aux|grep ]
root         1  0.0  0.0  10368   760 ?        Ss   10:44   0:03 init [3]                               
root         2  0.0  0.0      0     0 ?        S<   10:44   0:00 [migration/0]
root         3  0.0  0.0      0     0 ?        SN   10:44   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S<   10:44   0:00 [watchdog/0]
root         5  0.0  0.0      0     0 ?        S<   10:44   0:00 [migration/1]
root         6  0.0  0.0      0     0 ?        SN   10:44   0:00 [ksoftirqd/1]
root         7  0.0  0.0      0     0 ?        S<   10:44   0:00 [watchdog/1]
root         8  0.0  0.0      0     0 ?        S<   10:44   0:00 [migration/2]
root         9  0.0  0.0      0     0 ?        SN   10:44   0:00 [ksoftirqd/2]
root        10  0.0  0.0      0     0 ?        S<   10:44   0:00 [watchdog/2]
root        11  0.0  0.0      0     0 ?        S<   10:44   0:00 [migration/3]
root        12  0.0  0.0      0     0 ?        SN   10:44   0:00 [ksoftirqd/3]
root        13  0.0  0.0      0     0 ?        S<   10:44   0:00 [watchdog/3]
root        14  0.0  0.0      0     0 ?        S<   10:44   0:00 [migration/4]
root        15  0.0  0.0      0     0 ?        SN   10:44   0:00 [ksoftirqd/4]
root        16  0.0  0.0      0     0 ?        S<   10:44   0:00 [watchdog/4]
root        17  0.0  0.0      0     0 ?        S<   10:44   0:00 [migration/5]
root        18  0.0  0.0      0     0 ?        SN   10:44   0:00 [ksoftirqd/5]
root        19  0.0  0.0      0     0 ?        S<   10:44   0:00 [watchdog/5]
root        20  0.0  0.0      0     0 ?        S<   10:44   0:00 [migration/6]
root        21  0.0  0.0      0     0 ?        SN   10:44   0:00 [ksoftirqd/6]
root        22  0.0  0.0      0     0 ?        S<   10:44   0:00 [watchdog/6]
root        23  0.0  0.0      0     0 ?        S<   10:44   0:00 [migration/7]
root        24  0.0  0.0      0     0 ?        SN   10:44   0:00 [ksoftirqd/7]
root        25  0.0  0.0      0     0 ?        S<   10:44   0:00 [watchdog/7]
root        26  0.0  0.0      0     0 ?        S<   10:44   0:00 [migration/8]
root        27  0.0  0.0      0     0 ?        SN   10:44   0:00 [ksoftirqd/8]
root        28  0.0  0.0      0     0 ?        S<   10:44   0:00 [watchdog/8]
root        29  0.0  0.0      0     0 ?        S<   10:44   0:00 [migration/9]
root        30  0.0  0.0      0     0 ?        SN   10:44   0:00 [ksoftirqd/9]
root        31  0.0  0.0      0     0 ?        S<   10:44   0:00 [watchdog/9]
root        32  0.0  0.0      0     0 ?        S<   10:44   0:00 [migration/10]
root        33  0.0  0.0      0     0 ?        SN   10:44   0:00 [ksoftirqd/10]
root        34  0.0  0.0      0     0 ?        S<   10:44   0:00 [watchdog/10]
root        35  0.0  0.0      0     0 ?        S<   10:44   0:00 [migration/11]
root        36  0.0  0.0      0     0 ?        SN   10:44   0:00 [ksoftirqd/11]
root        37  0.0  0.0      0     0 ?        S<   10:44   0:00 [watchdog/11]
root        38  0.0  0.0      0     0 ?        S<   10:44   0:00 [events/0]
root        39  0.0  0.0      0     0 ?        S<   10:44   0:00 [events/1]
root        40  0.0  0.0      0     0 ?        S<   10:44   0:00 [events/2]
root        41  0.0  0.0      0     0 ?        S<   10:44   0:00 [events/3]
root        42  0.0  0.0      0     0 ?        S<   10:44   0:00 [events/4]
root        43  0.0  0.0      0     0 ?        S<   10:44   0:00 [events/5]
root        44  0.0  0.0      0     0 ?        S<   10:44   0:00 [events/6]
root        45  0.0  0.0      0     0 ?        S<   10:44   0:00 [events/7]
root        46  0.0  0.0      0     0 ?        S<   10:44   0:00 [events/8]
root        47  0.0  0.0      0     0 ?        S<   10:44   0:00 [events/9]
root        48  0.0  0.0      0     0 ?        S<   10:44   0:00 [events/10]
root        49  0.0  0.0      0     0 ?        S<   10:44   0:00 [events/11]
root        50  0.0  0.0      0     0 ?        S<   10:44   0:00 [khelper]
root       313  0.0  0.0      0     0 ?        S<   10:44   0:00 [kthread]
root       329  0.0  0.0      0     0 ?        S<   10:44   0:00 [kblockd/0]
root       330  0.0  0.0      0     0 ?        S<   10:44   0:00 [kblockd/1]
root       331  0.0  0.0      0     0 ?        S<   10:44   0:00 [kblockd/2]
root       332  0.0  0.0      0     0 ?        S<   10:44   0:00 [kblockd/3]
root       333  0.0  0.0      0     0 ?        S<   10:44   0:00 [kblockd/4]
root       334  0.0  0.0      0     0 ?        S<   10:44   0:00 [kblockd/5]
root       335  0.0  0.0      0     0 ?        S<   10:44   0:00 [kblockd/6]
root       336  0.0  0.0      0     0 ?        S<   10:44   0:00 [kblockd/7]
root       337  0.0  0.0      0     0 ?        S<   10:44   0:00 [kblockd/8]
root       338  0.0  0.0      0     0 ?        S<   10:44   0:00 [kblockd/9]
root       339  0.0  0.0      0     0 ?        S<   10:44   0:00 [kblockd/10]
root       340  0.0  0.0      0     0 ?        S<   10:44   0:00 [kblockd/11]
root       341  0.0  0.0      0     0 ?        S<   10:44   0:00 [kacpid]
root       509  0.0  0.0      0     0 ?        S<   10:44   0:00 [cqueue/0]
root       510  0.0  0.0      0     0 ?        S<   10:44   0:00 [cqueue/1]
root       511  0.0  0.0      0     0 ?        S<   10:44   0:00 [cqueue/2]
root       512  0.0  0.0      0     0 ?        S<   10:44   0:00 [cqueue/3]
root       513  0.0  0.0      0     0 ?        S<   10:44   0:00 [cqueue/4]
root       514  0.0  0.0      0     0 ?        S<   10:44   0:00 [cqueue/5]
root       515  0.0  0.0      0     0 ?        S<   10:44   0:00 [cqueue/6]
root       516  0.0  0.0      0     0 ?        S<   10:44   0:00 [cqueue/7]
root       517  0.0  0.0      0     0 ?        S<   10:44   0:00 [cqueue/8]
root       518  0.0  0.0      0     0 ?        S<   10:44   0:00 [cqueue/9]
root       519  0.0  0.0      0     0 ?        S<   10:44   0:00 [cqueue/10]
root       520  0.0  0.0      0     0 ?        S<   10:44   0:00 [cqueue/11]
root       523  0.0  0.0      0     0 ?        S<   10:44   0:00 [khubd]
root       525  0.0  0.0      0     0 ?        S<   10:44   0:00 [kseriod]
root       684  0.0  0.0      0     0 ?        S    10:44   0:00 [khungtaskd]
root       685  0.0  0.0      0     0 ?        S    10:44   0:00 [pdflush]
root       686  0.2  0.0      0     0 ?        S    10:44   0:14 [pdflush]
root       687  0.0  0.0      0     0 ?        S<   10:44   0:00 [kswapd0]
root       688  0.0  0.0      0     0 ?        S<   10:44   0:00 [kswapd1]
root       689  0.0  0.0      0     0 ?        S<   10:44   0:00 [aio/0]
root       690  0.0  0.0      0     0 ?        S<   10:44   0:00 [aio/1]
root       691  0.0  0.0      0     0 ?        S<   10:44   0:00 [aio/2]
root       692  0.0  0.0      0     0 ?        S<   10:44   0:00 [aio/3]
root       693  0.0  0.0      0     0 ?        S<   10:44   0:00 [aio/4]
root       694  0.0  0.0      0     0 ?        S<   10:44   0:00 [aio/5]
root       695  0.0  0.0      0     0 ?        S<   10:44   0:00 [aio/6]
root       696  0.0  0.0      0     0 ?        S<   10:44   0:00 [aio/7]
root       697  0.0  0.0      0     0 ?        S<   10:44   0:00 [aio/8]
root       698  0.0  0.0      0     0 ?        S<   10:44   0:00 [aio/9]
root       699  0.0  0.0      0     0 ?        S<   10:44   0:00 [aio/10]
root       700  0.0  0.0      0     0 ?        S<   10:44   0:00 [aio/11]
root       855  0.0  0.0      0     0 ?        S<   10:44   0:00 [kpsmoused]
root       990  0.0  0.0      0     0 ?        S<   10:44   0:00 [scsi_eh_0]
root       991  0.0  0.0      0     0 ?        S<   10:44   0:00 [qla2xxx_0_dpc]
root       992  0.0  0.0      0     0 ?        S<   10:44   0:00 [scsi_wq_0]
root       993  0.0  0.0      0     0 ?        S<   10:44   0:00 [fc_wq_0]
root       994  0.0  0.0      0     0 ?        S<   10:44   0:00 [fc_dl_0]
root       995  0.0  0.0      0     0 ?        S<   10:44   0:00 [scsi_eh_1]
root       996  0.0  0.0      0     0 ?        S<   10:44   0:00 [qla2xxx_1_dpc]
root       997  0.0  0.0      0     0 ?        S<   10:44   0:00 [scsi_wq_1]
root       998  0.0  0.0      0     0 ?        S<   10:44   0:00 [fc_wq_1]
root       999  0.0  0.0      0     0 ?        S<   10:44   0:00 [fc_dl_1]
root      1026  0.0  0.0      0     0 ?        S<   10:44   0:00 [kstriped]
root      1079  0.0  0.0      0     0 ?        S<   10:44   0:00 [ksnapd]
root      1134  0.0  0.0      0     0 ?        S<   10:44   0:05 [kjournald]
root      1159  0.0  0.0      0     0 ?        S<   10:44   0:00 [kauditd]
root      1995  0.0  0.0      0     0 ?        S<   10:44   0:00 [kedac]
root      2826  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpathd/0]
root      2827  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpathd/1]
root      2828  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpathd/2]
root      2829  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpathd/3]
root      2830  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpathd/4]
root      2831  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpathd/5]
root      2832  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpathd/6]
root      2833  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpathd/7]
root      2834  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpathd/8]
root      2835  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpathd/9]
root      2836  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpathd/10]
root      2837  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpathd/11]
root      2838  0.0  0.0      0     0 ?        S<   10:44   0:00 [kmpath_handlerd]
root      2916  0.3  0.0      0     0 ?        S<   10:44   0:20 [kjournald]
root      2919  0.0  0.0      0     0 ?        S<   10:44   0:00 [kjournald]
root      3044  0.0  0.0      0     0 ?        S<   10:44   0:00 [iscsi_eh]
root      3193  0.0  0.0      0     0 ?        S<   10:45   0:00 [cnic_wq]
root      3251  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_addr]
root      3296  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_mcast]
root      3298  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_inform]
root      3299  0.0  0.0      0     0 ?        S<   10:45   0:00 [local_sa]
root      3313  0.0  0.0      0     0 ?        S<   10:45   0:00 [iw_cm_wq]
root      3327  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_cm/0]
root      3329  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_cm/1]
root      3330  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_cm/2]
root      3331  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_cm/3]
root      3332  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_cm/4]
root      3333  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_cm/5]
root      3334  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_cm/6]
root      3335  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_cm/7]
root      3336  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_cm/8]
root      3337  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_cm/9]
root      3338  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_cm/10]
root      3339  0.0  0.0      0     0 ?        S<   10:45   0:00 [ib_cm/11]
root      3352  0.0  0.0      0     0 ?        S<   10:45   0:00 [rdma_cm]
root      3563  0.0  0.0      0     0 ?        S<   10:45   0:00 [bond0]
root      3975  0.0  0.0      0     0 ?        S<   10:45   0:00 [rpciod/0]
root      3976  0.0  0.0      0     0 ?        S<   10:45   0:00 [rpciod/1]
root      3977  0.0  0.0      0     0 ?        S<   10:45   0:00 [rpciod/2]
root      3978  0.0  0.0      0     0 ?        S<   10:45   0:00 [rpciod/3]
root      3979  0.0  0.0      0     0 ?        S<   10:45   0:00 [rpciod/4]
root      3980  0.0  0.0      0     0 ?        S<   10:45   0:00 [rpciod/5]
root      3981  0.0  0.0      0     0 ?        S<   10:45   0:00 [rpciod/6]
root      3982  0.0  0.0      0     0 ?        S<   10:45   0:00 [rpciod/7]
root      3983  0.0  0.0      0     0 ?        S<   10:45   0:00 [rpciod/8]
root      3984  0.0  0.0      0     0 ?        S<   10:45   0:00 [rpciod/9]
root      3985  0.0  0.0      0     0 ?        S<   10:45   0:00 [rpciod/10]
root      3986  0.0  0.0      0     0 ?        S<   10:45   0:00 [rpciod/11]
root      4349  0.0  0.0 110968  6620 ?        Ss   11:03   0:00 sshd: guzenkovs [priv]
root      4933  0.0  0.0      0     0 ?        SN   10:45   0:00 [kipmi0]
avahi     7932  0.0  0.0  21628  1312 ?        Ss   10:46   0:00 avahi-daemon: running [monitor02.local]
root      8361  0.0  0.0 110376  6608 ?        Ss   10:46   0:00 sshd: guzenkovs [priv]
root      9321  0.0  0.0 110376  6596 ?        Ss   10:46   0:00 sshd: guzenkovs [priv]
root      9472  0.0  0.0 110376  6592 ?        Ss   10:46   0:00 sshd: guzenkovs [priv]
root     10294  0.0  0.0 110376  6608 ?        Ss   10:47   0:00 sshd: guzenkovs [priv]
root     19592  0.0  0.0  61204   828 pts/3    S+   12:21   0:00 grep ]


[root@coke ~]# ps aux | grep ] | awk -F[ '{print $2}' | awk -F/ '{print $1}' |awk -F] '{print $1}'| sort| uniq

3
aio
bond0
cnic_wq
cqueue
events
fc_dl_0
fc_dl_1
fc_wq_0
fc_wq_1
ib_addr
ib_cm
ib_inform
ib_mcast
iscsi_eh
iw_cm_wq
kacpid
kauditd
kblockd
kedac
khelper
khubd
khungtaskd
kipmi0
kjournald
kmpathd
kmpath_handlerd
kpsmoused
kseriod
ksnapd
ksoftirqd
kstriped
kswapd0
kswapd1
kthread
local_sa
migration
monitor02.local
pdflush
priv
qla2xxx_0_dpc
qla2xxx_1_dpc
rdma_cm
rpciod
scsi_eh_0
scsi_eh_1
scsi_wq_0
scsi_wq_1
watchdog

 

 

Another question is on naming convention for multiple threads of the same name. In the output above I can see three different naming conventions:

1) _0, _1, ... e.g. [scsi_wq_0], [scsi_wq_1]

2) /1, /2, ... e.g. [aio/0], [aio/1]

3) 1, 2, ... e.g. [kswapd0], [kswapd1] --note that there is only one swap disk on that system.

Is there any semantic difference between them?

gu
Log in to join the conversation

Responses

rh Red Hat Guru 4540 points
24 October 2012 6:04 PM

The kernel source browser provides a good starting point for getting an overview of the various kernel helper threads, in particular when you start with the calls to the functions that create them, like kthread_create, kthread_run, and create_workqueue.

EA Newbie 10 points
4 July 2017 4:13 PM

Almost as usefull as reading vi source code to know what kind of software it is.

gu Guru Pro 2378 points
25 October 2012 1:46 PM

Thanks Ray,

Source browser lools great for working with the source. But I am looking for something sysadmin-friendly, not programmer-friendly. A _short_ summary of what is what. 

rh Red Hat Guru 4540 points
25 October 2012 2:37 PM

Sergey, can you help me understand what you would like to use this information for?

 

The kernel helper threads are essentially routines that are internal to the kernel which (for technical reasons) are implemented quite similarly to user-space processes (and, as such, show up in ps, top etc.). They are of interest to kernel developers, but there is little to them that I can imagine is of real interest to sysadmins. The behaviour of some of the helper threads may be tunable to some degree, but that should largely be covered by existing kernel documentation like Documentation/sysctl/ and section 7 manual pages already.

EA Newbie 10 points
4 July 2017 4:14 PM

dead link

TT Pro 427 points
23 November 2017 11:37 AM

It is probably not meant as a link (URL), but a directory on disk, if you have "installed" the source or related to the source browser at https://access.redhat.com/labs/psb/

E.g. for 3.10: https://access.redhat.com/labs/psb/versions/kernel-3.10.0-693.5.2.el7/Documentation/sysctl

You can also browse it for various versions of the Linux kernel at e.g. http://elixir.free-electrons.com/linux/latest/source/Documentation/sysctl

Marc Milgram's picture Red Hat Guru 1324 points
26 October 2012 1:56 AM

Here is a list of what some of those kernel threads do:

migration: Used to move processes between CPUs - for load balancing
ksoftirqd: Kernel healper thread to handle softirqs that can't be handled immediately
watchdog: Notices if the system appears to be hung
events: Handles kernel workqueues that aren't handled by other threads
khelper: used as a schedulable context for stuff that call_user_mode_helper wants to run; like /sbin/hotplug or modprobe
kblockd: used for running low-level disk operations
kacpid: deals with ACPI - such as for power saving functions
khubd: used for configuring USB
kseriod: used for handling serial io
khungtaskd: used for detecting hung tasks
pdflush: sends memory to disk - to free RAM
kswapd: swaps processes to disk
aio: handles asynchronous IO
kpsmoused: handles mouse IO
scsi_eh: SCSI event handler
qla2xxx_0_dpc: qla device driver thread
scsi_wq: scsi work queue handler
fc_wq: fiber chanel work queue handler
fc_dl: fiber chanel handler
kstriped: helper thread for striped volumes
ksnapd: helper thread for handling dm-snaps (disk snapshots)
kjournald: helper thread for handling filesystem journals
kauditd: kernel thread for handling audit events
kmpathd: multipath daemon helper thread
ib_*: infiniband threads

As to the question about the semantic differences between the different ways of adding numbers... The ones that have /number at the end are one per core. The other numbering is up to the code that creates the thread.

Information on some other threads:

kworker: handles delayed work from throughout the kernel where the work is not being handled by one of the other queues. The number of kworker threads vary over time based on the amount of work in the work queues.
kondemand: Used by ondemand CPU frequency govenor
YB Red Hat Pro 774 points
22 November 2017 2:24 AM

Hello Mark,

Can you update it for 'kworker' thread please ?

Marc Milgram's picture Red Hat Guru 1324 points
22 November 2017 8:24 PM

kworker threads handle delayed work fro throughout the kernel where the work is not being handled by one of the other queues. The number of kworker threads vary over time based on the amount of work in the work queues.

gu Guru Pro 2378 points
31 October 2012 12:53 AM

Marc,

thank's a lot. That's exactly what I was looking for.

gu Guru Pro 2378 points
31 October 2012 12:59 AM

Ray,

Sysadmins should know what processes are running on their systems, both for security and performance (where memory and CPU cyclesgoes) reasons.

Marc provided an excellent comment below.

gu Guru Pro 2378 points
31 October 2012 1:29 AM

Based on Marc Milgram's input, I started a spreadsheet  https://docs.google.com/spreadsheet/ccc?key=0AnWaJdi9mw2BdE53T1B6bUlUekdaa3NkQWZLMU9rdUE

 

Please feel free to review and edit it. 

 

Later on I will edit the initial article itself to make it look more like a Knowledge base article, and will attach the spreadsheet.

gu Guru Pro 2378 points
8 November 2012 9:36 AM

I added a few more columns to the spreadsheet.

First column has one of three values. K for Kernel thread, U for user process/thread, or S for init.d service. Thus from kernel-thread-specific this spreadsheet becomes running process/thread generic. Those who want to view only kernel threads can still do that by sorting  by the first column, or filtering by its content :-)

 

There are also RHEL5 and RHEL6 columns to denote whether RHEL5 or RHEL6 accordingly supply these packages. This allows us to describe services that are not supplied by Red Hat at the moment.

 

Please feel free to download the spreadsheet and keep it to yourself. But I would appreciate if you could make the spreadsheet it a little bit better.

AA Red Hat Newbie 10 points
18 October 2018 5:26 AM

Hello Mark, Can you please update it for 'kondemand' thread ? We know it deals with CPU frequency but in which situation/condition it will get invoked will be more useful. Thanks

Marc Milgram's picture Red Hat Guru 1324 points
19 October 2018 2:14 PM

Hello Amit, It looks like kondemand is used to have work that is scheduled periodically for the ondemand cpu frequency governor. This govenor can change the frequency of the cpu, so that when the system is busy, the cpu runs at maximum frequency, but if the system is idle, the cpu will scale down to a lower frequency.

SP Community Member 31 points
16 March 2019 4:25 AM

Hello Marc Milgram

Can you please update it for 'ksnapd' thread ? with little details

Marc Milgram's picture Red Hat Guru 1324 points
20 March 2019 3:43 PM

Hello Suraj,

ksnapd is no longer used. It was removed from the kernel in RHEL-6.2. It was used for dealing with disk snapshots.

SP Community Member 31 points
18 July 2019 8:15 PM

Hello Marc Milgram

Can you please update it for 'ksoftireqd' thread ? with little details And why ksoftireq usage 100% CPU ? What is this thread exactly ??

Marc Milgram's picture Red Hat Guru 1324 points
19 July 2019 5:08 PM

Suraj,

None of the kernel threads should be using 100% CPU most of the time. Check for warning messages in /var/log/messages. If you can open a case, a Red Hat associate can investigate what is causing a kernel thread ot use 100% CPU.