looking for a summary of what are different kernel threads
Hi everyone,
has anyone come upon a good summary of what different kernel threads are? aoi, events, fc_dl_1, ib_addr, ib_cm, iw_cm_wc, kblockd, khelper, kthread, local_sa, migration, kjournald, watchdog, etc. It would be handy to have them described in one document - at least the most common kernel threads.
Kernel threads are marked in the ps output with square brackets. An example of output from a RHEL5 system:
[root@coke ~]# ps aux|grep ] root 1 0.0 0.0 10368 760 ? Ss 10:44 0:03 init [3] root 2 0.0 0.0 0 0 ? S< 10:44 0:00 [migration/0] root 3 0.0 0.0 0 0 ? SN 10:44 0:00 [ksoftirqd/0] root 4 0.0 0.0 0 0 ? S< 10:44 0:00 [watchdog/0] root 5 0.0 0.0 0 0 ? S< 10:44 0:00 [migration/1] root 6 0.0 0.0 0 0 ? SN 10:44 0:00 [ksoftirqd/1] root 7 0.0 0.0 0 0 ? S< 10:44 0:00 [watchdog/1] root 8 0.0 0.0 0 0 ? S< 10:44 0:00 [migration/2] root 9 0.0 0.0 0 0 ? SN 10:44 0:00 [ksoftirqd/2] root 10 0.0 0.0 0 0 ? S< 10:44 0:00 [watchdog/2] root 11 0.0 0.0 0 0 ? S< 10:44 0:00 [migration/3] root 12 0.0 0.0 0 0 ? SN 10:44 0:00 [ksoftirqd/3] root 13 0.0 0.0 0 0 ? S< 10:44 0:00 [watchdog/3] root 14 0.0 0.0 0 0 ? S< 10:44 0:00 [migration/4] root 15 0.0 0.0 0 0 ? SN 10:44 0:00 [ksoftirqd/4] root 16 0.0 0.0 0 0 ? S< 10:44 0:00 [watchdog/4] root 17 0.0 0.0 0 0 ? S< 10:44 0:00 [migration/5] root 18 0.0 0.0 0 0 ? SN 10:44 0:00 [ksoftirqd/5] root 19 0.0 0.0 0 0 ? S< 10:44 0:00 [watchdog/5] root 20 0.0 0.0 0 0 ? S< 10:44 0:00 [migration/6] root 21 0.0 0.0 0 0 ? SN 10:44 0:00 [ksoftirqd/6] root 22 0.0 0.0 0 0 ? S< 10:44 0:00 [watchdog/6] root 23 0.0 0.0 0 0 ? S< 10:44 0:00 [migration/7] root 24 0.0 0.0 0 0 ? SN 10:44 0:00 [ksoftirqd/7] root 25 0.0 0.0 0 0 ? S< 10:44 0:00 [watchdog/7] root 26 0.0 0.0 0 0 ? S< 10:44 0:00 [migration/8] root 27 0.0 0.0 0 0 ? SN 10:44 0:00 [ksoftirqd/8] root 28 0.0 0.0 0 0 ? S< 10:44 0:00 [watchdog/8] root 29 0.0 0.0 0 0 ? S< 10:44 0:00 [migration/9] root 30 0.0 0.0 0 0 ? SN 10:44 0:00 [ksoftirqd/9] root 31 0.0 0.0 0 0 ? S< 10:44 0:00 [watchdog/9] root 32 0.0 0.0 0 0 ? S< 10:44 0:00 [migration/10] root 33 0.0 0.0 0 0 ? SN 10:44 0:00 [ksoftirqd/10] root 34 0.0 0.0 0 0 ? S< 10:44 0:00 [watchdog/10] root 35 0.0 0.0 0 0 ? S< 10:44 0:00 [migration/11] root 36 0.0 0.0 0 0 ? SN 10:44 0:00 [ksoftirqd/11] root 37 0.0 0.0 0 0 ? S< 10:44 0:00 [watchdog/11] root 38 0.0 0.0 0 0 ? S< 10:44 0:00 [events/0] root 39 0.0 0.0 0 0 ? S< 10:44 0:00 [events/1] root 40 0.0 0.0 0 0 ? S< 10:44 0:00 [events/2] root 41 0.0 0.0 0 0 ? S< 10:44 0:00 [events/3] root 42 0.0 0.0 0 0 ? S< 10:44 0:00 [events/4] root 43 0.0 0.0 0 0 ? S< 10:44 0:00 [events/5] root 44 0.0 0.0 0 0 ? S< 10:44 0:00 [events/6] root 45 0.0 0.0 0 0 ? S< 10:44 0:00 [events/7] root 46 0.0 0.0 0 0 ? S< 10:44 0:00 [events/8] root 47 0.0 0.0 0 0 ? S< 10:44 0:00 [events/9] root 48 0.0 0.0 0 0 ? S< 10:44 0:00 [events/10] root 49 0.0 0.0 0 0 ? S< 10:44 0:00 [events/11] root 50 0.0 0.0 0 0 ? S< 10:44 0:00 [khelper] root 313 0.0 0.0 0 0 ? S< 10:44 0:00 [kthread] root 329 0.0 0.0 0 0 ? S< 10:44 0:00 [kblockd/0] root 330 0.0 0.0 0 0 ? S< 10:44 0:00 [kblockd/1] root 331 0.0 0.0 0 0 ? S< 10:44 0:00 [kblockd/2] root 332 0.0 0.0 0 0 ? S< 10:44 0:00 [kblockd/3] root 333 0.0 0.0 0 0 ? S< 10:44 0:00 [kblockd/4] root 334 0.0 0.0 0 0 ? S< 10:44 0:00 [kblockd/5] root 335 0.0 0.0 0 0 ? S< 10:44 0:00 [kblockd/6] root 336 0.0 0.0 0 0 ? S< 10:44 0:00 [kblockd/7] root 337 0.0 0.0 0 0 ? S< 10:44 0:00 [kblockd/8] root 338 0.0 0.0 0 0 ? S< 10:44 0:00 [kblockd/9] root 339 0.0 0.0 0 0 ? S< 10:44 0:00 [kblockd/10] root 340 0.0 0.0 0 0 ? S< 10:44 0:00 [kblockd/11] root 341 0.0 0.0 0 0 ? S< 10:44 0:00 [kacpid] root 509 0.0 0.0 0 0 ? S< 10:44 0:00 [cqueue/0] root 510 0.0 0.0 0 0 ? S< 10:44 0:00 [cqueue/1] root 511 0.0 0.0 0 0 ? S< 10:44 0:00 [cqueue/2] root 512 0.0 0.0 0 0 ? S< 10:44 0:00 [cqueue/3] root 513 0.0 0.0 0 0 ? S< 10:44 0:00 [cqueue/4] root 514 0.0 0.0 0 0 ? S< 10:44 0:00 [cqueue/5] root 515 0.0 0.0 0 0 ? S< 10:44 0:00 [cqueue/6] root 516 0.0 0.0 0 0 ? S< 10:44 0:00 [cqueue/7] root 517 0.0 0.0 0 0 ? S< 10:44 0:00 [cqueue/8] root 518 0.0 0.0 0 0 ? S< 10:44 0:00 [cqueue/9] root 519 0.0 0.0 0 0 ? S< 10:44 0:00 [cqueue/10] root 520 0.0 0.0 0 0 ? S< 10:44 0:00 [cqueue/11] root 523 0.0 0.0 0 0 ? S< 10:44 0:00 [khubd] root 525 0.0 0.0 0 0 ? S< 10:44 0:00 [kseriod] root 684 0.0 0.0 0 0 ? S 10:44 0:00 [khungtaskd] root 685 0.0 0.0 0 0 ? S 10:44 0:00 [pdflush] root 686 0.2 0.0 0 0 ? S 10:44 0:14 [pdflush] root 687 0.0 0.0 0 0 ? S< 10:44 0:00 [kswapd0] root 688 0.0 0.0 0 0 ? S< 10:44 0:00 [kswapd1] root 689 0.0 0.0 0 0 ? S< 10:44 0:00 [aio/0] root 690 0.0 0.0 0 0 ? S< 10:44 0:00 [aio/1] root 691 0.0 0.0 0 0 ? S< 10:44 0:00 [aio/2] root 692 0.0 0.0 0 0 ? S< 10:44 0:00 [aio/3] root 693 0.0 0.0 0 0 ? S< 10:44 0:00 [aio/4] root 694 0.0 0.0 0 0 ? S< 10:44 0:00 [aio/5] root 695 0.0 0.0 0 0 ? S< 10:44 0:00 [aio/6] root 696 0.0 0.0 0 0 ? S< 10:44 0:00 [aio/7] root 697 0.0 0.0 0 0 ? S< 10:44 0:00 [aio/8] root 698 0.0 0.0 0 0 ? S< 10:44 0:00 [aio/9] root 699 0.0 0.0 0 0 ? S< 10:44 0:00 [aio/10] root 700 0.0 0.0 0 0 ? S< 10:44 0:00 [aio/11] root 855 0.0 0.0 0 0 ? S< 10:44 0:00 [kpsmoused] root 990 0.0 0.0 0 0 ? S< 10:44 0:00 [scsi_eh_0] root 991 0.0 0.0 0 0 ? S< 10:44 0:00 [qla2xxx_0_dpc] root 992 0.0 0.0 0 0 ? S< 10:44 0:00 [scsi_wq_0] root 993 0.0 0.0 0 0 ? S< 10:44 0:00 [fc_wq_0] root 994 0.0 0.0 0 0 ? S< 10:44 0:00 [fc_dl_0] root 995 0.0 0.0 0 0 ? S< 10:44 0:00 [scsi_eh_1] root 996 0.0 0.0 0 0 ? S< 10:44 0:00 [qla2xxx_1_dpc] root 997 0.0 0.0 0 0 ? S< 10:44 0:00 [scsi_wq_1] root 998 0.0 0.0 0 0 ? S< 10:44 0:00 [fc_wq_1] root 999 0.0 0.0 0 0 ? S< 10:44 0:00 [fc_dl_1] root 1026 0.0 0.0 0 0 ? S< 10:44 0:00 [kstriped] root 1079 0.0 0.0 0 0 ? S< 10:44 0:00 [ksnapd] root 1134 0.0 0.0 0 0 ? S< 10:44 0:05 [kjournald] root 1159 0.0 0.0 0 0 ? S< 10:44 0:00 [kauditd] root 1995 0.0 0.0 0 0 ? S< 10:44 0:00 [kedac] root 2826 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpathd/0] root 2827 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpathd/1] root 2828 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpathd/2] root 2829 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpathd/3] root 2830 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpathd/4] root 2831 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpathd/5] root 2832 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpathd/6] root 2833 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpathd/7] root 2834 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpathd/8] root 2835 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpathd/9] root 2836 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpathd/10] root 2837 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpathd/11] root 2838 0.0 0.0 0 0 ? S< 10:44 0:00 [kmpath_handlerd] root 2916 0.3 0.0 0 0 ? S< 10:44 0:20 [kjournald] root 2919 0.0 0.0 0 0 ? S< 10:44 0:00 [kjournald] root 3044 0.0 0.0 0 0 ? S< 10:44 0:00 [iscsi_eh] root 3193 0.0 0.0 0 0 ? S< 10:45 0:00 [cnic_wq] root 3251 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_addr] root 3296 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_mcast] root 3298 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_inform] root 3299 0.0 0.0 0 0 ? S< 10:45 0:00 [local_sa] root 3313 0.0 0.0 0 0 ? S< 10:45 0:00 [iw_cm_wq] root 3327 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_cm/0] root 3329 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_cm/1] root 3330 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_cm/2] root 3331 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_cm/3] root 3332 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_cm/4] root 3333 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_cm/5] root 3334 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_cm/6] root 3335 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_cm/7] root 3336 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_cm/8] root 3337 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_cm/9] root 3338 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_cm/10] root 3339 0.0 0.0 0 0 ? S< 10:45 0:00 [ib_cm/11] root 3352 0.0 0.0 0 0 ? S< 10:45 0:00 [rdma_cm] root 3563 0.0 0.0 0 0 ? S< 10:45 0:00 [bond0] root 3975 0.0 0.0 0 0 ? S< 10:45 0:00 [rpciod/0] root 3976 0.0 0.0 0 0 ? S< 10:45 0:00 [rpciod/1] root 3977 0.0 0.0 0 0 ? S< 10:45 0:00 [rpciod/2] root 3978 0.0 0.0 0 0 ? S< 10:45 0:00 [rpciod/3] root 3979 0.0 0.0 0 0 ? S< 10:45 0:00 [rpciod/4] root 3980 0.0 0.0 0 0 ? S< 10:45 0:00 [rpciod/5] root 3981 0.0 0.0 0 0 ? S< 10:45 0:00 [rpciod/6] root 3982 0.0 0.0 0 0 ? S< 10:45 0:00 [rpciod/7] root 3983 0.0 0.0 0 0 ? S< 10:45 0:00 [rpciod/8] root 3984 0.0 0.0 0 0 ? S< 10:45 0:00 [rpciod/9] root 3985 0.0 0.0 0 0 ? S< 10:45 0:00 [rpciod/10] root 3986 0.0 0.0 0 0 ? S< 10:45 0:00 [rpciod/11] root 4349 0.0 0.0 110968 6620 ? Ss 11:03 0:00 sshd: guzenkovs [priv] root 4933 0.0 0.0 0 0 ? SN 10:45 0:00 [kipmi0] avahi 7932 0.0 0.0 21628 1312 ? Ss 10:46 0:00 avahi-daemon: running [monitor02.local] root 8361 0.0 0.0 110376 6608 ? Ss 10:46 0:00 sshd: guzenkovs [priv] root 9321 0.0 0.0 110376 6596 ? Ss 10:46 0:00 sshd: guzenkovs [priv] root 9472 0.0 0.0 110376 6592 ? Ss 10:46 0:00 sshd: guzenkovs [priv] root 10294 0.0 0.0 110376 6608 ? Ss 10:47 0:00 sshd: guzenkovs [priv] root 19592 0.0 0.0 61204 828 pts/3 S+ 12:21 0:00 grep ] [root@coke ~]# ps aux | grep ] | awk -F[ '{print $2}' | awk -F/ '{print $1}' |awk -F] '{print $1}'| sort| uniq 3 aio bond0 cnic_wq cqueue events fc_dl_0 fc_dl_1 fc_wq_0 fc_wq_1 ib_addr ib_cm ib_inform ib_mcast iscsi_eh iw_cm_wq kacpid kauditd kblockd kedac khelper khubd khungtaskd kipmi0 kjournald kmpathd kmpath_handlerd kpsmoused kseriod ksnapd ksoftirqd kstriped kswapd0 kswapd1 kthread local_sa migration monitor02.local pdflush priv qla2xxx_0_dpc qla2xxx_1_dpc rdma_cm rpciod scsi_eh_0 scsi_eh_1 scsi_wq_0 scsi_wq_1 watchdog
Another question is on naming convention for multiple threads of the same name. In the output above I can see three different naming conventions:
1) _0, _1, ... e.g. [scsi_wq_0], [scsi_wq_1]
2) /1, /2, ... e.g. [aio/0], [aio/1]
3) 1, 2, ... e.g. [kswapd0], [kswapd1] --note that there is only one swap disk on that system.
Is there any semantic difference between them?
Responses
The kernel source browser provides a good starting point for getting an overview of the various kernel helper threads, in particular when you start with the calls to the functions that create them, like kthread_create, kthread_run, and create_workqueue.
Sergey, can you help me understand what you would like to use this information for?
The kernel helper threads are essentially routines that are internal to the kernel which (for technical reasons) are implemented quite similarly to user-space processes (and, as such, show up in ps, top etc.). They are of interest to kernel developers, but there is little to them that I can imagine is of real interest to sysadmins. The behaviour of some of the helper threads may be tunable to some degree, but that should largely be covered by existing kernel documentation like Documentation/sysctl/ and section 7 manual pages already.
dead link
It is probably not meant as a link (URL), but a directory on disk, if you have "installed" the source or related to the source browser at https://access.redhat.com/labs/psb/
E.g. for 3.10: https://access.redhat.com/labs/psb/versions/kernel-3.10.0-693.5.2.el7/Documentation/sysctl
You can also browse it for various versions of the Linux kernel at e.g. http://elixir.free-electrons.com/linux/latest/source/Documentation/sysctl
Here is a list of what some of those kernel threads do:
migration: Used to move processes between CPUs - for load balancing
ksoftirqd: Kernel healper thread to handle softirqs that can't be handled immediately
watchdog: Notices if the system appears to be hung
events: Handles kernel workqueues that aren't handled by other threads
khelper: used as a schedulable context for stuff that call_user_mode_helper wants to run – like /sbin/hotplug or modprobe
kblockd: used for running low-level disk operations
kacpid: deals with ACPI - such as for power saving functions
khubd: used for configuring USB
kseriod: used for handling serial io
khungtaskd: used for detecting hung tasks
pdflush: sends memory to disk - to free RAM
kswapd: swaps processes to disk
aio: handles asynchronous IO
kpsmoused: handles mouse IO
scsi_eh: SCSI event handler
qla2xxx_0_dpc: qla device driver thread
scsi_wq: scsi work queue handler
fc_wq: fiber chanel work queue handler
fc_dl: fiber chanel handler
kstriped: helper thread for striped volumes
ksnapd: helper thread for handling dm-snaps
kjournald: helper thread for handling filesystem journals
kauditd: kernel thread for handling audit events
kmpathd: multipath daemon helper thread
ib_*: infiniband threads
As to the question about the semantic differences between the different ways of adding numbers... The ones that have /number at the end are one per core. The other numbering is up to the code that creates the thread.
Information on some other threads:
kworker: handles delayed work from throughout the kernel where the work is not being handled by one of the other queues. The number of kworker threads vary over time based on the amount of work in the work queues.
kondemand: Used by ondemand CPU frequency govenor
Hello Mark, Can you please update it for 'kondemand' thread ? We know it deals with CPU frequency but in which situation/condition it will get invoked will be more useful. Thanks
Hello Amit,
It looks like kondemand
is used to have work that is scheduled periodically for the ondemand cpu frequency governor. This govenor can change the frequency of the cpu, so that when the system is busy, the cpu runs at maximum frequency, but if the system is idle, the cpu will scale down to a lower frequency.