OpenShift CoreOS node performs "native_queued_spin_lock_slowpath" soft lockup in "worker-##" process

Solution Unverified - Updated -

Issue

  • OpenShift CoreOS node performs native_queued_spin_lock_slowpath soft lockup in worker-## process (where ## is a number)
  • pstree shows worker processes owned by slkd-secrets-va with parent /opt/aquasec/slkd
  • Processes like coredns hang when closing a network socket
  • RHOCP kernel log contains softlockup logs like the following:
watchdog: BUG: soft lockup - CPU#x stuck for 22s! [worker-##:PID]

CPU: x PID: x Comm: worker-11 Not tainted 4.18.0-305.19.1.el8_4.x86_64 #1
RIP: 0010:native_queued_spin_lock_slowpath+0x20/0x1d0

Call Trace:
 queued_write_lock_slowpath+0x77/0x80
 bpf_sk_reuseport_detach+0x19/0x50
 reuseport_detach_sock+0x25/0xb0
 sk_destruct+0x43/0x60
 nf_queue_entry_release_refs+0x74/0x80
 nf_reinject+0x6e/0x1a0
 nfqnl_recv_verdict+0x29f/0x4b0 [nfnetlink_queue]
 ? __nla_validate_parse+0x12d/0x1a0
 nfnetlink_rcv_msg+0x13e/0x260 [nfnetlink]
 ? cred_has_capability+0x83/0x140
 ? __alloc_skb+0x57/0x1c0
 ? schedule+0x43/0xa0
 ? nfnetlink_net_exit_batch+0x60/0x60 [nfnetlink]
 netlink_rcv_skb+0x4c/0x120
 nfnetlink_rcv+0x68/0x15d [nfnetlink]
 netlink_unicast+0x19e/0x260
 netlink_sendmsg+0x204/0x3d0
 sock_sendmsg+0x4c/0x50
 ____sys_sendmsg+0x1eb/0x250
 ? copy_msghdr_from_user+0x5c/0x90
 ? locks_alloc_lock+0x17/0x70
 ? kmem_cache_alloc+0x12e/0x270
 ___sys_sendmsg+0x7c/0xc0
 ? do_lock_file_wait.part.32+0x38/0xc0
 ? fcntl_setlk+0x1bd/0x2f0
 __sys_sendmsg+0x57/0xa0
 do_syscall_64+0x5b/0x1a0
watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [coredns:PID]

CPU: x PID: x Comm: coredns Tainted: G        W    L   --------- -  - 4.18.0-305.19.1.el8_4.x86_64 #1
RIP: 0010:native_queued_spin_lock_slowpath+0x20/0x1d0

Call Trace:
 _raw_spin_lock_bh+0x27/0x30
 reuseport_detach_sock+0x16/0xb0
 inet_unhash+0x10c/0x190
 tcp_set_state+0x69/0x1d0
 __tcp_close+0x3a8/0x400
 tcp_close+0x1f/0x50
 inet_release+0x42/0x80
 __sock_release+0x3d/0xa0
 sock_close+0x11/0x20
 __fput+0xbe/0x250
 task_work_run+0x8a/0xb0
 exit_to_usermode_loop+0xeb/0xf0
 do_syscall_64+0x198/0x1a0
 entry_SYSCALL_64_after_hwframe+0x65/0xca

Environment

  • Red Hat OpenShift Container Platform 4
  • Aquasec cloud security product

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content