slow ssh login with soft lockups in selinux code

Solution Verified - Updated -

Issue

  • ssh login is delayed 30 seconds or more.
  • soft lockups appear in sshd or selinuxdefcon near the function security_get_user_sids.
    Example:

    Jul 26 18:08:53 host1 kernel: NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [sshd:14871]
    

    or

    Oct 17 11:20:27 host1 kernel: NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [selinuxdefcon:14224]
    Oct 17 11:20:27 host1 kernel: Modules linked in: ip6table_filter ip6_tables sctp_diag sctp dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag vxlan ip6_udp_tunnel udp_tunnel ip_set_hash_net xt_comment xt_ipvs xt_set ip_set nfnetlink ip_vs_wlc ip_vs dummy ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock xfs libcrc32c ppdev vmw_balloon coretemp joydev pcspkr sg vmw_vmci shpchp i2c_piix4 parport_pc parport binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mptspi ahci ttm
    Oct 17 11:20:27 host1 kernel: scsi_transport_spi libahci ata_piix mptscsih drm libata mptbase crc32c_intel serio_raw vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
    Oct 17 11:20:27 host1 kernel: CPU: 2 PID: 14224 Comm: selinuxdefcon Tainted: G             L ------------ T 3.10.0-693.21.1.el7.x86_64 #1
    Oct 17 11:20:27 host1 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
    Oct 17 11:20:27 host1 kernel: task: ffff88081c6d2f70 ti: ffff880110b68000 task.ti: ffff880110b68000
    Oct 17 11:20:27 host1 kernel: RIP: 0010:[<ffffffff812c8103>]  [<ffffffff812c8103>] sidtab_context_to_sid+0xb3/0x480
    Oct 17 11:20:27 host1 kernel: RSP: 0018:ffff880110b6bcf8  EFLAGS: 00000286
    Oct 17 11:20:27 host1 kernel: RAX: 0000000000000000 RBX: ffff880110b6bdb8 RCX: 0000000000000000
    Oct 17 11:20:27 host1 kernel: RDX: ffff880110b6bda0 RSI: 0000000000000000 RDI: ffffffff81fddb40
    Oct 17 11:20:27 host1 kernel: RBP: ffff880110b6bd40 R08: ffff880110b6bdb8 R09: 0000000000000300
    Oct 17 11:20:27 host1 kernel: R10: ffffffffffffffff R11: ffffffffffffffff R12: ffffffffffffffff
    Oct 17 11:20:27 host1 kernel: R13: ffffffffffffffff R14: 0000000000000137 R15: ffff880110b6bce8
    Oct 17 11:20:27 host1 kernel: FS:  00007f2f3d561800(0000) GS:ffff88083fc80000(0000) knlGS:0000000000000000
    Oct 17 11:20:27 host1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Oct 17 11:20:27 host1 kernel: CR2: 0000000001d8acf8 CR3: 000000010fe92000 CR4: 00000000000007e0
    Oct 17 11:20:27 host1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    Oct 17 11:20:27 host1 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
    Oct 17 11:20:27 host1 kernel: Call Trace:
    Oct 17 11:20:27 host1 kernel: [<ffffffff812d4ca5>] ? mls_setup_user_range+0x145/0x250
    Oct 17 11:20:27 host1 kernel: [<ffffffff812d1407>] security_get_user_sids+0x3f7/0x550
    Oct 17 11:20:27 host1 kernel: [<ffffffff812c59bb>] sel_write_user+0x12b/0x210
    Oct 17 11:20:27 host1 kernel: [<ffffffff8118bb56>] ? get_zeroed_page+0x16/0x20
    Oct 17 11:20:27 host1 kernel: [<ffffffff812c5890>] ? sel_write_member+0x200/0x200
    Oct 17 11:20:27 host1 kernel: [<ffffffff812c40fa>] selinux_transaction_write+0x4a/0x80
    Oct 17 11:20:27 host1 kernel: [<ffffffff81205680>] vfs_write+0xc0/0x1f0
    Oct 17 11:20:27 host1 kernel: [<ffffffff816c0661>] ? system_call_after_swapgs+0xae/0x146
    Oct 17 11:20:27 host1 kernel: [<ffffffff8120649f>] SyS_write+0x7f/0xe0
    Oct 17 11:20:27 host1 kernel: [<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
    Oct 17 11:20:27 host1 kernel: [<ffffffff816c0661>] ? system_call_after_swapgs+0xae/0x146
    Oct 17 11:20:27 host1 kernel: Code: 0f 1f 84 00 00 00 00 00 41 8b 50 0c 85 d2 74 08 39 d0 0f 84 80 02 00 00 4d 8b 64 24 50 4d 85 e4 0f 84 a2 02 00 00 41 8b 44 24 14 <85> c0 75 d9 41 8b 48 0c 85 c9 75 e1 49 8b 00 49 39 44 24 08 75 
    

Environment

  • Red Hat Enterprise Linux 7
  • Docker

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content