slow ssh login with soft lockups in selinux code

Solution Verified - Updated -

Issue

  • ssh login is delayed 30 seconds or more.
  • soft lockups appear in sshd or selinuxdefcon near the function security_get_user_sids.
    Example:

    Jul 26 18:08:53 host1 kernel: NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [sshd:14871]
    

    or

    Oct 17 11:20:27 host1 kernel: NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [selinuxdefcon:14224]
    Oct 17 11:20:27 host1 kernel: Modules linked in: ip6table_filter ip6_tables sctp_diag sctp dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag vxlan ip6_udp_tunnel udp_tunnel ip_set_hash_net xt_comment xt_ipvs xt_set ip_set nfnetlink ip_vs_wlc ip_vs dummy ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock xfs libcrc32c ppdev vmw_balloon coretemp joydev pcspkr sg vmw_vmci shpchp i2c_piix4 parport_pc parport binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mptspi ahci ttm
    Oct 17 11:20:27 host1 kernel: scsi_transport_spi libahci ata_piix mptscsih drm libata mptbase crc32c_intel serio_raw vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
    Oct 17 11:20:27 host1 kernel: CPU: 2 PID: 14224 Comm: selinuxdefcon Tainted: G             L ------------ T 3.10.0-693.21.1.el7.x86_64 #1
    Oct 17 11:20:27 host1 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
    Oct 17 11:20:27 host1 kernel: task: ffff88081c6d2f70 ti: ffff880110b68000 task.ti: ffff880110b68000
    Oct 17 11:20:27 host1 kernel: RIP: 0010:[<ffffffff812c8103>]  [<ffffffff812c8103>] sidtab_context_to_sid+0xb3/0x480
    Oct 17 11:20:27 host1 kernel: RSP: 0018:ffff880110b6bcf8  EFLAGS: 00000286
    Oct 17 11:20:27 host1 kernel: RAX: 0000000000000000 RBX: ffff880110b6bdb8 RCX: 0000000000000000
    Oct 17 11:20:27 host1 kernel: RDX: ffff880110b6bda0 RSI: 0000000000000000 RDI: ffffffff81fddb40
    Oct 17 11:20:27 host1 kernel: RBP: ffff880110b6bd40 R08: ffff880110b6bdb8 R09: 0000000000000300
    Oct 17 11:20:27 host1 kernel: R10: ffffffffffffffff R11: ffffffffffffffff R12: ffffffffffffffff
    Oct 17 11:20:27 host1 kernel: R13: ffffffffffffffff R14: 0000000000000137 R15: ffff880110b6bce8
    Oct 17 11:20:27 host1 kernel: FS:  00007f2f3d561800(0000) GS:ffff88083fc80000(0000) knlGS:0000000000000000
    Oct 17 11:20:27 host1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Oct 17 11:20:27 host1 kernel: CR2: 0000000001d8acf8 CR3: 000000010fe92000 CR4: 00000000000007e0
    Oct 17 11:20:27 host1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    Oct 17 11:20:27 host1 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
    Oct 17 11:20:27 host1 kernel: Call Trace:
    Oct 17 11:20:27 host1 kernel: [<ffffffff812d4ca5>] ? mls_setup_user_range+0x145/0x250
    Oct 17 11:20:27 host1 kernel: [<ffffffff812d1407>] security_get_user_sids+0x3f7/0x550
    Oct 17 11:20:27 host1 kernel: [<ffffffff812c59bb>] sel_write_user+0x12b/0x210
    Oct 17 11:20:27 host1 kernel: [<ffffffff8118bb56>] ? get_zeroed_page+0x16/0x20
    Oct 17 11:20:27 host1 kernel: [<ffffffff812c5890>] ? sel_write_member+0x200/0x200
    Oct 17 11:20:27 host1 kernel: [<ffffffff812c40fa>] selinux_transaction_write+0x4a/0x80
    Oct 17 11:20:27 host1 kernel: [<ffffffff81205680>] vfs_write+0xc0/0x1f0
    Oct 17 11:20:27 host1 kernel: [<ffffffff816c0661>] ? system_call_after_swapgs+0xae/0x146
    Oct 17 11:20:27 host1 kernel: [<ffffffff8120649f>] SyS_write+0x7f/0xe0
    Oct 17 11:20:27 host1 kernel: [<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
    Oct 17 11:20:27 host1 kernel: [<ffffffff816c0661>] ? system_call_after_swapgs+0xae/0x146
    Oct 17 11:20:27 host1 kernel: Code: 0f 1f 84 00 00 00 00 00 41 8b 50 0c 85 d2 74 08 39 d0 0f 84 80 02 00 00 4d 8b 64 24 50 4d 85 e4 0f 84 a2 02 00 00 41 8b 44 24 14 <85> c0 75 d9 41 8b 48 0c 85 c9 75 e1 49 8b 00 49 39 44 24 08 75 
    

Environment

  • Red Hat Enterprise Linux 7
  • Docker

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In