System crashed at cshook_network_ops_inet6_sockraw_release+0x171a9

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux

    • Observed but not limited to release 6 and 7

Issue

  • System crashed

  • The vmcore-dmesg.txt file or the serial console shows a backtrace containing the following line:

        Aug 15 07:11:14 HOSTNAME kernel: IP: [<ADDRESS>] cshook_network_ops_inet6_sockraw_release+0x171a9/0x1a650 [falcon_lsm_serviceable]

Resolution

Engage your respective CrowdStrike support vendor for assistance with troubleshooting potential issues with the falcon_lsm_serviceable kernel module provided from the CrowdStrike Falcon Sensor/Agent security software suite.

However, please note, if the support vendor finds an interaction with their module and the kernel results in undefined behavior, please do not hesitate to engage your respective Red Hat support vendor or open a case with us if your subscriptions entitle you to such benefits. From here, please provide the full details of their findings so we may assist.

Workaround

If your respective Information Security team allows disabling the CrowdStrike Falcon Sensor/Agent software suite, doing so will mitigate the crashes and provide temporary stability to the system in question while the issue is investigated.

Root Cause

Troubleshooting third-party software is outside of the scope of support and thus Red Hat can not provide direct assistance in troubleshooting and determining the cause of failure in third-party software. The limitation is due to Red Hat simply not having the deeper knowledge and expertise required to effectively troubleshoot third-party software which the vendor would have.

For more information on supportability of third-party software, please review the following:

Diagnostic Steps

  1. Ensure kdump is installed and functional on the target system. Please refer to the following knowledgebase article for more information on setting up kdump:

  2. On a system crash, check for the file vmcore-dmesg.txt.

    • In the event of a system crash where a vmcore is successfully produced, a file named vmcore-dmesg.txt should also be produced in the same directory as the vmcore.
    • vmcore-dmesg.txt contains the same output as the dmesg command but specifically at the time of the crash.
    • For this specific issue, review of vmcore-dmesg.txt is sufficient in determining if the crash matches the symptoms noted in this knowledgebase article.

  3. Within vmcore-dmesg.txt check for a backtrace containing IP: [<ADDRESS>] cshook_network_ops_inet6_sockraw_release+0x171a9/0x1a650 [falcon_lsm_serviceable] near the end of the file where <ADDRESS> is some 64-bit hexadecimal value. For example;

    [ 6351.785505] BUG: unable to handle kernel paging request at ffffb89cc5f98f39
--->[ 6351.785550] IP: [<ffffffffc07699a9>] cshook_network_ops_inet6_sockraw_release+0x171a9/0x1a650 [falcon_lsm_serviceable]       <---
    [ 6351.785590] PGD 17fd7a067 PUD 17fd7b067 PMD 81b0aa067 PTE 0
    [ 6351.785613] Oops: 0000 [#1] SMP  
    [ 6351.785628] Modules linked in: [...]
    [ 6351.785901] CPU: 6 PID: 8777 Comm: <COMMAND>. Kdump: loaded Tainted: P        W  OE  ------------   3.10.0-1160.53.1.el7.x86_64 #1
    [ 6351.785937] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
    [ 6351.785970] task: ffff99b70b41a100 ti: ffff99b7168dc000 task.ti: ffff99b7168dc000
    [ 6351.785993] RIP: 0010:[<ffffffffc07699a9>]  [<ffffffffc07699a9>] cshook_network_ops_inet6_sockraw_release+0x171a9/0x1a650 [falcon_lsm_serviceable]
    [ 6351.786035] RSP: 0018:ffff99b7168df508  EFLAGS: 00010202
    [ 6351.786053] RAX: 000000000000ff29 RBX: ffff99b789cef328 RCX: 0000000000000000
    [ 6351.786075] RDX: 000000000000ff29 RSI: 000000000000ff2a RDI: ffffb89cc5f89010
    [ 6351.786097] RBP: ffff99b7168df508 R08: ffff99b7168df5a8 R09: ffffb89ccd382088
    [ 6351.786119] R10: 0000000000002841 R11: 441c7fc358b0d800 R12: ffffb89cc5f89010
    [ 6351.786144] R13: ffff99b998e82010 R14: 0000000000010911 R15: ffff99b72a573840
    [ 6351.786168] FS:  00007f59c5114740(0000) GS:ffff99b99f380000(0000) knlGS:0000000000000000
    [ 6351.786193] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    [ 6351.786211] CR2: ffffb89cc5f98f39 CR3: 000000061069a000 CR4: 00000000001607e0
    [ 6351.786267] Call Trace:
    [ 6351.786282]  [<ffffffffc076ad77>] cshook_network_ops_inet6_sockraw_release+0x18577/0x1a650 [falcon_lsm_serviceable]
    [ 6351.786316]  [<ffffffffc076ae45>] cshook_network_ops_inet6_sockraw_release+0x18645/0x1a650 [falcon_lsm_serviceable]
    [ 6351.786348]  [<ffffffffc07312bf>] _ZdlPv+0x2ae4f/0x38c30 [falcon_lsm_serviceable]
    [ 6351.786372]  [<ffffffffc073194e>] _ZdlPv+0x2b4de/0x38c30 [falcon_lsm_serviceable]
    [ 6351.786397]  [<ffffffffc07319d9>] _ZdlPv+0x2b569/0x38c30 [falcon_lsm_serviceable]
    [ 6351.786423]  [<ffffffffc077077c>] cshook_security_sb_free_security+0x277c/0x2d10 [falcon_lsm_serviceable]
    [ 6351.786453]  [<ffffffffc076ffee>] cshook_security_sb_free_security+0x1fee/0x2d10 [falcon_lsm_serviceable]
    [ 6351.786484]  [<ffffffffc077047a>] cshook_security_sb_free_security+0x247a/0x2d10 [falcon_lsm_serviceable]
    [ 6351.786514]  [<ffffffffc076d99b>] cshook_security_file_permission+0x85b/0xd60 [falcon_lsm_serviceable]
    [ 6351.786544]  [<ffffffffc076dd11>] cshook_security_file_permission+0xbd1/0xd60 [falcon_lsm_serviceable]
    [ 6351.787357]  [<ffffffffc076df05>] cshook_security_bprm_check_security+0x65/0xb0 [falcon_lsm_serviceable]
    [ 6351.788166]  [<ffffffffc05195ec>] pinnedhook_security_bprm_check_security+0x5c/0x90 [falcon_lsm_pinned_14005]
    [ 6351.788988]  [<ffffffff9d308a9d>] security_bprm_check+0x1d/0x30
    [ 6351.789794]  [<ffffffff9d25587a>] search_binary_handler+0x2a/0x1c0
    [ 6351.790611]  [<ffffffff9d2b3055>] load_script+0x265/0x2a0
    [ 6351.791417]  [<ffffffff9d32deb9>] ? ima_bprm_check+0x49/0x50
    [ 6351.792227]  [<ffffffff9d2558ea>] search_binary_handler+0x9a/0x1c0
    [ 6351.793042]  [<ffffffff9d256dd6>] do_execve_common.isra.23+0x616/0x880
    [ 6351.793875]  [<ffffffff9d2572e9>] SyS_execve+0x29/0x30
    [ 6351.794694]  [<ffffffff9d796538>] stub_execve+0x48/0x80
  1. In the above backtrace, the system crashed specifically traversing code paths within the falcon_lsm_serviceable kernel module.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments