Logs show kernel: kernel_read failed

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux

    • Observed but not limited to Red Hat Enterprise Linux 6 and above
  • Crowdstrike Falcon endpoint protection services. In particular, with one or more of the following modules;

    $ grep ")" proc/modules 
    falcon_lsm_serviceable 1087049 1 - Live 0xffffffffa08ec000 (PE)
    falcon_nf_netcontain 16495 1 - Live 0xffffffffa04a6000 (PE)
    falcon_kal 45235 1 falcon_lsm_serviceable, Live 0xffffffffa0538000 (E)
    falcon_lsm_pinned_12904 84817 1 - Live 0xffffffffa0522000 (E)
    

Issue

  • The error kernel: kernel_read failed seen across different logs across different systems.
  • The error ends with a number but the number can vary.
  • Example dmesg output;

    [547352.111070] kernel_read failed: 8
    [633743.312493] kernel_read failed: 8
    
  • Example journalctl and rsyslog logs;

    Dec 16 09:48:35 localhost kernel: kernel_read failed: 4
    Dec 16 09:48:37 localhost kernel: kernel_read failed: 10
    Dec 16 09:48:40 localhost kernel: kernel_read failed: 4
    Dec 16 09:48:40 localhost kernel: kernel_read failed: 12
    Dec 16 09:48:41 localhost kernel: kernel_read failed: 4
    Dec 16 09:48:41 localhost kernel: kernel_read failed: 10
    Dec 16 09:48:41 localhost kernel: kernel_read failed: 4
    Dec 16 09:48:41 localhost kernel: kernel_read failed: 12
    

Resolution

  • Engage your respective Crowdstrike support representative for assistance with their kernel modules.
  • If possible, blacklist the third-party kernel modules if any exist on the system and reboot to eliminate the third-party kernel modules from being a possible source of the errors.

    • Note Some core system functionality may be dependent on the third-party modules installed. Blacklisting such modules may cause the system to fail to boot at all. To avoid such a scenario, engage the support representative for the relevant third-party kernel modules.

Root Cause

  • A root cause is unknown at the time of writing, however, the errors are extremely strongly correlated with systems running Crowdstrike Falcon endpoint protection services.

Diagnostic Steps

  • Attempt blacklisting any third-party kernel modules on the system and reboot the system.

    • Note Some core system functionality may be dependent on the third-party modules installed. Blacklisting such modules may cause the system to fail to boot at all. To avoid such a scenario, engage the support representative for the relevant third-party kernel modules.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

2 Comments

Is it possible to suppress these messages so they no longer appear in our log files (/var/log/messages)? We are seeing thousands of these messages (and yes they are being caused by Crowdstrike). Makes looking through logs a bit cumbersome. The vendor is "looking into it" but they advise these messages are just informational and not an actual error, so it would be nice to just suppress them.

Never mind, I answered my own question. Simple add this line to /etc/rsyslog.conf just before the line defining /var/log/messages:

"if $msg contains "kernel_read failed" then stop"

As outlined at: https://access.redhat.com/solutions/3556491. Of course these still appear in the systemd journal logs.