Server crashed with "PANIC: "sysrq: SysRq : Trigger a crash"" by Opensvc cluster.

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL)
  • Opensvc Clusterware v2.1

Issue

  • Red Hat Enterprise Linux Server which is a part of Opensvc Clusterware is getting crashed or rebooted.

Resolution

  • OpenSVC can trigger a node crash or reboot for 2 reasons :
    1/ In split-brain situation, where the node can no longer communicate with other nodes using any of its configured heartbeat.
    2/ In a critical resource failure situation, if the resource restart failed and if monitor_action is set to crash or reboot in the service configuration.

    As the crash/reboot has been triggered by the hb#1 heartbeat thread, the first situation occured.

    To make the cluster less prone to this crash/reboot decision, OpenSVC recommends:
    1/ Define more heartbeat links, using different hardware and different hb drivers. For example, 1 disk hb + 1 unicast hb on ethX + 1 relay heartbeat with the relay on a third site.
    2/ Set quorum=true in the cluster configuration, and define an arbitrator on the 3rd site to participate in the quorum vote.

For a simple analysis, the /var/log/opensvc/node.log contains traces of:
* The split brain situation and decisions (trigger of the node.split_action). See https://docs.opensvc.com/latest/agent.templates/template.node.node.html#split-action for possible values.
* Quorum vote and sysrq crash/reboot decision. See https://docs.opensvc.com/latest/agent.daemon.quorum.html for more information on quorum setup.

You can engage the OpenSVC support for further investigation and troubleshooting of the issue.

Root Cause

  • The Opensvc Clusterware agent hb#1 triggered the SysRq panic event (c) and rebooted this node.

Diagnostic Steps

  • System Information:
crash> sys

        CPUS: 8
        DATE: Wed Mar 15 11:26:55 EDT 2023
      UPTIME: 9 days, 02:32:19
LOAD AVERAGE: 2.96, 2.74, 2.41
       TASKS: 800
    NODENAME: Hostname
     RELEASE: 4.18.0-305.62.1.el8_4.x86_64
     VERSION: #1 SMP Thu Aug 11 12:07:27 EDT 2022
     MACHINE: x86_64  (2399 Mhz)
      MEMORY: 16 GB
       PANIC: "sysrq: SysRq : Trigger a crash"         <<<---
  • Panic task

crash> set
    PID: 2852
COMMAND: "hb#1"
   TASK: ffff8a38c5dd97c0  [THREAD_INFO: ffff8a38c5dd97c0]
    CPU: 2
  STATE: TASK_RUNNING (SYSRQ)
  • Backtrace

crash> bt
PID: 2852     TASK: ffff8a38c5dd97c0  CPU: 2    COMMAND: "hb#1"
 #0 [ffffa5e6c2877b98] machine_kexec at ffffffffb0e6095e
 #1 [ffffa5e6c2877bf0] __crash_kexec at ffffffffb0f8f7fd
 #2 [ffffa5e6c2877cb8] crash_kexec at ffffffffb0f906ed
 #3 [ffffa5e6c2877cd0] oops_end at ffffffffb0e2437d
 #4 [ffffa5e6c2877cf0] no_context at ffffffffb0e715ff
 #5 [ffffa5e6c2877d48] __bad_area_nosemaphore at ffffffffb0e7195c
 #6 [ffffa5e6c2877d90] do_page_fault at ffffffffb0e72237
 #7 [ffffa5e6c2877dc0] page_fault at ffffffffb18010fe
    [exception RIP: sysrq_handle_crash+18]
    RIP: ffffffffb1391652  RSP: ffffa5e6c2877e78  RFLAGS: 00010246
    RAX: ffffffffb1391640  RBX: 0000000000000063  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffff8a38eee96808  RDI: 0000000000000063
    RBP: 0000000000000007   R8: 00000000000007d3   R9: 0000000000aaaaaa
    R10: 0000000000000000  R11: ffffa5e6c3609600  R12: 0000000000000000
    R13: 0000000000000000  R14: ffffffffb1cab8e0  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ffffa5e6c2877e78] __handle_sysrq.cold.11 at ffffffffb13921e8
 #9 [ffffa5e6c2877ea8] write_sysrq_trigger at ffffffffb13920ab
#10 [ffffa5e6c2877eb8] proc_reg_write at ffffffffb119c109
#11 [ffffa5e6c2877ed0] vfs_write at ffffffffb11199f5
#12 [ffffa5e6c2877f00] ksys_write at ffffffffb1119c6f
#13 [ffffa5e6c2877f38] do_syscall_64 at ffffffffb0e0420b
#14 [ffffa5e6c2877f50] entry_SYSCALL_64_after_hwframe at ffffffffb18000ad
    RIP: 00007ff45b5f8a07  RSP: 00007ff450e6e560  RFLAGS: 00000293
    RAX: ffffffffffffffda  RBX: 000000000000000c  RCX: 00007ff45b5f8a07
    RDX: 0000000000000001  RSI: 00007ff428042a00  RDI: 000000000000000c
    RBP: 00007ff428042a00   R8: 0000000000000000   R9: 00007ff45ba19e6d
    R10: 0000000000000000  R11: 0000000000000293  R12: 0000000000000001
    R13: 000000000000000c  R14: 00007ff428042a00  R15: 00005614791a1af0
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
  • The panic task issued sysrq to the system.

crash> files -R sysrq
PID: 2852     TASK: ffff8a38c5dd97c0  CPU: 2    COMMAND: "hb#1"
ROOT: /    CWD: / 
 FD       FILE            DENTRY           INODE       TYPE PATH
 12 ffff8a36ea5a8a00 ffff8a374a46e840 ffff8a35d85859e0 REG  /proc/sysrq-trigger
  • The panic task appears to be a wrapper script executed by the python.

crash> task -R mm ffff8a38c5dd97c0
PID: 2852     TASK: ffff8a38c5dd97c0  CPU: 2    COMMAND: "hb#1"        <<<----- 
  mm = 0xffff8a38dd868000,

crash> mm_struct.exe_file 0xffff8a38dd868000
    exe_file = 0xffff8a38e7944c00,

crash> struct file.f_path 0xffff8a38e7944c00
  f_path = {
    mnt = 0xffff8a38420121a0,
    dentry = 0xffff8a38d5d190c0
  },

crash> files -d 0xffff8a38d5d190c0
     DENTRY           INODE           SUPERBLK     TYPE PATH
ffff8a38d5d190c0 ffff8a38d5d29c78 ffff8a38ed764000 REG  /usr/libexec/platform-python3.6    <<<--- executed by python 

crash> ps -g 2852
PID: 2643     TASK: ffff8a38c8f8c740  CPU: 1    COMMAND: "python3"
  PID: 2847     TASK: ffff8a38e3d60000  CPU: 3    COMMAND: "listener"
  PID: 2848     TASK: ffff8a38e3d62f80  CPU: 1    COMMAND: "collector"
  PID: 2849     TASK: ffff8a38c5dddf00  CPU: 7    COMMAND: "monitor"
  PID: 2850     TASK: ffff8a38c5ddc740  CPU: 2    COMMAND: "scheduler"
  PID: 2852     TASK: ffff8a38c5dd97c0  CPU: 2    COMMAND: "hb#1"
  PID: 2853     TASK: ffff8a38c5ddaf80  CPU: 6    COMMAND: "hb#1"
  PID: 2856     TASK: ffff8a38c5dd8000  CPU: 7    COMMAND: "hb#2"
  PID: 2857     TASK: ffff8a38e79597c0  CPU: 0    COMMAND: "hb#2"
  • The hb#1 is a hearbeat monitoring thread in the cluster and it can panic the system by issuing sysrq.

crash> files 2643
PID: 2643     TASK: ffff8a38c8f8c740  CPU: 1    COMMAND: "python3"
ROOT: /    CWD: /
 FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff8a38e613c100 ffff8a35c786ab40 ffff8a38df9cc370 CHR  /dev/null
  1 ffff8a38e613c100 ffff8a35c786ab40 ffff8a38df9cc370 CHR  /dev/null
  2 ffff8a38e613c100 ffff8a35c786ab40 ffff8a38df9cc370 CHR  /dev/null
  3 ffff8a38c5c44900 ffff8a38c8a71a80 ffff8a38c8aecf00 REG  /var/log/opensvc/node.log
  4 ffff8a38c5c44c00 ffff8a38c8b686c0 ffff8a38c89fa6b0 SOCK UNIX
  5 ffff8a38c8dfac00 ffff8a389c9b1840 ffff8a35c787b1b0 SOCK TCPv6
  6 ffff8a38e6638c00 ffff8a389c89bcc0 ffff8a38ed94a110 REG  /var/log/opensvc/node.scheduler.log
  7 ffff8a38c8dfa500 ffff8a389c9b1900 ffff8a35c787ac30 SOCK UNIX
  8 ffff8a38c8dfae00 ffff8a389c9b19c0 ffff8a35c7878030 SOCK UNIX
  9 ffff8a38c8c53400 ffff8a389c8c7840 ffff8a35c787b9f0 SOCK TCPv6
 10 ffff8a38dec41c00 ffff8a35c786ab40 ffff8a38df9cc370 CHR  /dev/null
 11 ffff8a389e934c00 ffff8a38304ccd80 ffff8a3880c9e5f8 REG  /var/log/opensvc/xmlrpc.log
 12 ffff8a36ea5a8a00 ffff8a374a46e840 ffff8a35d85859e0 REG  /proc/sysrq-trigger

crash> files 
PID: 2852     TASK: ffff8a38c5dd97c0  CPU: 2    COMMAND: "hb#1"
ROOT: /    CWD: /
 FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff8a38e613c100 ffff8a35c786ab40 ffff8a38df9cc370 CHR  /dev/null
  1 ffff8a38e613c100 ffff8a35c786ab40 ffff8a38df9cc370 CHR  /dev/null
  2 ffff8a38e613c100 ffff8a35c786ab40 ffff8a38df9cc370 CHR  /dev/null
  3 ffff8a38c5c44900 ffff8a38c8a71a80 ffff8a38c8aecf00 REG  /var/log/opensvc/node.log
  4 ffff8a38c5c44c00 ffff8a38c8b686c0 ffff8a38c89fa6b0 SOCK UNIX
  5 ffff8a38c8dfac00 ffff8a389c9b1840 ffff8a35c787b1b0 SOCK TCPv6
  6 ffff8a38e6638c00 ffff8a389c89bcc0 ffff8a38ed94a110 REG  /var/log/opensvc/node.scheduler.log
  7 ffff8a38c8dfa500 ffff8a389c9b1900 ffff8a35c787ac30 SOCK UNIX
  8 ffff8a38c8dfae00 ffff8a389c9b19c0 ffff8a35c7878030 SOCK UNIX
  9 ffff8a38c8c53400 ffff8a389c8c7840 ffff8a35c787b9f0 SOCK TCPv6
 10 ffff8a38dec41c00 ffff8a35c786ab40 ffff8a38df9cc370 CHR  /dev/null
 11 ffff8a389e934c00 ffff8a38304ccd80 ffff8a3880c9e5f8 REG  /var/log/opensvc/xmlrpc.log
 12 ffff8a36ea5a8a00 ffff8a374a46e840 ffff8a35d85859e0 REG  /proc/sysrq-trigger       <<<------

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments