rsyslog crashes on segfault when disk-assisted queues are corrupted

Solution Verified - Updated -

Issue

  • rsyslog segfaults at startup, and systemd restarts it in a forever loop

    # systemctl start rsyslog
    Job for rsyslog.service failed because a fatal signal was delivered to the control process. See "systemctl status rsyslog.service" and "journalctl -xe" for details.
    
    # journalctl | grep rsyslog
    Nov 11 13:15:08 server rsyslogd[928855]: action-0-builtin:omfwd queue[DA]: error dequeueing element - ignoring, but strange things may happen
    Nov 11 13:15:08 server rsyslogd[928855]: action-0-builtin:omfwd queue[DA]: qDeqDisk error happened at around offset 84191 
    Nov 11 13:15:08 server rsyslogd[928855]: action-0-builtin:omfwd queue[DA]: error dequeueing element - ignoring, but strange things may happen
    Nov 11 13:15:08 server rsyslogd[928855]: action-0-builtin:omfwd queue[DA]: qDeqDisk error happened at around offset 84344 
    Nov 11 13:15:08 server rsyslogd[928855]: action-0-builtin:omfwd queue[DA]: error dequeueing element - ignoring, but strange things may happen
    Nov 11 13:15:08 server kernel: traps: rs:action-0-bui[928866] general protection fault ip:55f977d51779 sp:7fb6251eb9b0 error:0 in rsyslogd[55f977d0c000+6b000]
    Nov 11 13:15:08 server systemd-coredump[928963]: Resource limits disable core dumping for process 928855 (rsyslogd).
    Nov 11 13:15:08 server systemd-coredump[928963]: Process 928855 (rsyslogd) of user 0 dumped core.
    Nov 11 13:15:08 server systemd[1]: rsyslog.service: Main process exited, code=dumped, status=11/SEGV
    Nov 11 13:15:08 server systemd[1]: rsyslog.service: Failed with result 'core-dump'.
    Nov 11 13:15:08 server systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 5.
    Nov 11 13:15:08 server systemd[1]: Stopped System Logging Service.
    Nov 11 13:15:08 server systemd[1]: rsyslog.service: Start request repeated too quickly.
    Nov 11 13:15:08 server systemd[1]: rsyslog.service: Failed with result 'core-dump'.
    Nov 11 13:15:08 server systemd[1]: Failed to start System Logging Service.
    
  • rsyslog shows this message in debug mode on stderr

    <Obj:1:msg:1:
    +iProtocolVersion:2:1:0:
    +iSeverity:2:1:5:
    +iFacility:2:2:16:
    +m'
    3817.443895127:action 0 queue[DA]:Reg/w0: error property name: 'iFacilit<Obj'
    3817.443898262:action 0 queue[DA]:Reg/w0: error var type: '1'
    Segmentation fault
    
  • The following message is seen in dmesg output

    kernel: rs:action 0 que[29724]: segfault at 5 ip 0000558247132cec sp 00007f75b3cbfb50 error 4 in rsyslogd[55824710c000+95000]
    

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 10
  • rsyslog

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content