rsyslog fails to start or cannot write logs when there are too many journal files

Issue

• The rsyslog service fails to start with log messages in the journal similar to the following:

  Oct 04 08:10:52 <hostname> systemd[1]: Starting System Logging Service...
  Oct 04 08:10:52 <hostname> rsyslogd[<PID>]:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="<PID>" x-info="http://www.rsyslog.com"] start
  Oct 04 08:10:52 <hostname> rsyslogd[<PID>]: imjournal: open on state file `/var/lib/rsyslog/imjournal.state' failed  [v8.24.0 try http://www.rsyslog.com/e/2013 ]
  Oct 04 08:10:52 <hostname> systemd[1]: rsyslog.service: main process exited, code=exited, status=1/FAILURE
  Oct 04 08:10:52 <hostname> systemd[1]: Failed to start System Logging Service.
  Oct 04 08:10:52 <hostname> systemd[1]: Unit rsyslog.service entered failed state.
  Oct 04 08:10:52 <hostname> systemd[1]: rsyslog.service failed.
  Oct 04 08:10:52 <hostname> systemd[1]: rsyslog.service holdoff time over, scheduling restart.
  

• The /var/lib/rsyslog/imjournal.state file is present on the system and correct permissions and SELinux contexts are set on the path:

  # ls -ldZ / /var; ls -lZa /var/lib/rsyslog
  dr-xr-xr-x. root root system_u:object_r:root_t:s0      /
  drwxr-xr-x. root root system_u:object_r:var_t:s0       /var
  drwx------. root root system_u:object_r:syslogd_var_lib_t:s0 .
  drwxr-xr-x. root root system_u:object_r:var_lib_t:s0   ..
  -rw-------. root root system_u:object_r:syslogd_var_lib_t:s0 imjournal.state
  

• The rsyslog service record no messages after log rotation. lsof shows that the rsyslog opens many journal files, but no log files.

  rsyslogd    3260        0    0r      CHR                1,3        0t0       1028 /dev/null
  rsyslogd    3260        0    1w      CHR                1,3        0t0       1028 /dev/null
  rsyslogd    3260        0    2w      CHR                1,3        0t0       1028 /dev/null
  rsyslogd    3260        0    3r  a_inode               0,10          0       6466 inotify
  rsyslogd    3260        0    4u     unix 0xffff9dbebb8f8cc0        0t0      38568 socket
  rsyslogd    3260        0   15r      REG              253,0    8388608   34641044 /var/log/journal/f2823e12c3df47db94bd4d47d2f58ef7/system@28a4e96b1dea4824b02ce8304d7b69e4-0000000002b9489e-00061dac772486ea.journal (deleted)
  rsyslogd    3260        0   16r      REG              253,0    8388608   33699395 /var/log/journal/f2823e12c3df47db94bd4d47d2f58ef7/user-1001.journal
  rsyslogd    3260        0   17r      REG              253,0    8388608   33908061 /var/log/journal/f2823e12c3df47db94bd4d47d2f58ef7/system@28a4e96b1dea4824b02ce8304d7b69e4-0000000002ba0db6-00061dbfbf5f2bab.journal (deleted)
  rsyslogd    3260        0   18r      REG              253,0    8388608   33890116 /var/log/journal/f2823e12c3df47db94bd4d47d2f58ef7/system@28a4e96b1dea4824b02ce8304d7b69e4-0000000002ba0f87-00061dc020bea245.journal (deleted)
  (...omit...)
  rsyslogd    3260        0 1021r      REG              253,0    8388608   35292705 /var/log/journal/f2823e12c3df47db94bd4d47d2f58ef7/system@28a4e96b1dea4824b02ce8304d7b69e4-0000000002bd8947-00061e1e8daf8fed.journal (deleted)
  rsyslogd    3260        0 1022r      REG              253,0    8388608   34636221 /var/log/journal/f2823e12c3df47db94bd4d47d2f58ef7/system@28a4e96b1dea4824b02ce8304d7b69e4-0000000002bd9af2-00061e1e908314cc.journal (deleted)
  rsyslogd    3260        0 1023r      REG              253,0    8388608   35292818 /var/log/journal/f2823e12c3df47db94bd4d47d2f58ef7/system@28a4e96b1dea4824b02ce8304d7b69e4-0000000002bdacec-00061e1e936f738f.journal (deleted)