systemd-journald: Failed

Latest response

Hello,

We have a problem with an up-to-date RHEL 7.2 Server.

This server hosts a proxy server (squid-3.3.8) and we send all access log to syslog (rsyslog-7.4.7).

After a while, we get no more log and instead we have thousands of journald error lines :

#dmesg
...
 [333311.366625] systemd-journald[810]: Failed to write entry (20 items, 616 bytes), ignoring: Cannot assign requested address
 [333311.401662] systemd-journald[810]: Failed to write entry (20 items, 633 bytes), ignoring: Cannot assign requested address
 [333311.476621] systemd-journald[810]: Failed to write entry (20 items, 615 bytes), ignoring: Cannot assign requested address
...

# systemctl status systemd-journald
● systemd-journald.service - Journal Service
  Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled)
  Active: active (running) since Thu 2015-12-17 14:06:59 CET; 3 days ago
    Docs: man:systemd-journald.service(8)
          man:journald.conf(5)
Main PID: 810 (systemd-journal)
  Status: "Processing requests..."
  CGroup: /system.slice/systemd-journald.service
          └─810 /usr/lib/systemd/systemd-journald
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

# journalctl --verify
 4f7160: invalid object
 File corruption detected at /run/log/journal/54ddca845fdd4b4c95e9da7b01a9b28f/system.journal:4f7160 (of 8388608 bytes, 62%).
 FAIL: /run/log/journal/54ddca845fdd4b4c95e9da7b01a9b28f/system.journal (Cannot assign requested address)
 PASS: /run/log/journal/54ddca845fdd4b4c95e9da7b01a9b28f/system@52f4a87054a6433f911ceb46fbfd3876-000000000005f2b4-00052719d37be23f.journal
...

If we removed the corrupted file and restart the service, we receive again logs but the problem appears again after a while

# rm /run/log/journal/54ddca845fdd4b4c95e9da7b01a9b28f/system.journal
# systemctl restart systemd-journald.service

Do you have an idea to permanently correct this dysfunction?

Best Regards,

Equipe Systeme's picture

Responses

Hi there. I'm looking into this as I've got another customer seeing it as well.

Any input (from anyone) is welcome though!

In the mean time, I'll say that you should be able to correct the problem (until it happens again) without deleting the journal and doing a full restart. Simply triggering a log-rotate should work. You could even set it up to run on an interval until we figure this out, e.g.:

echo "*/10 * * * * root systemctl kill -s SIGUSR2 systemd-journald" >/etc/cron.d/journal-reload

That sets up cron to tell journald to rotate the journal every 10 minutes.

Hi, same issue here, and reproduced with ansible.

Please be aware of this stackoverflow post:
http://stackoverflow.com/questions/36334412/ansible-hangs-systemd-journald-stops-logging-and-floods-dmesg-with-ansible-tas

And this redhat related post :
https://access.redhat.com/solutions/2117311

Hello Gwenael. I cannot reproduce it in RHEL7.2 using that same playbook. There's got to be some other problem on that system if simply ssh'ing in and adding files to ~cloud/.ssh/authorized_keys causes journald to freak out. I really can't it explain it.

I noticed it is related to my specific files read by ansible. How can i send my archive to you?

Hello everyone. We just released systemd errata to fix this issue. See: https://access.redhat.com/solutions/2117311

Was this helpful?

We appreciate your feedback. Leave a comment if you would like to provide more detail.
It looks like we have some work to do. Leave a comment to let us know how we could improve.
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.