Directory /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/ is very big because of many .jrnl files

Solution Verified - Updated -

Environment

Red Hat Satellite 6.3.x and later versions

Issue

The amount of .jrnl files in /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/ is too high.

Resolution

  • Deleting the journal files will free some space and will make qpidd to start faster.

  • Before deleting the files it is important to stop qpidd:

    # service qpidd stop
# rm -rf /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/*.jrnl
# service qpidd start

  • Files under /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/ must not be deleted. Avoid doing rm over /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/.

Root Cause

When a host is managed by gofer, using katello-agent, a gofer queue and .jrnl journal file are created. When those hosts are no longer managed by gofer the queues are deleted but .jrnl files are moved to the Empty File Pool (EFP) for future usage. This way, instead of deleting lot of 2M files and having to recreate them in case host is again managed with gofer, they're moved to EFP rather than being deleted.

Goferless Host Management is a feature available since 6.2.11.

Diagnostic Steps

[root@satellite ~]# ls /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/*.jrnl |wc -l
    10415

[root@satellite ~]# systemctl -l status qpidd
● qpidd.service - An AMQP message broker daemon.
   Loaded: loaded (/usr/lib/systemd/system/qpidd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/qpidd.service.d
           └─limits.conf
   Active: active (running) since Thu 2019-03-28 15:03:05 CET; 13min ago
     Docs: man:qpidd(1)
           http://qpid.apache.org/
 Main PID: 10187 (qpidd)
   CGroup: /system.slice/qpidd.service
           └─10187 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf

Mar 28 15:03:05 satellite.example.com systemd[1]: Started An AMQP message broker daemon..
Mar 28 15:03:06 satellite.example.com qpidd[10187]: 2019-03-28 15:03:06 [Store] error Linear Store: ERROR: File /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/230f89cf-a11c-4c49-a801-6da4b5704d4f.jrnl: Incorrect size: Expected=2101248; actual=2105344
Mar 28 15:03:06 satellite.example.com qpidd[10187]: 2019-03-28 15:03:06 [Store] error Linear Store: ERROR: File /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/230f89cf-a11c-4c49-a801-6da4b5704d4f.jrnl: Incorrect size: Expected=2101248; actual=2105344
Mar 28 15:03:06 satellite.example.com qpidd[10187]: 2019-03-28 15:03:06 [Store] error Linear Store: ERROR: File /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/b08a65ef-aab0-47ff-a324-76616c3d0dd9.jrnl: Incorrect size: Expected=2101248; actual=2105344
Mar 28 15:03:06 satellite.example.com qpidd[10187]: 2019-03-28 15:03:06 [Store] error Linear Store: ERROR: File /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/b08a65ef-aab0-47ff-a324-76616c3d0dd9.jrnl: Incorrect size: Expected=2101248; actual=2105344

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments