foreman-maintain upgrade check command fails with an error "Could not acquire change log lock" on Red Hat Satellite server.

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6

Issue

  • Execution of foreman-maintain upgrade check command on Red Hat Satellite server ends with below error message:

    Command output: Liquibase update Failed: Could not acquire change log lock.  Currently locked by host.example.com (192.XXX.X.X) since 6/13/19 3:32 PM
SEVERE 6/13/19 4:13 PM:liquibase: Could not acquire change log lock.  Currently locked by host.example.com (192.XXX.X.X) since 6/13/19 3:32 PM
liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by host.example.com (192.XXX.X.X) since 6/13/19 3:32 PM

Resolution

  • Check locks on the candlepin database.

    # su - postgres -c "psql candlepin -c \"select * from public.databasechangeloglock;\""

id | locked |        lockgranted         |                                lockedby 
----+--------+----------------------------+------------------------------------------------------------------------
 1 | t      | 2018-01-01 00:00:00.000+00 | host.example.com (details)
(1 row)

    Locked t means that table is really locked.

  • Remove lock.

    # su - postgres -c "psql candlepin -c \"UPDATE DATABASECHANGELOGLOCK SET LOCKED=FALSE, LOCKGRANTED=null, LOCKEDBY=null where ID=1;\""

UPDATE 1

  • Restart postgresql

    # systemctl restart postgresql

  • Check if the lock is gone.

    # su - postgres -c "psql candlepin -c \"select * from public.databasechangeloglock;\""

id | locked | lockgranted | lockedby 
----+--------+-------------+----------
 1 | f      |             | 
(1 row)

    Locked f signalize that table is not locked.

Root Cause

  • Usually is table locked by some application to prevent concurrent changes, when that application crashes or is abruptly stopped lock will stay.

Diagnostic Steps

  • Check locks on candlepin database

    su - postgres -c "psql candlepin -c \"select * from public.databasechangeloglock;\""

 id | locked |        lockgranted         |                                lockedby                                
----+--------+----------------------------+------------------------------------------------------------------------
  1 | t      | 2018-01-01 00:00:00.000+00 | host.example.com (details)
(1 row)

    Locked t means that table is really locked

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments