WAL segment has already been removed error when running database backup on CloudForms

Solution In Progress - Updated -

Environment

  • CloudForms 5.0

Issue

  • Getting requested WAL segment 00000001000000140000004F has already been removed when running pg_basebackup on CloudForms.

Resolution

  • Adjust the wal_keep_segments setting in the postgresql.conf file.

Instructions:

  • SSL into your DB server CloudForms appliance
  • Set wal_keep_segments to (<pg-volume-free-space>/4)/16MB
  • Run: psql -c "ALTER SYSTEM SET wal_keep_segments = '<DESIRED_NUMBER>'; -d vmdb_production ##This is going to make the change
  • Run: psql -c "SELECT pg_reload_conf();" -d vmdb_production ## To reload the conf file
  • Run: psql -c "show wal_keep_segments;" -d vmdb_production ## Show what the current setting is (should show )

Please note that when calculating the that you want to keep it as low as possible, as this will take up space on the database

Root Cause

  • The server recycles old WAL segments before the backup can finish.
  • Depending on how long the pg_basebackup is taking it is common knowledge that the segments are generated about every 5 minutes, so if the backup takes an hour, you need at least 12 segments stored.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.