Chapter 5. CloudForms 4.1 Cluster Resource Configuration
With the release of CloudForms 4.1, the underlying database replication technology was changed from using rubyrep.[6] to pglogical.[7]. This technology change provides better performance as it takes advantage of the logical decoding concept versus using triggers and external process for replicating changes.
This change however poses some differences in the High Availability configuration. To implement HA using pglogical , the following steps are performed:
- Postgres DB HA has already been configured and failover has been tested. postgresql.log should show no errors; refer to Section 3: PostgreSQL Configuration
-
pglogicalreplication has been setup and working properly from the current primary database -
Update
corosyncpriority and timeout - Create a pgsql resource agent for pacemaker
-
Update the
postgresql-mscluster resource to supportpglogical
With the release of CloudForms 4.2, repmgr becomes the default HA technology and this document’s architecture using pacemaker is deprecated.
Post cluster deployment, prior to validating, ensure that pglogical is configured on the database servers. Refer to CloudForms 4.1: General Configuration Guide.[8].
5.1. pglogical Replication Validation
Before continuing, we need to confirm that pglogical replication is working properly. Validation for pglogical replication is performed on the CloudForms appliance for the Global and Regional databases by performing the following:
Global
Monitor the state of the subscription in global by watching its sync status on the database. The sync status can be i, d, or r. A successful sync will list all tables with a status of “r” for “ready”.
$ psql -d vmdb_production psql> select sync_relname, sync_status from \ pglogical.local_sync_status;
Region
Check in the regional database and confirm that the subscription is active.
$ psql -d vmdb_production
psql> select slot_name, plugin, database, active from pg_replication_slots;
slot_name | plugin | database | active
pgl_vmdb_production_region_10_region_1b30d4cc | pglogical_output | vmdb_production | t
(1 row)Check progress in region by finding out how many bytes of wal archives still need to be replicated. Also you can check how much wal archives currently still exist.
irb> MiqPglogical.new.replication_lag PostgreSQLAdapter#log_after_checkout, connection_pool: size: 1, connections: 1, in use: 1, waiting_in_queue: 0 => [{"lag_bytes"=>"1698248", "application_name"=>"region_10_subscription"}]
irb> MiqPglogical.new.replication_wal_retained => [{"retained_bytes"=>"13170184", "slot_name"=>"pgl_vmdb_production_region_10_region_1b30d4cc"}]5.2. corosync Updates
Update the totem token timeout to 5000 milliseconds or 5 seconds. Default setting is 1000 milliseconds or 1 second.
Update /etc/corosync/corosync.conf :
totem {
...
token: 5000
}Restart corosync:
# systemctl restart corosync
5.3. pacemaker Resource Agents
Several changes are required for pacemaker to support pglogical . A new cluster resource is created and the existing postgresql-ms resource is modified to include pglogical options for Postgres.
5.3.1. pgsql Resource Agent
Update the pgsql cluster resource agent and place it in /var/lib/ocf/resource.d/_heartbeat/ .
Refer to Appendix E: Configuration Scripts , pgsql for the cluster resource agent example. Optionally create this executable as a convenience method to watch the pglogical failover actions. If opting out, comment out the calls to this file in pgsql resource agent file.
$ cat > /usr/local/bin/pg_logical_failover << 'EOF'
#!/bin/bash
DT=`date`
echo "${DT} $1" >> /tmp/pg_logical_failover.log
EOF$ chmod 755 /usr/local/bin/pg_logical_failover
5.3.2. postgresql-ms Resource Agent Update
Update the resource agent configurations to account for the new parameters and operations.
- pglogical_user: is the postgres database user that owns vmdb database
- pglogical_db: is the name of the vmdb database that contains the replication slots information
- pglogical_shared_dir: is a shared read/write location available to all nodes. If this parameter is not provided, it is assumed that a passwordless scp is available to write a subscriptions.dat file to each of the secondary nodes in the tmp directory
- syncsubs: is a new operation defined that will pull subscription data from the primary and write to a location available to the secondary nodes. If the secondary node is promoted to primary, it will read this subscription data and ensure that the slot is created in the same manner as before
$ pcs resource update postgresql-ms pglogical_user="root" $ pcs resource update postgresql-ms pglogical_db="vmdb_production" $ pcs resource update postgresql-ms \ pglogical_shared_dir="/var/opt/rh/rh-postgresql94/lib/pgsql/wal-archive/" $ pcs resource op add postgresql-ms syncsubs timeout=5s interval=600s role="Master"
5.3.2.1. Under the Hood (informational only)
The handling of pglogical during a failover is entirely within the resource agent definition. The primary database is responsible for constantly providing all the secondary databases the information necessary to keep up the replication slots info. That is the purpose of the syncsubs operation. The interval setting is set to 600s which means this operation will update the secondary databases every 10 minutes.
The syncsub operation runs on the Master (primary) database and writes it to the designated tmp directory locally using the write_subs method.
write_subs `() {
runasowner "pg_logical_failover master_writesubs"
GET_SUBS_SQL="select slot_name from pg_replication_slots
where plugin =
'pglogical_output'"
`su $OCF_RESKEY_pgdba -c "cd $OCF_RESKEY_pgdata; $OCF_RESKEY_psql $psql_options -U
$OCF_RESKEY_pglogical_user -d
$OCF_RESKEY_pglogical_db -Atc \"${GET_SUBS_SQL}\" > ${OCF_RESKEY_tmpdir}/subscriptions.dat"
if [ $? != 0 ];
then
ocf_exit_reason "Failed to select slot_name from ${OCF_RESKEY_pglogical_db}."
return ${OCF_FAILED_MASTER}
fi
ocf_log info "Completed pg_logical_failover write subs script."
return $OCF_RUNNING_MASTER
}
The syncsub operation then calls the xfer_subs method to make a decision how and where to deliver the subscriptions data. The data needs to immediately be on all secondary nodes to be ready for promotion. The two methods to transfer are write to a locally available shared filesystem (like wal-archives) or scp with password-less keys.
xfer_subs() {
runasowner "pg_logical_failover master_xfersubs ${OCF_RESKEY_node_list}"
for standby in ${OCF_RESKEY_node_list};
do
current_host=`hostname`
if [ ${current_host} != ${standby} ];
then
if [ -n "$OCF_RESKEY_pglogical_shared_dir" ];
then
runasowner "cp -f ${OCF_RESKEY_tmpdir}/subscriptions.da ${OCF_RESKEY_pglogical_shared_dir}"
else
runasowner "scp ${OCF_RESKEY_tmpdir}/subscriptions.dat ${OCF_RESKEY_pgdba}@${standby}:${OCF_RESKEY_tmpdir}"
fi
fi
if [ $? != 0 ];
then
ocf_exit_reason "Failed to write subscriptions.dat on ${OCF_RESKEY_pgdba}@${standby} or${OCF_RESKEY_pglogical_shared_dir}."
return ${OCF_FAILED_MASTER}
fi
done
ocf_log info "Completed pg_logical_failover transfer subs script."
return $OCF_RUNNING_MASTER
}
Assuming that the subscription file is in place, if a node suddenly finds itself promoted to the primary, the promotion operation will call read_subs after it has switched to non-recovery mode. This read_subs method will create a pg logical replication slot of the same name as before if it doesn’t already exist.
read_subs() {
# read in the pg_logical subscription info
runasowner "pg_logical_failover read_subs"
if [ -n "$OCF_RESKEY_pglogical_shared_dir" ];
then
SLOT_NAME=$( <${OCF_RESKEY_pglogical_shared_dir}/subscriptions.dat )
PUT_SUBS_SQL="select * from pg_create_logical_replication_slot('${SLOT_NAME}','pglogical_output')"
else
SLOT_NAME=$( <${OCF_RESKEY_tmpdir}/subscriptions.dat )
PUT_SUBS_SQL="select * from pg_create_logical_replication_slot('${SLOT_NAME}','pglogical_output')"
fi
if [ -n "${SLOT_NAME}" ];
then
`su $OCF_RESKEY_pgdba -c "cd $OCF_RESKEY_pgdata; $OCF_RESKEY_psql $psql_options -U $OCF_RESKEY_pglogical_user -d
$OCF_RESKEY_pglogical_db -Atc \"${PUT_SUBS_SQL}\" "`
else
ocf_exit_reason "Failed to read subscriptions.dat."
return ${OCF_FAILED_MASTER}
fi
if [ $? != 0 ];
then
ocf_exit_reason "Failed to insert slot_name in #{OCF_RESKEY_pglogical_db}."
return ${OCF_FAILED_MASTER}
fi
ocf_log info "Completed pg_logical_failover read subs script."
}
Where did the comment section go?
Red Hat's documentation publication system recently went through an upgrade to enable speedier, more mobile-friendly content. We decided to re-evaluate our commenting platform to ensure that it meets your expectations and serves as an optimal feedback mechanism. During this redesign, we invite your input on providing feedback on Red Hat documentation via the discussion platform.