2.7. Cloning
2.7.1. About Cloning
Planning for high availability reduces unplanned outages and other problems by making one or more subsystem clones available. When a host machine goes down, the cloned subsystems can handle requests and perform services, taking over from the master (original) subsystem seamlessly and keeping uninterrupted service.
Using cloned subsystems also allows systems to be taken offline for repair, troubleshooting, or other administrative tasks without interrupting the services of the overall PKI system.
Note
All of the subsystems except the TPS can be cloned.
Cloning is one method of providing scalability to the PKI by assigning the same task, such as handling certificate requests, to separate instances on different machines. The internal databases for the master and its clones are replicated between each other, so the information about certificate requests or archived keys on one subsystem is available on all the others.
Typically, master and cloned instances are installed on different machines, and those machines are placed behind a load balancer. The load balancer accepts HTTP and HTTPS requests made to the Certificate System subsystems and directs those requests appropriately between the master and cloned instances. In the event that one machine fails, the load balancer transparently redirects all requests to the machine that is still running until the other machine is brought back online.
Figure 2.5. Cloning Example
The load balancer in front of a Certificate System subsystem is what provides the actual failover support in a high availability system. A load balancer can also provide the following advantages as part of a Certificate System subsystem:
- DNS round-robin, a feature for managing network congestion that distributes load across several different servers.
- Sticky SSL/TLS, which makes it possible for a user returning to the system to be routed the same host used previously.
Normally, when a system is cloned, the configuration servlet sets up the replication agreements between the internal LDAP databases. However, some users may want to establish and manage their own replication agreements. The pkispawn executable has been modified to allow this by adding a
[Tomcat] section to the PKI instance override configuration file and adding the following two name=value pairs under that section:
[Tomcat] pki_clone_setup_replication=False pki_clone_reindex_data=False
Important
Data must be replicated before
pkispawn is invoked when specifying this kind of installation.
2.7.2. Preparing Clones
You need to export certificates to a
p12 file which can then be used while installing a clone subsystem. This command does not install or setup a clone subsystem, but prepares it for installation. Here is an example how to export subsystem certificates using the pki-server SUBSYSTEM-clone-prepare command.
Example 2.3. Exporting Subsystem Certificates
[root@pki1 ~]$pki-server ca-clone-prepare --i topology-02-CA --pkcs12-file /tmp/caclone.p12 --pkcs12-password SECret.123----------------------------------------------------- Added certificate "subsystemCert cert-topology-02-CA" ----------------------------------------------------- -------------------------------------------------------- Added certificate "caSigningCert cert-topology-02-CA CA" -------------------------------------------------------- ---------------------------------------------------------- Added certificate "ocspSigningCert cert-topology-02-CA CA" ---------------------------------------------------------- ----------------------------------------------------------- Added certificate "auditSigningCert cert-topology-02-CA CA" -----------------------------------------------------------
You can verify that the exported
p12 file contains the certificates. In the following example, the output of the pki pkcs12-cert-find returns four certificates:
[root@pki1 ~]$pki pkcs12-cert-find --pkcs12-file /tmp/caclone.p12 --pkcs12-password SECret.123--------------- 4 entries found --------------- Certificate ID: 4649cef11b90c78d126874b91654de98ded54073 Serial Number: 0x4 Nickname: subsystemCert cert-topology-02-CA Subject DN: CN=Subsystem Certificate,OU=topology-02-CA,O=topology-02_Foobarmaster.org Issuer DN: CN=CA Signing Certificate,OU=topology-02-CA,O=topology-02_Foobarmaster.org Trust Flags: u,u,u Has Key: true Certificate ID: a304cf107abd79fbda06d887cd279fb02cefe438 Serial Number: 0x1 Nickname: caSigningCert cert-topology-02-CA CA Subject DN: CN=CA Signing Certificate,OU=topology-02-CA,O=topology-02_Foobarmaster.org Issuer DN: CN=CA Signing Certificate,OU=topology-02-CA,O=topology-02_Foobarmaster.org Trust Flags: CTu,Cu,Cu Has Key: true Certificate ID: 4a09e057c1edfee40f412551db1959831abe117d Serial Number: 0x2 Nickname: ocspSigningCert cert-topology-02-CA CA Subject DN: CN=CA OCSP Signing Certificate,OU=topology-02-CA,O=topology-02_Foobarmaster.org Issuer DN: CN=CA Signing Certificate,OU=topology-02-CA,O=topology-02_Foobarmaster.org Trust Flags: u,u,u Has Key: true Certificate ID: 3f42f88026267f90f59631d38805cc60ee4c711a Serial Number: 0x5 Nickname: auditSigningCert cert-topology-02-CA CA Subject DN: CN=CA Audit Signing Certificate,OU=topology-02-CA,O=topology-02_Foobarmaster.org Issuer DN: CN=CA Signing Certificate,OU=topology-02-CA,O=topology-02_Foobarmaster.org Trust Flags: u,u,Pu Has Key: true
With the exported
p12 file, you can now set up a clone subsystem.
2.7.3. Cloning for CAs
Cloned instances have the exact same private keys as the master, so their certificates are identical. For CAs, that means that the CA signing certificates are identical for the original master CA and its cloned CAs. From the perspectives of clients, these look like a single CA.
Every CA, both cloned and master, can issue certificates and process revocation requests.
The main issue with managing replicated CAs is how to assign serial numbers to the certificates they issue. Different replicated CAs can have different levels of traffic, using serial numbers at different rates, and it is imperative that no two replicated CAs issue certificates with the same serial number. These serial number ranges are assigned and managed dynamically by using a shared, replicated entry that defines the ranges for each CA and the next available range to reassign when one CA range runs low.
The serial number ranges with cloned CAs are fluid. All replicated CAs share a common configuration entry which defines the next available range. When one CA starts running low on available numbers, it checks this configuration entry and claims the next range. The entry is automatically updated, so that the next CA gets a new range.
The ranges are defined in
begin*Number and end*Number attributes, with separate ranges defined for requests and certificate serial numbers. For example:
dbs.beginRequestNumber=1 dbs.beginSerialNumber=1 dbs.enableSerialManagement=true dbs.endRequestNumber=9980000 dbs.endSerialNumber=ffe0000 dbs.replicaCloneTransferNumber=5
Only one replicated CA can generate, cache, and publish CRLs; this is the CRL CA. CRL requests submitted to other replicated CAs are immediately redirected to the CRL CA. While other replicated CAs can revoke, display, import, and download CRLs previously generated by the CRL CA, synchronization problems might occur if they generate new CRLs. For instructions on how to move CRL generation to a different replicated CA, see Section 10.7.1, “Converting CA Clones and Masters”.
Master CAs also manage the relationships and information sharing between the cloned CAs by monitoring replication changes to the internal databases.
Note
If a CA which is a security domain master is cloned, then that cloned CA is also a security domain master. In that case, both the original CA and its clone share the same security domain configuration.
Important
As covered in Section 2.7.9, “Custom Configuration and Clones”, the data within the LDAP database is replicated among a master and clones, but the configuration files for the different instances are not replicated. This means that any changes which affect the
CS.cfg file — such as adding a KRA connection or creating a custom profile — are not copied into a clone's configuration if the change occurs after the clone is created.
Any changes to the CA configuration should either be made to the master before any clones are created (so the custom changes are included in the cloning process) or any configuration changes must be copied over manually to the clone instances after they are created.
2.7.4. Cloning for KRAs
With KRAs, all keys archived in one KRA are replicated to the internal databases of the other KRAs. This allows a key recovery to be initiated on any clone KRA, regardless of which KRA the key was originally archived on.
After a key recovery is processed, the record of the recovery is stored in the internal database of all of the cloned KRAs.
With synchronous key recovery, although the recovery process can be initiated on any clone, it must be completed on the same single KRA on which it was initiated. This is because a recovery operation is recorded in the replicated database only after the appropriate number of approvals have been obtained from the KRA agents. Until then, the KRA on which the recovery is initiated is the only one which knows about the recovery operation.
Important
There exists a synchronous key recovery mechanism that has been deprecated in Red Hat Certificate System 9 and is not discussed in this documentation. Red Hat recommends the use of asynchronous key recovery instead.
2.7.5. Cloning for Other Subsystems
There is no real operational difference between replicated TKSs; the information created or maintained on one is replicated along the other servers.
For OCSPs, only one replicated OCSP receives CRL updates, and then the published CRLs are replicated to the clones.
2.7.6. Cloning and Key Stores
Cloning a subsystem creates two server processes performing the same functions: another new instance of the subsystem is created and configured to use the same keys and certificates to perform its operations. Depending on where the keys are stored for the master clone, the method for the clone to access the keys is very different.
If the keys and certificates are stored in the internal software token, then they must be exported from the master subsystem when it is first configured. When configuring the master instance, it is possible to backup the system keys and certificates to a PKCS #12 file by specifying the
pki_backup_keys and pki_backup_password parameters in the pkispawn configuration file. See the BACKUP PARAMETERS section in the pki_default.cfg(5) man page for more details.
If the keys were not backed up during the initial configuration, you can extract them to a PKCS #12 file using the
PKCS12Export utility, as described in Section 10.1, “Backing up Subsystem Keys from a Software Database”.
Then copy the PKCS #12 file over to the clone subsystem, and define its location and password in the
pkispawn configuration file using the pki_clone_pkcs12_password and pki_clone_pkcs12_path parameters For more information, see the Installing a Clone section in the pkispawn(8) man page. In particular, make sure that the PKCS#12 file is accessible by the pkiuser user and that it has the correct SELinux label.
If the keys and certificates are stored on a hardware token, then the keys and certificates must be copied using hardware token specific utilities or referenced directly in the token:
- Duplicate all the required keys and certificates, except the SSL/TLS server key and certificate to the clone instance. Keep the nicknames for those certificates the same. Additionally, copy all the necessary trusted root certificates from the master instance to the clone instance, such as chains or cross-pair certificates.
- If the token is network-based, then the keys and certificates simply need to be available to the token; the keys and certificates do not need to be copied.
- When using a network-based hardware token, make sure the high-availability feature is enabled on the hardware token to avoid single point of failure.
2.7.7. LDAP and Port Considerations
As mentioned in Section 2.7.1, “About Cloning”, part of the behavior of cloning is to replicate information between replicated subsystems, so that they work from an identical set of data and records. This means that the LDAP servers for the replicated subsystems need to be able to communicate.
If the Directory Server instances are on different hosts, then make sure that there is appropriate firewall access to allow the Directory Server instances to connect with each other.
Note
Cloned subsystems and their masters must use separate LDAP servers while they replicate data between common suffixes.
A subsystem can connect to its internal database using either SSL/TLS over an LDAPS port or over a standard connection over an LDAP port. When a subsystem is cloned, the clone instance uses the same connection method (SSL/TLS or standard) as its master to connect to the database. With cloning, there is an additional database connection though: the master Directory Server database to the clone Directory Server database. For that connection, there are three connection options:
- If the master uses SSL/TLS to connect to its database, then the clone uses SSL/TLS, and the master/clone Directory Server databases use SSL/TLS connections for replication.
- If the master uses a standard connection to its database, then the clone must use a standard connection, and the Directory Server databases can use unencrypted connections for replication.
- If the master uses a standard connection to its database, then the clone must use a standard connection, but there is an option to use Start TLS for the master/clone Directory Server databases for replication. Start TLS opens a secure connection over a standard port.
Note
To use Start TLS, the Directory Server must still be configured to accept SSL/TLS connections. This means that prior to configuring the clone, a server certificate and a CA certificate must be installed on the Directory Server, and SSL/TLS must be enabled.
Whatever connection method (secure or standard) used by the master must be used by the clone and must be properly configured for the Directory Server databases prior to configuring the clone.
Important
Even if the clone connects to the master over a secure connection, the standard LDAP port (389 by default) must still be open and enabled on the LDAP server while cloning is configured.
For secure environments, the standard LDAP port can be disabled on the master's Directory Server instance once the clone is configured.
2.7.8. Replica ID Numbers
Cloning is based on setting up a replication agreement between the Directory Server for the master instance and the Directory Server for the cloned instance.
Servers involved together with replication are in the same replication topology. Every time a subsystem instance is cloned, it is added to the overall topology. Directory Server discerns between different servers in the topology based on their replica ID number. This replica ID must be unique among all of the servers in the topology.
As with the serial number ranges used for requests and certificates (covered in Section 2.7.3, “Cloning for CAs”), every subsystem is assigned a range of allowed replica IDs. When the subsystem is cloned, it assigns one of the replica IDs from its range to the new clone instance.
dbs.beginReplicaNumber=1 dbs.endReplicaNumber=95
The replica ID range can be refreshed with new numbers if an instance begins to exhaust its current range.
2.7.9. Custom Configuration and Clones
After a clone is created, configuration changes are not replicated between clones or between a master and a clone. The instance configuration is in the
CS.cfg file — outside the replicated database.
For example, there are two CAs, a master and a clone. A new KRA is installed which is associated, at its configuration, with the master CA. The CA-KRA connector information is stored in the master CA's
CS.cfg file, but this connector information is not added to the clone CA configuration. If a certificate request that includes a key archival is submitted to the master CA, then the key archival is forwarded to the KRA using the CA-KRA connector information. If the request is submitted to the clone CA, no KRA is recognized, and the key archival request is disallowed.
Changes made to the configuration of a master server or to a clone server are not replicated to other cloned instances. Any critical settings must be added manually to clones.
Note
You can set all required, custom configuration for a master server before configuring any clones. For example, install all KRAs so that all the connector information is in the master CA configuration file, create any custom profiles, or configure all publishing points for a master OCSP responder. Note that if the LDAP profiles are stored in the Directory Server, they are replicated and kept in sync across servers.
Any custom settings in the master instance will be included in the cloned instances at the time they are cloned (but not after).
