Chapter 8. Ensuring system integrity with Keylime

Procedure

1. Install the Keylime verifier:

   # dnf install keylime-verifier

2. Define the IP address and port of the registrar and verifier in the verifier configuration.

   1. Create a new .conf file in the /etc/keylime/verifier.conf.d/ directory, for example, /etc/keylime/verifier.conf.d/00-verifier-ip.conf, with the following content:

      [verifier]
      ip = <verifier_IP_address>

      • Replace <verifier_IP_address> with the verifier’s IP address. Alternatively, use ip = * or ip = 0.0.0.0 to bind the verifier to all available IP addresses.
      • Optionally, you can also change the verifier’s port from the default value 8881 by using the port = <verifier_port> option.

   2. Create a new .conf file in the /etc/keylime/verifier.conf.d/ directory, for example, /etc/keylime/verifier.conf.d/00-registrar-ip.conf, with the following content:

      [verifier]
      registrar_ip = <registrar_IP_address>

      • Replace <registrar_IP_address> with the registrar’s IP address.
      • If the registrar uses a different port than the default value 8891, add the registrar_port = <registrar_port> setting.

3. Optional: Configure the verifier’s database for the list of agents. The default configuration uses an SQLite database in the verifiers’s /var/lib/keylime/cv_data.sqlite/ directory. You can define a different database by creating a new .conf file in the /etc/keylime/verifier.conf.d/ directory, for example, /etc/keylime/verifier.conf.d/00-db-url.conf, with the following content:

   [verifier]
   database_url = <protocol>://<name>:<password>@<ip_address_or_hostname>/<properties>

   Replace <protocol>://<name>:<password>@<ip_address_or_hostname>/<properties> with the URL of the database, for example, postgresql://verifier:UQ?nRNY9g7GZzN7@198.51.100.1/verifierdb.

   Ensure that the credentials you use have the permissions for Keylime to create the database structure.

4. Add certificates and keys to the verifier. You can either let Keylime generate them, or use existing keys and certificates:

   • With the default tls_dir = generate option, Keylime generates new certificates for the verifier, registrar, and tenant in the /var/lib/keylime/cv_ca/ directory.

   • To load existing keys and certificates in the configuration, define their location in the verifier configuration.

     Note

     Certificates must be accessible by the keylime user, under which the Keylime services are running.

     Create a new .conf file in the /etc/keylime/verifier.conf.d/ directory, for example, /etc/keylime/verifier.conf.d/00-keys-and-certs.conf, with the following content:

     [verifier]
     tls_dir = /var/lib/keylime/cv_ca
     server_key = </path/to/server_key>
     server_key_password = <passphrase1>
     server_cert = </path/to/server_cert>
     trusted_client_ca = ['</path/to/ca/cert1>', '</path/to/ca/cert2>']
     client_key = </path/to/client_key>
     client_key_password = <passphrase2>
     client_cert = </path/to/client_cert>
     trusted_server_ca = ['</path/to/ca/cert3>', '</path/to/ca/cert4>']

     Note

     Use absolute paths to define key and certificate locations. Alternatively, relative paths are resolved from the directory defined in the tls_dir option.

5. Open the port in firewall:

   # firewall-cmd --add-port 8881/tcp
   # firewall-cmd --runtime-to-permanent

   If you use a different port, replace 8881 with the port number defined in the .conf file.

6. Start the verifier service:

   # systemctl enable --now keylime_verifier

   Note

   In the default configuration, start the keylime_verifier before starting the keylime_registrar service because the verifier creates the CA and certificates for the other Keylime components. This order is not necessary when you use custom certificates.

Procedure

1. Install the Keylime registrar:

   # dnf install keylime-registrar

2. Define the IP address and port of the registrar:

   1. Create a new .conf file in the /etc/keylime/registrar.conf.d/ directory, for example, /etc/keylime/registrar.conf.d/00-registrar-ip.conf, with the following content:

      [registrar]
      ip = <registrar_IP_address>

      • Replace <registrar_IP_address> with the registrar’s IP address. Alternatively, use ip = * or ip = 0.0.0.0 to bind the registrar to all available IP addresses.
      • Optionally, change the port to which the Keylime agents connect by using the port = option. The default value is 8890.
      • Optionally, change the TLS port to which the Keylime verifier and tenant connect by using the tls_port = option. The default value is 8891.

3. Optional: Configure the registrar’s database for the list of agents. The default configuration uses an SQLite database in the registrar’s /var/lib/keylime/reg_data.sqlite directory. You can create a new .conf file in the /etc/keylime/registrar.conf.d/ directory, for example, /etc/keylime/registrar.conf.d/00-db-url.conf, with the following content:

   [registrar]
   database_url = <protocol>://<name>:<password>@<ip_address_or_hostname>/<properties>

   Replace <protocol>://<name>:<password>@<ip_address_or_hostname>/<properties> with the URL of the database, for example, postgresql://registrar:EKYYX-bqY2?#raXm@198.51.100.1/registrardb.

   Ensure that the credentials you use have the permissions for Keylime to create the database structure.

4. Add certificates and keys to the registrar:

   • You can use the default configuration and load the keys and certificates to the /var/lib/keylime/reg_ca/ directory.

   • Alternatively, you can define the location of the keys and certificates in the configuration. Create a new .conf file in the /etc/keylime/registrar.conf.d/ directory, for example, /etc/keylime/registrar.conf.d/00-keys-and-certs.conf, with the following content:

     Important

     Do not use the tls_dir = default setting with custom CA certificates. Instead, specify a path to the location of the certificates. For more information, see RHELPLAN-157337.

     [registrar]
     tls_dir = /var/lib/keylime/reg_ca
     server_key = </path/to/server_key>
     server_key_password = <passphrase1>
     server_cert = </path/to/server_cert>
     trusted_client_ca = ['</path/to/ca/cert1>', '</path/to/ca/cert2>']

     Note

     Use absolute paths to define key and certificate locations. Alternatively, you can define a directory in the tls_dir option and use paths relative to that directory.

5. Open the ports in firewall:

   # firewall-cmd --add-port 8890/tcp --add-port 8891/tcp
   # firewall-cmd --runtime-to-permanent

   If you use a different port, replace 8890 or 8891 with the port number defined in the .conf file.

6. Start the keylime_registrar service:

   # systemctl enable --now keylime_registrar

   Note

   In the default configuration, start the keylime_verifier before starting the keylime_registrar service because the verifier creates the CA and certificates for the other Keylime components. This order is not necessary when you use custom certificates.

Procedure

1. Install the Keylime tenant:

   # dnf install keylime-tenant

2. Define the tenant’s connection to the Keylime verifier by editing the /etc/keylime/tenant.conf.d/00-verifier-ip.conf file:

   [tenant]
   verifier_ip = <verifier_ip>

   • Replace <verifier_ip> with the IP address to the verifier’s system. Alternatively, use ip = * or ip = 0.0.0.0 to bind the tenant to all available IP addresses.
   • If the verifier uses a different port than the default value 8881, add the verifier_port = <verifier_port> setting.

3. Define the tenant’s connection to the Keylime registrar by editing the /etc/keylime/tenant.conf.d/00-registrar-ip.conf file:

   [tenant]
   registrar_ip = <registrar_ip>
   registrar_port = <registrar_port>

   • Replace <registrar_ip> with the IP address to the registrar’s system.
   • If the registrar uses a different port than the default value 8891, add the registrar_port = <registrar_port> setting.

4. Add certificates and keys to the tenant:

   1. You can use the default configuration and load the keys and certificates to the /var/lib/keylime/cv_ca directory.

   2. Alternatively, you can define the location of the keys and certificates in the configuration. Create a new .conf file in the /etc/keylime/tenant.conf.d/ directory, for example, /etc/keylime/tenant.conf.d/00-keys-and-certs.conf, with the following content:

      [tenant]
      tls_dir = /var/lib/keylime/cv_ca
      client_key = tenant-key.pem
      client_key_password = <passphrase1>
      client_cert = tenant-cert.pem
      trusted_server_ca = ['/var/lib/keylime/cv_ca/cacert.pem']

      The trusted_server_ca parameter accepts paths to the verifier and registrar server CA certificate. You can provide multiple comma-separated paths, for example if the verifier and registrar use different CAs.

      Note

      Use absolute paths to define key and certificate locations. Alternatively, you can define a directory in the tls_dir option and use paths relative to that directory.

5. Optional: If the trusted platform module (TPM) endorsement key (EK) cannot be verified by using certificates in the /var/lib/keylime/tpm_cert_store directory, add the certificate to that directory. This can occur particularly when using virtual machines with emulated TPMs.

Verification

1. Check the status of the verifier:

   3 keylime_tenant -c cvstatus
   Reading configuration from ['/etc/keylime/logging.conf']
   2022-10-14 12:56:08.155 - keylime.tpm - INFO - TPM2-TOOLS Version: 5.2
   Reading configuration from ['/etc/keylime/tenant.conf']
   2022-10-14 12:56:08.157 - keylime.tenant - INFO - Setting up client TLS...
   2022-10-14 12:56:08.158 - keylime.tenant - INFO - Using default client_cert option for tenant
   2022-10-14 12:56:08.158 - keylime.tenant - INFO - Using default client_key option for tenant
   2022-10-14 12:56:08.178 - keylime.tenant - INFO - TLS is enabled.
   2022-10-14 12:56:08.178 - keylime.tenant - WARNING - Using default UUID d432fbb3-d2f1-4a97-9ef7-75bd81c00000
   2022-10-14 12:56:08.221 - keylime.tenant - INFO - Verifier at 127.0.0.1 with Port 8881 does not have agent d432fbb3-d2f1-4a97-9ef7-75bd81c00000.

   If correctly set up, and if no agent is configured, the verifier responds that it does not recognize the default agent UUID.

2. Check the status of the registrar:

   # keylime_tenant -c regstatus
   Reading configuration from ['/etc/keylime/logging.conf']
   2022-10-14 12:56:02.114 - keylime.tpm - INFO - TPM2-TOOLS Version: 5.2
   Reading configuration from ['/etc/keylime/tenant.conf']
   2022-10-14 12:56:02.116 - keylime.tenant - INFO - Setting up client TLS...
   2022-10-14 12:56:02.116 - keylime.tenant - INFO - Using default client_cert option for tenant
   2022-10-14 12:56:02.116 - keylime.tenant - INFO - Using default client_key option for tenant
   2022-10-14 12:56:02.137 - keylime.tenant - INFO - TLS is enabled.
   2022-10-14 12:56:02.137 - keylime.tenant - WARNING - Using default UUID d432fbb3-d2f1-4a97-9ef7-75bd81c00000
   2022-10-14 12:56:02.171 - keylime.registrar_client - CRITICAL - Error: could not get agent d432fbb3-d2f1-4a97-9ef7-75bd81c00000 data from Registrar Server: 404
   2022-10-14 12:56:02.172 - keylime.registrar_client - CRITICAL - Response code 404: agent d432fbb3-d2f1-4a97-9ef7-75bd81c00000 not found
   2022-10-14 12:56:02.172 - keylime.tenant - INFO - Agent d432fbb3-d2f1-4a97-9ef7-75bd81c00000 does not exist on the registrar. Please register the agent with the registrar.
   2022-10-14 12:56:02.172 - keylime.tenant - INFO - {"code": 404, "status": "Agent d432fbb3-d2f1-4a97-9ef7-75bd81c00000 does not exist on registrar 127.0.0.1 port 8891.", "results": {}}

   If correctly set up, and if no agent is configured, the registrar responds that it does not recognize the default agent UUID.

Procedure

1. Install the Keylime agent:

   # dnf install keylime-agent

   This command installs the keylime-agent-rust package.

2. Define the agent’s IP address and port in the configuration files. Create a new .conf file in the /etc/keylime/agent.conf.d/ directory, for example, /etc/keylime/agent.conf.d/00-agent-ip.conf, with the following content:

   [agent]
   ip = '<agent_ip>'

   Note

   Because the Keylime agent configuration uses the TOML format, which is different from the INI format used for configuration of the other components, the values must be in single quotation marks.

   • Replace <agent_IP_address> with the agent’s IP address. Alternatively, use ip = '*' or ip = '0.0.0.0' to bind the agent to all available IP addresses.
   • Optionally, you can also change the agent’s port from the default value 9002 by using the port = '<agent_port>' option.

3. Define the registrar’s IP address and port in the configuration files. Create a new .conf file in the /etc/keylime/agent.conf.d/ directory, for example, /etc/keylime/agent.conf.d/00-registrar-ip.conf, with the following content:

   [agent]
   registrar_ip = '<registrar_IP_address>'

   • Replace <registrar_IP_address> with the registrar’s IP address.
   • Optionally, you can also change the registrar’s port from the default value 8890 by using the registrar_port = '<registrar_port>' option.

4. Optional: Define the agent’s universally unique identifier (UUID). If it is not defined, the default UUID is used. Create a new .conf file in the /etc/keylime/agent.conf.d/ directory, for example, /etc/keylime/agent.conf.d/00-agent-uuid.conf, with the following content:

   [agent]
   uuid = '<agent_UUID>'

   • Replace <agent_UUID> with the agent’s UUID, for example d432fbb3-d2f1-4a97-9ef7-abcdef012345. You can use the uuidgen utility to generate a UUID.

5. Optional: Load existing keys and certificates for the agent. If the agent receives no server_key and server_cert, it generates its own key and a self-signed certificate.

   Important

   Do not use certificate chains. Keylime currently does not correctly use all the provided certificates during signature verification, which results in a TLS handshake failure. For more information, see RHEL-396.

   Define the location of the keys and certificates in the configuration. Create a new .conf file in the /etc/keylime/agent.conf.d/ directory, for example, /etc/keylime/agent.conf.d/00-keys-and-certs.conf, with the following content:

   [agent]
   server_key = '</path/to/server_key>'
   server_key_password = '<passphrase1>'
   server_cert = '</path/to/server_cert>'
   trusted_client_ca = '</path/to/ca/cert>'

   Note

   Use absolute paths to define key and certificate locations. The Keylime agent does not accept relative paths.

6. Open the port in firewall:

   # firewall-cmd --add-port 9002/tcp
   # firewall-cmd --runtime-to-permanent

   If you use a different port, replace 9002 with the port number defined in the .conf file.

7. Enable and start the keylime_agent service:

   # systemctl enable --now keylime_agent

8. Optional: From the system where the Keylime tenant is configured, verify that the agent is correctly configured and can connect to the registrar.

   # keylime_tenant -c regstatus --uuid <agent_uuid>
   Reading configuration from ['/etc/keylime/logging.conf']
   ...
   ==\n-----END CERTIFICATE-----\n", "ip": "127.0.0.1", "port": 9002, "regcount": 1, "operational_state": "Registered"}}}

   • Replace <agent_uuid> with the agent’s UUID.

     If the registrar and agent are correctly configured, the output displays the agent’s IP address and port, followed by "operational_state": "Registered".

9. Create a new IMA policy by entering the following content into the /etc/ima/ima-policy file:

   # PROC_SUPER_MAGIC
   dont_measure fsmagic=0x9fa0
   # SYSFS_MAGIC
   dont_measure fsmagic=0x62656572
   # DEBUGFS_MAGIC
   dont_measure fsmagic=0x64626720
   # TMPFS_MAGIC
   dont_measure fsmagic=0x01021994
   # RAMFS_MAGIC
   dont_measure fsmagic=0x858458f6
   # SECURITYFS_MAGIC
   dont_measure fsmagic=0x73636673
   # SELINUX_MAGIC
   dont_measure fsmagic=0xf97cff8c
   # CGROUP_SUPER_MAGIC
   dont_measure fsmagic=0x27e0eb
   # OVERLAYFS_MAGIC
   dont_measure fsmagic=0x794c7630
   # Don't measure log, audit or tmp files
   dont_measure obj_type=var_log_t
   dont_measure obj_type=auditd_log_t
   dont_measure obj_type=tmp_t
   # MEASUREMENTS
   measure func=BPRM_CHECK
   measure func=FILE_MMAP mask=MAY_EXEC
   measure func=MODULE_CHECK uid=0

10. Reboot the system to apply the new IMA policy.

Verification

1. Verify that the agent is running:

   # systemctl status keylime_agent
   ● keylime_agent.service - The Keylime compute agent
        Loaded: loaded (/usr/lib/systemd/system/keylime_agent.service; enabled; preset: disabled)
        Active: active (running) since ...

Procedure

1. On the monitored system where the Keylime agent is configured and running, generate an allowlist from the current state of the system:

   # /usr/share/keylime/scripts/create_allowlist.sh -o <allowlist.txt> -h sha256sum

   Replace <allowlist.txt> with the file name of the allowlist.

   Important

   Use the SHA-256 hash function. SHA-1 is not secure and has been deprecated in RHEL 9. For additional information, see SHA-1 deprecation in Red Hat Enterprise Linux 9.

2. Copy the generated allowlist to the system where the keylime_tenant utility is configured, for example:

   # scp allowlist.txt root@<tenant_._ip>:/root/allowlist.txt

3. Optional: You can define a list of files or directories excluded from Keylime measurements by creating a file on the tenant system and entering the files and directories to exclude. The excludelist accepts Python regular expressions with one regular expression per line. For more information, see Regular expression operations at docs.python.org. For example, to exclude all files in the /tmp/ directory from Keylime measurements, create a /root/excludelist.txt file with the following content:

   /tmp/.*

   Save the excludelist on the tenant system.

4. On the system where the Keylime tenant is configured, provision the agent by using the keylime_tenant utility:

   # keylime_tenant -c add -t <agent_ip> -u <agent_uuid> --allowlist <allowlist.txt> --exclude <excludelist> --cert default

   • Replace <agent_ip> with the agent’s IP address.
   • Replace <agent_uuid> with the agent’s UUID.
   • Replace <allowlist.txt> with the path to the allowlist file.
   • Replace <excludelist> with the path to the excludelist file. The --exclude option is optional; provisioning the agent works even without delivering a file.

   • With the --cert option, the tenant generates and signs a certificate for the agent by using the CA certificates and keys located in the specified directory, or the default /var/lib/keylime/ca/ directory. If the directory contains no CA certificates and keys, the tenant will generate them automatically according to the configuration in the /etc/keylime/ca.conf file and save them to the specified directory. The tenant then sends these keys and certificates to the agent.

     When generating CA certificates or signing agent certificates, you may be prompted for the password to access the CA private key: Please enter the password to decrypt your keystore:.

     If you do not want to use a certificate, use the -f option instead for delivering a file to the agent. Provisioning an agent requires sending any file, even an empty file.

     Note

     Keylime encrypts the file sent to the agent, and decrypts it only if the agent’s system complies with the TPM policy and the IMA allowlist. By default, Keylime decompresses .zip files.

   As an example, with the following command, keylime_tenant provisions a new Keylime agent at 127.0.0.1 with UUID d432fbb3-d2f1-4a97-9ef7-75bd81c00000 and loads an allowlist named allowlist.txt. It also generates a certificate into the default directory and sends it to the agent. Keylime decrypts the file only if the TPM policy configured in /etc/keylime/verifier.conf is satisfied:

   # keylime_tenant -c add -t 127.0.0.1 -u d432fbb3-d2f1-4a97-9ef7-75bd81c00000 -cert default --allowlist allowlist.txt --exclude excludelist.txt

   Note

   You can stop Keylime from monitoring a node by using the # keylime_tenant -c delete -u <agent_uuid> command.

   You can modify the configuration of an already registered agent by using the keylime_tenant -c update command.

Verification

1. Optional: Reboot the monitored system before verification to verify that the settings are persistent.

2. Verify a successful attestation of the agent:

   # keylime_tenant -c cvstatus -u <agent.uuid>
   ...
   {"<agent.uuid>": {"operational_state": "Get Quote"..."attestation_count": 5
   ...

   Replace <agent.uuid> with the agent’s UUID.

   If the value of operational_state is Get Quote and attestation_count is non-zero, the attestation of this agent is successful.

   If the value of operational_state is Invalid Quote or Failed attestation fails, the command displays output similar to the following:

   {"<agent.uuid>": {"operational_state": "Invalid Quote", ... "ima.validation.ima-ng.not_in_allowlist", "attestation_count": 5, "last_received_quote": 1684150329, "last_successful_attestation": 1684150327}}

3. If the attestation fails, display more details in the verifier log:

   # journalctl -u keylime_verifier
   keylime.tpm - INFO - Checking IMA measurement list...
   keylime.ima - WARNING - File not found in allowlist: /root/bad-script.sh
   keylime.ima - ERROR - IMA ERRORS: template-hash 0 fnf 1 hash 0 good 781
   keylime.cloudverifier - WARNING - agent D432FBB3-D2F1-4A97-9EF7-75BD81C00000 failed, stopping polling

Procedure

1. On the monitored system where the Keylime agent is configured and running, install the python3-keylime, which contains create_mb_refstate utility:

   # dnf -y install python3-keylime

2. On the monitored system, generate a policy from the measured boot log of the current state of the system by using the create_mb_refstate script:

   # /usr/share/keylime/scripts/create_mb_refstate /sys/kernel/security/tpm0/binary_bios_measurements <./measured_boot_reference_state.json>

   Replace <./measured_boot_reference_state.json> with the path where the script saves the generated policy.

   Important

   The policy generated with the create_mb_refstate script is based on the current state of the system and is very strict. Any modifications of the system including kernel updates and system updates will change the boot process and the system will fail the attestation.

3. Copy the generated policy to the system where the keylime_tenant utility is configured, for example:

   # scp root@<agent_ip>:/root/measured_boot_reference_state.json /root/measured_boot_reference_state.json

4. On the system where the Keylime tenant is configured, provision the agent by using the keylime_tenant utility:

   # keylime_tenant -c add -t <agent_ip> -u <agent_uuid> --mb_refstate <./measured_boot_reference_state.json>

   • Replace <agent_ip> with the agent’s IP address.
   • Replace <agent_uuid> with the agent’s UUID.
   • Replace <./measured_boot_reference_state.json> with the path to the measured boot policy.

   If you configure measured boot in combination with runtime monitoring, provide all the options from both use cases when entering the keylime_tenant -c add command.

   Note

   You can stop Keylime from monitoring a node by using the # keylime_tenant -c delete -t <agent_ip> -u <agent_uuid> command.

   You can modify the configuration of an already registered agent by using the keylime_tenant -c update command.

Verification

1. Reboot the monitored system and verify a successful attestation of the agent:

   # keylime_tenant -c cvstatus -u <agent_uuid>
   ...
   {"<agent.uuid>": {"operational_state": "Get Quote"..."attestation_count": 5
   ...

   Replace <agent_uuid> with the agent’s UUID.

   If the value of operational_state is Get Quote and attestation_count is non-zero, the attestation of this agent is successful.

   If the value of operational_state is Invalid Quote or Failed attestation fails, the command displays output similar to the following:

   {"<agent.uuid>": {"operational_state": "Invalid Quote", ... "ima.validation.ima-ng.not_in_allowlist", "attestation_count": 5, "last_received_quote": 1684150329, "last_successful_attestation": 1684150327}}

2. If the attestation fails, display more details in the verifier log:

   # journalctl -u keylime_verifier
   {"d432fbb3-d2f1-4a97-9ef7-75bd81c00000": {"operational_state": "Tenant Quote Failed", ... "last_event_id": "measured_boot.invalid_pcr_0", "attestation_count": 0, "last_received_quote": 1684487093, "last_successful_attestation": 0}}