How to compact and defrag etcd to decrease the database size in OpenShift 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.X

Issue

  • What are the steps to compact and defrag the etcd database in OCP 4.
  • When etcd needs defragmentation?

Resolution

Prerequisites

Important: etcd defragmentation can impact cluster performance. Before compacting or defragmenting, check that the dbSize and the dbSizeInUse differ in more than 40-50% in the output of the etcdctl endpoint status --cluster -w json command. Check also what member is the etcd leader in above commands, as it needs to be the last member to run the commands against.

$ oc get pods -n openshift-etcd | grep etcd
[...]
$ oc rsh  -n openshift-etcd [etcd-pod] etcdctl endpoint status --cluster -w table
[...]
$ oc rsh  -n openshift-etcd [etcd-pod] etcdctl endpoint status --cluster -w json
[...]

NOTE: If the above does not work, compact operation can be performed first and then defrag.

In case the oc doesn't work, the commands can be executed directly in the container on the master. If you have SSH access to the master, you can connect to the container using the command below.

# crictl exec -it $(crictl ps | grep etcdctl | awk '{print $1}') bash

Etcd compact operation

# Get the current revision number
$ etcdctl endpoint status --write-out json | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*'
440161
440161
440161

# Compact the database
$etcdctl --command-timeout=600s compact <rev_number>

Defrag

For Defrag of the etcd database, please refer to the OpenShift 4 documentation: Defragmenting etcd data.

To verify the defragmentation, run the steps from "Diagnostic Steps" section.

Additional notes

  • Make sure of leaving the leader instance to be the last one to run the command against.
  • If a timeout occurs please increase --command-timeout until success.
  • It is important to note that the defrag action is blocking. The member will not respond until the defrag is complete. For this reason, defrag should be a rolling action.
  • Allow 30 seconds to 1 minute in between defrag actions on each of the etcd pods for the cluster to recover.
  • Note that defrag operation needs to be run against each member individually. Make sure to unset ETCDCTL_ENDPOINTS before defrag and run the command with --endpoints=https://localhost:2379 as per documentation.

Root Cause

Compacting old revisions internally fragments etcd by leaving gaps in the backend database. Fragmented space is available for use by etcd but unavailable to the host filesystem. In other words, deleting application data does not reclaim the space on the disk.
The process of defragmentation releases this storage space back to the file system. Defragmentation is issued for each etcd member individually, so that cluster-wide latency spikes may be avoided as the defrag action is blocking.

For OCP 3, refer to How to defrag Etcd to decrease DB size in OpenShift 3.

Diagnostic Steps

Verify that the defrag was successful in reducing the database space on all the etcd instances:

$ oc get pods -n openshift-etcd | grep etcd

$ oc rsh  -n openshift-etcd [etcd-pod]
sh-4.2# etcdctl endpoint status -w table --cluster
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.0.000.000:2379 | 2be524b7dda9390d |   3.4.9 |   70 MB |     false |      false |         7 |     657422 |             657422 |        |
| https://10.0.001.010:2379 | b2d7a157e701c9e5 |   3.4.9 |  70 MB |      true |      false |         7 |     657422 |             657422 |        |
|  https://10.0.002.02:2379 | d5ae36565eff47a1 |   3.4.9 |  70 MB |     false |      false |         7 |     657422 |             657422 |        |
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments