Tectonic clusters contain three distinct types of nodes:
- master node(s) which carry the master control plane and core Kubernetes components (such as the scheduler or the api controller).
- worker node(s) which carry the user applications
- etcd node(s) which acts as a key-value storage for the cluster.
In a normal production environment, master nodes are usually run in HA (high availability) mode, so that the control plane is available at any given time, even in the event of a master failure. In this article, I'll explain and provide troubleshooting tips on how to rescue a Tectonic cluster with all masters down.