Chapter 4. Troubleshooting Kubernetes

All Applications are not equally kuberizable, because there are limitations on the type of applications that can be run as Kubernetes services. In Kubernetes, a service is a load balanced proxy whose IP address is injected into the iptables of clients of that service. Therefore, you should verify that the application you intend to "kuberize":

If neither of these restrictions apply, or if the user can disable these checks, you can continue on.

Depending on the type of software you are running you may wish to take advantage of some predefined environment variables that are provided for clients of Kubernetes services.

For example, given a service named db, if you launch a Pod in Kubernetes that uses that service, Kubernetes will inject the following environment variables into the containers in that pod:

DB_SERVICE_PORT_3306_TCP_PORT=3306
DB_SERVICE_SERVICE_HOST=10.254.100.1
DB_SERVICE_PORT_3306_TCP_PROTO=tcp
DB_SERVICE_PORT_3306_TCP_ADDR=10.254.100.1
DB_SERVICE_PORT_3306_TCP=tcp://10.254.100.1:3306
DB_SERVICE_PORT=tcp://10.254.100.1:3306
DB_SERVICE_SERVICE_PORT=3306

NOTE: Notice that the service name (db) is capitalized in the variables (DB). If there were dashes (-) in the name, they would be converted to underscores (_).

To see these and other shell variables, use docker exec to open a shell to the active container and run env to see the shell variables:

# docker exec -it <container_ID> /bin/bash
[root@e7ea67..]# env
...
WEBSERVER_SERVICE_SERVICE_PORT=80
KUBERNETES_RO_SERVICE_PORT=80
KUBERNETES_SERVICE_PORT=443
KUBERNETES_RO_PORT_80_TCP_PORT=80
KUBERNETES_SERVICE_HOST=10.254.255.128
DB_SERVICE_PORT_3306_TCP_PORT=3306
DB_SERVICE_SERVICE_HOST=10.254.100.1
WEBSERVER_SERVICE_PORT_80_TCP_ADDR=10.254.100.50
...

When starting your client applications you may want to leverage those variables. If you are debugging communications problems between containers, viewing these shell variables is a great way to see each container’s view of the addresses of services and ports.

From the Kubernetes master, inspect the running Kubernetes configuration. We’ll start by showing you how this configuration should look when everything is working. Then we’ll show you how to the setup might break in various ways and how you can go about fixing it.

Using kubectl is the simplest way to manually debug the process of application submission, service creation, and pod assignment. To see what pods, services, and replication controllers are active, run these commands on the master:

# kubectl get pods
POD             IP          CONTAINER(S)     IMAGE(S)  HOST                LABELS                                         STATUS
4e04dd3b-c...   10.20.29.3  apache-frontend  webwithdb node2.example.com/  name=webserver,selectorname=webserver,uses=db  Running
5544eab2-c...   10.20.48.15 apache-frontend  webwithdb node1.example.com/  name=webserver,selectorname=webserver,uses=db  Running
1c971a09-c...   10.20.29.2  db               dbforweb  node2.example.com   name=db,selectorname=db                        Running
1c97a755-c...   10.20.48.14 db               dbforweb  node1.example.com/  name=db,selectorname=db                        Running
# kubectl get services
NAME               LABELS                                   SELECTOR        IP                  PORT
webserver-service  name=webserver                           name=webserver  10.254.100.50       80
db-service         name=db                                  name=db         10.254.100.1        3306
kubernetes         component=apiserver,provider=kubernetes            10.254.92.19        443
kubernetes-ro      component=apiserver,provider=kubernetes            10.254.206.141      80
# kubectl get replicationControllers
CONTROLLER             CONTAINER(S)        IMAGE(S)            SELECTOR                 REPLICAS
webserver-controller   apache-frontend     webwithdb           selectorname=webserver   2
db-controller          db                  dbforweb            selectorname=db          2

Here’s information to help you interpret this output:

If something goes wrong in the process of getting to this state, the following sections will help you troubleshoot problems.

A Kubernetes cluster that consists of a master and one or more nodes (minions) needs to initialize a particular set of systemd services. You should verify that the following services are running on the master and on each node:

Here’s how you verify those services on the master and each node:

Master: On your kubernetes master server, this will tell you if the proper services are active and enabled (flanneld may not be configured on your system):

# for SERVICES in etcd flanneld kube-apiserver kube-controller-manager kube-scheduler;
    do echo --- $SERVICES --- ; systemctl is-active $SERVICES ;
    systemctl is-enabled $SERVICES ; echo "";  done
--- etcd ---
active
enabled
--- flanneld ---
active
enabled
--- kube-apiserver ---
active
enabled
--- kube-controller-manager ---
active
enabled
--- kube-scheduler ---
active
enabled

Nodes: On each node, make sure the proper services are active and enabled:

# for SERVICES in flanneld docker kube-proxy.service kubelet.service; \
do echo --- $SERVICES --- ; systemctl is-active $SERVICES ; \
systemctl is-enabled $SERVICES ; echo "";  done
--- flanneld ---
active
enabled
--- docker ---
active
enabled
--- kube-proxy.service ---
active
enabled
--- kubelet.service ---
active
enabled

If any of the master or node systemd services are disabled or failed, here’s what to do:

There is no iptables or firewalld service installed on RHEL Atomic Host. So, by default, there are no firewall filter rules blocking access to Kubernetes services. However, if you have a firewall running on a RHEL host or if you have added iptables firewall rules to your Kubernetes master or nodes to filter incoming ports, you need to make sure that the ports that need to be exposed on those systems are not blocked.

The following is the output of a netstat command, showing which ports Kubernetes and related services are listening on a Kubernetes nodes:

# netstat -tupln
tcp6       0      0 :::10249          :::*          LISTEN      125528/kube-proxy
tcp6       0      0 :::10250          :::*          LISTEN      125536/kubelet

NOTE: The kube-proxy service listens on random ports. This is not a problem on RHEL Atomic systems, since there is on filtering firewall service used by default. However, if you add a firewall to RHEL Atomic or use a default RHEL system, you can request that kube-proxy listen on specific ports in the service definition and then open those ports in the firewall.

Here is netstat output on a Kubernetes master:

tcp        0      0 192.168.122.249:7080  0.0.0.0:* LISTEN   636/kube-apiserver
tcp6       0      0 :::8080               :::*      LISTEN   636/kube-apiserver
tcp        0      0 127.0.0.1:10252       0.0.0.0:* LISTEN   7541/kube-controller
tcp        0      0 127.0.0.1:10251       0.0.0.0:* LISTEN   7590/kube-scheduler
tcp6       0      0 :::4001               :::*      LISTEN   941/etcd
tcp6       0      0 :::7001               :::*      LISTEN   941/etcd

The output in the third column shows the IP addresses and port number that each service is listening on. (::: represents all interfaces) Open ports to each of those services.

You set up your Kubernetes environment (pods, services, and replication controllers) by loading information from yaml or json files using the kubectl create command. Failures can result from those files being improperly formatted or missing needed information.

The following sections contain tips for fixing problems that occur from broken yaml or json files.

id: "db-service"
kind: "Service"
apiVersion: "v1"
port: 3306
portalIP: "10.254.100.1"
selector:
  name: "db"
labels:
  name: "db"

# kubectl create -f db-service
# kubectl get services
NAME          LABELS                                  SELECTOR  IP             PORT
db-service    name=db                                 name=db   10.254.100.1   3306
kubernetes-ro component=apiserver,provider=kubernetes     10.254.186.33  80
kubernetes    component=apiserver,provider=kubernetes     10.254.198.9   443

id: "webserver-controller"
kind: "ReplicationController"
apiVersion: "v1"
metadata:
  name: "webserver-controller"
spec:
  replicas: 1
  selector:
    name: "webserver"
  template:
    spec:
      containers:
        - name: "apache-frontend"
          image: "webwithdb"
          ports:
            - containerPort: 80
    metadata:
      labels:
        name: "webserver"
        uses: db
  labels:
    name: "webserver"

# kubectl create -f webserver-service.yaml
# kubectl get pods
POD         IP          CONTAINER(S)     IMAGE(S)    HOST
       LABELS                                             STATUS
f28980d...  10.20.48.4  apache-frontend  webwithdb   node1.example.com/
       name=webserver,selectorname=webserver,uses=db      Running
f28a0a8...  10.20.29.9  apache-frontend  webwithdb   node2.example.com/
       name=webserver,selectorname=webserver,uses=db      Running
# kubectl get replicationControllers
CONTROLLER           CONTAINER(S)    IMAGE(S)   SELECTOR               REPLICAS

webserver-controller apache-frontend webwithdb  selectorname=webserver 2

The etcd service provides the database that Kubernetes uses to coordinate information across the cluster. There are ways to view the database directly and fix problems in it (or clear the database if it is beyond repair).

Displaying data from the etcd database: You can query most information you need from the etcd database using kubectl get commands. However, if this database seems to be inconsistent with the way you believe your configuration should be, you can directly query the etcd database using the etcdctl command.

Use the etcdctl command with the ls option to list the directory structure of the database. To get values, use the get option. For example, to see the root of the database, type the following:

# etcdctl ls /
/registry

To list information associated with the etcd database, type this:

# etcdctl ls /registry/

/registry/namespaces
/registry/ranges
/registry/serviceaccounts
/registry/services

...

To see the data associated with a particular entry, type the following:

# etcdctl get /registry/namespaces/default | python -mjson.tool
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2016-10-24T12:05:11Z",
        "name": "default",
        "uid": "1d6efb5f-99e2-11e6-8f4b-525400585a9f"
    },
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },
    "status": {
        "phase": "Active"
    }
}

The output above is piped to a python json.tool formatting module, to make it easier to read.

NOTE: Instead of the etcdctl command, you can use the curl. For example, to see the root of the database with curl, use this instead of the etcdctl ls / command: curl -L http://localhost:2379/v2/keys/ | python -mjson.tool. Use that form of the curl command to display both directory and key values. If you believe that a node is not able to connect to the etcd service on the master, you could use the following curl command to test that connection from the node:

# curl -s -L http://localhost:2379/version
{"etcdserver":"2.3.7","etcdcluster":"2.3.0"}

Fixing the etcd database: It is possible to correct problems with your etcd database if information gets out of sync. There is are etcdctl update and etcdctl set commands for changing the contents of a key. However, if you are not careful, changing these values can cause more problems than they fix.

However, if your etcd database become completely unuseable, you can clear it and start over again. The way to do that is to run the etcd daemon with the -f option.

WARNING: Before you clear the etcd database, try using kubectl delete command to try to remove the offending services, pods, replicationControllers or minions. If you still feel you need to clear the database, keep in mind that if you do so, you need to recreate everything from scratch.

To clear the etcd database, type the following:

# etcd -f

How you stop and delete components in Kubernetes matters. Because Kubernetes is designed to get things to a particular state, simply deleting a container or a pod will often just cause another one to be started.

If you do delete components out of order, here’s what you can expect:

PODS can be stuck in the waiting state for some time period. Here are some possible causes:

You should be able to see the entire output from the container session. So, for example, if you opened a shell in a container, you will see all the commands you ran from that shell when you run docker logs.