Readiness and Liveness probe: initialDelaySeconds in RHOCP 4

Solution Verified - Updated -

Environment

  • RedHat Openshift Container Platform 4

Issue

  • What is liveness probe, readiness probe and initialDelaySeconds ?
  • At what time app will come into ready state if the below mentioned variables in deployment/pod yaml file:
LivenessProbe: initialDelaySeconds: 10s
ReadinessProbe: initialDelaySeconds: 15s
  • Will app become ready in 15s or (10+15)25s or some other time?
  • What will be the dependency?

Resolution

  • Pod will get into ready state after 15s.
  • Readiness and liveness probes can be used in parallel for the same container. Using both can ensure that traffic does not reach a container that is not ready for it, and that containers are restarted when they fail.
  • The kubelet uses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a container in such a state can help to make the application more available despite bugs.
  • The kubelet uses readiness probes to know when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.
  • The kubelet uses startup probes to know when a container application has started. If such a probe is configured, it disables liveness and readiness checks until it succeeds, making sure those probes don't interfere with the application startup. This can be used to adopt liveness checks on slow starting containers, avoiding them getting killed by the kubelet before they are up and running.
  • Please refer below link as it explains the same example: Define a TCP liveness probe

Root Cause

  • Because of below mentioned environment variable , pod is getting into ready state after 15s.
LivenessProbe: initialDelaySeconds: 10s
ReadinessProbe: initialDelaySeconds: 15s

Diagnostic Steps

  • Create a sample project for test:
$ oc new-project test
  • Create a new sample-app
$ oc new-app --name=livenesstest /path/to/source/code
  • Edit the pod file of newly created app and put variables in spec section under containers:
$ oc edit pod livenesstest

  • Prefer document Configure Liveness, Readiness and Startup Probes in order to know the exact format of yaml file.

  • In the configuration file, Pod has a single Container. The periodSeconds field specifies that the kubelet should perform a liveness probe every 20 seconds. The initialDelaySeconds field tells the kubelet that it should wait 10 seconds before performing the first probe. To perform a probe, the kubelet executes the command cat /tmp/healthy in the target container. If the command succeeds, it returns 0, and the kubelet considers the container to be alive and healthy. If the command returns a non-zero value, the kubelet kills the container and restarts it.

       readinessProbe:
         tcpSocket:
           port: 8080
         initialDelaySeconds: 15
         periodSeconds: 10
       livenessProbe:
         tcpSocket:
           port: 8080
         initialDelaySeconds: 10
         periodSeconds: 20
  • Output states that pod will be in ready state after 15s of readiness checks.
$ oc get pods 
    NAME               READY   STATUS    RESTARTS   AGE
    podname-xxx    1/1         Running           0               xxx

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments