Chapter 8. Deployments

OpenShift Container Platform deployments provide fine-grained management over common user applications. They are described using three separate API objects:

When you create a deployment configuration, a replication controller is created representing the deployment configuration’s pod template. If the deployment configuration changes, a new replication controller is created with the latest pod template, and a deployment process runs to scale down the old replication controller and scale up the new replication controller.

Instances of your application are automatically added and removed from both service load balancers and routers as they are created. As long as your application supports graceful shutdown when it receives the TERM signal, you can ensure that running user connections are given a chance to complete normally.

Features provided by the deployment system:

Deployment configurations are deploymentConfig OpenShift Container Platform API resources which can be managed with the oc command like any other resource. The following is an example of a deploymentConfig resource:

kind: "DeploymentConfig"
apiVersion: "v1"
metadata:
  name: "frontend"
spec:
  template: 1
    metadata:
      labels:
        name: "frontend"
    spec:
      containers:
        - name: "helloworld"
          image: "openshift/origin-ruby-sample"
          ports:
            - containerPort: 8080
              protocol: "TCP"
  replicas: 5 2
  triggers:
    - type: "ConfigChange" 3
    - type: "ImageChange" 4
      imageChangeParams:
        automatic: true
        containerNames:
          - "helloworld"
        from:
          kind: "ImageStreamTag"
          name: "origin-ruby-sample:latest"
  strategy: 5
    type: "Rolling"
  paused: false 6
  revisionHistoryLimit: 2 7
  minReadySeconds: 0 8

You can start a new deployment process manually using the web console, or from the CLI:

$ oc deploy --latest dc/<name>

To get basic information about all the available revisions of your application:

$ oc rollout history dc/<name>

This will show details about all recently created replication controllers for the provided deployment configuration, including any currently running deployment process.

You can view details specific to a revision by using the --revision flag:

$ oc rollout history dc/<name> --revision=1

For more detailed information about a deployment configuration and its latest revision:

$ oc describe dc <name>

To cancel a running or stuck deployment process:

$ oc deploy --cancel dc/<name>

If the current revision of your deployment configuration failed to deploy, you can restart the deployment process with:

$ oc deploy --retry dc/<name>

If the latest revision of it was deployed successfully, the command will display a message and the deployment process will not be retried.

Rollbacks revert an application back to a previous revision and can be performed using the REST API, the CLI, or the web console.

To rollback to the last successful deployed revision of your configuration:

$ oc rollout undo dc/<name>

The deployment configuration’s template will be reverted to match the deployment revision specified in the undo command, and a new replication controller will be started. If no revision is specified with --to-revision, then the last successfully deployed revision will be used.

Image change triggers on the deployment configuration are disabled as part of the rollback to prevent accidentally starting a new deployment process soon after the rollback is complete. To re-enable the image change triggers:

$ oc set triggers dc/<name> --auto

You can add a command to a container, which modifies the container’s startup behavior by overruling the image’s ENTRYPOINT. This is different from a lifecycle hook, which instead can be run once per deployment at a specified time.

Add the command parameters to the spec field of the deployment configuration. You can also add an args field, which modifies the command (or the ENTRYPOINT if command does not exist).

...
spec:
  containers:
    -
    name: <container_name>
    image: 'image'
    command:
      - '<command>'
    args:
      - '<argument_1>'
      - '<argument_2>'
      - '<argument_3>'
...

For example, to execute the java command with the -jar and /opt/app-root/springboots2idemo.jar arguments:

...
spec:
  containers:
    -
    name: example-spring-boot
    image: 'image'
    command:
      - java
    args:
      - '-jar'
      - /opt/app-root/springboots2idemo.jar
...

To stream the logs of the latest revision for a given deployment configuration:

$ oc logs -f dc/<name>

If the latest revision is running or failed, oc logs will return the logs of the process that is responsible for deploying your pods. If it is successful, oc logs will return the logs from a pod of your application.

You can also view logs from older failed deployment processes, if and only if these processes (old replication controllers and their deployer pods) exist and have not been pruned or deleted manually:

$ oc logs --version=1 dc/<name>

For more options on retrieving logs see:

$ oc logs --help

A deployment configuration can contain triggers, which drive the creation of new deployment processes in response to events inside the cluster.

triggers:
  - type: "ConfigChange"

triggers:
  - type: "ImageChange"
    imageChangeParams:
      automatic: true 1
      from:
        kind: "ImageStreamTag"
        name: "origin-ruby-sample:latest"
        namespace: "myproject"
      containerNames:
        - "helloworld"

$ oc set triggers dc/frontend --from-image=myproject/origin-ruby-sample:latest -c helloworld

$ oc set triggers --help

A deployment is completed by a pod that consumes resources (memory and CPU) on a node. By default, pods consume unbounded node resources. However, if a project specifies default container limits, then pods consume resources up to those limits.

You can also limit resource use by specifying resource limits as part of the deployment strategy. Deployment resources can be used with the Recreate, Rolling, or Custom deployment strategies.

In the following example, each of resources, cpu, and memory is optional:

type: "Recreate"
resources:
  limits:
    cpu: "100m" 1
    memory: "256Mi" 2

However, if a quota has been defined for your project, one of the following two items is required:

See Quotas and Limit Ranges to learn more about compute resources and the differences between requests and limits.

Otherwise, deploy pod creation will fail, citing a failure to satisfy quota.

In addition to rollbacks, you can exercise fine-grained control over the number of replicas from the web console, or by using the oc scale command. For example, the following command sets the replicas in the deployment configuration frontend to 3.

$ oc scale dc frontend --replicas=3

The number of replicas eventually propagates to the desired and current state of the deployment configured by the deployment configuration frontend.

You can use node selectors in conjunction with labeled nodes to control pod placement.

Cluster administrators can set the default node selector for your project in order to restrict pod placement to specific nodes. As an OpenShift Container Platform developer, you can set a node selector on a pod configuration to restrict nodes even further.

To add a node selector when creating a pod, edit the pod configuration, and add the nodeSelector value. This can be added to a single pod configuration, or in a pod template:

apiVersion: v1
kind: Pod
spec:
  nodeSelector:
    disktype: ssd
...

Pods created when the node selector is in place are assigned to nodes with the specified labels.

The labels specified here are used in conjunction with the labels added by a cluster administrator.

For example, if a project has the type=user-node and region=east labels added to a project by the cluster administrator, and you add the above disktype: ssd label to a pod, the pod will only ever be scheduled on nodes that have all three labels.

You can run a pod with a service account other than the default:

Add a secret to your deployment configuration so that it can access a private repository.

A deployment strategy determines the deployment process, and is defined by the deployment configuration. Each application has different requirements for availability (and other considerations) during deployments. OpenShift Container Platform provides strategies to support a variety of deployment scenarios.

A deployment strategy uses readiness checks to determine if a new pod is ready for use. If a readiness check fails, the deployment configuration will retry to run the pod until it times out. The default timeout is 10m, a value set in TimeoutSeconds in dc.spec.strategy.*params.

The Rolling strategy is the default strategy used if no strategy is specified on a deployment configuration.

A rolling deployment slowly replaces instances of the previous version of an application with instances of the new version of the application. A rolling deployment typically waits for new pods to become ready via a readiness check before scaling down the old components. If a significant issue occurs, the rolling deployment can be aborted.

strategy:
  type: Rolling
  rollingParams:
    updatePeriodSeconds: 1 1
    intervalSeconds: 1 2
    timeoutSeconds: 120 3
    maxSurge: "20%" 4
    maxUnavailable: "10%" 5
    pre: {} 6
    post: {}

The Recreate strategy has basic rollout behavior and supports lifecycle hooks for injecting code into the deployment process.

The following is an example of the Recreate strategy:

strategy:
  type: Recreate
  recreateParams: 1
    pre: {} 2
    mid: {}
    post: {}

The Recreate strategy will:

The Custom strategy allows you to provide your own deployment behavior.

The following is an example of the Custom strategy:

strategy:
  type: Custom
  customParams:
    image: organization/strategy
    command: [ "command", "arg1" ]
    environment:
      - name: ENV_1
        value: VALUE_1

In the above example, the organization/strategy container image provides the deployment behavior. The optional command array overrides any CMD directive specified in the image’s Dockerfile. The optional environment variables provided are added to the execution environment of the strategy process.

Additionally, OpenShift Container Platform provides the following environment variables to the deployment process:

The replica count of the new deployment will initially be zero. The responsibility of the strategy is to make the new deployment active using the logic that best serves the needs of the user.

Learn more about advanced deployment strategies.

Alternatively, use customParams to inject the custom deployment logic into the existing deployment strategies. Provide a custom shell script logic and call the openshift-deploy binary. Users do not have to supply their custom deployer container image, but the default OpenShift Container Platform deployer image will be used instead:

strategy:
  type: Rolling
  customParams:
    command:
    - /bin/sh
    - -c
    - |
      set -e
      openshift-deploy --until=50%
      echo Halfway there
      openshift-deploy
      echo Complete

This will result in following deployment:

Started deployment #2
--> Scaling up custom-deployment-2 from 0 to 2, scaling down custom-deployment-1 from 2 to 0 (keep 2 pods available, don't exceed 3 pods)
    Scaling custom-deployment-2 up to 1
--> Reached 50% (currently 50%)
Halfway there
--> Scaling up custom-deployment-2 from 1 to 2, scaling down custom-deployment-1 from 2 to 0 (keep 2 pods available, don't exceed 3 pods)
    Scaling custom-deployment-1 down to 1
    Scaling custom-deployment-2 up to 2
    Scaling custom-deployment-1 down to 0
--> Success
Complete

If the custom deployment strategy process requires access to the OpenShift Container Platform API or the Kubernetes API the container that executes the strategy can use the service account token available inside the container for authentication.

The Recreate and Rolling strategies support lifecycle hooks, which allow behavior to be injected into the deployment process at predefined points within the strategy:

The following is an example of a pre lifecycle hook:

pre:
  failurePolicy: Abort
  execNewPod: {} 1

Every hook has a failurePolicy, which defines the action the strategy should take when a hook failure is encountered:

Hooks have a type-specific field that describes how to execute the hook. Currently, pod-based hooks are the only supported hook type, specified by the execNewPod field.

kind: DeploymentConfig
apiVersion: v1
metadata:
  name: frontend
spec:
  template:
    metadata:
      labels:
        name: frontend
    spec:
      containers:
        - name: helloworld
          image: openshift/origin-ruby-sample
  replicas: 5
  selector:
    name: frontend
  strategy:
    type: Rolling
    rollingParams:
      pre:
        failurePolicy: Abort
        execNewPod:
          containerName: helloworld 1
          command: [ "/usr/bin/command", "arg1", "arg2" ] 2
          env: 3
            - name: CUSTOM_VAR1
              value: custom_value1
          volumes:
            - data 4

$ oc set deployment-hook dc/frontend --pre -c helloworld -e CUSTOM_VAR1=custom_value1 \
  -v data --failure-policy=abort -- /usr/bin/command arg1 arg2

Blue-green deployments involve running two versions of an application at the same time and moving production traffic from the old version to the new version. There are several ways to implement a blue-green deployment in OpenShift Container Platform.

A/B deployments generally imply running two (or more) versions of the application code or application configuration at the same time for testing or experimentation purposes.

The simplest form of an A/B deployment is to divide production traffic between two or more distinct shards — a single group of instances with homogeneous configuration and code.

More complicated A/B deployments may involve a specialized proxy or load balancer that assigns traffic to specific shards based on information about the user or application (all "test" users get sent to the B shard, but regular users get sent to the A shard).

A/B deployments can be considered similar to A/B testing, although an A/B deployment implies multiple versions of code and configuration, where as A/B testing often uses one code base with application specific checks.

In production environments, you can precisely control the distribution of traffic that lands on a particular shard. When dealing with large numbers of instances, you can use the relative scale of individual shards to implement percentage based traffic. That combines well with a proxy shard, which forwards or splits the traffic it receives to a separate service or application running elsewhere.

In the simplest configuration, the proxy would forward requests unchanged. In more complex setups, you can duplicate the incoming requests and send to both a separate cluster as well as to a local instance of the application, and compare the result. Other patterns include keeping the caches of a DR installation warm, or sampling incoming traffic for analysis purposes.

While an implementation is beyond the scope of this example, any TCP (or UDP) proxy could be run under the desired shard. Use the oc scale command to alter the relative number of instances serving requests under the proxy shard. For more complex traffic management, consider customizing the OpenShift Container Platform router with proportional balancing capabilities.

Applications that have new code and old code running at the same time must be careful to ensure that data written by the new code can be read by the old code. This is sometimes called schema evolution and is a complex problem.

For some applications, the period of time that old code and new code is running side by side is short, so bugs or some failed user transactions are acceptable. For others, the failure pattern may result in the entire application becoming non-functional.

One way to validate N-1 compatibility is to use an A/B deployment. Run the old code and new code at the same time in a controlled way in a test environment, and verify that traffic that flows to the new deployment does not cause failures in the old deployment.

OpenShift Container Platform and Kubernetes give application instances time to shut down before removing them from load balancing rotations. However, applications must ensure they cleanly terminate user connections as well before they exit.

On shutdown, OpenShift Container Platform will send a TERM signal to the processes in the container. Application code, on receiving SIGTERM, should stop accepting new connections. This will ensure that load balancers route traffic to other active instances. The application code should then wait until all open connections are closed (or gracefully terminate individual connections at the next opportunity) before exiting.

After the graceful termination period expires, a process that has not exited will be sent the KILL signal, which immediately ends the process. The terminationGracePeriodSeconds attribute of a pod or pod template controls the graceful termination period (default 30 seconds) and may be customized per application as necessary.

In the upstream Kubernetes project, a new first-class object type called deployments was added in version 1.2. This object type (referred to here as Kubernetes deployments for distinction) serves as a descendant of the deployment configuration object type.

Support for Kubernetes deployments is available as a Technology Preview feature.

Like deployment configurations, Kubernetes deployments describe the desired state of a particular component of an application as a pod template. Kubernetes deployments create replica sets (an iteration of replication controllers), which orchestrate pod lifecycles.

For example, this definition of a Kubernetes deployment creates a replica set to bring up one hello-openshift pod:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: hello-openshift
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: hello-openshift
    spec:
      containers:
      - name: hello-openshift
        image: openshift/hello-openshift:latest
        ports:
        - containerPort: 80

After saving the definition to a local file, you could then use it to create a Kubernetes deployment:

$ oc create -f hello-openshift-deployment.yaml

You can use the CLI to inspect and operate on Kubernetes deployments and replica sets like other object types, as described in Common Operations, like get and describe. For the object type, use deployments or deploy for Kubernetes deployments and replicasets or rs for replica sets.

See the Kubernetes documentation for more details about Deployments and Replica Sets, substituting oc for kubectl in CLI usage examples.

Because deployment configurations existed in OpenShift Container Platform prior to deployments being added in Kubernetes 1.2, the latter object type naturally diverges slightly from the former. The long-term goal in OpenShift Container Platform is to reach full feature parity in Kubernetes deployments and switch to using them as a single object type that provides fine-grained management over applications.

Kubernetes deployments are supported to ensure upstream projects and examples that use the new object type can run smoothly on OpenShift Container Platform. Given the current feature set of Kubernetes deployments, you may want to use them instead of deployment configurations in OpenShift Container Platform if you do not plan to use any of the following in particular:

The following sections go into more details on the differences between the two object types to further help you decide when you might want to use Kubernetes deployments over deployment configurations.

$ oc rollout pause deployments/<name>