Chapter 4. Developing Operators

4.1. Getting started with the Operator SDK

This guide outlines the basics of the Operator SDK and walks Operator authors with cluster administrator access to a Kubernetes-based cluster (such as OpenShift Container Platform) through an example of building a simple Go-based Memcached Operator and managing its lifecycle from installation to upgrade.

This is accomplished using two centerpieces of the Operator Framework: Operator SDK (the operator-sdk CLI tool and controller-runtime library API) and Operator Lifecycle Manager (OLM).

Note

OpenShift Container Platform 4.5 supports Operator SDK v0.17.2.

4.1.1. Architecture of the Operator SDK

The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way. Operators take advantage of Kubernetes extensibility to deliver the automation advantages of cloud services like provisioning, scaling, and backup and restore, while being able to run anywhere that Kubernetes can run.

Operators make it easy to manage complex, stateful applications on top of Kubernetes. However, writing an Operator today can be difficult because of challenges such as using low-level APIs, writing boilerplate, and a lack of modularity, which leads to duplication.

The Operator SDK is a framework designed to make writing Operators easier by providing:

  • High-level APIs and abstractions to write the operational logic more intuitively
  • Tools for scaffolding and code generation to quickly bootstrap a new project
  • Extensions to cover common Operator use cases

4.1.1.1. Workflow

The Operator SDK provides the following workflow to develop a new Operator:

  1. Create a new Operator project using the Operator SDK command line interface (CLI).
  2. Define new resource APIs by adding custom resource definitions (CRDs).
  3. Specify resources to watch using the Operator SDK API.
  4. Define the Operator reconciling logic in a designated handler and use the Operator SDK API to interact with resources.
  5. Use the Operator SDK CLI to build and generate the Operator deployment manifests.

Figure 4.1. Operator SDK workflow

osdk workflow

At a high level, an Operator using the Operator SDK processes events for watched resources in an Operator author-defined handler and takes actions to reconcile the state of the application.

4.1.1.2. Manager file

The main program for the Operator is the manager file at cmd/manager/main.go. The manager automatically registers the scheme for all custom resources (CRs) defined under pkg/apis/ and runs all controllers under pkg/controller/.

The manager can restrict the namespace that all controllers watch for resources:

mgr, err := manager.New(cfg, manager.Options{Namespace: namespace})

By default, this is the namespace that the Operator is running in. To watch all namespaces, you can leave the namespace option empty:

mgr, err := manager.New(cfg, manager.Options{Namespace: ""})

4.1.1.3. Prometheus Operator support

Prometheus is an open-source systems monitoring and alerting toolkit. The Prometheus Operator creates, configures, and manages Prometheus clusters running on Kubernetes-based clusters, such as OpenShift Container Platform.

Helper functions exist in the Operator SDK by default to automatically set up metrics in any generated Go-based Operator for use on clusters where the Prometheus Operator is deployed.

4.1.2. Installing the Operator SDK CLI

The Operator SDK has a CLI tool that assists developers in creating, building, and deploying a new Operator project. You can install the SDK CLI on your workstation so you are prepared to start authoring your own Operators.

Note

This guide uses minikube v0.25.0+ as the local Kubernetes cluster and Quay.io for the public registry.

4.1.2.1. Installing from GitHub release

You can download and install a pre-built release binary of the Operator SDK CLI from the project on GitHub.

Prerequisites

  • Go v1.13+
  • docker v17.03+, podman v1.2.0+, or buildah v1.7+
  • OpenShift CLI (oc) v4.5+ installed
  • Access to a cluster based on Kubernetes v1.12.0+
  • Access to a container registry

Procedure

  1. Set the release version variable:

    $ RELEASE_VERSION=v0.17.2
  2. Download the release binary.

    • For Linux:

      $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
    • For macOS:

      $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin
  3. Verify the downloaded release binary.

    1. Download the provided .asc file.

      • For Linux:

        $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu.asc
      • For macOS:

        $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin.asc
    2. Place the binary and corresponding .asc file into the same directory and run the following command to verify the binary:

      • For Linux:

        $ gpg --verify operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu.asc
      • For macOS:

        $ gpg --verify operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin.asc

      If you do not have the public key of the maintainer on your workstation, you will get the following error:

      Example output with error

      $ gpg: assuming signed data in 'operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin'
      $ gpg: Signature made Fri Apr  5 20:03:22 2019 CEST
      $ gpg:                using RSA key <key_id> 1
      $ gpg: Can't check signature: No public key

      1
      RSA key string.

      To download the key, run the following command, replacing <key_id> with the RSA key string provided in the output of the previous command:

      $ gpg [--keyserver keys.gnupg.net] --recv-key "<key_id>" 1
      1
      If you do not have a key server configured, specify one with the --keyserver option.
  4. Install the release binary in your PATH:

    • For Linux:

      $ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
      $ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu /usr/local/bin/operator-sdk
      $ rm operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
    • For macOS:

      $ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin
      $ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin /usr/local/bin/operator-sdk
      $ rm operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin
  5. Verify that the CLI tool was installed correctly:

    $ operator-sdk version

4.1.2.2. Installing from Homebrew

You can install the SDK CLI using Homebrew.

Prerequisites

  • Homebrew
  • docker v17.03+, podman v1.2.0+, or buildah v1.7+
  • OpenShift CLI (oc) v4.5+ installed
  • Access to a cluster based on Kubernetes v1.12.0+
  • Access to a container registry

Procedure

  1. Install the SDK CLI using the brew command:

    $ brew install operator-sdk
  2. Verify that the CLI tool was installed correctly:

    $ operator-sdk version

4.1.2.3. Compiling and installing from source

You can obtain the Operator SDK source code to compile and install the SDK CLI.

Prerequisites

  • Git
  • Go v1.13+
  • docker v17.03+, podman v1.2.0+, or buildah v1.7+
  • OpenShift CLI (oc) v4.5+ installed
  • Access to a cluster based on Kubernetes v1.12.0+
  • Access to a container registry

Procedure

  1. Clone the operator-sdk repository:

    $ mkdir -p $GOPATH/src/github.com/operator-framework
    $ cd $GOPATH/src/github.com/operator-framework
    $ git clone https://github.com/operator-framework/operator-sdk
    $ cd operator-sdk
  2. Check out the desired release branch:

    $ git checkout master
  3. Compile and install the SDK CLI:

    $ make dep
    $ make install

    This installs the CLI binary operator-sdk at $GOPATH/bin.

  4. Verify that the CLI tool was installed correctly:

    $ operator-sdk version

4.1.3. Building a Go-based Operator using the Operator SDK

The Operator SDK makes it easier to build Kubernetes native applications, a process that can require deep, application-specific operational knowledge. The SDK not only lowers that barrier, but it also helps reduce the amount of boilerplate code needed for many common management capabilities, such as metering or monitoring.

This procedure walks through an example of building a simple Memcached Operator using tools and libraries provided by the SDK.

Prerequisites

  • Operator SDK CLI installed on the development workstation
  • Operator Lifecycle Manager (OLM) installed on a Kubernetes-based cluster (v1.8 or above to support the apps/v1beta2 API group), for example OpenShift Container Platform 4.5
  • Access to the cluster using an account with cluster-admin permissions
  • OpenShift CLI (oc) v4.5+ installed

Procedure

  1. Create a new project.

    Use the CLI to create a new memcached-operator project:

    $ mkdir -p $GOPATH/src/github.com/example-inc/
    $ cd $GOPATH/src/github.com/example-inc/
    $ operator-sdk new memcached-operator
    $ cd memcached-operator
  2. Add a new custom resource definition (CRD).

    1. Use the CLI to add a new CRD API called Memcached, with APIVersion set to cache.example.com/v1apha1 and Kind set to Memcached:

      $ operator-sdk add api \
          --api-version=cache.example.com/v1alpha1 \
          --kind=Memcached

      This scaffolds the Memcached resource API under pkg/apis/cache/v1alpha1/.

    2. Modify the spec and status of the Memcached custom resource (CR) at the pkg/apis/cache/v1alpha1/memcached_types.go file:

      type MemcachedSpec struct {
      	// Size is the size of the memcached deployment
      	Size int32 `json:"size"`
      }
      type MemcachedStatus struct {
      	// Nodes are the names of the memcached pods
      	Nodes []string `json:"nodes"`
      }
    3. After modifying the *_types.go file, always run the following command to update the generated code for that resource type:

      $ operator-sdk generate k8s
  3. Optional: Add custom validation to your CRD.

    OpenAPI v3.0 schemas are added to CRD manifests in the spec.validation block when the manifests are generated. This validation block allows Kubernetes to validate the properties in a Memcached CR when it is created or updated.

    Additionally, a pkg/apis/<group>/<version>/zz_generated.openapi.go file is generated. This file contains the Go representation of this validation block if the +k8s:openapi-gen=true annotation is present above the Kind type declaration, which is present by default. This auto-generated code is the OpenAPI model of your Go Kind type, from which you can create a full OpenAPI Specification and generate a client.

    As an Operator author, you can use Kubebuilder markers (annotations) to configure custom validations for your API. These markers must always have a +kubebuilder:validation prefix. For example, adding an enum-type specification can be done by adding the following marker:

    // +kubebuilder:validation:Enum=Lion;Wolf;Dragon
    type Alias string

    Usage of markers in API code is discussed in the Kubebuilder Generating CRDs and Markers for Config/Code Generation documentation. A full list of OpenAPIv3 validation markers is also available in the Kubebuilder CRD Validation documentation.

    If you add any custom validations, run the following command to update the OpenAPI validation section in the deploy/crds/cache.example.com_memcacheds_crd.yaml file for the CRD:

    $ operator-sdk generate crds

    Example generated YAML

    spec:
      validation:
        openAPIV3Schema:
          properties:
            spec:
              properties:
                size:
                  format: int32
                  type: integer

  4. Add a new controller.

    1. Add a new controller to the project to watch and reconcile the Memcached resource:

      $ operator-sdk add controller \
          --api-version=cache.example.com/v1alpha1 \
          --kind=Memcached

      This scaffolds a new controller implementation under pkg/controller/memcached/.

    2. For this example, replace the generated controller file pkg/controller/memcached/memcached_controller.go with the example implementation.

      The example controller executes the following reconciliation logic for each Memcached resource:

      • Create a Memcached deployment if it does not exist.
      • Ensure that the Deployment size is the same as specified by the Memcached CR spec.
      • Update the Memcached resource status with the names of the Memcached pods.

      The next two sub-steps inspect how the controller watches resources and how the reconcile loop is triggered. You can skip these steps to go directly to building and running the Operator.

    3. Inspect the controller implementation at the pkg/controller/memcached/memcached_controller.go file to see how the controller watches resources.

      The first watch is for the Memcached type as the primary resource. For each add, update, or delete event, the reconcile loop is sent a reconcile Request (a <namespace>:<name> key) for that Memcached object:

      err := c.Watch(
        &source.Kind{Type: &cachev1alpha1.Memcached{}}, &handler.EnqueueRequestForObject{})

      The next watch is for Deployment objects, but the event handler maps each event to a reconcile Request for the owner of the deployment. In this case, this is the Memcached object for which the deployment was created. This allows the controller to watch deployments as a secondary resource:

      err := c.Watch(&source.Kind{Type: &appsv1.Deployment{}}, &handler.EnqueueRequestForOwner{
      		IsController: true,
      		OwnerType:    &cachev1alpha1.Memcached{},
      	})
    4. Every controller has a Reconciler object with a Reconcile() method that implements the reconcile loop. The reconcile loop is passed the Request argument which is a <namespace>:<name> key used to lookup the primary resource object, Memcached, from the cache:

      func (r *ReconcileMemcached) Reconcile(request reconcile.Request) (reconcile.Result, error) {
        // Lookup the Memcached instance for this reconcile request
        memcached := &cachev1alpha1.Memcached{}
        err := r.client.Get(context.TODO(), request.NamespacedName, memcached)
        ...
      }

      Based on the return value of the Reconcile() function, the reconcile Request might be requeued, and the loop might be triggered again:

      // Reconcile successful - don't requeue
      return reconcile.Result{}, nil
      // Reconcile failed due to error - requeue
      return reconcile.Result{}, err
      // Requeue for any reason other than error
      return reconcile.Result{Requeue: true}, nil
  5. Build and run the Operator.

    1. Before running the Operator, the CRD must be registered with the Kubernetes API server:

      $ oc create \
          -f deploy/crds/cache_v1alpha1_memcached_crd.yaml
    2. After registering the CRD, there are two options for running the Operator:

      • As a Deployment inside a Kubernetes cluster
      • As Go program outside a cluster

      Choose one of the following methods.

      1. Option A: Running as a deployment inside the cluster.

        1. Build the memcached-operator image and push it to a registry:

          $ operator-sdk build quay.io/example/memcached-operator:v0.0.1
        2. The deployment manifest is generated at deploy/operator.yaml. Update the deployment image as follows since the default is just a placeholder:

          $ sed -i 's|REPLACE_IMAGE|quay.io/example/memcached-operator:v0.0.1|g' deploy/operator.yaml
        3. Ensure you have an account on Quay.io for the next step, or substitute your preferred container registry. On the registry, create a new public image repository named memcached-operator.
        4. Push the image to the registry:

          $ podman push quay.io/example/memcached-operator:v0.0.1
        5. Set up RBAC and create the memcached-operator manifests:

          $ oc create -f deploy/role.yaml
          $ oc create -f deploy/role_binding.yaml
          $ oc create -f deploy/service_account.yaml
          $ oc create -f deploy/operator.yaml
        6. Verify that the memcached-operator deploy is up and running:

          $ oc get deployment

          Example output

          NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
          memcached-operator       1         1         1            1           1m

      2. Option B: Running locally outside the cluster.

        This method is preferred during development cycle to deploy and test faster.

        Run the Operator locally with the default Kubernetes configuration file present at $HOME/.kube/config:

        $ operator-sdk run --local --namespace=default

        You can use a specific kubeconfig using the flag --kubeconfig=<path/to/kubeconfig>.

  6. Verify that the Operator can deploy a Memcached application by creating a Memcached CR.

    1. Create the example Memcached CR that was generated at deploy/crds/cache_v1alpha1_memcached_cr.yaml.
    2. View the file:

      $ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml

      Example output

      apiVersion: "cache.example.com/v1alpha1"
      kind: "Memcached"
      metadata:
        name: "example-memcached"
      spec:
        size: 3

    3. Create the object:

      $ oc apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml
    4. Ensure that memcached-operator creates the deployment for the CR:

      $ oc get deployment

      Example output

      NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      memcached-operator       1         1         1            1           2m
      example-memcached        3         3         3            3           1m

    5. Check the pods and CR to confirm the CR status is updated with the pod names:

      $ oc get pods

      Example output

      NAME                                  READY     STATUS    RESTARTS   AGE
      example-memcached-6fd7c98d8-7dqdr     1/1       Running   0          1m
      example-memcached-6fd7c98d8-g5k7v     1/1       Running   0          1m
      example-memcached-6fd7c98d8-m7vn7     1/1       Running   0          1m
      memcached-operator-7cc7cfdf86-vvjqk   1/1       Running   0          2m

      $ oc get memcached/example-memcached -o yaml

      Example output

      apiVersion: cache.example.com/v1alpha1
      kind: Memcached
      metadata:
        clusterName: ""
        creationTimestamp: 2018-03-31T22:51:08Z
        generation: 0
        name: example-memcached
        namespace: default
        resourceVersion: "245453"
        selfLink: /apis/cache.example.com/v1alpha1/namespaces/default/memcacheds/example-memcached
        uid: 0026cc97-3536-11e8-bd83-0800274106a1
      spec:
        size: 3
      status:
        nodes:
        - example-memcached-6fd7c98d8-7dqdr
        - example-memcached-6fd7c98d8-g5k7v
        - example-memcached-6fd7c98d8-m7vn7

  7. Verify that the Operator can manage a deployed Memcached application by updating the size of the deployment.

    1. Change the spec.size field in the memcached CR from 3 to 4:

      $ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml

      Example output

      apiVersion: "cache.example.com/v1alpha1"
      kind: "Memcached"
      metadata:
        name: "example-memcached"
      spec:
        size: 4

    2. Apply the change:

      $ oc apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml
    3. Confirm that the Operator changes the deployment size:

      $ oc get deployment

      Example output

      NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      example-memcached    4         4         4            4           5m

  8. Clean up the resources:

    $ oc delete -f deploy/crds/cache_v1alpha1_memcached_cr.yaml
    $ oc delete -f deploy/crds/cache_v1alpha1_memcached_crd.yaml
    $ oc delete -f deploy/operator.yaml
    $ oc delete -f deploy/role.yaml
    $ oc delete -f deploy/role_binding.yaml
    $ oc delete -f deploy/service_account.yaml

Additional resources

4.1.4. Managing a Go-based Operator using Operator Lifecycle Manager

The previous section has covered manually running an Operator. The next sections explore using Operator Lifecycle Manager (OLM), which is what enables a more robust deployment model for Operators being run in production environments.

OLM helps you to install, update, and generally manage the lifecycle of all of the Operators and their associated services on a Kubernetes cluster. It runs as an Kubernetes extension and lets you use oc for all the lifecycle management functions without any additional tools.

Prerequisites

  • OLM installed on a Kubernetes-based cluster (v1.8 or above to support the apps/v1beta2 API group), for example OpenShift Container Platform 4.5
  • Memcached Operator built

Procedure

  1. Generate an Operator manifest.

    An Operator manifest describes how to display, create, and manage the application, in this case Memcached, as a whole. It is defined by a ClusterServiceVersion (CSV) object and is required for OLM to function.

    From the memcached-operator/ directory that was created when you built the Memcached Operator, generate the CSV manifest:

    $ operator-sdk generate csv --csv-version 0.0.1
    Note

    See Building a CSV for the Operator Framework for more information on manually defining a manifest file.

  2. Create an Operator group that specifies the namespaces that the Operator will target. Create the following Operator group in the namespace where you will create the CSV. In this example, the default namespace is used:

    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: memcached-operator-group
      namespace: default
    spec:
      targetNamespaces:
      - default
  3. Deploy the Operator. Use the files that were generated into the deploy/ directory by the Operator SDK when you built the Memcached Operator.

    1. Edit the generated CSV manifest file by adding displayName fields for each custom resource definition (CRD) kind in the spec.customresourcedefinitions.owned section:

      deploy/olm-catalog/memcached-operator/0.0.1/memcached-operator.v0.0.1.clusterserviceversion.yaml file

      ...
      spec:
        customresourcedefinitions:
          owned:
          - kind: Memcached
            name: memcacheds.cache.example.com
            version: v1alpha1
            description: Memcached is the Schema for the memcacheds API
            displayName: Memcached 1
      ...

      1
      Specify a display name for the CRD.
    2. Apply the CSV manifest to the specified namespace in the cluster:

      $ oc apply -f deploy/olm-catalog/memcached-operator/0.0.1/memcached-operator.v0.0.1.clusterserviceversion.yaml

      When you apply this manifest, the cluster does not immediately update because it does not yet meet the requirements specified in the manifest.

    3. Create the role, role binding, and service account to grant resource permissions to the Operator, and the custom resource definition (CRD) to create the Memcached custom resource that the Operator manages:

      $ oc create -f deploy/crds/cache.example.com_memcacheds_crd.yaml
      $ oc create -f deploy/service_account.yaml
      $ oc create -f deploy/role.yaml
      $ oc create -f deploy/role_binding.yaml

      Because OLM creates Operators in a particular namespace when a manifest is applied, administrators can leverage the native Kubernetes RBAC permission model to restrict which users are allowed to install Operators.

  4. Create an application instance.

    The Memcached Operator is now running in the default namespace. Users interact with Operators via instances of custom resources; in this case, the resource has the kind Memcached. Native Kubernetes RBAC also applies to custom resources, providing administrators control over who can interact with each Operator.

    Creating instances of Memcached objects in this namespace will now trigger the Memcached Operator to instantiate pods running the memcached server that are managed by the Operator. The more custom resources you create, the more unique Memcached application instances are managed by the Memcached Operator running in this namespace.

    $ cat <<EOF | oc apply -f -
    apiVersion: "cache.example.com/v1alpha1"
    kind: "Memcached"
    metadata:
      name: "memcached-for-wordpress"
    spec:
      size: 1
    EOF
    $ cat <<EOF | oc apply -f -
    apiVersion: "cache.example.com/v1alpha1"
    kind: "Memcached"
    metadata:
      name: "memcached-for-drupal"
    spec:
      size: 1
    EOF
    $ oc get Memcached

    Example output

    NAME                      AGE
    memcached-for-drupal      22s
    memcached-for-wordpress   27s

    $ oc get pods

    Example output

    NAME                                       READY     STATUS    RESTARTS   AGE
    memcached-app-operator-66b5777b79-pnsfj    1/1       Running   0          14m
    memcached-for-drupal-5476487c46-qbd66      1/1       Running   0          3s
    memcached-for-wordpress-65b75fd8c9-7b9x7   1/1       Running   0          8s

4.1.5. Additional resources

4.2. Creating Ansible-based Operators

This guide outlines Ansible support in the Operator SDK and walks Operator authors through examples building and running Ansible-based Operators with the operator-sdk CLI tool that use Ansible playbooks and modules.

4.2.1. Ansible support in the Operator SDK

The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way. This framework includes the Operator SDK, which assists developers in bootstrapping and building an Operator based on their expertise without requiring knowledge of Kubernetes API complexities.

One of the Operator SDK options for generating an Operator project includes leveraging existing Ansible playbooks and modules to deploy Kubernetes resources as a unified application, without having to write any Go code.

4.2.1.1. Custom resource files

Operators use the Kubernetes extension mechanism, custom resource definitions (CRDs), so your custom resource (CR) looks and acts just like the built-in, native Kubernetes objects.

The CR file format is a Kubernetes resource file. The object has mandatory and optional fields:

Table 4.1. Custom resource fields

FieldDescription

apiVersion

Version of the CR to be created.

kind

Kind of the CR to be created.

metadata

Kubernetes-specific metadata to be created.

spec (optional)

Key-value list of variables which are passed to Ansible. This field is empty by default.

status

Summarizes the current state of the object. For Ansible-based Operators, the status subresource is enabled for CRDs and managed by the operator_sdk.util.k8s_status Ansible module by default, which includes condition information to the CR status.

annotations

Kubernetes-specific annotations to be appended to the CR.

The following list of CR annotations modify the behavior of the Operator:

Table 4.2. Ansible-based Operator annotations

AnnotationDescription

ansible.operator-sdk/reconcile-period

Specifies the reconciliation interval for the CR. This value is parsed using the standard Golang package time. Specifically, ParseDuration is used which applies the default suffix of s, giving the value in seconds.

Example Ansible-based Operator annotation

apiVersion: "test1.example.com/v1alpha1"
kind: "Test1"
metadata:
  name: "example"
annotations:
  ansible.operator-sdk/reconcile-period: "30s"

4.2.1.2. watches.yaml file

A group/version/kind (GVK) is a unique identifier for a Kubernetes API. The watches.yaml file contains a list of mappings from custom resources (CRs), identified by its GVK, to an Ansible role or playbook. The Operator expects this mapping file in a predefined location at /opt/ansible/watches.yaml.

Table 4.3. watches.yaml file mappings

FieldDescription

group

Group of CR to watch.

version

Version of CR to watch.

kind

Kind of CR to watch

role (default)

Path to the Ansible role added to the container. For example, if your roles directory is at /opt/ansible/roles/ and your role is named busybox, this value would be /opt/ansible/roles/busybox. This field is mutually exclusive with the playbook field.

playbook

Path to the Ansible playbook added to the container. This playbook is expected to be a way to call roles. This field is mutually exclusive with the role field.

reconcilePeriod (optional)

The reconciliation interval, how often the role or playbook is run, for a given CR.

manageStatus (optional)

When set to true (default), the Operator manages the status of the CR generically. When set to false, the status of the CR is managed elsewhere, by the specified role or playbook or in a separate controller.

Example watches.yaml file

- version: v1alpha1 1
  group: test1.example.com
  kind: Test1
  role: /opt/ansible/roles/Test1

- version: v1alpha1 2
  group: test2.example.com
  kind: Test2
  playbook: /opt/ansible/playbook.yml

- version: v1alpha1 3
  group: test3.example.com
  kind: Test3
  playbook: /opt/ansible/test3.yml
  reconcilePeriod: 0
  manageStatus: false

1
Simple example mapping Test1 to the test1 role.
2
Simple example mapping Test2 to a playbook.
3
More complex example for the Test3 kind. Disables re-queuing and managing the CR status in the playbook.
4.2.1.2.1. Advanced options

Advanced features can be enabled by adding them to your watches.yaml file per GVK. They can go below the group, version, kind and playbook or role fields.

Some features can be overridden per resource using an annotation on that CR. The options that can be overridden have the annotation specified below.

Table 4.4. Advanced watches.yaml file options

FeatureYAML keyDescriptionAnnotation for overrideDefault value

Reconcile period

reconcilePeriod

Time between reconcile runs for a particular CR.

ansbile.operator-sdk/reconcile-period

1m

Manage status

manageStatus

Allows the Operator to manage the conditions section of each CR status section.

 

true

Watch dependent resources

watchDependentResources

Allows the Operator to dynamically watch resources that are created by Ansible.

 

true

Watch cluster-scoped resources

watchClusterScopedResources

Allows the Operator to watch cluster-scoped resources that are created by Ansible.

 

false

Max runner artifacts

maxRunnerArtifacts

Manages the number of artifact directories that Ansible Runner keeps in the Operator container for each individual resource.

ansible.operator-sdk/max-runner-artifacts

20

Example watches.yml file with advanced options

- version: v1alpha1
  group: app.example.com
  kind: AppService
  playbook: /opt/ansible/playbook.yml
  maxRunnerArtifacts: 30
  reconcilePeriod: 5s
  manageStatus: False
  watchDependentResources: False

4.2.1.3. Extra variables sent to Ansible

Extra variables can be sent to Ansible, which are then managed by the Operator. The spec section of the custom resource (CR) passes along the key-value pairs as extra variables. This is equivalent to extra variables passed in to the ansible-playbook command.

The Operator also passes along additional variables under the meta field for the name of the CR and the namespace of the CR.

For the following CR example:

apiVersion: "app.example.com/v1alpha1"
kind: "Database"
metadata:
  name: "example"
spec:
  message:"Hello world 2"
  newParameter: "newParam"

The structure passed to Ansible as extra variables is:

{ "meta": {
        "name": "<cr_name>",
        "namespace": "<cr_namespace>",
  },
  "message": "Hello world 2",
  "new_parameter": "newParam",
  "_app_example_com_database": {
     <full_crd>
   },
}

The message and newParameter fields are set in the top level as extra variables, and meta provides the relevant metadata for the CR as defined in the Operator. The meta fields can be accessed using dot notation in Ansible, for example:

- debug:
    msg: "name: {{ meta.name }}, {{ meta.namespace }}"

4.2.1.4. Ansible Runner directory

Ansible Runner keeps information about Ansible runs in the container. This is located at /tmp/ansible-operator/runner/<group>/<version>/<kind>/<namespace>/<name>.

Additional resources

4.2.2. Installing the Operator SDK CLI

The Operator SDK has a CLI tool that assists developers in creating, building, and deploying a new Operator project. You can install the SDK CLI on your workstation so you are prepared to start authoring your own Operators.

Note

This guide uses minikube v0.25.0+ as the local Kubernetes cluster and Quay.io for the public registry.

4.2.2.1. Installing from GitHub release

You can download and install a pre-built release binary of the Operator SDK CLI from the project on GitHub.

Prerequisites

  • Go v1.13+
  • docker v17.03+, podman v1.2.0+, or buildah v1.7+
  • OpenShift CLI (oc) v4.5+ installed
  • Access to a cluster based on Kubernetes v1.12.0+
  • Access to a container registry

Procedure

  1. Set the release version variable:

    $ RELEASE_VERSION=v0.17.2
  2. Download the release binary.

    • For Linux:

      $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
    • For macOS:

      $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin
  3. Verify the downloaded release binary.

    1. Download the provided .asc file.

      • For Linux:

        $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu.asc
      • For macOS:

        $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin.asc
    2. Place the binary and corresponding .asc file into the same directory and run the following command to verify the binary:

      • For Linux:

        $ gpg --verify operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu.asc
      • For macOS:

        $ gpg --verify operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin.asc

      If you do not have the public key of the maintainer on your workstation, you will get the following error:

      Example output with error

      $ gpg: assuming signed data in 'operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin'
      $ gpg: Signature made Fri Apr  5 20:03:22 2019 CEST
      $ gpg:                using RSA key <key_id> 1
      $ gpg: Can't check signature: No public key

      1
      RSA key string.

      To download the key, run the following command, replacing <key_id> with the RSA key string provided in the output of the previous command:

      $ gpg [--keyserver keys.gnupg.net] --recv-key "<key_id>" 1
      1
      If you do not have a key server configured, specify one with the --keyserver option.
  4. Install the release binary in your PATH:

    • For Linux:

      $ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
      $ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu /usr/local/bin/operator-sdk
      $ rm operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
    • For macOS:

      $ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin
      $ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin /usr/local/bin/operator-sdk
      $ rm operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin
  5. Verify that the CLI tool was installed correctly:

    $ operator-sdk version

4.2.2.2. Installing from Homebrew

You can install the SDK CLI using Homebrew.

Prerequisites

  • Homebrew
  • docker v17.03+, podman v1.2.0+, or buildah v1.7+
  • OpenShift CLI (oc) v4.5+ installed
  • Access to a cluster based on Kubernetes v1.12.0+
  • Access to a container registry

Procedure

  1. Install the SDK CLI using the brew command:

    $ brew install operator-sdk
  2. Verify that the CLI tool was installed correctly:

    $ operator-sdk version

4.2.2.3. Compiling and installing from source

You can obtain the Operator SDK source code to compile and install the SDK CLI.

Prerequisites

  • Git
  • Go v1.13+
  • docker v17.03+, podman v1.2.0+, or buildah v1.7+
  • OpenShift CLI (oc) v4.5+ installed
  • Access to a cluster based on Kubernetes v1.12.0+
  • Access to a container registry

Procedure

  1. Clone the operator-sdk repository:

    $ mkdir -p $GOPATH/src/github.com/operator-framework
    $ cd $GOPATH/src/github.com/operator-framework
    $ git clone https://github.com/operator-framework/operator-sdk
    $ cd operator-sdk
  2. Check out the desired release branch:

    $ git checkout master
  3. Compile and install the SDK CLI:

    $ make dep
    $ make install

    This installs the CLI binary operator-sdk at $GOPATH/bin.

  4. Verify that the CLI tool was installed correctly:

    $ operator-sdk version

4.2.3. Building an Ansible-based Operator using the Operator SDK

This procedure walks through an example of building a simple Memcached Operator powered by Ansible playbooks and modules using tools and libraries provided by the Operator SDK.

Prerequisites

  • Operator SDK CLI installed on the development workstation
  • Access to a Kubernetes-based cluster v1.11.3+ (for example OpenShift Container Platform 4.5) using an account with cluster-admin permissions
  • OpenShift CLI (oc) v4.5+ installed
  • ansible v2.9.0+
  • ansible-runner v1.1.0+
  • ansible-runner-http v1.0.0+

Procedure

  1. Create a new Operator project. A namespace-scoped Operator watches and manages resources in a single namespace. Namespace-scoped Operators are preferred because of their flexibility. They enable decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions.

    To create a new Ansible-based, namespace-scoped memcached-operator project and change to the new directory, use the following commands:

    $ operator-sdk new memcached-operator \
        --api-version=cache.example.com/v1alpha1 \
        --kind=Memcached \
        --type=ansible
    $ cd memcached-operator

    This creates the memcached-operator project specifically for watching the Memcached resource with API version example.com/v1apha1 and kind Memcached.

  2. Customize the Operator logic.

    For this example, the memcached-operator executes the following reconciliation logic for each Memcached custom resource (CR):

    • Create a memcached deployment if it does not exist.
    • Ensure that the deployment size is the same as specified by the Memcached CR.

    By default, the memcached-operator watches Memcached resource events as shown in the watches.yaml file and executes the Ansible role Memcached:

    - version: v1alpha1
      group: cache.example.com
      kind: Memcached

    You can optionally customize the following logic in the watches.yaml file:

    1. Specifying a role option configures the Operator to use this specified path when launching ansible-runner with an Ansible role. By default, the operator-sdk new command fills in an absolute path to where your role should go:

      - version: v1alpha1
        group: cache.example.com
        kind: Memcached
        role: /opt/ansible/roles/memcached
    2. Specifying a playbook option in the watches.yaml file configures the Operator to use this specified path when launching ansible-runner with an Ansible playbook:

      - version: v1alpha1
        group: cache.example.com
        kind: Memcached
        playbook: /opt/ansible/playbook.yaml
  3. Build the Memcached Ansible role.

    Modify the generated Ansible role under the roles/memcached/ directory. This Ansible role controls the logic that is executed when a resource is modified.

    1. Define the Memcached spec.

      Defining the spec for an Ansible-based Operator can be done entirely in Ansible. The Ansible Operator passes all key-value pairs listed in the CR spec field along to Ansible as variables. The names of all variables in the spec field are converted to snake case (lowercase with an underscore) by the Operator before running Ansible. For example, serviceAccount in the spec becomes service_account in Ansible.

      Tip

      You should perform some type validation in Ansible on the variables to ensure that your application is receiving expected input.

      In case the user does not set the spec field, set a default by modifying the roles/memcached/defaults/main.yml file:

      size: 1
    2. Define the Memcached deployment.

      With the Memcached spec now defined, you can define what Ansible is actually executed on resource changes. Because this is an Ansible role, the default behavior executes the tasks in the roles/memcached/tasks/main.yml file.

      The goal is for Ansible to create a deployment if it does not exist, which runs the memcached:1.4.36-alpine image. Ansible 2.7+ supports the k8s Ansible module, which this example leverages to control the deployment definition.

      Modify the roles/memcached/tasks/main.yml to match the following:

      - name: start memcached
        k8s:
          definition:
            kind: Deployment
            apiVersion: apps/v1
            metadata:
              name: '{{ meta.name }}-memcached'
              namespace: '{{ meta.namespace }}'
            spec:
              replicas: "{{size}}"
              selector:
                matchLabels:
                  app: memcached
              template:
                metadata:
                  labels:
                    app: memcached
                spec:
                  containers:
                  - name: memcached
                    command:
                    - memcached
                    - -m=64
                    - -o
                    - modern
                    - -v
                    image: "docker.io/memcached:1.4.36-alpine"
                    ports:
                      - containerPort: 11211
      Note

      This example used the size variable to control the number of replicas of the Memcached deployment. This example sets the default to 1, but any user can create a CR that overwrites the default.

  4. Deploy the CRD.

    Before running the Operator, Kubernetes needs to know about the new custom resource definition (CRD) that the Operator will be watching. Deploy the Memcached CRD:

    $ oc create -f deploy/crds/cache.example.com_memcacheds_crd.yaml
  5. Build and run the Operator.

    There are two ways to build and run the Operator:

    • As a pod inside a Kubernetes cluster.
    • As a Go program outside the cluster using the operator-sdk up command.

    Choose one of the following methods:

    1. Run as a pod inside a Kubernetes cluster. This is the preferred method for production use.

      1. Build the memcached-operator image and push it to a registry:

        $ operator-sdk build quay.io/example/memcached-operator:v0.0.1
        $ podman push quay.io/example/memcached-operator:v0.0.1
      2. Deployment manifests are generated in the deploy/operator.yaml file. The deployment image in this file needs to be modified from the placeholder REPLACE_IMAGE to the previous built image. To do this, run:

        $ sed -i 's|REPLACE_IMAGE|quay.io/example/memcached-operator:v0.0.1|g' deploy/operator.yaml
      3. Deploy the memcached-operator manifests:

        $ oc create -f deploy/service_account.yaml
        $ oc create -f deploy/role.yaml
        $ oc create -f deploy/role_binding.yaml
        $ oc create -f deploy/operator.yaml
      4. Verify that the memcached-operator deployment is up and running:

        $ oc get deployment
        NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
        memcached-operator       1         1         1            1           1m
    2. Run outside the cluster. This method is preferred during the development cycle to speed up deployment and testing.

      Ensure that Ansible Runner and Ansible Runner HTTP Plug-in are installed or else you will see unexpected errors from Ansible Runner when a CR is created.

      It is also important that the role path referenced in the watches.yaml file exists on your machine. Because normally a container is used where the role is put on disk, the role must be manually copied to the configured Ansible roles path (for example /etc/ansible/roles).

      1. To run the Operator locally with the default Kubernetes configuration file present at $HOME/.kube/config:

        $ operator-sdk run --local

        To run the Operator locally with a provided Kubernetes configuration file:

        $ operator-sdk run --local --kubeconfig=config
  6. Create a Memcached CR.

    1. Modify the deploy/crds/cache_v1alpha1_memcached_cr.yaml file as shown and create a Memcached CR:

      $ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml

      Example output

      apiVersion: "cache.example.com/v1alpha1"
      kind: "Memcached"
      metadata:
        name: "example-memcached"
      spec:
        size: 3

      $ oc apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml
    2. Ensure that the memcached-operator creates the deployment for the CR:

      $ oc get deployment

      Example output

      NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      memcached-operator       1         1         1            1           2m
      example-memcached        3         3         3            3           1m

    3. Check the pods to confirm three replicas were created:

      $ oc get pods
      NAME                                  READY     STATUS    RESTARTS   AGE
      example-memcached-6fd7c98d8-7dqdr     1/1       Running   0          1m
      example-memcached-6fd7c98d8-g5k7v     1/1       Running   0          1m
      example-memcached-6fd7c98d8-m7vn7     1/1       Running   0          1m
      memcached-operator-7cc7cfdf86-vvjqk   1/1       Running   0          2m
  7. Update the size.

    1. Change the spec.size field in the memcached CR from 3 to 4 and apply the change:

      $ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml

      Example output

      apiVersion: "cache.example.com/v1alpha1"
      kind: "Memcached"
      metadata:
        name: "example-memcached"
      spec:
        size: 4

      $ oc apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml
    2. Confirm that the Operator changes the deployment size:

      $ oc get deployment

      Example output

      NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      example-memcached    4         4         4            4           5m

  8. Clean up the resources:

    $ oc delete -f deploy/crds/cache_v1alpha1_memcached_cr.yaml
    $ oc delete -f deploy/operator.yaml
    $ oc delete -f deploy/role_binding.yaml
    $ oc delete -f deploy/role.yaml
    $ oc delete -f deploy/service_account.yaml
    $ oc delete -f deploy/crds/cache_v1alpha1_memcached_crd.yaml

4.2.4. Managing application lifecycle using the k8s Ansible module

To manage the lifecycle of your application on Kubernetes using Ansible, you can use the k8s Ansible module. This Ansible module allows a developer to either leverage their existing Kubernetes resource files (written in YAML) or express the lifecycle management in native Ansible.

One of the biggest benefits of using Ansible in conjunction with existing Kubernetes resource files is the ability to use Jinja templating so that you can customize resources with the simplicity of a few variables in Ansible.

This section goes into detail on usage of the k8s Ansible module. To get started, install the module on your local workstation and test it using a playbook before moving on to using it within an Operator.

4.2.4.1. Installing the k8s Ansible module

To install the k8s Ansible module on your local workstation:

Procedure

  1. Install Ansible 2.9+:

    $ sudo yum install ansible
  2. Install the OpenShift python client package using pip:

    $ sudo pip install openshift
    $ sudo pip install kubernetes

4.2.4.2. Testing the k8s Ansible module locally

Sometimes, it is beneficial for a developer to run the Ansible code from their local machine as opposed to running and rebuilding the Operator each time.

Procedure

  1. Install the community.kubernetes collection:

    $ ansible-galaxy collection install community.kubernetes
  2. Initialize a new Ansible-based Operator project:

    $ operator-sdk new --type ansible \
        --kind Test1 \
        --api-version test1.example.com/v1alpha1 test1-operator

    Example output

    Create test1-operator/tmp/init/galaxy-init.sh
    Create test1-operator/tmp/build/Dockerfile
    Create test1-operator/tmp/build/test-framework/Dockerfile
    Create test1-operator/tmp/build/go-test.sh
    Rendering Ansible Galaxy role [test1-operator/roles/test1]...
    Cleaning up test1-operator/tmp/init
    Create test1-operator/watches.yaml
    Create test1-operator/deploy/rbac.yaml
    Create test1-operator/deploy/crd.yaml
    Create test1-operator/deploy/cr.yaml
    Create test1-operator/deploy/operator.yaml
    Run git init ...
    Initialized empty Git repository in /home/user/go/src/github.com/user/opsdk/test1-operator/.git/
    Run git init done

    $ cd test1-operator
  3. Modify the roles/test1/tasks/main.yml file with the Ansible logic that you want. This example creates and deletes a namespace with the switch of a variable.

    - name: set test namespace to "{{ state }}"
      community.kubernetes.k8s:
        api_version: v1
        kind: Namespace
        state: "{{ state }}"
        name: test
      ignore_errors: true 1
    1
    Setting ignore_errors: true ensures that deleting a nonexistent project does not fail.
  4. Modify the roles/test1/defaults/main.yml file to set state to present by default:

    state: present
  5. Create an Ansible playbook playbook.yml in the top-level directory, which includes the test1 role:

    - hosts: localhost
      roles:
        - test1
  6. Run the playbook:

    $ ansible-playbook playbook.yml

    Example output

     [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
    
    PLAY [localhost] ***************************************************************************
    
    PROCEDURE [Gathering Facts] *********************************************************************
    ok: [localhost]
    
    Task [test1 : set test namespace to present]
    changed: [localhost]
    
    PLAY RECAP *********************************************************************************
    localhost                  : ok=2    changed=1    unreachable=0    failed=0

  7. Check that the namespace was created:

    $ oc get namespace

    Example output

    NAME          STATUS    AGE
    default       Active    28d
    kube-public   Active    28d
    kube-system   Active    28d
    test          Active    3s

  8. Rerun the playbook setting state to absent:

    $ ansible-playbook playbook.yml --extra-vars state=absent

    Example output

     [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
    
    PLAY [localhost] ***************************************************************************
    
    PROCEDURE [Gathering Facts] *********************************************************************
    ok: [localhost]
    
    Task [test1 : set test namespace to absent]
    changed: [localhost]
    
    PLAY RECAP *********************************************************************************
    localhost                  : ok=2    changed=1    unreachable=0    failed=0

  9. Check that the namespace was deleted:

    $ oc get namespace

    Example output

    NAME          STATUS    AGE
    default       Active    28d
    kube-public   Active    28d
    kube-system   Active    28d

4.2.4.3. Testing the k8s Ansible module inside an Operator

After you are familiar with using the k8s Ansible module locally, you can trigger the same Ansible logic inside of an Operator when a custom resource (CR) changes. This example maps an Ansible role to a specific Kubernetes resource that the Operator watches. This mapping is done in the watches.yaml file.

4.2.4.3.1. Testing an Ansible-based Operator locally

After getting comfortable testing Ansible workflows locally, you can test the logic inside of an Ansible-based Operator running locally.

To do so, use the operator-sdk run --local command from the top-level directory of your Operator project. This command reads from the watches.yaml file and uses the ~/.kube/config file to communicate with a Kubernetes cluster just as the k8s Ansible module does.

Procedure

  1. Because the run --local command reads from the watches.yaml file, there are options available to the Operator author. If role is left alone (by default, /opt/ansible/roles/<name>) you must copy the role over to the /opt/ansible/roles/ directory from the Operator directly.

    This is cumbersome because changes are not reflected from the current directory. Instead, change the role field to point to the current directory and comment out the existing line:

    - version: v1alpha1
      group: test1.example.com
      kind: Test1
      #  role: /opt/ansible/roles/Test1
      role: /home/user/test1-operator/Test1
  2. Create a custom resource definition (CRD) and proper role-based access control (RBAC) definitions for the custom resource (CR) Test1. The operator-sdk command autogenerates these files inside of the deploy/ directory:

    $ oc create -f deploy/crds/test1_v1alpha1_test1_crd.yaml
    $ oc create -f deploy/service_account.yaml
    $ oc create -f deploy/role.yaml
    $ oc create -f deploy/role_binding.yaml
  3. Run the run --local command:

    $ operator-sdk run --local

    Example output

    [...]
    INFO[0000] Starting to serve on 127.0.0.1:8888
    INFO[0000] Watching test1.example.com/v1alpha1, Test1, default

  4. Now that the Operator is watching the resource Test1 for events, the creation of a CR triggers your Ansible role to execute. View the deploy/cr.yaml file:

    apiVersion: "test1.example.com/v1alpha1"
    kind: "Test1"
    metadata:
      name: "example"

    Because the spec field is not set, Ansible is invoked with no extra variables. The next section covers how extra variables are passed from a CR to Ansible. This is why it is important to set reasonable defaults for the Operator.

  5. Create a CR instance of Test1 with the default variable state set to present:

    $ oc create -f deploy/cr.yaml
  6. Check that the namespace test was created:

    $ oc get namespace

    Example output

    NAME          STATUS    AGE
    default       Active    28d
    kube-public   Active    28d
    kube-system   Active    28d
    test          Active    3s

  7. Modify the deploy/cr.yaml file to set the state field to absent:

    apiVersion: "test1.example.com/v1alpha1"
    kind: "Test1"
    metadata:
      name: "example"
    spec:
      state: "absent"
  8. Apply the changes and confirm that the namespace is deleted:

    $ oc apply -f deploy/cr.yaml
    $ oc get namespace

    Example output

    NAME          STATUS    AGE
    default       Active    28d
    kube-public   Active    28d
    kube-system   Active    28d

4.2.4.3.2. Testing an Ansible-based Operator on a cluster

After getting familiar running Ansible logic inside of an Ansible-based Operator locally, you can test the Operator inside of a pod on a Kubernetes cluster, such as OpenShift Container Platform. Running as a pod on a cluster is preferred for production use.

Procedure

  1. Build the test1-operator image and push it to a registry:

    $ operator-sdk build quay.io/example/test1-operator:v0.0.1
    $ podman push quay.io/example/test1-operator:v0.0.1
  2. Deployment manifests are generated in the deploy/operator.yaml file. The deployment image in this file must be modified from the placeholder REPLACE_IMAGE to the previously-built image. To do so, run the following command:

    $ sed -i 's|REPLACE_IMAGE|quay.io/example/test1-operator:v0.0.1|g' deploy/operator.yaml

    If you are performing these steps on macOS, use the following command instead:

    $ sed -i "" 's|REPLACE_IMAGE|quay.io/example/test1-operator:v0.0.1|g' deploy/operator.yaml
  3. Deploy the test1-operator:

    $ oc create -f deploy/crds/test1_v1alpha1_test1_crd.yaml 1
    1
    Only required if the CRD does not exist already.
    $ oc create -f deploy/service_account.yaml
    $ oc create -f deploy/role.yaml
    $ oc create -f deploy/role_binding.yaml
    $ oc create -f deploy/operator.yaml
  4. Verify that the test1-operator is up and running:

    $ oc get deployment

    Example output

    NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    test1-operator       1         1         1            1           1m

  5. You can now view the Ansible logs for the test1-operator:

    $ oc logs deployment/test1-operator

4.2.5. Managing custom resource status using the operator_sdk.util Ansible collection

Ansible-based Operators automatically update custom resource (CR) status subresources with generic information about the previous Ansible run. This includes the number of successful and failed tasks and relevant error messages as shown:

status:
  conditions:
    - ansibleResult:
      changed: 3
      completion: 2018-12-03T13:45:57.13329
      failures: 1
      ok: 6
      skipped: 0
    lastTransitionTime: 2018-12-03T13:45:57Z
    message: 'Status code was -1 and not [200]: Request failed: <urlopen error [Errno
      113] No route to host>'
    reason: Failed
    status: "True"
    type: Failure
  - lastTransitionTime: 2018-12-03T13:46:13Z
    message: Running reconciliation
    reason: Running
    status: "True"
    type: Running

Ansible-based Operators also allow Operator authors to supply custom status values with the k8s_status Ansible module, which is included in the operator_sdk.util collection. This allows the author to update the status from within Ansible with any key-value pair as desired.

By default, Ansible-based Operators always include the generic Ansible run output as shown above. If you would prefer your application did not update the status with Ansible output, you can track the status manually from your application.

Procedure

  1. To track CR status manually from your application, update the watches.yaml file with a manageStatus field set to false:

    - version: v1
      group: api.example.com
      kind: Test1
      role: Test1
      manageStatus: false
  2. Use the operator_sdk.util.k8s_status Ansible module to update the subresource. For example, to update with key test1 and value test2, operator_sdk.util can be used as shown:

    - operator_sdk.util.k8s_status:
        api_version: app.example.com/v1
        kind: Test1
        name: "{{ meta.name }}"
        namespace: "{{ meta.namespace }}"
        status:
          test1: test2

    Collections can also be declared in the meta/main.yml for the role, which is included for new scaffolded Ansible Operators:

    collections:
      - operator_sdk.util

    Declaring collections in the role meta allows you to invoke the k8s_status module directly:

    k8s_status:
      <snip>
      status:
        test1: test2

Additional resources

4.2.6. Additional resources

4.3. Creating Helm-based Operators

This guide outlines Helm chart support in the Operator SDK and walks Operator authors through an example of building and running an Nginx Operator with the operator-sdk CLI tool that uses an existing Helm chart.

4.3.1. Helm chart support in the Operator SDK

The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way. This framework includes the Operator SDK, which assists developers in bootstrapping and building an Operator based on their expertise without requiring knowledge of Kubernetes API complexities.

One of the Operator SDK options for generating an Operator project includes leveraging an existing Helm chart to deploy Kubernetes resources as a unified application, without having to write any Go code. Such Helm-based Operators are designed to excel at stateless applications that require very little logic when rolled out, because changes should be applied to the Kubernetes objects that are generated as part of the chart. This may sound limiting, but can be sufficient for a surprising amount of use-cases as shown by the proliferation of Helm charts built by the Kubernetes community.

The main function of an Operator is to read from a custom object that represents your application instance and have its desired state match what is running. In the case of a Helm-based Operator, the spec field of the object is a list of configuration options that are typically described in the Helm values.yaml file. Instead of setting these values with flags using the Helm CLI (for example, helm install -f values.yaml), you can express them within a custom resource (CR), which, as a native Kubernetes object, enables the benefits of RBAC applied to it and an audit trail.

For an example of a simple CR called Tomcat:

apiVersion: apache.org/v1alpha1
kind: Tomcat
metadata:
  name: example-app
spec:
  replicaCount: 2

The replicaCount value, 2 in this case, is propagated into the template of the chart where the following is used:

{{ .Values.replicaCount }}

After an Operator is built and deployed, you can deploy a new instance of an app by creating a new instance of a CR, or list the different instances running in all environments using the oc command:

$ oc get Tomcats --all-namespaces

There is no requirement use the Helm CLI or install Tiller; Helm-based Operators import code from the Helm project. All you have to do is have an instance of the Operator running and register the CR with a custom resource definition (CRD). Because it obeys RBAC, you can more easily prevent production changes.

4.3.2. Installing the Operator SDK CLI

The Operator SDK has a CLI tool that assists developers in creating, building, and deploying a new Operator project. You can install the SDK CLI on your workstation so you are prepared to start authoring your own Operators.

Note

This guide uses minikube v0.25.0+ as the local Kubernetes cluster and Quay.io for the public registry.

4.3.2.1. Installing from GitHub release

You can download and install a pre-built release binary of the Operator SDK CLI from the project on GitHub.

Prerequisites

  • Go v1.13+
  • docker v17.03+, podman v1.2.0+, or buildah v1.7+
  • OpenShift CLI (oc) v4.5+ installed
  • Access to a cluster based on Kubernetes v1.12.0+
  • Access to a container registry

Procedure

  1. Set the release version variable:

    $ RELEASE_VERSION=v0.17.2
  2. Download the release binary.

    • For Linux:

      $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
    • For macOS:

      $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin
  3. Verify the downloaded release binary.

    1. Download the provided .asc file.

      • For Linux:

        $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu.asc
      • For macOS:

        $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin.asc
    2. Place the binary and corresponding .asc file into the same directory and run the following command to verify the binary:

      • For Linux:

        $ gpg --verify operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu.asc
      • For macOS:

        $ gpg --verify operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin.asc

      If you do not have the public key of the maintainer on your workstation, you will get the following error:

      Example output with error

      $ gpg: assuming signed data in 'operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin'
      $ gpg: Signature made Fri Apr  5 20:03:22 2019 CEST
      $ gpg:                using RSA key <key_id> 1
      $ gpg: Can't check signature: No public key

      1
      RSA key string.

      To download the key, run the following command, replacing <key_id> with the RSA key string provided in the output of the previous command:

      $ gpg [--keyserver keys.gnupg.net] --recv-key "<key_id>" 1
      1
      If you do not have a key server configured, specify one with the --keyserver option.
  4. Install the release binary in your PATH:

    • For Linux:

      $ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
      $ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu /usr/local/bin/operator-sdk
      $ rm operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
    • For macOS:

      $ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin
      $ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin /usr/local/bin/operator-sdk
      $ rm operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin
  5. Verify that the CLI tool was installed correctly:

    $ operator-sdk version

4.3.2.2. Installing from Homebrew

You can install the SDK CLI using Homebrew.

Prerequisites

  • Homebrew
  • docker v17.03+, podman v1.2.0+, or buildah v1.7+
  • OpenShift CLI (oc) v4.5+ installed
  • Access to a cluster based on Kubernetes v1.12.0+
  • Access to a container registry

Procedure

  1. Install the SDK CLI using the brew command:

    $ brew install operator-sdk
  2. Verify that the CLI tool was installed correctly:

    $ operator-sdk version

4.3.2.3. Compiling and installing from source

You can obtain the Operator SDK source code to compile and install the SDK CLI.

Prerequisites

  • Git
  • Go v1.13+
  • docker v17.03+, podman v1.2.0+, or buildah v1.7+
  • OpenShift CLI (oc) v4.5+ installed
  • Access to a cluster based on Kubernetes v1.12.0+
  • Access to a container registry

Procedure

  1. Clone the operator-sdk repository:

    $ mkdir -p $GOPATH/src/github.com/operator-framework
    $ cd $GOPATH/src/github.com/operator-framework
    $ git clone https://github.com/operator-framework/operator-sdk
    $ cd operator-sdk
  2. Check out the desired release branch:

    $ git checkout master
  3. Compile and install the SDK CLI:

    $ make dep
    $ make install

    This installs the CLI binary operator-sdk at $GOPATH/bin.

  4. Verify that the CLI tool was installed correctly:

    $ operator-sdk version

4.3.3. Building a Helm-based Operator using the Operator SDK

This procedure walks through an example of building a simple Nginx Operator powered by a Helm chart using tools and libraries provided by the Operator SDK.

Tip

It is best practice to build a new Operator for each chart. This can allow for more native-behaving Kubernetes APIs (for example, oc get Nginx) and flexibility if you ever want to write a fully-fledged Operator in Go, migrating away from a Helm-based Operator.

Prerequisites

  • Operator SDK CLI installed on the development workstation
  • Access to a Kubernetes-based cluster v1.11.3+ (for example OpenShift Container Platform 4.5) using an account with cluster-admin permissions
  • OpenShift CLI (oc) v4.5+ installed

Procedure

  1. Create a new Operator project. A namespace-scoped Operator watches and manages resources in a single namespace. Namespace-scoped Operators are preferred because of their flexibility. They enable decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions.

    To create a new Helm-based, namespace-scoped nginx-operator project, use the following command:

    $ operator-sdk new nginx-operator \
      --api-version=example.com/v1alpha1 \
      --kind=Nginx \
      --type=helm
    $ cd nginx-operator

    This creates the nginx-operator project specifically for watching the Nginx resource with API version example.com/v1apha1 and kind Nginx.

  2. Customize the Operator logic.

    For this example, the nginx-operator executes the following reconciliation logic for each Nginx custom resource (CR):

    • Create an Nginx deployment if it does not exist.
    • Create an Nginx service if it does not exist.
    • Create an Nginx ingress if it is enabled and does not exist.
    • Ensure that the deployment, service, and optional ingress match the desired configuration (for example, replica count, image, service type) as specified by the Nginx CR.

    By default, the nginx-operator watches Nginx resource events as shown in the watches.yaml file and executes Helm releases using the specified chart:

    - version: v1alpha1
      group: example.com
      kind: Nginx
      chart: /opt/helm/helm-charts/nginx
    1. Review the Nginx Helm chart.

      When a Helm Operator project is created, the Operator SDK creates an example Helm chart that contains a set of templates for a simple Nginx release.

      For this example, templates are available for deployment, service, and ingress resources, along with a NOTES.txt template, which Helm chart developers use to convey helpful information about a release.

      If you are not already familiar with Helm Charts, review the Helm Chart developer documentation.

    2. Understand the Nginx CR spec.

      Helm uses a concept called values to provide customizations to the defaults of a Helm chart, which are defined in the values.yaml file.

      Override these defaults by setting the desired values in the CR spec. You can use the number of replicas as an example:

      1. First, inspect the helm-charts/nginx/values.yaml file to find that the chart has a value called replicaCount and it is set to 1 by default. To have 2 Nginx instances in your deployment, your CR spec must contain replicaCount: 2.

        Update the deploy/crds/example.com_v1alpha1_nginx_cr.yaml file to look like the following:

        apiVersion: example.com/v1alpha1
        kind: Nginx
        metadata:
          name: example-nginx
        spec:
          replicaCount: 2
      2. Similarly, the default service port is set to 80. To instead use 8080, update the deploy/crds/example.com_v1alpha1_nginx_cr.yaml file again by adding the service port override:

        apiVersion: example.com/v1alpha1
        kind: Nginx
        metadata:
          name: example-nginx
        spec:
          replicaCount: 2
          service:
            port: 8080

        The Helm Operator applies the entire spec as if it was the contents of a values file, just like the helm install -f ./overrides.yaml command works.

  3. Deploy the CRD.

    Before running the Operator, Kubernetes must know about the new custom resource definition (CRD) that the Operator will be watching. Deploy the following CRD:

    $ oc create -f deploy/crds/example_v1alpha1_nginx_crd.yaml
  4. Build and run the Operator.

    There are two ways to build and run the Operator:

    • As a pod inside a Kubernetes cluster.
    • As a Go program outside the cluster using the operator-sdk up command.

    Choose one of the following methods:

    1. Run as a pod inside a Kubernetes cluster. This is the preferred method for production use.

      1. Build the nginx-operator image and push it to a registry:

        $ operator-sdk build quay.io/example/nginx-operator:v0.0.1
        $ podman push quay.io/example/nginx-operator:v0.0.1
      2. Deployment manifests are generated in the deploy/operator.yaml file. The deployment image in this file needs to be modified from the placeholder REPLACE_IMAGE to the previous built image. To do this, run:

        $ sed -i 's|REPLACE_IMAGE|quay.io/example/nginx-operator:v0.0.1|g' deploy/operator.yaml
      3. Deploy the nginx-operator manifests:

        $ oc create -f deploy/service_account.yaml
        $ oc create -f deploy/role.yaml
        $ oc create -f deploy/role_binding.yaml
        $ oc create -f deploy/operator.yaml
      4. Verify that the nginx-operator deployment is up and running:

        $ oc get deployment

        Example output

        NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
        nginx-operator       1         1         1            1           1m

    2. Run outside the cluster. This method is preferred during the development cycle to speed up deployment and testing.

      It is important that the chart path referenced in the watches.yaml file exists on your machine. By default, the watches.yaml file is scaffolded to work with an Operator image built with the operator-sdk build command. When developing and testing your Operator with the operator-sdk run --local command, the SDK looks in your local file system for this path.

      1. Create a symlink at this location to point to the path of your Helm chart:

        $ sudo mkdir -p /opt/helm/helm-charts
        $ sudo ln -s $PWD/helm-charts/nginx /opt/helm/helm-charts/nginx
      2. To run the Operator locally with the default Kubernetes configuration file present at $HOME/.kube/config:

        $ operator-sdk run --local

        To run the Operator locally with a provided Kubernetes configuration file:

        $ operator-sdk run --local --kubeconfig=<path_to_config>
  5. Deploy the Nginx CR.

    Apply the Nginx CR that you modified earlier:

    $ oc apply -f deploy/crds/example.com_v1alpha1_nginx_cr.yaml

    Ensure that the nginx-operator creates the deployment for the CR:

    $ oc get deployment

    Example output

    NAME                                           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    example-nginx-b9phnoz9spckcrua7ihrbkrt1        2         2         2            2           1m

    Check the pods to confirm two replicas were created:

    $ oc get pods

    Example output

    NAME                                                      READY     STATUS    RESTARTS   AGE
    example-nginx-b9phnoz9spckcrua7ihrbkrt1-f8f9c875d-fjcr9   1/1       Running   0          1m
    example-nginx-b9phnoz9spckcrua7ihrbkrt1-f8f9c875d-ljbzl   1/1       Running   0          1m

    Check that the service port is set to 8080:

    $ oc get service

    Example output

    NAME                                      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
    example-nginx-b9phnoz9spckcrua7ihrbkrt1   ClusterIP   10.96.26.3   <none>        8080/TCP   1m

  6. Update the replicaCount and remove the port.

    Change the spec.replicaCount field from 2 to 3, remove the spec.service field, and apply the change:

    $ cat deploy/crds/example.com_v1alpha1_nginx_cr.yaml

    Example output

    apiVersion: "example.com/v1alpha1"
    kind: "Nginx"
    metadata:
      name: "example-nginx"
    spec:
      replicaCount: 3

    $ oc apply -f deploy/crds/example.com_v1alpha1_nginx_cr.yaml

    Confirm that the Operator changes the deployment size:

    $ oc get deployment

    Example output

    NAME                                           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    example-nginx-b9phnoz9spckcrua7ihrbkrt1        3         3         3            3           1m

    Check that the service port is set to the default 80:

    $ oc get service

    Example output

    NAME                                      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)  AGE
    example-nginx-b9phnoz9spckcrua7ihrbkrt1   ClusterIP   10.96.26.3   <none>        80/TCP   1m

  7. Clean up the resources:

    $ oc delete -f deploy/crds/example.com_v1alpha1_nginx_cr.yaml
    $ oc delete -f deploy/operator.yaml
    $ oc delete -f deploy/role_binding.yaml
    $ oc delete -f deploy/role.yaml
    $ oc delete -f deploy/service_account.yaml
    $ oc delete -f deploy/crds/example_v1alpha1_nginx_crd.yaml

4.3.4. Additional resources

4.4. Generating a cluster service version (CSV)

A cluster service version (CSV), defined by a ClusterServiceVersion object, is a YAML manifest created from Operator metadata that assists Operator Lifecycle Manager (OLM) in running the Operator in a cluster. It is the metadata that accompanies an Operator container image, used to populate user interfaces with information such as its logo, description, and version. It is also a source of technical information that is required to run the Operator, like the RBAC rules it requires and which custom resources (CRs) it manages or depends on.

The Operator SDK includes the generate csv subcommand to generate a CSV for the current Operator project customized using information contained in manually-defined YAML manifests and Operator source files.

A CSV-generating command removes the responsibility of Operator authors having in-depth OLM knowledge in order for their Operator to interact with OLM or publish metadata to the Catalog Registry. Further, because the CSV spec will likely change over time as new Kubernetes and OLM features are implemented, the Operator SDK is equipped to easily extend its update system to handle new CSV features going forward.

The CSV version is the same as the Operator version, and a new CSV is generated when upgrading Operator versions. Operator authors can use the --csv-version flag to have their Operator state encapsulated in a CSV with the supplied semantic version:

$ operator-sdk generate csv --csv-version <version>

This action is idempotent and only updates the CSV file when a new version is supplied, or a YAML manifest or source file is changed. Operator authors should not have to directly modify most fields in a CSV manifest. Those that require modification are defined in this guide. For example, the CSV version must be included in metadata.name.

4.4.1. How CSV generation works

The deploy/ directory of an Operator project is the standard location for all manifests required to deploy an Operator. The Operator SDK can use data from manifests in deploy/ to write a cluster service version (CSV).

The following command:

$ operator-sdk generate csv --csv-version <version>

writes a CSV YAML file to the deploy/olm-catalog/ directory by default.

Exactly three types of manifests are required to generate a CSV:

  • operator.yaml
  • *_{crd,cr}.yaml
  • RBAC role files, for example role.yaml

Operator authors may have different versioning requirements for these files and can configure which specific files are included in the deploy/olm-catalog/csv-config.yaml file.

Workflow

Depending on whether an existing CSV is detected, and assuming all configuration defaults are used, the generate csv subcommand either:

  • Creates a new CSV, with the same location and naming convention as exists currently, using available data in YAML manifests and source files.

    1. The update mechanism checks for an existing CSV in deploy/. When one is not found, it creates a ClusterServiceVersion object, referred to here as a cache, and populates fields easily derived from Operator metadata, such as Kubernetes API ObjectMeta.
    2. The update mechanism searches deploy/ for manifests that contain data a CSV uses, such as a Deployment resource, and sets the appropriate CSV fields in the cache with this data.
    3. After the search completes, every cache field populated is written back to a CSV YAML file.

or:

  • Updates an existing CSV at the currently pre-defined location, using available data in YAML manifests and source files.

    1. The update mechanism checks for an existing CSV in deploy/. When one is found, the CSV YAML file contents are marshaled into a CSV cache.
    2. The update mechanism searches deploy/ for manifests that contain data a CSV uses, such as a Deployment resource, and sets the appropriate CSV fields in the cache with this data.
    3. After the search completes, every cache field populated is written back to a CSV YAML file.
Note

Individual YAML fields are overwritten and not the entire file, as descriptions and other non-generated parts of a CSV should be preserved.

4.4.2. CSV composition configuration

Operator authors can configure CSV composition by populating several fields in the deploy/olm-catalog/csv-config.yaml file:

FieldDescription

operator-path (string)

The Operator resource manifest file path. Default: deploy/operator.yaml.

crd-cr-path-list (string(, string)*)

A list of CRD and CR manifest file paths. Default: [deploy/crds/*_{crd,cr}.yaml].

rbac-path-list (string(, string)*)

A list of RBAC role manifest file paths. Default: [deploy/role.yaml].

4.4.3. Manually-defined CSV fields

Many CSV fields cannot be populated using generated, generic manifests that are not specific to Operator SDK. These fields are mostly human-written, English metadata about the Operator and various custom resource definitions (CRDs).

Operator authors must directly modify their cluster service version (CSV) YAML file, adding personalized data to the following required fields. The Operator SDK gives a warning during CSV generation when a lack of data in any of the required fields is detected.

Table 4.5. Required

FieldDescription

metadata.name

A unique name for this CSV. Operator version should be included in the name to ensure uniqueness, for example app-operator.v0.1.1.

metadata.capabilities

The capability level according to the Operator maturity model. Options include Basic Install, Seamless Upgrades, Full Lifecycle, Deep Insights, and Auto Pilot.

spec.displayName

A public name to identify the Operator.

spec.description

A short description of the functionality of the Operator.

spec.keywords

Keywords describing the Operator.

spec.maintainers

Human or organizational entities maintaining the Operator, with a name and email.

spec.provider

The provider of the Operator (usually an organization), with a name.

spec.labels

Key-value pairs to be used by Operator internals.

spec.version

Semantic version of the Operator, for example 0.1.1.

spec.customresourcedefinitions

Any CRDs the Operator uses. This field is populated automatically by the Operator SDK if any CRD YAML files are present in deploy/. However, several fields not in the CRD manifest spec require user input:

  • description: description of the CRD.
  • resources: any Kubernetes resources leveraged by the CRD, for example Pod and StatefulSet objects.
  • specDescriptors: UI hints for inputs and outputs of the Operator.

Table 4.6. Optional

FieldDescription

spec.replaces

The name of the CSV being replaced by this CSV.

spec.links

URLs (for example, websites and documentation) pertaining to the Operator or application being managed, each with a name and url.

spec.selector

Selectors by which the Operator can pair resources in a cluster.

spec.icon

A base64-encoded icon unique to the Operator, set in a base64data field with a mediatype.

spec.maturity

The level of maturity the software has achieved at this version. Options include planning, pre-alpha, alpha, beta, stable, mature, inactive, and deprecated.

Further details on what data each field above should hold are found in the CSV spec.

Note

Several YAML fields currently requiring user intervention can potentially be parsed from Operator code.

Additional resources

4.4.4. Generating a CSV

Prerequisites

  • An Operator project generated using the Operator SDK

Procedure

  1. In your Operator project, configure your CSV composition by modifying the deploy/olm-catalog/csv-config.yaml file, if desired.
  2. Generate the CSV:

    $ operator-sdk generate csv --csv-version <version>
  3. In the new CSV generated in the deploy/olm-catalog/ directory, ensure all required, manually-defined fields are set appropriately.

4.4.5. Enabling your Operator for restricted network environments

As an Operator author, your CSV must meet the following additional requirements for your Operator to run properly in a restricted network environment:

  • List any related images, or other container images that your Operator might require to perform their functions.
  • Reference all specified images by a digest (SHA) and not by a tag.

You must use SHA references to related images in two places in the Operator’s CSV:

  • in spec.relatedImages:

    ...
    spec:
      relatedImages: 1
        - name: etcd-operator 2
          image: quay.io/etcd-operator/operator@sha256:d134a9865524c29fcf75bbc4469013bc38d8a15cb5f41acfddb6b9e492f556e4 3
        - name: etcd-image
          image: quay.io/etcd-operator/etcd@sha256:13348c15263bd8838ec1d5fc4550ede9860fcbb0f843e48cbccec07810eebb68
    ...
    1
    Create a relatedImages section and set the list of related images.
    2
    Specify a unique identifier for the image.
    3
    Specify each image by a digest (SHA), not by an image tag.
  • in the env section of the Operators Deployments when declaring environment variables that inject the image that the Operator should use:

    spec:
      install:
        spec:
          deployments:
          - name: etcd-operator-v3.1.1
            spec:
              replicas: 1
              selector:
                matchLabels:
                  name: etcd-operator
              strategy:
                type: Recreate
              template:
                metadata:
                  labels:
                    name: etcd-operator
                spec:
                  containers:
                  - args:
                    - /opt/etcd/bin/etcd_operator_run.sh
                    env:
                    - name: WATCH_NAMESPACE
                      valueFrom:
                        fieldRef:
                          fieldPath: metadata.annotations['olm.targetNamespaces']
                    - name: ETCD_OPERATOR_DEFAULT_ETCD_IMAGE 1
                      value: quay.io/etcd-operator/etcd@sha256:13348c15263bd8838ec1d5fc4550ede9860fcbb0f843e48cbccec07810eebb68 2
                    - name: ETCD_LOG_LEVEL
                      value: INFO
                    image: quay.io/etcd-operator/operator@sha256:d134a9865524c29fcf75bbc4469013bc38d8a15cb5f41acfddb6b9e492f556e4 3
                    imagePullPolicy: IfNotPresent
                    livenessProbe:
                      httpGet:
                        path: /healthy
                        port: 8080
                      initialDelaySeconds: 10
                      periodSeconds: 30
                    name: etcd-operator
                    readinessProbe:
                      httpGet:
                        path: /ready
                        port: 8080
                      initialDelaySeconds: 10
                      periodSeconds: 30
                    resources: {}
                  serviceAccountName: etcd-operator
        strategy: deployment
    1
    Inject the images referenced by the Operator by using environment variables.
    2
    Specify each image by a digest (SHA), not by an image tag.
    3
    Also reference the Operator container image by a digest (SHA), not by an image tag.
    Note

    When configuring probes, the timeoutSeconds value must be lower than the periodSeconds value. The timeoutSeconds default value is 1. The periodSeconds default value is 10.

  • Look for the Disconnected annotation, which indicates that the Operator works in a disconnected environment:

    metadata:
      annotations:
        operators.openshift.io/infrastructure-features: '["Disconnected"]'

    Operators can be filtered in OperatorHub by this infrastructure feature.

4.4.6. Enabling your Operator for multiple architectures and operating systems

Operator Lifecycle Manager (OLM) assumes that all Operators run on Linux hosts. However, as an Operator author, you can specify whether your Operator supports managing workloads on other architectures, if worker nodes are available in the OpenShift Container Platform cluster.

If your Operator supports variants other than AMD64 and Linux, you can add labels to the cluster service version (CSV) that provides the Operator to list the supported variants. Labels indicating supported architectures and operating systems are defined by the following:

labels:
    operatorframework.io/arch.<arch>: supported 1
    operatorframework.io/os.<os>: supported 2
1
Set <arch> to a supported string.
2
Set <os> to a supported string.
Note

Only the labels on the channel head of the default channel are considered for filtering package manifests by label. This means, for example, that providing an additional architecture for an Operator in the non-default channel is possible, but that architecture is not available for filtering in the PackageManifest API.

If a CSV does not include an os label, it is treated as if it has the following Linux support label by default:

labels:
    operatorframework.io/os.linux: supported

If a CSV does not include an arch label, it is treated as if it has the following AMD64 support label by default:

labels:
    operatorframework.io/arch.amd64: supported

If an Operator supports multiple node architectures or operating systems, you can add multiple labels, as well.

Prerequisites

  • An Operator project with a CSV.
  • To support listing multiple architectures and operating systems, your Operator image referenced in the CSV must be a manifest list image.
  • For the Operator to work properly in restricted network, or disconnected, environments, the image referenced must also be specified using a digest (SHA) and not by a tag.

Procedure

  • Add a label in the metadata.labels of your CSV for each supported architecture and operating system that your Operator supports:

    labels:
      operatorframework.io/arch.s390x: supported
      operatorframework.io/os.zos: supported
      operatorframework.io/os.linux: supported 1
      operatorframework.io/arch.amd64: supported 2
    1 2
    After you add a new architecture or operating system, you must also now include the default os.linux and arch.amd64 variants explicitly.

Additional resources

4.4.6.1. Architecture and operating system support for Operators

The following strings are supported in Operator Lifecycle Manager (OLM) on OpenShift Container Platform when labeling or filtering Operators that support multiple architectures and operating systems:

Table 4.7. Architectures supported on OpenShift Container Platform

ArchitectureString

AMD64

amd64

64-bit PowerPC little-endian

ppc64le

IBM Z

s390x

Table 4.8. Operating systems supported on OpenShift Container Platform

Operating systemString

Linux

linux

z/OS

zos

Note

Different versions of OpenShift Container Platform and other Kubernetes-based distributions might support a different set of architectures and operating systems.

4.4.7. Setting a suggested namespace

Some Operators must be deployed in a specific namespace, or with ancillary resources in specific namespaces, in order to work properly. If resolved from a subscription, Operator Lifecycle Manager (OLM) defaults the namespaced resources of an Operator to the namespace of its subscription.

As an Operator author, you can instead express a desired target namespace as part of your cluster service version (CSV) to maintain control over the final namespaces of the resources installed for their Operators. When adding the Operator to a cluster using OperatorHub, this enables the web console to autopopulate the suggested namespace for the cluster administrator during the installation process.

Procedure

  • In your CSV, set the operatorframework.io/suggested-namespace annotation to your suggested namespace:

    metadata:
      annotations:
        operatorframework.io/suggested-namespace: <namespace> 1
    1
    Set your suggested namespace.

4.4.8. Understanding your custom resource definitions (CRDs)

There are two types of custom resource definitions (CRDs) that your Operator can use: ones that are owned by it and ones that it depends on, which are required.

4.4.8.1. Owned CRDs

The custom resource definitions (CRDs) owned by your Operator are the most important part of your CSV. This establishes the link between your Operator and the required RBAC rules, dependency management, and other Kubernetes concepts.

It is common for your Operator to use multiple CRDs to link together concepts, such as top-level database configuration in one object and a representation of replica sets in another. Each one should be listed out in the CSV file.

Table 4.9. Owned CRD fields

FieldDescriptionRequired/optional

Name

The full name of your CRD.

Required

Version

The version of that object API.

Required

Kind

The machine readable name of your CRD.

Required

DisplayName

A human readable version of your CRD name, for example MongoDB Standalone.

Required

Description

A short description of how this CRD is used by the Operator or a description of the functionality provided by the CRD.

Required

Group

The API group that this CRD belongs to, for example database.example.com.

Optional

Resources

Your CRDs own one or more types of Kubernetes objects. These are listed in the resources section to inform your users of the objects they might need to troubleshoot or how to connect to the application, such as the service or ingress rule that exposes a database.

It is recommended to only list out the objects that are important to a human, not an exhaustive list of everything you orchestrate. For example, do not list config maps that store internal state that are not meant to be modified by a user.

Optional

SpecDescriptors, StatusDescriptors, and ActionDescriptors

These descriptors are a way to hint UIs with certain inputs or outputs of your Operator that are most important to an end user. If your CRD contains the name of a secret or config map that the user must provide, you can specify that here. These items are linked and highlighted in compatible UIs.

There are three types of descriptors:

  • SpecDescriptors: A reference to fields in the spec block of an object.
  • StatusDescriptors: A reference to fields in the status block of an object.
  • ActionDescriptors: A reference to actions that can be performed on an object.

All descriptors accept the following fields:

  • DisplayName: A human readable name for the Spec, Status, or Action.
  • Description: A short description of the Spec, Status, or Action and how it is used by the Operator.
  • Path: A dot-delimited path of the field on the object that this descriptor describes.
  • X-Descriptors: Used to determine which "capabilities" this descriptor has and which UI component to use. See the openshift/console project for a canonical list of React UI X-Descriptors for OpenShift Container Platform.

Also see the openshift/console project for more information on Descriptors in general.

Optional

The following example depicts a MongoDB Standalone CRD that requires some user input in the form of a secret and config map, and orchestrates services, stateful sets, pods and config maps:

Example owned CRD

      - displayName: MongoDB Standalone
        group: mongodb.com
        kind: MongoDbStandalone
        name: mongodbstandalones.mongodb.com
        resources:
          - kind: Service
            name: ''
            version: v1
          - kind: StatefulSet
            name: ''
            version: v1beta2
          - kind: Pod
            name: ''
            version: v1
          - kind: ConfigMap
            name: ''
            version: v1
        specDescriptors:
          - description: Credentials for Ops Manager or Cloud Manager.
            displayName: Credentials
            path: credentials
            x-descriptors:
              - 'urn:alm:descriptor:com.tectonic.ui:selector:core:v1:Secret'
          - description: Project this deployment belongs to.
            displayName: Project
            path: project
            x-descriptors:
              - 'urn:alm:descriptor:com.tectonic.ui:selector:core:v1:ConfigMap'
          - description: MongoDB version to be installed.
            displayName: Version
            path: version
            x-descriptors:
              - 'urn:alm:descriptor:com.tectonic.ui:label'
        statusDescriptors:
          - description: The status of each of the pods for the MongoDB cluster.
            displayName: Pod Status
            path: pods
            x-descriptors:
              - 'urn:alm:descriptor:com.tectonic.ui:podStatuses'
        version: v1
        description: >-
          MongoDB Deployment consisting of only one host. No replication of
          data.

4.4.8.2. Required CRDs

Relying on other required CRDs is completely optional and only exists to reduce the scope of individual Operators and provide a way to compose multiple Operators together to solve an end-to-end use case.

An example of this is an Operator that might set up an application and install an etcd cluster (from an etcd Operator) to use for distributed locking and a Postgres database (from a Postgres Operator) for data storage.

Operator Lifecycle Manager (OLM) checks against the available CRDs and Operators in the cluster to fulfill these requirements. If suitable versions are found, the Operators are started within the desired namespace and a service account created for each Operator to create, watch, and modify the Kubernetes resources required.

Table 4.10. Required CRD fields

FieldDescriptionRequired/optional

Name

The full name of the CRD you require.

Required

Version

The version of that object API.

Required

Kind

The Kubernetes object kind.

Required

DisplayName

A human readable version of the CRD.

Required

Description

A summary of how the component fits in your larger architecture.

Required

Example required CRD

    required:
    - name: etcdclusters.etcd.database.coreos.com
      version: v1beta2
      kind: EtcdCluster
      displayName: etcd Cluster
      description: Represents a cluster of etcd nodes.

4.4.8.3. CRD templates

Users of your Operator must be made aware of which options are required versus optional. You can provide templates for each of your custom resource definitions (CRDs) with a minimum set of configuration as an annotation named alm-examples. Compatible UIs will pre-fill this template for users to further customize.

The annotation consists of a list of the kind, for example, the CRD name and the corresponding metadata and spec of the Kubernetes object.

The following full example provides templates for EtcdCluster, EtcdBackup and EtcdRestore:

metadata:
  annotations:
    alm-examples: >-
      [{"apiVersion":"etcd.database.coreos.com/v1beta2","kind":"EtcdCluster","metadata":{"name":"example","namespace":"default"},"spec":{"size":3,"version":"3.2.13"}},{"apiVersion":"etcd.database.coreos.com/v1beta2","kind":"EtcdRestore","metadata":{"name":"example-etcd-cluster"},"spec":{"etcdCluster":{"name":"example-etcd-cluster"},"backupStorageType":"S3","s3":{"path":"<full-s3-path>","awsSecret":"<aws-secret>"}}},{"apiVersion":"etcd.database.coreos.com/v1beta2","kind":"EtcdBackup","metadata":{"name":"example-etcd-cluster-backup"},"spec":{"etcdEndpoints":["<etcd-cluster-endpoints>"],"storageType":"S3","s3":{"path":"<full-s3-path>","awsSecret":"<aws-secret>"}}}]

4.4.8.4. Hiding internal objects

It is common practice for Operators to use custom resource definitions (CRDs) internally to accomplish a task. These objects are not meant for users to manipulate and can be confusing to users of the Operator. For example, a database Operator might have a Replication CRD that is created whenever a user creates a Database object with replication: true.

As an Operator author, you can hide any CRDs in the user interface that are not meant for user manipulation by adding the operators.operatorframework.io/internal-objects annotation to the cluster service version (CSV) of your Operator.

Procedure

  1. Before marking one of your CRDs as internal, ensure that any debugging information or configuration that might be required to manage the application is reflected on the status or spec block of your CR, if applicable to your Operator.
  2. Add the operators.operatorframework.io/internal-objects annotation to the CSV of your Operator to specify any internal objects to hide in the user interface:

    Internal object annotation

    apiVersion: operators.coreos.com/v1alpha1
    kind: ClusterServiceVersion
    metadata:
      name: my-operator-v1.2.3
      annotations:
        operators.operatorframework.io/internal-objects: '["my.internal.crd1.io","my.internal.crd2.io"]' 1
    ...

    1
    Set any internal CRDs as an array of strings.

4.4.9. Understanding your API services

As with CRDs, there are two types of API services that your Operator may use: owned and required.

4.4.9.1. Owned API services

When a CSV owns an API service, it is responsible for describing the deployment of the extension api-server that backs it and the group/version/kind (GVK) it provides.

An API service is uniquely identified by the group/version it provides and can be listed multiple times to denote the different kinds it is expected to provide.

Table 4.11. Owned API service fields

FieldDescriptionRequired/optional

Group

Group that the API service provides, for example database.example.com.

Required

Version

Version of the API service, for example v1alpha1.

Required

Kind

A kind that the API service is expected to provide.

Required

Name

The plural name for the API service provided.

Required

DeploymentName

Name of the deployment defined by your CSV that corresponds to your API service (required for owned API services). During the CSV pending phase, the OLM Operator searches the InstallStrategy of your CSV for a Deployment spec with a matching name, and if not found, does not transition the CSV to the "Install Ready" phase.

Required

DisplayName

A human readable version of your API service name, for example MongoDB Standalone.

Required

Description

A short description of how this API service is used by the Operator or a description of the functionality provided by the API service.

Required

Resources

Your API services own one or more types of Kubernetes objects. These are listed in the resources section to inform your users of the objects they might need to troubleshoot or how to connect to the application, such as the service or ingress rule that exposes a database.

It is recommended to only list out the objects that are important to a human, not an exhaustive list of everything you orchestrate. For example, do not list config maps that store internal state that are not meant to be modified by a user.

Optional

SpecDescriptors, StatusDescriptors, and ActionDescriptors

Essentially the same as for owned CRDs.

Optional

4.4.9.1.1. API service resource creation

Operator Lifecycle Manager (OLM) is responsible for creating or replacing the service and API service resources for each unique owned API service:

  • Service pod selectors are copied from the CSV deployment matching the DeploymentName field of the API service description.
  • A new CA key/certificate pair is generated for each installation and the base64-encoded CA bundle is embedded in the respective API service resource.
4.4.9.1.2. API service serving certificates

OLM handles generating a serving key/certificate pair whenever an owned API service is being installed. The serving certificate has a common name (CN) containing the host name of the generated Service resource and is signed by the private key of the CA bundle embedded in the corresponding API service resource.

The certificate is stored as a type kubernetes.io/tls secret in the deployment namespace, and a volume named apiservice-cert is automatically appended to the volumes section of the deployment in the CSV matching the DeploymentName field of the API service description.

If one does not already exist, a volume mount with a matching name is also appended to all containers of that deployment. This allows users to define a volume mount with the expected name to accommodate any custom path requirements. The path of the generated volume mount defaults to /apiserver.local.config/certificates and any existing volume mounts with the same path are replaced.

4.4.9.2. Required API services

OLM ensures all required CSVs have an API service that is available and all expected GVKs are discoverable before attempting installation. This allows a CSV to rely on specific kinds provided by API services it does not own.

Table 4.12. Required API service fields

FieldDescriptionRequired/optional

Group

Group that the API service provides, for example database.example.com.

Required

Version

Version of the API service, for example v1alpha1.

Required

Kind

A kind that the API service is expected to provide.

Required

DisplayName

A human readable version of your API service name, for example MongoDB Standalone.

Required

Description

A short description of how this API service is used by the Operator or a description of the functionality provided by the API service.

Required

4.5. Working with bundle images

You can use the Operator SDK to package Operators using the Bundle Format.

4.5.1. Building a bundle image

You can build, push, and validate an Operator bundle image using the Operator SDK.

Prerequisites

  • Operator SDK version 0.17.2
  • podman version 1.4.4+
  • An Operator project generated using the Operator SDK

Procedure

  1. From your Operator project directory, build the bundle image using the Operator SDK:

    $ operator-sdk bundle create \
        quay.io/<namespace>/test-operator:v0.1.0 \1
        -b podman 2
    1
    The image tag that you want the bundle image to have.
    2
    The CLI tool to use for building the container image, either docker (default), podman, or buildah. This example uses podman.
    Note

    If your local manifests are not located in the default <project_root>/deploy/olm-catalog/test-operator/manifests, specify the location with the --directory flag.

  2. Log in to the registry where you want to push the bundle image. For example:

    $ podman login quay.io
  3. Push the bundle image to the registry:

    $ podman push quay.io/<namespace>/test-operator:v0.1.0
  4. Validate the bundle image in the remote registry:

    $ operator-sdk bundle validate \
        quay.io/<namespace>/test-operator:v0.1.0 \
        -b podman

    Example output

    INFO[0000] Unpacked image layers                                 bundle-dir=/tmp/bundle-041168359 container-tool=podman
    INFO[0000] running podman pull                                   bundle-dir=/tmp/bundle-041168359 container-tool=podman
    INFO[0002] running podman save                                   bundle-dir=/tmp/bundle-041168359 container-tool=podman
    INFO[0002] All validation tests have completed successfully      bundle-dir=/tmp/bundle-041168359 container-tool=podman

4.5.2. Additional resources

4.6. Validating Operators using the scorecard

Operator authors should validate that their Operator is packaged correctly and free of syntax errors. As an Operator author, you can use the Operator SDK scorecard tool to validate your Operator packaging and run tests.

Note

OpenShift Container Platform 4.5 supports Operator SDK v0.17.2.

4.6.1. About the scorecard tool

To validate an Operator, the scorecard tool provided by the Operator SDK begins by creating all resources required by any related custom resources (CRs) and the Operator. The scorecard then creates a proxy container in the deployment of the Operator which is used to record calls to the API server and run some of the tests. The tests performed also examine some of the parameters in the CRs.

4.6.2. Scorecard configuration

The scorecard tool uses a configuration file that allows you to configure internal plug-ins, as well as several global configuration options.

4.6.2.1. Configuration file

The default location for the scorecard tool configuration is the <project_dir>/.osdk-scorecard.*. The following is an example of a YAML-formatted configuration file:

Scorecard configuration file

scorecard:
  output: json
  plugins:
    - basic: 1
        cr-manifest:
          - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml"
          - "deploy/crds/cache.example.com_v1alpha1_memcachedrs_cr.yaml"
    - olm: 2
        cr-manifest:
          - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml"
          - "deploy/crds/cache.example.com_v1alpha1_memcachedrs_cr.yaml"
        csv-path: "deploy/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml"

1
basic tests configured to test two custom resources (CRs).
2
olm tests configured to test two CRs.

Configuration methods for global options take the following priority, highest to lowest:

Command arguments (if available) → configuration file → default

The configuration file must be in YAML format. As the configuration file might be extended to allow configuration of all operator-sdk subcommands in the future, the scorecard configuration must be under a scorecard subsection.

Note

Configuration file support is provided by the viper package. For more info on how viper configuration works, see the README.

4.6.2.2. Command arguments

While most of the scorecard tool configuration is done using a configuration file, you can also use the following arguments:

Table 4.13. Scorecard tool arguments

FlagTypeDescription

--bundle, -b

string

The path to a bundle directory used for the bundle validation test.

--config

string

The path to the scorecard configuration file. The default is <project_dir>/.osdk-scorecard. The file type and extension must be .yaml. If a configuration file is not provided or found at the default location, the scorecard exits with an error.

--output, -o

string

Output format. Valid options are text and json. The default format is text, which is designed to be a human readable format. The json format uses the JSON schema output format used for plug-ins defined later.

--kubeconfig, -o

string

The path to the kubeconfig file. It sets the kubeconfig for internal plug-ins.

--version

string

The version of scorecard to run. The default and only valid option is v1alpha2.

--selector, -l

string

The label selector to filter tests on.

--list, -L

bool

If true, only print the test names that would be run based on selector filtering.

4.6.2.3. Configuration file options

The scorecard configuration file provides the following options:

Table 4.14. Scorecard configuration file options

OptionTypeDescription

bundle

string

Equivalent of the --bundle flag. Operator Lifecycle Manager (OLM) bundle directory path, when specified, runs bundle validation.

output

string

Equivalent of the --output flag. If this option is defined by both the configuration file and the flag, the flag value takes priority.

kubeconfig

string

Equivalent of the --kubeconfig flag. If this option is defined by both the configuration file and the flag, the flag value takes priority.

plugins

array

An array of plug-in names.

4.6.2.3.1. Basic and OLM plug-ins

The scorecard supports the internal basic and olm plug-ins, which are configured by a plugins section in the configuration file.

Table 4.15. Plug-in options

OptionTypeDescription

cr-manifest

[]string

The path(s) for CRs being tested. Required if olm-deployed is unset or false.

csv-path

string

The path to the cluster service version (CSV) for the Operator. Required for OLM tests or if olm-deployed is set to true.

olm-deployed

bool

Indicates that the CSV and relevant CRDs have been deployed onto the cluster by OLM.

kubeconfig

string

The path to the kubeconfig file. If both the global kubeconfig and this field are set, this field is used for the plug-in.

namespace

string

The namespace to run the plug-ins in. If unset, the default specified by the kubeconfig file is used.

init-timeout

int

Time in seconds until a timeout during initialization of the Operator.

crds-dir

string

The path to the directory containing CRDs that must be deployed to the cluster.

namespaced-manifest

string

The manifest file with all resources that run within a namespace. By default, the scorecard combines the service_account.yaml, role.yaml, role_binding.yaml, and operator.yaml files from the deploy directory into a temporary manifest to use as the namespaced manifest.

global-manifest

string

The manifest containing required resources that run globally (not namespaced). By default, the scorecard combines all CRDs in the crds-dir directory into a temporary manifest to use as the global manifest.

Note

Currently, using the scorecard with a CSV does not permit multiple CR manifests to be set through the CLI, configuration file, or CSV annotations. You must tear down your Operator in the cluster, re-deploy, and re-run the scorecard for each CR that is tested.

Additional resources

  • You can either set cr-manifest or your CSV metadata.annotations['alm-examples'] to provide CRs to the scorecard, but not both. See CRD templates for details.

4.6.3. Tests performed

By default, the scorecard tool has a set of internal tests it can run available across two internal plug-ins. If multiple CRs are specified for a plug-in, the test environment is fully cleaned up after each CR so that each CR gets a clean testing environment.

Each test has a short name that uniquely identifies the test. This is useful when selecting a specific test or tests to run. For example:

$ operator-sdk scorecard -o text --selector=test=checkspectest
$ operator-sdk scorecard -o text --selector='test in (checkspectest,checkstatustest)'

4.6.3.1. Basic plug-in

The following basic Operator tests are available from the basic plug-in:

Table 4.16. basic plug-in tests

TestDescriptionShort name

Spec Block Exists

This test checks the custom resources (CRs) created in the cluster to make sure that all CRs have a spec block. This test has a maximum score of 1.

checkspectest

Status Block Exists

This test checks the CRs created in the cluster to make sure that all CRs have a status block. This test has a maximum score of 1.

checkstatustest

Writing Into CRs Has An Effect

This test reads the scorecard proxy logs to verify that the Operator is making PUT or POST, or both, requests to the API server, indicating that it is modifying resources. This test has a maximum score of 1.

writingintocrshaseffecttest

4.6.3.2. OLM plug-in

The following Operator Lifecycle Manager (OLM) integration tests are available from the olm plug-in:

Table 4.17. olm plug-in tests

TestDescriptionShort name

OLM Bundle Validation

This test validates the OLM bundle manifests found in the bundle directory as specified by the bundle flag. If the bundle contents contain errors, then the test result output includes the validator log as well as error messages from the validation library.

bundlevalidationtest

Provided APIs Have Validation

This test verifies that the CRDs for the provided CRs contain a validation section and that there is validation for each spec and status field detected in the CR. This test has a maximum score equal to the number of CRs provided by the cr-manifest option.

crdshavevalidationtest

Owned CRDs Have Resources Listed

This test makes sure that the CRDs for each CR provided by the cr-manifest option have a resources subsection in the owned CRDs section of the CSV. If the test detects used resources that are not listed in the resources section, it lists them in the suggestions at the end of the test. This test has a maximum score equal to the number of CRs provided by the cr-manifest option.

crdshaveresourcestest

Spec Fields With Descriptors

This test verifies that every field in the spec sections of custom resources have a corresponding descriptor listed in the CSV. This test has a maximum score equal to the total number of fields in the spec sections of each custom resource passed in by the cr-manifest option.

specdescriptorstest

Status Fields With Descriptors

This test verifies that every field in the status sections of custom resources have a corresponding descriptor listed in the CSV. This test has a maximum score equal to the total number of fields in the status sections of each custom resource passed in by the cr-manifest option.

statusdescriptorstest

Additional resources

4.6.4. Running the scorecard

Prerequisites

The following prerequisites for the Operator project are checked by the scorecard tool:

  • Access to a cluster running Kubernetes 1.11.3 or later.
  • If you want to use the scorecard to check the integration of your Operator project with Operator Lifecycle Manager (OLM), then a cluster service version (CSV) file is also required. This is a requirement when the olm-deployed option is used.
  • For Operators that were not generated using the Operator SDK (non-SDK Operators):

    • Resource manifests for installing and configuring the Operator and custom resources (CRs).
    • Configuration getter that supports reading from the KUBECONFIG environment variable, such as the clientcmd or controller-runtime configuration getters. This is required for the scorecard proxy to work correctly.

Procedure

  1. Define a .osdk-scorecard.yaml configuration file in your Operator project.
  2. Create the namespace defined in the RBAC files (role_binding).
  3. Run the scorecard from the root directory of your Operator project:

    $ operator-sdk scorecard

    The scorecard return code is 1 if any of the executed texts did not pass and 0 if all selected tests passed.

4.6.5. Running the scorecard with an OLM-managed Operator

The scorecard can be run using a cluster service version (CSV), providing a way to test cluster-ready and non-Operator SDK Operators.

Procedure

  1. The scorecard requires a proxy container in the deployment pod of the Operator to read Operator logs. A few modifications to your CSV and creation of one extra object are required to run the proxy before deploying your Operator with Operator Lifecycle Manager (OLM).

    This step can be performed manually or automated using bash functions. Choose one of the following methods.

    • Manual method:

      1. Create a proxy server secret containing a local kubeconfig file`.

        1. Generate a user name using the namespaced owner reference of the scorecard proxy.

          $ echo '{"apiVersion":"","kind":"","name":"scorecard","uid":"","Namespace":"'<namespace>'"}' | base64 -w 0 1
          1
          Replace <namespace> with the namespace your Operator will deploy in.
        2. Write a Config manifest scorecard-config.yaml using the following template, replacing <username> with the base64 user name generated in the previous step:

          apiVersion: v1
          kind: Config
          clusters:
          - cluster:
              insecure-skip-tls-verify: true
              server: http://<username>@localhost:8889
            name: proxy-server
          contexts:
          - context:
              cluster: proxy-server
              user: admin/proxy-server
            name: <namespace>/proxy-server
          current-context: <namespace>/proxy-server
          preferences: {}
          users:
          - name: admin/proxy-server
            user:
              username: <username>
              password: unused
        3. Encode the Config as base64:

          $ cat scorecard-config.yaml | base64 -w 0
        4. Create a Secret manifest scorecard-secret.yaml:

          apiVersion: v1
          kind: Secret
          metadata:
            name: scorecard-kubeconfig
            namespace: <namespace> 1
          data:
            kubeconfig: <kubeconfig_base64> 2
          1
          Replace <namespace> with the namespace your Operator will deploy in.
          2
          Replace <kubeconfig_base64> with the Config encoded as base64.
        5. Apply the secret:

          $ oc apply -f scorecard-secret.yaml
        6. Insert a volume referring to the secret into the deployment for the Operator:

          spec:
            install:
              spec:
                deployments:
                - name: memcached-operator
                  spec:
                    ...
                    template:
                      ...
                      spec:
                        containers:
                        ...
                        volumes:
                        - name: scorecard-kubeconfig 1
                          secret:
                            secretName: scorecard-kubeconfig
                            items:
                            - key: kubeconfig
                              path: config
          1
          Scorecard kubeconfig volume.
      2. Insert a volume mount and KUBECONFIG environment variable into each container in the deployment of your Operator:

        spec:
          install:
            spec:
              deployments:
              - name: memcached-operator
                spec:
                  ...
                  template:
                    ...
                    spec:
                      containers:
                      - name: container1
                        ...
                        volumeMounts:
                        - name: scorecard-kubeconfig 1
                          mountPath: /scorecard-secret
                        env:
                        - name: KUBECONFIG 2
                          value: /scorecard-secret/config
                      - name: container2 3
                        ...
        1
        Scorecard kubeconfig volume mount.
        2
        Scorecard kubeconfig environment variable.
        3
        Repeat the same for this and all other containers.
      3. Insert the scorecard proxy container into the deployment of your Operator:

        spec:
          install:
            spec:
              deployments:
              - name: memcached-operator
                spec:
                  ...
                  template:
                    ...
                    spec:
                      containers:
                      ...
                      - name: scorecard-proxy 1
                        command:
                        - scorecard-proxy
                        env:
                        - name: WATCH_NAMESPACE
                          valueFrom:
                            fieldRef:
                              apiVersion: v1
                              fieldPath: metadata.namespace
                        image: quay.io/operator-framework/scorecard-proxy:master
                        imagePullPolicy: Always
                        ports:
                        - name: proxy
                          containerPort: 8889
        1
        Scorecard proxy container.
    • Automated method:

      The community-operators repository has several bash functions that can perform the previous steps in the procedure for you.

      1. Run the following curl command:

        $ curl -Lo csv-manifest-modifiers.sh \
            https://raw.githubusercontent.com/operator-framework/community-operators/master/scripts/lib/file
      2. Source the csv-manifest-modifiers.sh file:

        $ . ./csv-manifest-modifiers.sh
      3. Create the kubeconfig secret file:

        $ create_kubeconfig_secret_file scorecard-secret.yaml "<namespace>" 1
        1
        Replace <namespace> with the namespace your Operator will deploy in.
      4. Apply the secret:

        $ oc apply -f scorecard-secret.yaml
      5. Insert the kubeconfig volume:

        $ insert_kubeconfig_volume "<csv_file>" 1
        1
        Replace <csv_file> with the path to your CSV manifest.
      6. Insert the kubeconfig secret mount:

        $ insert_kubeconfig_secret_mount "<csv_file>"
      7. Insert the proxy container:

        $ insert_proxy_container "<csv_file>" "quay.io/operator-framework/scorecard-proxy:master"
  2. After inserting the proxy container, follow the steps in the Getting started with the Operator SDK guide to bundle your CSV and custom resource definitions (CRDs) and deploy your Operator on OLM.
  3. After your Operator has been deployed on OLM, define a .osdk-scorecard.yaml configuration file in your Operator project and ensure both the csv-path: <csv_manifest_path> and olm-deployed options are set.
  4. Run the scorecard with both the csv-path: <csv_manifest_path> and olm-deployed options set in your scorecard configuration file:

    $ operator-sdk scorecard

4.7. Configuring built-in monitoring with Prometheus

This guide describes the built-in monitoring support provided by the Operator SDK using the Prometheus Operator and details usage for Operator authors.

4.7.1. Prometheus Operator support

Prometheus is an open-source systems monitoring and alerting toolkit. The Prometheus Operator creates, configures, and manages Prometheus clusters running on Kubernetes-based clusters, such as OpenShift Container Platform.

Helper functions exist in the Operator SDK by default to automatically set up metrics in any generated Go-based Operator for use on clusters where the Prometheus Operator is deployed.

4.7.2. Metrics helper

In Go-based Operators generated using the Operator SDK, the following function exposes general metrics about the running program:

func ExposeMetricsPort(ctx context.Context, port int32) (*v1.Service, error)

These metrics are inherited from the controller-runtime library API. By default, the metrics are served on 0.0.0.0:8383/metrics.

A Service object is created with the metrics port exposed, which can be then accessed by Prometheus. The Service object is garbage collected when the leader pod’s root owner is deleted.

The following example is present in the cmd/manager/main.go file in all Operators generated using the Operator SDK:

import(
    "github.com/operator-framework/operator-sdk/pkg/metrics"
    "machine.openshift.io/controller-runtime/pkg/manager"
)

var (
    // Change the below variables to serve metrics on a different host or port.
    metricsHost       = "0.0.0.0" 1
    metricsPort int32 = 8383 2
)
...
func main() {
    ...
    // Pass metrics address to controller-runtime manager
    mgr, err := manager.New(cfg, manager.Options{
        Namespace:          namespace,
        MetricsBindAddress: fmt.Sprintf("%s:%d", metricsHost, metricsPort),
    })

    ...
    // Create Service object to expose the metrics port.
    _, err = metrics.ExposeMetricsPort(ctx, metricsPort)
    if err != nil {
        // handle error
        log.Info(err.Error())
    }
    ...
}
1
The host that the metrics are exposed on.
2
The port that the metrics are exposed on.

4.7.2.1. Modifying the metrics port

Operator authors can modify the port that metrics are exposed on.

Prerequisites

  • Go-based Operator generated using the Operator SDK
  • Kubernetes-based cluster with the Prometheus Operator deployed

Procedure

  • In the cmd/manager/main.go file of the generated Operator, change the value of metricsPort in the following line:

    var metricsPort int32 = 8383

4.7.3. Service monitors

A ServiceMonitor is a custom resource provided by the Prometheus Operator that discovers the Endpoints in Service objects and configures Prometheus to monitor those pods.

In Go-based Operators generated using the Operator SDK, the GenerateServiceMonitor() helper function can take a Service object and generate a ServiceMonitor object based on it.

Additional resources

4.7.3.1. Creating service monitors

Operator authors can add service target discovery of created monitoring services using the metrics.CreateServiceMonitor() helper function, which accepts the newly created service.

Prerequisites

  • Go-based Operator generated using the Operator SDK
  • Kubernetes-based cluster with the Prometheus Operator deployed

Procedure

  • Add the metrics.CreateServiceMonitor() helper function to your Operator code:

    import(
        "k8s.io/api/core/v1"
        "github.com/operator-framework/operator-sdk/pkg/metrics"
        "machine.openshift.io/controller-runtime/pkg/client/config"
    )
    func main() {
    
        ...
        // Populate below with the Service(s) for which you want to create ServiceMonitors.
        services := []*v1.Service{}
        // Create one ServiceMonitor per application per namespace.
        // Change the below value to name of the Namespace you want the ServiceMonitor to be created in.
        ns := "default"
        // restConfig is used for talking to the Kubernetes apiserver
        restConfig := config.GetConfig()
    
        // Pass the Service(s) to the helper function, which in turn returns the array of ServiceMonitor objects.
        serviceMonitors, err := metrics.CreateServiceMonitors(restConfig, ns, services)
        if err != nil {
            // Handle errors here.
        }
        ...
    }

4.8. Configuring leader election

During the lifecycle of an Operator, it is possible that there may be more than one instance running at any given time, for example when rolling out an upgrade for the Operator. In such a scenario, it is necessary to avoid contention between multiple Operator instances using leader election. This ensures only one leader instance handles the reconciliation while the other instances are inactive but ready to take over when the leader steps down.

There are two different leader election implementations to choose from, each with its own trade-off:

Leader-for-life
The leader pod only gives up leadership, using garbage collection, when it is deleted. This implementation precludes the possibility of two instances mistakenly running as leaders, a state also known as split brain. However, this method can be subject to a delay in electing a new leader. For example, when the leader pod is on an unresponsive or partitioned node, the pod-eviction-timeout dictates long how it takes for the leader pod to be deleted from the node and step down, with a default of 5m. See the Leader-for-life Go documentation for more.
Leader-with-lease
The leader pod periodically renews the leader lease and gives up leadership when it cannot renew the lease. This implementation allows for a faster transition to a new leader when the existing leader is isolated, but there is a possibility of split brain in certain situations. See the Leader-with-lease Go documentation for more.

By default, the Operator SDK enables the Leader-for-life implementation. Consult the related Go documentation for both approaches to consider the trade-offs that make sense for your use case.

The following examples illustrate how to use the two options.

4.8.1. Using Leader-for-life election

With the Leader-for-life election implementation, a call to leader.Become() blocks the Operator as it retries until it can become the leader by creating the config map named memcached-operator-lock:

import (
  ...
  "github.com/operator-framework/operator-sdk/pkg/leader"
)

func main() {
  ...
  err = leader.Become(context.TODO(), "memcached-operator-lock")
  if err != nil {
    log.Error(err, "Failed to retry for leader lock")
    os.Exit(1)
  }
  ...
}

If the Operator is not running inside a cluster, leader.Become() simply returns without error to skip the leader election since it cannot detect the name of the Operator.

4.8.2. Using Leader-with-lease election

The Leader-with-lease implementation can be enabled using the Manager Options for leader election:

import (
  ...
  "sigs.k8s.io/controller-runtime/pkg/manager"
)

func main() {
  ...
  opts := manager.Options{
    ...
    LeaderElection: true,
    LeaderElectionID: "memcached-operator-lock"
  }
  mgr, err := manager.New(cfg, opts)
  ...
}

When the Operator is not running in a cluster, the Manager returns an error when starting because it cannot detect the namespace of the Operator in order to create the config map for leader election. You can override this namespace by setting the LeaderElectionNamespace option for the Manager.

4.9. Operator SDK CLI reference

This guide documents the Operator SDK CLI commands and their syntax:

$ operator-sdk <command> [<subcommand>] [<argument>] [<flags>]

4.9.1. build

The operator-sdk build command compiles the code and builds the executables. After build completes, the image is built using a local container engine. It must then be pushed to a remote registry.

Table 4.18. build arguments

ArgumentDescription

<image>

The container image to be built, for example quay.io/example/operator:v0.0.1.

Table 4.19. build flags

FlagDescription

--enable-tests (bool)

Enable in-cluster testing by adding test binary to the image.

--namespaced-manifest (string)

Path of namespaced resources manifest for tests. Default: deploy/operator.yaml.

--test-location (string)

Location of tests. Default: ./test/e2e.

-h, --help

Usage help output.

If --enable-tests is set, the build command also builds the testing binary, adds it to the container image, and generates a deploy/test-pod.yaml file that allows a user to run the tests as a pod on a cluster.

For example:

$ operator-sdk build quay.io/example/operator:v0.0.1

Example output

building example-operator...

building container quay.io/example/operator:v0.0.1...
Sending build context to Docker daemon  163.9MB
Step 1/4 : FROM alpine:3.6
 ---> 77144d8c6bdc
Step 2/4 : ADD tmp/_output/bin/example-operator /usr/local/bin/example-operator
 ---> 2ada0d6ca93c
Step 3/4 : RUN adduser -D example-operator
 ---> Running in 34b4bb507c14
Removing intermediate container 34b4bb507c14
 ---> c671ec1cff03
Step 4/4 : USER example-operator
 ---> Running in bd336926317c
Removing intermediate container bd336926317c
 ---> d6b58a0fcb8c
Successfully built d6b58a0fcb8c
Successfully tagged quay.io/example/operator:v0.0.1

4.9.2. completion

The operator-sdk completion command generates shell completions to make issuing CLI commands quicker and easier.

Table 4.20. completion subcommands

SubcommandDescription

bash

Generate bash completions.

zsh

Generate zsh completions.

Table 4.21. completion flags

FlagDescription

-h, --help

Usage help output.

For example:

$ operator-sdk completion bash

Example output

# bash completion for operator-sdk                         -*- shell-script -*-
...
# ex: ts=4 sw=4 et filetype=sh

4.9.3. print-deps

The operator-sdk print-deps command prints the most recent Golang packages and versions required by Operators. It prints in columnar format by default.

Table 4.22. print-deps flags

FlagDescription

--as-file

Print packages and versions in Gopkg.toml format.

For example:

$ operator-sdk print-deps --as-file

Example output

required = [
  "k8s.io/code-generator/cmd/defaulter-gen",
  "k8s.io/code-generator/cmd/deepcopy-gen",
  "k8s.io/code-generator/cmd/conversion-gen",
  "k8s.io/code-generator/cmd/client-gen",
  "k8s.io/code-generator/cmd/lister-gen",
  "k8s.io/code-generator/cmd/informer-gen",
  "k8s.io/code-generator/cmd/openapi-gen",
  "k8s.io/gengo/args",
]

[[override]]
  name = "k8s.io/code-generator"
  revision = "6702109cc68eb6fe6350b83e14407c8d7309fd1a"
...

4.9.4. generate

The operator-sdk generate command invokes a specific generator to generate code as needed.

4.9.4.1. crds

The generate crds subcommand generates custom resource definitions (CRDs) or updates them if they exist, under deploy/crds/__crd.yaml. OpenAPI V3 validation YAML is generated as a validation object.

Table 4.23. generate crds flags

FlagDescription

--crd-version (string)

CRD version to generate. Default: v1beta1

-h, --help

Help for generate crds

For example:

$ operator-sdk generate crds
$ tree deploy/crds

Example output

├── deploy/crds/app.example.com_v1alpha1_appservice_cr.yaml
└── deploy/crds/app.example.com_appservices_crd.yaml

4.9.4.2. csv

The csv subcommand writes a cluster service version (CSV) manifest for use with Operator Lifecycle Manager (OLM). It also optionally writes CRD files to deploy/olm-catalog/<operator_name>/<csv_version>.

Table 4.24. generate csv flags

FlagDescription

--csv-channel (string)

The channel the CSV should be registered under in the package manifest.

--csv-config (string)

The path to the CSV configuration file. Default: deploy/olm-catalog/csv-config.yaml.

--csv-version (string)

The semantic version of the CSV manifest. Required.

--default-channel

Use the channel passed to --csv-channel as the default channel of the package manifests. Only valid when --csv-channel is set.

--from-version (string)

The semantic version of CSV manifest to use as a base for a new version.

--operator-name

The Operator name to use while generating the CSV.

--update-crds

Updates CRD manifests in deploy/<operator_name>/<csv_version> using the latest CRD manifests.

For example:

$ operator-sdk generate csv \
    --csv-version 0.1.0 \
		--update-crds

Example output

INFO[0000] Generating CSV manifest version 0.1.0
INFO[0000] Fill in the following required fields in file deploy/olm-catalog/operator-name/0.1.0/operator-name.v0.1.0.clusterserviceversion.yaml:
	spec.keywords
	spec.maintainers
	spec.provider
	spec.labels
INFO[0000] Created deploy/olm-catalog/operator-name/0.1.0/operator-name.v0.1.0.clusterserviceversion.yaml

4.9.4.3. k8s

The k8s subcommand runs the Kubernetes code-generators for all CRD APIs under pkg/apis/. Currently, k8s only runs deepcopy-gen to generate the required DeepCopy() functions for all custom resource (CR) types.

Note

This command must be run every time the API (spec and status) for a custom resource type is updated.

For example:

$ tree pkg/apis/app/v1alpha1/

Example output

pkg/apis/app/v1alpha1/
├── appservice_types.go
├── doc.go
└── register.go

$ operator-sdk generate k8s

Example output

Running code-generation for Custom Resource (CR) group versions: [app:v1alpha1]
Generating deepcopy funcs

$ tree pkg/apis/app/v1alpha1/

Example output

pkg/apis/app/v1alpha1/
├── appservice_types.go
├── doc.go
├── register.go
└── zz_generated.deepcopy.go

4.9.5. new

The operator-sdk new command creates a new Operator application and generates (or scaffolds) a default project directory layout based on the input <project_name>.

Table 4.25. new arguments

ArgumentDescription

<project_name>

Name of the new project.

Table 4.26. new flags

FlagDescription

--api-version

CRD API version in the format <group_name>/<version>, for example app.example.com/v1alpha1. Used with ansible or helm types.

--generate-playbook

Generate an Ansible playbook skeleton. Used with ansible type.

--header-file <string>

Path to file containing headers for generated Go files. Copied to hack/boilerplate.go.txt.

--helm-chart <string>

Initialize Helm Operator with existing Helm chart: <url>, <repo>/<name>, or local path.

--helm-chart-repo <string>

Chart repository URL for the requested Helm chart.

--helm-chart-version <string>

Specific version of the Helm chart. Default: latest version.

--help, -h

Usage and help output.

--kind <string>

CRD kind, for example AppService. Used with ansible or helm types.

--skip-git-init

Do not initialize the directory as a Git repository.

--type

Type of Operator to initialize: go, ansible or helm. Default: go.

Note

Starting with Operator SDK v0.12.0, the --dep-manager flag and support for dep-based projects have been removed. Go projects are now scaffolded to use Go modules.

Example usage for Go project

$ mkdir $GOPATH/src/github.com/example.com/

$ cd $GOPATH/src/github.com/example.com/
$ operator-sdk new app-operator

Example usage for Ansible project

$ operator-sdk new app-operator \
    --type=ansible \
    --api-version=app.example.com/v1alpha1 \
    --kind=AppService

4.9.6. add

The operator-sdk add command adds a controller or resource to the project. The command must be run from the Operator project root directory.

Table 4.27. add subcommands

SubcommandDescription

api

Adds a new API definition for a new custom resource (CR) under pkg/apis and generates the custom resource definition (CRD) and CR files under deploy/crds/. If the API already exists at pkg/apis/<group>/<version>, then the command does not overwrite and returns an error.

controller

Adds a new controller under pkg/controller/<kind>/. The controller expects to use the CR type that should already be defined under pkg/apis/<group>/<version> via the operator-sdk add api --kind=<kind> --api-version=<group/version> command. If the controller package for that kind already exists at pkg/controller/<kind>, then the command does not overwrite and returns an error.

crd

Adds a CRD and the CR files. The <project_name>/deploy path must already exist. The --api-version and --kind flags are required to generate the new Operator application.

  • Generated CRD filename: <project_name>/deploy/crds/<group>_<version>_<kind>_crd.yaml
  • Generated CR filename: <project_name>/deploy/crds/<group>_<version>_<kind>_cr.yaml

Table 4.28. add api flags

FlagDescription

--api-version (string)

CRD API version in the format <group_name>/<version>, for example app.example.com/v1alpha1.

--kind (string)

CRD Kind (e.g., AppService).

For example:

$ operator-sdk add api \
    --api-version app.example.com/v1alpha1 \
    --kind AppService

Example output

Create pkg/apis/app/v1alpha1/appservice_types.go
Create pkg/apis/addtoscheme_app_v1alpha1.go
Create pkg/apis/app/v1alpha1/register.go
Create pkg/apis/app/v1alpha1/doc.go
Create deploy/crds/app_v1alpha1_appservice_cr.yaml
Create deploy/crds/app_v1alpha1_appservice_crd.yaml
Running code-generation for Custom Resource (CR) group versions: [app:v1alpha1]
Generating deepcopy funcs

$ tree pkg/apis

Example output

pkg/apis/
├── addtoscheme_app_appservice.go
├── apis.go
└── app
    └── v1alpha1
        ├── doc.go
        ├── register.go
        └── types.go

$ operator-sdk add controller \
    --api-version app.example.com/v1alpha1 \
    --kind AppService

Example output

Create pkg/controller/appservice/appservice_controller.go
Create pkg/controller/add_appservice.go

$ tree pkg/controller

Example output

pkg/controller/
├── add_appservice.go
├── appservice
│   └── appservice_controller.go
└── controller.go

$ operator-sdk add crd \
    --api-version app.example.com/v1alpha1 \
    --kind AppService

Example output

Generating Custom Resource Definition (CRD) files
Create deploy/crds/app_v1alpha1_appservice_crd.yaml
Create deploy/crds/app_v1alpha1_appservice_cr.yaml

4.9.7. test

The operator-sdk test command can test the Operator locally.

4.9.7.1. local

The local subcommand runs Go tests built using the test framework of the Operator SDK locally.

Table 4.29. test local arguments

ArgumentsDescription

<test_location> (string)

Location of end-to-end (e2e) test files, for example ./test/e2e/.

Table 4.30. test local flags

FlagsDescription

--kubeconfig (string)

Location of kubeconfig for a cluster. Default: ~/.kube/config.

--global-manifest (string)

Path to manifest for global resources. Default: deploy/crd.yaml.

--namespaced-manifest (string)

Path to manifest for per-test, namespaced resources. Default: combines deploy/service_account.yaml, deploy/rbac.yaml, and deploy/operator.yaml.

--namespace (string)

If non-empty, a single namespace to run tests in, for example operator-test. Default: "".

--go-test-flags (string)

Extra arguments to pass to go test, for example -f "-v -parallel=2".

--up-local

Enable running the Operator locally with go run instead of as an image in the cluster.

--no-setup

Disable test resource creation.

--image (string)

Use a different Operator image from the one specified in the namespaced manifest.

-h, --help

Usage help output.

For example:

$ operator-sdk test local ./test/e2e/

Example output

ok  	github.com/operator-framework/operator-sdk-samples/memcached-operator/test/e2e	20.410s

4.9.8. run

The operator-sdk run command provides options that can launch the Operator in various environments.

Table 4.31. run arguments

ArgumentsDescription

--kubeconfig (string)

The file path to a Kubernetes configuration file. Default: $HOME/.kube/config

--local

The Operator is run locally by building the Operator binary with the ability to access a Kubernetes cluster using a kubeconfig file.

--namespace (string)

The namespace where the Operator watches for changes. Default: default

--operator-flags

Flags that the local Operator might require. Example: --flag1 value1 --flag2=value2. For use with the --local flag only.

-h, --help

Usage help output.

4.9.8.1. --local

The --local flag launches the Operator on the local machine by building the Operator binary with the ability to access a Kubernetes cluster using a kubeconfig file.

For example:

$ operator-sdk run --local \
  --kubeconfig "mycluster.kubecfg" \
  --namespace "default" \
  --operator-flags "--flag1 value1 --flag2=value2"

The following example uses the default kubeconfig, the default namespace environment variable, and passes in flags for the Operator. To use the Operator flags, your Operator must know how to handle the option. For example, for an Operator that understands the resync-interval flag:

$ operator-sdk run --local --operator-flags "--resync-interval 10"

If you are planning on using a different namespace than the default, use the --namespace flag to change where the Operator is watching for custom resources (CRs) to be created:

$ operator-sdk run --local --namespace "testing"

For this to work, your Operator must handle the WATCH_NAMESPACE environment variable. This can be accomplished using the utility function k8sutil.GetWatchNamespace in your Operator.

4.10. Appendices

4.10.1. Operator project scaffolding layout

The operator-sdk CLI generates a number of packages for each Operator project. The following sections describes a basic rundown of each generated file and directory.

4.10.1.1. Go-based projects

Go-based Operator projects (the default type) generated using the operator-sdk new command contain the following directories and files:

File/foldersPurpose

cmd/

Contains manager/main.go file, which is the main program of the Operator. This instantiates a new manager which registers all custom resource defintitions (CRDs) under pkg/apis/ and starts all controllers under pkg/controllers/.

pkg/apis/

Contains the directory tree that defines the APIs of the CRDs. Users are expected to edit the pkg/apis/<group>/<version>/<kind>_types.go files to define the API for each resource type and import these packages in their controllers to watch for these resource types.

pkg/controller

This pkg contains the controller implementations. Users are expected to edit the pkg/controller/<kind>/<kind>_controller.go files to define the reconcile logic of the controller for handling a resource type of the specified kind.

build/

Contains the Dockerfile and build scripts used to build the Operator.

deploy/

Contains various YAML manifests for registering CRDs, setting up RBAC, and deploying the Operator as a deployment.

Gopkg.toml
Gopkg.lock

The Go Dep manifests that describe the external dependencies of this Operator.

vendor/

The Golang vendor folder that contains the local copies of the external dependencies that satisfy the imports of this project. Go Dep manages the vendor directly.

4.10.1.2. Helm-based projects

Helm-based Operator projects generated using the operator-sdk new --type helm command contain the following directories and files:

File/foldersPurpose

deploy/

Contains various YAML manifests for registering CRDs, setting up RBAC, and deploying the Operator as a Deployment.

helm-charts/<kind>

Contains a Helm chart initialized using the equivalent of the helm create command.

build/

Contains the Dockerfile and build scripts used to build the Operator.

watches.yaml

Contains group/version/kind (GVK) and Helm chart location.