4.3. Creating Helm-based Operators
This guide outlines Helm chart support in the Operator SDK and walks Operator authors through an example of building and running an Nginx Operator with the operator-sdk CLI tool that uses an existing Helm chart.
4.3.1. Helm chart support in the Operator SDK
The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way. This framework includes the Operator SDK, which assists developers in bootstrapping and building an Operator based on their expertise without requiring knowledge of Kubernetes API complexities.
One of the Operator SDK options for generating an Operator project includes leveraging an existing Helm chart to deploy Kubernetes resources as a unified application, without having to write any Go code. Such Helm-based Operators are designed to excel at stateless applications that require very little logic when rolled out, because changes should be applied to the Kubernetes objects that are generated as part of the chart. This may sound limiting, but can be sufficient for a surprising amount of use-cases as shown by the proliferation of Helm charts built by the Kubernetes community.
The main function of an Operator is to read from a custom object that represents your application instance and have its desired state match what is running. In the case of a Helm-based Operator, the spec field of the object is a list of configuration options that are typically described in the Helm values.yaml file. Instead of setting these values with flags using the Helm CLI (for example, helm install -f values.yaml), you can express them within a custom resource (CR), which, as a native Kubernetes object, enables the benefits of RBAC applied to it and an audit trail.
For an example of a simple CR called Tomcat:
apiVersion: apache.org/v1alpha1 kind: Tomcat metadata: name: example-app spec: replicaCount: 2
The replicaCount value, 2 in this case, is propagated into the template of the chart where the following is used:
{{ .Values.replicaCount }}
After an Operator is built and deployed, you can deploy a new instance of an app by creating a new instance of a CR, or list the different instances running in all environments using the oc command:
$ oc get Tomcats --all-namespaces
There is no requirement use the Helm CLI or install Tiller; Helm-based Operators import code from the Helm project. All you have to do is have an instance of the Operator running and register the CR with a custom resource definition (CRD). Because it obeys RBAC, you can more easily prevent production changes.
4.3.2. Installing the Operator SDK CLI
The Operator SDK has a CLI tool that assists developers in creating, building, and deploying a new Operator project. You can install the SDK CLI on your workstation so you are prepared to start authoring your own Operators.
4.3.2.1. Installing from GitHub release
You can download and install a pre-built release binary of the Operator SDK CLI from the project on GitHub.
Prerequisites
- Go v1.13+
-
dockerv17.03+,podmanv1.2.0+, orbuildahv1.7+ -
OpenShift CLI (
oc) v4.5+ installed - Access to a cluster based on Kubernetes v1.12.0+
- Access to a container registry
Procedure
Set the release version variable:
$ RELEASE_VERSION=v0.17.2
Download the release binary.
For Linux:
$ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnuFor macOS:
$ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin
Verify the downloaded release binary.
Download the provided
.ascfile.For Linux:
$ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu.ascFor macOS:
$ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin.asc
Place the binary and corresponding
.ascfile into the same directory and run the following command to verify the binary:For Linux:
$ gpg --verify operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu.ascFor macOS:
$ gpg --verify operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin.asc
If you do not have the public key of the maintainer on your workstation, you will get the following error:
Example output with error
$ gpg: assuming signed data in 'operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin' $ gpg: Signature made Fri Apr 5 20:03:22 2019 CEST $ gpg: using RSA key <key_id> 1 $ gpg: Can't check signature: No public key- 1
- RSA key string.
To download the key, run the following command, replacing
<key_id>with the RSA key string provided in the output of the previous command:$ gpg [--keyserver keys.gnupg.net] --recv-key "<key_id>" 1- 1
- If you do not have a key server configured, specify one with the
--keyserveroption.
Install the release binary in your
PATH:For Linux:
$ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu$ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu /usr/local/bin/operator-sdk$ rm operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnuFor macOS:
$ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin$ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin /usr/local/bin/operator-sdk$ rm operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin
Verify that the CLI tool was installed correctly:
$ operator-sdk version
4.3.2.2. Installing from Homebrew
You can install the SDK CLI using Homebrew.
Prerequisites
- Homebrew
-
dockerv17.03+,podmanv1.2.0+, orbuildahv1.7+ -
OpenShift CLI (
oc) v4.5+ installed - Access to a cluster based on Kubernetes v1.12.0+
- Access to a container registry
Procedure
Install the SDK CLI using the
brewcommand:$ brew install operator-sdk
Verify that the CLI tool was installed correctly:
$ operator-sdk version
4.3.2.3. Compiling and installing from source
You can obtain the Operator SDK source code to compile and install the SDK CLI.
Prerequisites
Procedure
Clone the
operator-sdkrepository:$ mkdir -p $GOPATH/src/github.com/operator-framework
$ cd $GOPATH/src/github.com/operator-framework
$ git clone https://github.com/operator-framework/operator-sdk
$ cd operator-sdk
Check out the desired release branch:
$ git checkout master
Compile and install the SDK CLI:
$ make dep
$ make install
This installs the CLI binary
operator-sdkat $GOPATH/bin.Verify that the CLI tool was installed correctly:
$ operator-sdk version
4.3.3. Building a Helm-based Operator using the Operator SDK
This procedure walks through an example of building a simple Nginx Operator powered by a Helm chart using tools and libraries provided by the Operator SDK.
It is best practice to build a new Operator for each chart. This can allow for more native-behaving Kubernetes APIs (for example, oc get Nginx) and flexibility if you ever want to write a fully-fledged Operator in Go, migrating away from a Helm-based Operator.
Prerequisites
- Operator SDK CLI installed on the development workstation
-
Access to a Kubernetes-based cluster v1.11.3+ (for example OpenShift Container Platform 4.5) using an account with
cluster-adminpermissions -
OpenShift CLI (
oc) v4.5+ installed
Procedure
Create a new Operator project. A namespace-scoped Operator watches and manages resources in a single namespace. Namespace-scoped Operators are preferred because of their flexibility. They enable decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions.
To create a new Helm-based, namespace-scoped
nginx-operatorproject, use the following command:$ operator-sdk new nginx-operator \ --api-version=example.com/v1alpha1 \ --kind=Nginx \ --type=helm
$ cd nginx-operator
This creates the
nginx-operatorproject specifically for watching the Nginx resource with API versionexample.com/v1apha1and kindNginx.Customize the Operator logic.
For this example, the
nginx-operatorexecutes the following reconciliation logic for eachNginxcustom resource (CR):- Create an Nginx deployment if it does not exist.
- Create an Nginx service if it does not exist.
- Create an Nginx ingress if it is enabled and does not exist.
- Ensure that the deployment, service, and optional ingress match the desired configuration (for example, replica count, image, service type) as specified by the Nginx CR.
By default, the
nginx-operatorwatchesNginxresource events as shown in thewatches.yamlfile and executes Helm releases using the specified chart:- version: v1alpha1 group: example.com kind: Nginx chart: /opt/helm/helm-charts/nginx
Review the Nginx Helm chart.
When a Helm Operator project is created, the Operator SDK creates an example Helm chart that contains a set of templates for a simple Nginx release.
For this example, templates are available for deployment, service, and ingress resources, along with a
NOTES.txttemplate, which Helm chart developers use to convey helpful information about a release.If you are not already familiar with Helm Charts, review the Helm Chart developer documentation.
Understand the Nginx CR spec.
Helm uses a concept called values to provide customizations to the defaults of a Helm chart, which are defined in the
values.yamlfile.Override these defaults by setting the desired values in the CR spec. You can use the number of replicas as an example:
First, inspect the
helm-charts/nginx/values.yamlfile to find that the chart has a value calledreplicaCountand it is set to1by default. To have 2 Nginx instances in your deployment, your CR spec must containreplicaCount: 2.Update the
deploy/crds/example.com_v1alpha1_nginx_cr.yamlfile to look like the following:apiVersion: example.com/v1alpha1 kind: Nginx metadata: name: example-nginx spec: replicaCount: 2
Similarly, the default service port is set to
80. To instead use8080, update thedeploy/crds/example.com_v1alpha1_nginx_cr.yamlfile again by adding the service port override:apiVersion: example.com/v1alpha1 kind: Nginx metadata: name: example-nginx spec: replicaCount: 2 service: port: 8080The Helm Operator applies the entire spec as if it was the contents of a values file, just like the
helm install -f ./overrides.yamlcommand works.
Deploy the CRD.
Before running the Operator, Kubernetes must know about the new custom resource definition (CRD) that the Operator will be watching. Deploy the following CRD:
$ oc create -f deploy/crds/example_v1alpha1_nginx_crd.yaml
Build and run the Operator.
There are two ways to build and run the Operator:
- As a pod inside a Kubernetes cluster.
-
As a Go program outside the cluster using the
operator-sdk upcommand.
Choose one of the following methods:
Run as a pod inside a Kubernetes cluster. This is the preferred method for production use.
Build the
nginx-operatorimage and push it to a registry:$ operator-sdk build quay.io/example/nginx-operator:v0.0.1
$ podman push quay.io/example/nginx-operator:v0.0.1
Deployment manifests are generated in the
deploy/operator.yamlfile. The deployment image in this file needs to be modified from the placeholderREPLACE_IMAGEto the previous built image. To do this, run:$ sed -i 's|REPLACE_IMAGE|quay.io/example/nginx-operator:v0.0.1|g' deploy/operator.yaml
Deploy the
nginx-operatormanifests:$ oc create -f deploy/service_account.yaml
$ oc create -f deploy/role.yaml
$ oc create -f deploy/role_binding.yaml
$ oc create -f deploy/operator.yaml
Verify that the
nginx-operatordeployment is up and running:$ oc get deployment
Example output
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-operator 1 1 1 1 1m
Run outside the cluster. This method is preferred during the development cycle to speed up deployment and testing.
It is important that the chart path referenced in the
watches.yamlfile exists on your machine. By default, thewatches.yamlfile is scaffolded to work with an Operator image built with theoperator-sdk buildcommand. When developing and testing your Operator with theoperator-sdk run --localcommand, the SDK looks in your local file system for this path.Create a symlink at this location to point to the path of your Helm chart:
$ sudo mkdir -p /opt/helm/helm-charts
$ sudo ln -s $PWD/helm-charts/nginx /opt/helm/helm-charts/nginx
To run the Operator locally with the default Kubernetes configuration file present at
$HOME/.kube/config:$ operator-sdk run --local
To run the Operator locally with a provided Kubernetes configuration file:
$ operator-sdk run --local --kubeconfig=<path_to_config>
Deploy the
NginxCR.Apply the
NginxCR that you modified earlier:$ oc apply -f deploy/crds/example.com_v1alpha1_nginx_cr.yaml
Ensure that the
nginx-operatorcreates the deployment for the CR:$ oc get deployment
Example output
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1 2 2 2 2 1m
Check the pods to confirm two replicas were created:
$ oc get pods
Example output
NAME READY STATUS RESTARTS AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1-f8f9c875d-fjcr9 1/1 Running 0 1m example-nginx-b9phnoz9spckcrua7ihrbkrt1-f8f9c875d-ljbzl 1/1 Running 0 1m
Check that the service port is set to
8080:$ oc get service
Example output
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1 ClusterIP 10.96.26.3 <none> 8080/TCP 1m
Update the
replicaCountand remove the port.Change the
spec.replicaCountfield from2to3, remove thespec.servicefield, and apply the change:$ cat deploy/crds/example.com_v1alpha1_nginx_cr.yaml
Example output
apiVersion: "example.com/v1alpha1" kind: "Nginx" metadata: name: "example-nginx" spec: replicaCount: 3
$ oc apply -f deploy/crds/example.com_v1alpha1_nginx_cr.yaml
Confirm that the Operator changes the deployment size:
$ oc get deployment
Example output
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1 3 3 3 3 1m
Check that the service port is set to the default
80:$ oc get service
Example output
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1 ClusterIP 10.96.26.3 <none> 80/TCP 1m
Clean up the resources:
$ oc delete -f deploy/crds/example.com_v1alpha1_nginx_cr.yaml
$ oc delete -f deploy/operator.yaml
$ oc delete -f deploy/role_binding.yaml
$ oc delete -f deploy/role.yaml
$ oc delete -f deploy/service_account.yaml
$ oc delete -f deploy/crds/example_v1alpha1_nginx_crd.yaml
4.3.4. Additional resources
- See Appendices to learn about the project directory structures created by the Operator SDK.
- Operator Development Guide for Red Hat Partners