Chapter 4. GitOps

4.1. Red Hat OpenShift GitOps release notes

Red Hat OpenShift GitOps is a declarative way to implement continuous deployment for cloud native applications. Red Hat OpenShift GitOps ensures consistency in applications when you deploy them to different clusters in different environments, such as: development, staging, and production. Red Hat OpenShift GitOps helps you automate the following tasks:

  • Ensure that the clusters have similar states for configuration, monitoring, and storage
  • Recover or recreate clusters from a known state
  • Apply or revert configuration changes to multiple OpenShift Container Platform clusters
  • Associate templated configuration with different environments
  • Promote applications across clusters, from staging to production

For an overview of Red Hat OpenShift GitOps, see Understanding OpenShift GitOps.

4.1.1. Compatibility and support matrix

Some features in this release are currently in Technology Preview. These experimental features are not intended for production use.

In the table, features are marked with the following statuses:

  • TP: Technology Preview
  • GA: General Availability
OpenShift GitOpsComponent VersionsOpenShift Versions

Version

kam

Helm

Kustomize

Argo CD

ApplicationSet

Dex

RH SSO

 

1.4.0

0.0.41 TP

3.7.1 GA

4.2.0 GA

2.2.2 GA

0.2.0 TP

2.30.0 GA

7.4.0 GA

4.7-4.9

1.3.0

0.0.40 TP

3.6.0 GA

4.2.0 GA

2.1.2 GA

0.2.0 TP

2.28.0 GA

7.4.0 GA

4.6-4.9

1.2.0

0.0.38 TP

3.5.0 GA

3.9.4 GA

2.0.5 GA

0.1.0 TP

N/A

7.4.0 GA

4.8

1.1.0

0.0.32 TP

3.5.0 GA

3.9.4 GA

2.0.0 GA

N/A

N/A

N/A

4.7

  • "kam" is an abbreviation for Red Hat OpenShift GitOps Application Manager (kam).
  • "RH SSO" is an abbreviation for Red Hat SSO.
  • The Environments page in the Developer perspective of the OpenShift Container Platform web console is also in Technology Preview.

4.1.2. Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.

4.1.3. Release notes for Red Hat OpenShift GitOps 1.4.0

Red Hat OpenShift GitOps 1.4.0 is now available on OpenShift Container Platform 4.7, 4.8, and 4.9.

4.1.3.1. New features

The current release adds the following improvements.

  • This enhancement upgrades Red Hat OpenShift GitOps Application Manager (kam) to version 0.0.41. GITOPS-1669
  • This enhancement upgrades Argo CD to version 2.2.2. GITOPS-1532
  • This enhancement upgrades Helm to version 3.7.1. GITOPS-1530
  • This enhancement adds the health status of the DeploymentConfig, Route, and OLM Operator items to the Argo CD Dashboard and OpenShift Container Platform web console. This information helps you monitor the overall health status of your application. GITOPS-655, GITOPS-915, GITOPS-916, GITOPS-1110
  • With this update, you can to specify the number of desired replicas for the argocd-server and argocd-repo-server components by setting the .spec.server.replicas and .spec.repo.replicas attributes in the Argo CD custom resource, respectively. If you configure the horizontal pod autoscaler (HPA) for the argocd-server components, it takes precedence over the Argo CD custom resource attributes. GITOPS-1245
  • As an administrative user, when you give Argo CD access to a namespace by using the argocd.argoproj.io/managed-by label, it assumes namespace-admin privileges. These privileges are an issue for administrators who provide namespaces to non-administrators, such as development teams, because the privileges enable non-administrators to modify objects such as network policies.

    With this update, administrators can configure a common cluster role for all the managed namespaces. In role bindings for the Argo CD application controller, the Operator refers to the CONTROLLER_CLUSTER_ROLE environment variable. In role bindings for the Argo CD server, the Operator refers to the SERVER_CLUSTER_ROLE environment variable. If these environment variables contain custom roles, the Operator doesn’t create the default admin role. Instead, it uses the existing custom role for all managed namespaces. GITOPS-1290

  • With this update, the Environment page in the OpenShift Container Platform Developer Console displays a broken heart icon to indicate degraded resources, excluding ones whose status is Progressing, Missing, and Unknown. The console displays a yellow yield sign icon to indicate out-of-sync resources. GITOPS-1307

4.1.3.2. Fixed issues

The following issues have been resolved in the current release:

  • Before this update, when the Route to the Red Hat OpenShift GitOps Application Manager (kam) was accessed without specifying a path in the URL, a default page without any helpful information was displayed to the user. This update fixes the issue so that the default page displays download links for kam. GITOPS-923
  • Before this update, setting a resource quota in the namespace of the Argo CD custom resource might cause the setup of the Red Hat SSO (RH SSO) instance to fail. This update fixes this issue by setting a minimum resource request for the RH SSO deployment pods. GITOPS-1297
  • Before this update, if you changed the log level for the argocd-repo-server workload, the Operator didn’t reconcile this setting. The workaround was to delete the deployment resource so that the Operator recreated it with the new log level. With this update, the log level is correctly reconciled for existing argocd-repo-server workloads. GITOPS-1387
  • Before this update, if the Operator managed an Argo CD instance that lacked the .data field in the argocd-secret Secret, the Operator on that instance crashed. This update fixes the issue so that the Operator doesn’t crash when the .data field is missing. Instead, the secret regenerates and the gitops-operator-controller-manager resource is redeployed. GITOPS-1402
  • Before this update, the gitopsservice service was annotated as an internal object. This update removes the annotation so you can update or delete the default Argo CD instance and run GitOps workloads on infrastructure nodes by using the UI. GITOPS-1429

4.1.3.3. Known issues

These are the known issues in the current release:

  • If you migrate from the Dex authentication provider to the Keycloak provider, you might experience login issues with Keycloak.

    To prevent this issue, when migrating, uninstall Dex by removing the .spec.dex section from the Argo CD custom resource. Allow a few minutes for Dex to uninstall completely. Then, install Keycloak by adding .spec.sso.provider: keycloak to the Argo CD custom resource.

    As a workaround, uninstall Keycloak by removing .spec.sso.provider: keycloak. Then, re-install it. GITOPS-1450, GITOPS-1331

4.1.4. Release notes for Red Hat OpenShift GitOps 1.3.2

Red Hat OpenShift GitOps 1.3.2 is now available on OpenShift Container Platform 4.6, 4.7, 4.8, and 4.9.

4.1.4.1. New features

In addition to the fixes and stability improvements, the following sections highlight what is new in Red Hat OpenShift GitOps 1.3.2:

  • Upgraded Argo CD to version 2.1.8
  • Upgraded Dex to version 2.30.0

4.1.4.2. Fixed issues

The following issues have been resolved in the current release:

  • Previously, in the OperatorHub UI under the Infrastructure Features section, when you filtered by Disconnected the Red Hat OpenShift GitOps Operator did not show in the search results, as the Operator did not have the related annotation set in its CSV file. With this update, the Disconnected Cluster annotation has been added to the Red Hat OpenShift GitOps Operator as an infrastructure feature. GITOPS-1539
  • When using an Namespace-scoped Argo CD instance, for example, an Argo CD instance that is not scoped to All Namepsaces in a cluster, Red Hat OpenShift GitOps dynamically maintains a list of managed namespaces. These namespaces include the argocd.argoproj.io/managed-by label. This list of namespaces is stored in a cache in Argo CD → Settings → Clusters → "in-cluster" → NAMESPACES. Before this update, if you deleted one of these namespaces, the Operator ignored that, and the namespace remained in the list. This behavior broke the CONNECTION STATE in that cluster configuration, and all sync attempts resulted in errors. For example:

    Argo service account does not have <random_verb> on <random_resource_type> in namespace <the_namespace_you_deleted>.

    This bug is fixed. GITOPS-1521

  • With this update, the Red Hat OpenShift GitOps Operator has been annotated with the Deep Insights capability level. GITOPS-1519
  • Previously, the Argo CD Operator managed the resource.exclusion field by itself but ignored the resource.inclusion field. This prevented the resource.inclusion field configured in the Argo CD CR to generate in the argocd-cm configuration map. This bug is fixed. GITOPS-1518

4.1.5. Release notes for Red Hat OpenShift GitOps 1.3.1

Red Hat OpenShift GitOps 1.3.1 is now available on OpenShift Container Platform 4.6, 4.7, 4.8, and 4.9.

4.1.5.1. Fixed issues

  • If you upgrade to v1.3.0, the Operator does not return an ordered slice of environment variables. As a result, the reconciler fails causing the frequent recreation of Argo CD pods in OpenShift Container Platform clusters running behind a proxy. This update fixes the issue so that Argo CD pods are not recreated. GITOPS-1489

4.1.6. Release notes for Red Hat OpenShift GitOps 1.3

Red Hat OpenShift GitOps 1.3 is now available on OpenShift Container Platform 4.7, 4.8, and 4.9.

4.1.6.1. New features

In addition to the fixes and stability improvements, the following sections highlight what is new in Red Hat OpenShift GitOps 1.3.0:

  • For a fresh install of v1.3.0, Dex is automatically configured. You can log into the default Argo CD instance in the openshift-gitops namespace using the OpenShift or kubeadmin credentials. As an admin you can disable the Dex installation after the Operator is installed which will remove the Dex deployment from the openshift-gitops namespace.
  • The default Argo CD instance installed by the Operator as well as accompanying controllers can now run on the infrastructure nodes of the cluster by setting a simple configuration toggle.
  • Internal communications in Argo CD can now be secured using the TLS and the OpenShift cluster certificates. The Argo CD routes can now leverage the OpenShift cluster certificates in addition to using external certificate managers such as the cert-manager.
  • Use the improved environment details view in the Developer perspective of the console 4.9 to gain insights into the GitOps environments.
  • You can now access custom health checks in Argo CD for DeploymentConfig resources, Route resources, and Operators installed using OLM.
  • The GitOps Operator now conforms to the naming conventions recommended by the latest Operator-SDK:

    • The prefix gitops-operator- is added to all resources
    • Service account is renamed to gitops-operator-controller-manager

4.1.6.2. Fixed issues

The following issues were resolved in the current release:

  • Previously, if you set up a new namespace to be managed by a new instance of Argo CD, it would immediately be Out Of Sync due to the new roles and bindings that the Operator creates to manage that new namespace. This behavior is fixed. GITOPS-1384

4.1.6.3. Known issues

  • While migrating from the Dex authentication provider to the Keycloak provider, you may experience login issues with Keycloak. GITOPS-1450

    To prevent the above issue, when migrating, uninstall Dex by removing the .spec.dex section found in the Argo CD custom resource. Allow a few minutes for Dex to uninstall completely, and then proceed to install Keycloak by adding .spec.sso.provider: keycloak to the Argo CD custom resource.

    As a workaround, uninstall Keycloak by removing .spec.sso.provider: keycloak and then re-install.

4.1.7. Release notes for Red Hat OpenShift GitOps 1.2.1

Red Hat OpenShift GitOps 1.2.1 is now available on OpenShift Container Platform 4.7 and 4.8.

4.1.7.1. Support matrix

Some features in this release are currently in Technology Preview. These experimental features are not intended for production use.

Technology Preview Features Support Scope

In the table below, features are marked with the following statuses:

  • TP: Technology Preview
  • GA: General Availability

Note the following scope of support on the Red Hat Customer Portal for these features:

Table 4.1. Support matrix

FeatureRed Hat OpenShift GitOps 1.2.1

Argo CD

GA

Argo CD ApplicationSet

TP

Red Hat OpenShift GitOps Application Manager (kam)

TP

4.1.7.2. Fixed issues

The following issues were resolved in the current release:

  • Previously, huge memory spikes were observed on the application controller on startup. The flag --kubectl-parallelism-limit for the application controller is now set to 10 by default, however this value can be overridden by specifying a number for .spec.controller.kubeParallelismLimit in the Argo CD CR specification. GITOPS-1255
  • The latest Triggers APIs caused Kubernetes build failure due to duplicate entries in the kustomization.yaml when using the kam bootstrap command. The Pipelines and Tekton triggers components have now been updated to v0.24.2 and v0.14.2, respectively, to address this issue. GITOPS-1273
  • Persisting RBAC roles and bindings are now automatically removed from the target namespace when the Argo CD instance from the source namespace is deleted. GITOPS-1228
  • Previously, when deploying an Argo CD instance into a namespace, the Argo CD instance would change the "managed-by" label to be its own namespace. This fix would make namespaces unlabelled while also making sure the required RBAC roles and bindings are created and deleted for the namespace. GITOPS-1247
  • Previously, the default resource request limits on Argo CD workloads, specifically for the repo-server and application controller, were found to be very restrictive. The existing resource quota has now been removed and the default memory limit has been increased to 1024M in the repo server. Please note that this change will only affect new installations; existing Argo CD instance workloads will not be affected. GITOPS-1274

4.1.8. Release notes for Red Hat OpenShift GitOps 1.2

Red Hat OpenShift GitOps 1.2 is now available on OpenShift Container Platform 4.7, 4.8, and 4.9.

4.1.8.1. Support matrix

Some features in this release are currently in Technology Preview. These experimental features are not intended for production use.

Technology Preview Features Support Scope

In the table below, features are marked with the following statuses:

  • TP: Technology Preview
  • GA: General Availability

Note the following scope of support on the Red Hat Customer Portal for these features:

Table 4.2. Support matrix

FeatureRed Hat OpenShift GitOps 1.2

Argo CD

GA

Argo CD ApplicationSet

TP

Red Hat OpenShift GitOps Application Manager (kam)

TP

4.1.8.2. New features

In addition to the fixes and stability improvements, the following sections highlight what is new in Red Hat OpenShift GitOps 1.2:

  • If you do not have read or write access to the openshift-gitops namespace, you can now use the DISABLE_DEFAULT_ARGOCD_INSTANCE environment variable in the GitOps Operator and set the value to TRUE to prevent the default Argo CD instance from starting in the openshift-gitops namespace.
  • Resource requests and limits are now configured in Argo CD workloads. Resource quota is enabled in the openshift-gitops namespace. As a result, out-of-band workloads deployed manually in the openshift-gitops namespace must be configured with resource requests and limits and the resource quota may need to be increased.
  • Argo CD authentication is now integrated with Red Hat SSO and it is automatically configured with OpenShift 4 Identity Provider on the cluster. This feature is disabled by default. To enable Red Hat SSO, add SSO configuration in ArgoCD CR as shown below. Currently,keycloak is the only supported provider.

    apiVersion: argoproj.io/v1alpha1
    kind: ArgoCD
    metadata:
      name: example-argocd
      labels:
        example: basic
    spec:
      sso:
        provider: keycloak
      server:
        route:
         enabled: true
  • You can now define hostnames using route labels to support router sharding. Support for setting labels on the server (argocd server), grafana, and prometheus routes is now available. To set labels on a route, add labels under the route configuration for a server in the ArgoCD CR.

    Example ArgoCD CR YAML to set labels on argocd server

    apiVersion: argoproj.io/v1alpha1
    kind: ArgoCD
    metadata:
      name: example-argocd
      labels:
        example: basic
    spec:
      server:
        route:
         enabled: true
         labels:
           key1: value1
           key2: value2

  • The GitOps Operator now automatically grants permissions to Argo CD instances to manage resources in target namespaces by applying labels. Users can label the target namespace with the label argocd.argoproj.io/managed-by: <source-namespace>, where the source-namespace is the namespace where the argocd instance is deployed.

4.1.8.3. Fixed issues

The following issues were resolved in the current release:

  • Previously, if a user created additional instances of Argo CD managed by the default cluster instance in the openshift-gitops namespace, the application responsible for the new Argo CD instance would get stuck in an OutOfSync status. This issue has now been resolved by adding an owner reference to the cluster secret. GITOPS-1025

4.1.8.4. Known issues

These are the known issues in Red Hat OpenShift GitOps 1.2:

  • When an Argo CD instance is deleted from the source namespace, the argocd.argoproj.io/managed-by labels in the target namespaces are not removed. GITOPS-1228
  • Resource quota has been enabled in the openshift-gitops namespace in Red Hat OpenShift GitOps 1.2. This can affect out-of-band workloads deployed manually and workloads deployed by the default Argo CD instance in the openshift-gitops namespace. When you upgrade from Red Hat OpenShift GitOps v1.1.2 to v1.2 such workloads must be configured with resource requests and limits. If there are any additional workloads, the resource quota in the openshift-gitops namespace must be increased.

    Current Resource Quota for openshift-gitops namespace.

    ResourceRequestsLimits

    CPU

    6688m

    13750m

    Memory

    4544Mi

    9070Mi

    You can use the below command to update the CPU limits.

    $ oc patch resourcequota openshift-gitops-compute-resources -n openshift-gitops --type='json' -p='[{"op": "replace", "path": "/spec/hard/limits.cpu", "value":"9000m"}]'

    You can use the below command to update the CPU requests.

    $ oc patch resourcequota openshift-gitops-compute-resources -n openshift-gitops --type='json' -p='[{"op": "replace", "path": "/spec/hard/cpu", "value":"7000m"}]

    You can replace the path in the above commands from cpu to memory to update the memory.

4.1.9. Release notes for Red Hat OpenShift GitOps 1.1

Red Hat OpenShift GitOps 1.1 is now available on OpenShift Container Platform 4.7.

4.1.9.1. Support matrix

Some features in this release are currently in Technology Preview. These experimental features are not intended for production use.

Technology Preview Features Support Scope

In the table below, features are marked with the following statuses:

  • TP: Technology Preview
  • GA: General Availability

Note the following scope of support on the Red Hat Customer Portal for these features:

Table 4.3. Support matrix

FeatureRed Hat OpenShift GitOps 1.1

Argo CD

GA

Argo CD ApplicationSet

TP

Red Hat OpenShift GitOps Application Manager (kam)

TP

4.1.9.2. New features

In addition to the fixes and stability improvements, the following sections highlight what is new in Red Hat OpenShift GitOps 1.1:

  • The ApplicationSet feature is now added (Technology Preview). The ApplicationSet feature enables both automation and greater flexibility when managing Argo CD applications across a large number of clusters and within monorepos. It also makes self-service usage possible on multitenant Kubernetes clusters.
  • Argo CD is now integrated with cluster logging stack and with the OpenShift Container Platform Monitoring and Alerting features.
  • Argo CD auth is now integrated with OpenShift Container Platform.
  • Argo CD applications controller now supports horizontal scaling.
  • Argo CD Redis servers now support high availability (HA).

4.1.9.3. Fixed issues

The following issues were resolved in the current release:

  • Previously, Red Hat OpenShift GitOps did not work as expected in a proxy server setup with active global proxy settings. This issue is fixed and now Argo CD is configured by the Red Hat OpenShift GitOps Operator using fully qualified domain names (FQDN) for the pods to enable communication between components. GITOPS-703
  • The Red Hat OpenShift GitOps backend relies on the ?ref= query parameter in the Red Hat OpenShift GitOps URL to make API calls. Previously, this parameter was not read from the URL, causing the backend to always consider the default reference. This issue is fixed and the Red Hat OpenShift GitOps backend now extracts the reference query parameter from the Red Hat OpenShift GitOps URL and only uses the default reference when there is no input reference provided. GITOPS-817
  • Previously, the Red Hat OpenShift GitOps backend failed to find the valid GitLab repository. This was because the Red Hat OpenShift GitOps backend checked for main as the branch reference, instead of master in the GitLab repository. This issue is fixed now. GITOPS-768
  • The Environments page in the Developer perspective of the OpenShift Container Platform web console now shows the list of applications and the number of environments. This page also displays an Argo CD link that directs you to the Argo CD Applications page that lists all the applications. The Argo CD Applications page has LABELS (for example, app.kubernetes.io/name=appName) that help you filter only the applications of your choice. GITOPS-544

4.1.9.4. Known issues

These are the known issues in Red Hat OpenShift GitOps 1.1:

  • Red Hat OpenShift GitOps does not support Helm v2 and ksonnet.
  • The Red Hat SSO (RH SSO) Operator is not supported in disconnected clusters. As a result, the Red Hat OpenShift GitOps Operator and RH SSO integration is not supported in disconnected clusters.
  • When you delete an Argo CD application from the OpenShift Container Platform web console, the Argo CD application gets deleted in the user interface, but the deployments are still present in the cluster. As a workaround, delete the Argo CD application from the Argo CD console. GITOPS-830

4.1.9.5. Breaking Change

4.1.9.5.1. Upgrading from Red Hat OpenShift GitOps v1.0.1

When you upgrade from Red Hat OpenShift GitOps v1.0.1 to v1.1, the Red Hat OpenShift GitOps Operator renames the default Argo CD instance created in the openshift-gitops namespace from argocd-cluster to openshift-gitops.

This is a breaking change and needs the following steps to be performed manually, before the upgrade:

  1. Go to the OpenShift Container Platform web console and copy the content of the argocd-cm.yml config map file in the openshift-gitops namespace to a local file. The content may look like the following example:

    Example argocd config map YAML

    kind: ConfigMap
    apiVersion: v1
    metadata:
    selfLink: /api/v1/namespaces/openshift-gitops/configmaps/argocd-cm
    resourceVersion: '112532'
    name: argocd-cm
    uid: f5226fbc-883d-47db-8b53-b5e363f007af
    creationTimestamp: '2021-04-16T19:24:08Z'
    managedFields:
    ...
    namespace: openshift-gitops
    labels:
      app.kubernetes.io/managed-by: argocd-cluster
      app.kubernetes.io/name: argocd-cm
      app.kubernetes.io/part-of: argocd
    data: "" 1
    admin.enabled: 'true'
    statusbadge.enabled: 'false'
    resource.exclusions: |
      - apiGroups:
        - tekton.dev
        clusters:
        - '*'
        kinds:
        - TaskRun
        - PipelineRun
    ga.trackingid: ''
    repositories: |
      - type: git
        url: https://github.com/user-name/argocd-example-apps
    ga.anonymizeusers: 'false'
    help.chatUrl: ''
    url: >-
      https://argocd-cluster-server-openshift-gitops.apps.dev-svc-4.7-041614.devcluster.openshift.com   "" 2
    help.chatText: ''
    kustomize.buildOptions: ''
    resource.inclusions: ''
    repository.credentials: ''
    users.anonymous.enabled: 'false'
    configManagementPlugins: ''
    application.instanceLabelKey: ''

    1
    Restore only the data section of the content in the argocd-cm.yml config map file manually.
    2
    Replace the URL value in the config map entry with the new instance name openshift-gitops.
  2. Delete the default argocd-cluster instance.
  3. Edit the new argocd-cm.yml config map file to restore the entire data section manually.
  4. Replace the URL value in the config map entry with the new instance name openshift-gitops. For example, in the preceding example, replace the URL value with the following URL value:

    url: >-
      https://openshift-gitops-server-openshift-gitops.apps.dev-svc-4.7-041614.devcluster.openshift.com
  5. Login to the Argo CD cluster and verify that the previous configurations are present.

4.2. Understanding OpenShift GitOps

4.2.1. About GitOps

GitOps is a declarative way to implement continuous deployment for cloud native applications. You can use GitOps to create repeatable processes for managing OpenShift Container Platform clusters and applications across multi-cluster Kubernetes environments. GitOps handles and automates complex deployments at a fast pace, saving time during deployment and release cycles.

The GitOps workflow pushes an application through development, testing, staging, and production. GitOps either deploys a new application or updates an existing one, so you only need to update the repository; GitOps automates everything else.

GitOps is a set of practices that use Git pull requests to manage infrastructure and application configurations. In GitOps, the Git repository is the only source of truth for system and application configuration. This Git repository contains a declarative description of the infrastructure you need in your specified environment and contains an automated process to make your environment match the described state. Also, it contains the entire state of the system so that the trail of changes to the system state are visible and auditable. By using GitOps, you resolve the issues of infrastructure and application configuration sprawl.

GitOps defines infrastructure and application definitions as code. Then, it uses this code to manage multiple workspaces and clusters to simplify the creation of infrastructure and application configurations. By following the principles of the code, you can store the configuration of clusters and applications in Git repositories, and then follow the Git workflow to apply these repositories to your chosen clusters. You can apply the core principles of developing and maintaining software in a Git repository to the creation and management of your cluster and application configuration files.

4.2.2. About Red Hat OpenShift GitOps

Red Hat OpenShift GitOps ensures consistency in applications when you deploy them to different clusters in different environments, such as: development, staging, and production. Red Hat OpenShift GitOps organizes the deployment process around the configuration repositories and makes them the central element. It always has at least two repositories:

  1. Application repository with the source code
  2. Environment configuration repository that defines the desired state of the application

These repositories contain a declarative description of the infrastructure you need in your specified environment. They also contain an automated process to make your environment match the described state.

Red Hat OpenShift GitOps uses Argo CD to maintain cluster resources. Argo CD is an open-source declarative tool for the continuous integration and continuous deployment (CI/CD) of applications. Red Hat OpenShift GitOps implements Argo CD as a controller so that it continuously monitors application definitions and configurations defined in a Git repository. Then, Argo CD compares the specified state of these configurations with their live state on the cluster.

Argo CD reports any configurations that deviate from their specified state. These reports allow administrators to automatically or manually resync configurations to the defined state. Therefore, Argo CD enables you to deliver global custom resources, like the resources that are used to configure OpenShift Container Platform clusters.

4.2.2.1. Key features

Red Hat OpenShift GitOps helps you automate the following tasks:

  • Ensure that the clusters have similar states for configuration, monitoring, and storage
  • Apply or revert configuration changes to multiple OpenShift Container Platform clusters
  • Associate templated configuration with different environments
  • Promote applications across clusters, from staging to production

4.3. Installing OpenShift GitOps

Red Hat Red Hat OpenShift GitOps uses Argo CD to manage specific cluster-scoped resources, including platform operators, optional Operator Lifecycle Manager (OLM) operators, and user management.

This guide explains how to install the Red Hat Red Hat OpenShift GitOps Operator to an OpenShift Container Platform cluster and logging in to the Argo CD instance.

4.3.1. Installing OpenShift GitOps Operator in web console

Prerequisites

  • Access to the OpenShift Container Platform web console.
  • An account with the cluster-admin role.
  • You are logged in to the OpenShift Container Platform cluster as an administrator.
Warning

If you have already installed the Community version of the Argo CD Operator, remove the Argo CD Community Operator before you install the Red Hat OpenShift GitOps Operator.

Procedure

  1. Open the Administrator perspective of the web console and navigate to OperatorsOperatorHub in the menu on the left.
  2. Search for OpenShift GitOps, click the Red Hat OpenShift GitOps tile, and then click Install.

    Red Hat OpenShift GitOps will be installed in all namespaces of the cluster.

After the Red Hat OpenShift GitOps Operator is installed, it automatically sets up a ready-to-use Argo CD instance that is available in the openshift-gitops namespace, and an Argo CD icon is displayed in the console toolbar. You can create subsequent Argo CD instances for your applications under your projects.

4.3.2. Logging in to the Argo CD instance by using your OpenShift Container Platform credentials

Red Hat OpenShift GitOps Operator automatically creates a ready-to-use Argo CD instance that is available in the openshift-gitops namespace.

Prerequisites

  • You have installed the Red Hat OpenShift GitOps Operator in your cluster.

Procedure

  1. In the Administrator perspective of the web console, navigate to OperatorsInstalled Operators to verify that the Red Hat OpenShift GitOps Operator is installed.
  2. Navigate to the red hat applications menu icon menu → OpenShift GitOpsCluster Argo CD. The login page of the Argo CD UI is displayed in a new window.
  3. Obtain the password for the Argo CD instance:

    1. In the left panel of the console, use the perspective switcher to switch to the Developer perspective.
    2. Use the Project drop-down list and select the openshift-gitops project.
    3. Use the left navigation panel to navigate to the Secrets page.
    4. Select the openshift-gitops-cluster instance to display the password.
    5. Copy the password.
  4. Use this password and admin as the username to log in to the Argo CD UI in the new window.
Note

You cannot create two Argo CD CRs in the same namespace.

4.4. Uninstalling OpenShift GitOps

Uninstalling the Red Hat OpenShift GitOps Operator is a two-step process:

  1. Delete the Argo CD instances that were added under the default namespace of the Red Hat OpenShift GitOps Operator.
  2. Uninstall the Red Hat OpenShift GitOps Operator.

Uninstalling only the Operator will not remove the Argo CD instances created.

4.4.1. Deleting the Argo CD instances

Delete the Argo CD instances added to the namespace of the GitOps Operator.

Procedure

  1. In the Terminal type the following command:
$ oc delete gitopsservice cluster -n openshift-gitops
Note

You cannot delete an Argo CD cluster from the web console UI.

After the command runs successfully all the Argo CD instances will be deleted from the openshift-gitops namespace.

Delete any other Argo CD instances from other namespaces using the same command:

$ oc delete gitopsservice cluster -n <namespace>

4.4.2. Uninstalling the GitOps Operator

Procedure

  1. From the OperatorsOperatorHub page, use the Filter by keyword box to search for Red Hat OpenShift GitOps Operator tile.
  2. Click the Red Hat OpenShift GitOps Operator tile. The Operator tile indicates it is installed.
  3. In the Red Hat OpenShift GitOps Operator descriptor page, click Uninstall.

Additional Resources

4.5. Configuring an OpenShift cluster by deploying an application with cluster configurations

With Red Hat OpenShift GitOps, you can configure Argo CD to recursively sync the content of a Git directory with an application that contains custom configurations for your cluster.

Prerequisites

  • Red Hat OpenShift GitOps is installed in your cluster.
  • Logged into Argo CD instance.

4.5.1. Running the Argo CD instance at the cluster-level

The default Argo CD instance and the accompanying controllers, installed by the Red Hat OpenShift GitOps Operator, can now run on the infrastructure nodes of the cluster by setting a simple configuration toggle.

Procedure

  1. Label the existing nodes:

    $ oc label node <node-name> node-role.kubernetes.io/infra=""
  2. Optional: If required, you can also apply taints and isolate the workloads on infrastructure nodes and prevent other workloads from scheduling on these nodes:

    $ oc adm taint nodes -l node-role.kubernetes.io/infra
    infra=reserved:NoSchedule infra=reserved:NoExecute
  3. Add the runOnInfra toggle in the GitOpsService custom resource:

    apiVersion: pipelines.openshift.io/v1alpha1
    kind: GitopsService
    metadata:
      name: cluster
    spec:
      runOnInfra: true
  4. Optional: If taints have been added to the nodes, then add tolerations to the GitOpsService custom resource, for example:

      spec:
        runOnInfra: true
        tolerations:
        - effect: NoSchedule
          key: infra
          value: reserved
        - effect: NoExecute
          key: infra
          value: reserved
  5. Verify that the workloads in the openshift-gitops namespace are now scheduled on the infrastructure nodes by viewing PodsPod details for any pod in the console UI.
Note

Any nodeSelectors and tolerations manually added to the default Argo CD custom resource are overwritten by the toggle and tolerations in the GitOpsService custom resource.

4.5.2. Creating an application by using the Argo CD dashboard

Argo CD provides a dashboard which allows you to create applications.

This sample workflow walks you through the process of configuring Argo CD to recursively sync the content of the cluster directory to the cluster-configs application. The directory defines the OpenShift Container Platform web console cluster configurations that add a link to the Red Hat Developer Blog - Kubernetes under the red hat applications menu icon menu in the web console, and defines a namespace spring-petclinic on the cluster.

Procedure

  1. In the Argo CD dashboard, click NEW APP to add a new Argo CD application.
  2. For this workflow, create a cluster-configs application with the following configurations:

    Application Name
    cluster-configs
    Project
    default
    Sync Policy
    Manual
    Repository URL
    https://github.com/redhat-developer/openshift-gitops-getting-started
    Revision
    HEAD
    Path
    cluster
    Destination
    https://kubernetes.default.svc
    Namespace
    spring-petclinic
    Directory Recurse
    checked
  3. Click CREATE to create your application.
  4. Open the Administrator perspective of the web console and navigate to AdministrationNamespaces in the menu on the left.
  5. Search for and select the namespace, then enter argocd.argoproj.io/managed-by=openshift-gitops in the Label field so that the Argo CD instance in the openshift-gitops namespace can manage your namespace.

4.5.3. Creating an application by using the oc tool

You can create Argo CD applications in your terminal by using the oc tool.

Procedure

  1. Download the sample application:

    $ git clone git@github.com:redhat-developer/openshift-gitops-getting-started.git
  2. Create the application:

    $ oc create -f openshift-gitops-getting-started/argo/cluster.yaml
  3. Run the oc get command to review the created application:

    $ oc get application -n openshift-gitops
  4. Add a label to the namespace your application is deployed in so that the Argo CD instance in the openshift-gitops namespace can manage it:

    $ oc label namespace spring-petclinic argocd.argoproj.io/managed-by=openshift-gitops

4.5.4. Synchronizing your application with your Git repository

Procedure

  1. In the Argo CD dashboard, notice that the cluster-configs Argo CD application has the statuses Missing and OutOfSync. Because the application was configured with a manual sync policy, Argo CD does not sync it automatically.
  2. Click SYNC on the cluster-configs tile, review the changes, and then click SYNCHRONIZE. Argo CD will detect any changes in the Git repository automatically. If the configurations are changed, Argo CD will change the status of the cluster-configs to OutOfSync. You can modify the synchronization policy for Argo CD to automatically apply changes from your Git repository to the cluster.
  3. Notice that the cluster-configs Argo CD application now has the statuses Healthy and Synced. Click the cluster-configs tile to check the details of the synchronized resources and their status on the cluster.
  4. Navigate to the OpenShift Container Platform web console and click red hat applications menu icon to verify that a link to the Red Hat Developer Blog - Kubernetes is now present there.
  5. Navigate to the Project page and search for the spring-petclinic namespace to verify that it has been added to the cluster.

    Your cluster configurations have been successfully synchronized to the cluster.

4.6. Deploying a Spring Boot application with Argo CD

With Argo CD, you can deploy your applications to the OpenShift cluster either by using the Argo CD dashboard or by using the oc tool.

Prerequisites

  • Red Hat OpenShift GitOps is installed in your cluster.
  • Logged into Argo CD instance.

4.6.1. Creating an application by using the Argo CD dashboard

Argo CD provides a dashboard which allows you to create applications.

This sample workflow walks you through the process of configuring Argo CD to recursively sync the content of the cluster directory to the cluster-configs application. The directory defines the OpenShift Container Platform web console cluster configurations that add a link to the Red Hat Developer Blog - Kubernetes under the red hat applications menu icon menu in the web console, and defines a namespace spring-petclinic on the cluster.

Procedure

  1. In the Argo CD dashboard, click NEW APP to add a new Argo CD application.
  2. For this workflow, create a cluster-configs application with the following configurations:

    Application Name
    cluster-configs
    Project
    default
    Sync Policy
    Manual
    Repository URL
    https://github.com/redhat-developer/openshift-gitops-getting-started
    Revision
    HEAD
    Path
    cluster
    Destination
    https://kubernetes.default.svc
    Namespace
    spring-petclinic
    Directory Recurse
    checked
  3. For this workflow, create a spring-petclinic application with the following configurations:

    Application Name
    spring-petclinic
    Project
    default
    Sync Policy
    Automatic
    Repository URL
    https://github.com/redhat-developer/openshift-gitops-getting-started
    Revision
    HEAD
    Path
    app
    Destination
    https://kubernetes.default.svc
    Namespace
    spring-petclinic
  4. Click CREATE to create your application.
  5. Open the Administrator perspective of the web console and navigate to AdministrationNamespaces in the menu on the left.
  6. Search for and select the namespace, then enter argocd.argoproj.io/managed-by=openshift-gitops in the Label field so that the Argo CD instance in the openshift-gitops namespace can manage your namespace.

4.6.2. Creating an application by using the oc tool

You can create Argo CD applications in your terminal by using the oc tool.

Procedure

  1. Download the sample application:

    $ git clone git@github.com:redhat-developer/openshift-gitops-getting-started.git
  2. Create the application:

    $ oc create -f openshift-gitops-getting-started/argo/app.yaml
    $ oc create -f openshift-gitops-getting-started/argo/cluster.yaml
  3. Run the oc get command to review the created application:

    $ oc get application -n openshift-gitops
  4. Add a label to the namespace your application is deployed in so that the Argo CD instance in the openshift-gitops namespace can manage it:

    $ oc label namespace spring-petclinic argocd.argoproj.io/managed-by=openshift-gitops
    $ oc label namespace spring-petclinic argocd.argoproj.io/managed-by=openshift-gitops

4.6.3. Verifying Argo CD self-healing behavior

Argo CD constantly monitors the state of deployed applications, detects differences between the specified manifests in Git and live changes in the cluster, and then automatically corrects them. This behavior is referred to as self-healing.

You can test and observe the self-healing behavior in Argo CD.

Prerequisites

  • The sample app-spring-petclinic application is deployed and configured.

Procedure

  1. In the Argo CD dashboard, verify that your application has the Synced status.
  2. Click the app-spring-petclinic tile in the Argo CD dashboard to view the application resources that are deployed to the cluster.
  3. In the OpenShift Container Platform web console, navigate to the Developer perspective.
  4. Modify the Spring PetClinic deployment and commit the changes to the app/ directory of the Git repository. Argo CD will automatically deploy the changes to the cluster.

    1. Fork the OpenShift GitOps getting started repository.
    2. In the deployment.yaml file, change the failureThreshold value to 5.
    3. In the deployment cluster, run the following command to verify the changed value of the failureThreshold field:

      $ oc edit deployment spring-petclinic -n spring-petclinic
  5. Test the self-healing behavior by modifying the deployment on the cluster and scaling it up to two pods while watching the application in the OpenShift Container Platform web console.

    1. Run the following command to modify the deployment:

      $ oc scale deployment spring-petclinic --replicas 2  -n spring-petclinic
    2. In the OpenShift Container Platform web console, notice that the deployment scales up to two pods and immediately scales down again to one pod. Argo CD detected a difference from the Git repository and auto-healed the application on the OpenShift Container Platform cluster.
  6. In the Argo CD dashboard, click the app-spring-petclinic tile → APP DETAILSEVENTS. The EVENTS tab displays the following events: Argo CD detecting out of sync deployment resources on the cluster and then resyncing the Git repository to correct it.

4.7. Monitoring health information for application resources and deployments

The environment details page displays the health status of the application resources, such as routes, synchronization status, deployment configuration and deployment history.

4.7.1. Checking health information

The Red Hat OpenShift GitOps Operator will install the GitOps backend service in the openshift-gitops namespace.

Prerequisites

  • The Red Hat OpenShift GitOps Operator is installed from OperatorHub.
  • Argo CD applications are in sync.

Procedure

  1. Click Environments under the Developer perspective. The Environments page shows the list of applications along with their Environment status.
  2. Hover over the icons under the Environment status column to see the synchronization status of all the environments.
  3. Click on the application name from the list to view the details of a specific application.
  4. If the application is out of sync or degraded applications the respecting icons are displayed under the Resources. Hover over the icons to see the health status and the sync status. The icons are:

    1. For degraded, the broken heart icon is displayed.
    2. For out of sync, the yellow yield icon is displayed.

4.8. Configuring SSO for Argo CD using Dex

After the Red Hat OpenShift GitOps Operator is installed, Argo CD automatically creates a user with admin permissions. To manage multiple users, cluster administrators can use Argo CD to configure Single Sign-On (SSO).

4.8.1. Enabling the Dex OpenShift OAuth Connector

Dex uses the users and groups defined within OpenShift by checking the OAuth server provided by the platform. The following example shows the properties of Dex along with example configurations:

apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
  name: example-argocd
  labels:
    example: openshift-oauth
spec:
  dex:
    openShiftOAuth: true 1
    groups:2
     - default
  rbac:3
    defaultPolicy: 'role:readonly'
    policy: |
      g, cluster-admins, role:admin
    scopes: '[groups]'
1
The openShiftOAuth property triggers the Operator to automatically configure the built-in OpenShift OAuth server when the value is set to true.
2
The groups property assigns users to one group or all groups in the groups list.
3
The RBAC policy property assigns the admin role in the Argo CD cluster to users in the OpenShift cluster-admins group.

4.8.1.1. Mapping users to specific roles

Argo CD cannot map users to specific roles if they have a direct ClusterRoleBinding role. You can manually change the role as role:admin on SSO through OpenShift.

Procedure

  1. Create a group named cluster-admins.

    $ oc adm groups new cluster-admins
  2. Add the user to the group.

    $ oc adm groups add-users cluster-admins USER
  3. Apply the cluster-admin ClusterRole to the group:

    $ oc adm policy add-cluster-role-to-group cluster-admin cluster-admins

4.8.2. Disabling Dex

Dex is installed by default for all the Argo CD instances created by the Operator. You can disable Dex.

Procedure

  • Set the environmental variable DISABLE_DEX to true in the YAML resource of the Operator:

      spec:
      config:
        env:
        - name: DISABLE_DEX
          value: "true"

4.9. Configuring SSO for Argo CD using Keycloak

After the Red Hat OpenShift GitOps Operator is installed, Argo CD automatically creates a user with admin permissions. To manage multiple users, cluster administrators can use Argo CD to configure Single Sign-On (SSO).

Prerequisites

  • Red Hat SSO is installed on the cluster.
  • Argo CD is installed on the cluster.

4.9.1. Configuring a new client in Keycloak

Dex is installed by default for all the Argo CD instances created by the Operator. However, you can delete the Dex configuration and add Keycloak instead to log in to Argo CD using your OpenShift credentials. Keycloak acts as an identity broker between Argo CD and OpenShift.

Procedure

To configure Keycloak, follow these steps:

  1. Delete the Dex configuration by removing the following section from the Argo CD Custom Resource (CR), and save the CR:

    dex:
        openShiftOAuth: true
        resources:
          limits:
            cpu:
            memory:
          requests:
            cpu:
            memory:
  2. Configure Keycloak by editing the Argo CD CR, and updating the value for the provider parameter as keycloak. For example:

    apiVersion: argoproj.io/v1alpha1
    kind: ArgoCD
    metadata:
      name: example-argocd
      labels:
        example: basic
    spec:
      sso:
        provider: keycloak
      server:
        route:
         enabled: true
Note

The Keycloak instance takes 2-3 minutes to install and run.

4.9.2. Logging in to Keycloak

Log in to the Keycloak console to manage identities or roles and define the permissions assigned to the various roles.

Prerequisites

  • The default configuration of Dex is removed.
  • Your Argo CD CR must be configured to use the Keycloak SSO provider.

Procedure

  1. Get the Keycloak route URL for login:

    $ oc -n argocd get route keycloak
    
    NAME        HOST/PORT                                                        PATH   SERVICES   PORT    TERMINATION   WILDCARD
    keycloak    keycloak-default.apps.ci-ln-******.origin-ci-int-aws.dev.**.com         keycloak   <all>    reencrypt     None
  2. Get the Keycloak pod name that stores the user name and password as environment variables:

    $ oc -n argocd get pods
    
    NAME                      READY   STATUS           RESTARTS   AGE
    keycloak-1-2sjcl           1/1    Running            0        45m
    1. Get the Keycloak user name:

      $ oc -n argocd exec keycloak-1-2sjcl -- "env" | grep SSO_ADMIN_USERNAME
      
      SSO_ADMIN_USERNAME=Cqid54Ih
    2. Get the Keycloak password:

      $ oc -n argocd exec keycloak-1-2sjcl -- "env" | grep SSO_ADMIN_PASSWORD
      
      SSO_ADMIN_PASSWORD=GVXxHifH
  3. On the login page, click LOG IN VIA KEYCLOAK.

    Note

    You only see the option LOGIN VIA KEYCLOAK after the Keycloak instance is ready.

  4. Click Login with OpenShift.

    Note

    Login using kubeadmin is not supported.

  5. Enter the OpenShift credentials to log in.
  6. Optional: By default, any user logged in to Argo CD has read-only access. You can manage the user level access by updating the argocd-rbac-cm config map:

    policy.csv:
    <name>, <email>, role:admin

4.9.3. Integrating Keycloak with the OpenShift OAuth server in a disconnected cluster

In a disconnected cluster, Keycloak communicates with the OpenShift OAuth server through a proxy.

Procedure

Follow these steps to integrate Keycloak with the OpenShift OAuth server:

  1. Log in to the Keycloak pod:

    $ oc exec -it dc/keycloak -n argocd -- /bin/bash
  2. Launch the JBoss CLI tool to set up the proxy mappings:

    /opt/eap/bin/jboss-cli.sh
  3. In the JBoss CLI tool, run the following command to start an embedded standalone server:

    embed-server --server-config=standalone-openshift.xml
  4. Set up proxy mappings for the OpenShift OAuth server host:

    /subsystem=keycloak-server/spi=connectionsHttpClient/provider=default:write-attribute(name=properties.proxy-mappings,value=["<oauth-server-hostname>;http://<proxy-server-host>:<proxy-server-port>"])
  5. Stop the embedded server:

    quit
  6. Reload the JBoss CLI tool to apply the proxy mappings:

    /opt/eap/bin/jboss-cli.sh --connect --command=:reload

4.9.4. Uninstalling Keycloak

You can delete the Keycloak resources and their relevant configurations by removing the SSO field from the Argo CD Custom Resource (CR) file. After you remove the SSO field, the values in the file look similar to the following:

  apiVersion: argoproj.io/v1alpha1
  kind: ArgoCD
  metadata:
    name: example-argocd
    labels:
      example: basic
  spec:
    server:
      route:
       enabled: true
Note

A Keycloak application created by using this method is currently not persistent. Additional configurations created in the Argo CD Keycloak realm are deleted when the server restarts.

4.10. Running GitOps control plane workloads on infrastructure nodes

You can use infrastructure nodes to isolate infrastructure workloads for two primary purposes:

  • Prevent incurring billing costs against subscription counts.
  • Separate maintenance and management.

4.10.1. Adding infrastructure nodes

Procedure

  1. Label existing nodes as infrastructure by running the following command:

    $ oc label node <node-name> node-role.kubernetes.io/infra
  2. Edit the GitOpsService Custom Resource (CR) to add the infrastructure node selector:

    $ oc edit gitopsservice -n openshift-gitops
  3. In the GitOpsService CR file, add runOnInfra toggle to the spec section and set it to true:

    apiVersion: pipelines.openshift.io/v1alpha1
    kind: GitopsService
    metadata:
      name: cluster
    spec:
      runOnInfra: true
  4. Optional: Apply taints and isolate the workloads on infrastructure nodes and prevent other workloads from scheduling on these nodes.

    $ oc adm taint nodes -l node-role.kubernetes.io/infra
    infra=reserved:NoSchedule infra=reserved:NoExecute
  5. Optional: If you apply taints to the nodes, you can add tolerations in the GitOpsService CR:

    spec:
      runOnInfra: true
      tolerations:
      - effect: NoSchedule
        key: infra
        value: reserved
      - effect: NoExecute
        key: infra
        value: reserved

To verify that the workloads are scheduled on infrastructure nodes in the Red Hat OpenShift GitOps namespace, click any of the pod names and ensure that the Node selector and Tolerations have been added.

Note

Any manually added Node selectors and Tolerations in the default Argo CD CR will be overwritten by the toggle and the tolerations in the GitOpsService CR.