Chapter 3. GitOps

3.1. Red Hat OpenShift GitOps release notes

Red Hat OpenShift GitOps is a declarative way to implement continuous deployment for cloud native applications. Red Hat OpenShift GitOps ensures consistency in applications when you deploy them to different clusters in different environments, such as: development, staging, and production. Red Hat OpenShift GitOps helps you automate the following tasks:

  • Ensure that the clusters have similar states for configuration, monitoring, and storage
  • Recover or recreate clusters from a known state
  • Apply or revert configuration changes to multiple OpenShift Container Platform clusters
  • Associate templated configuration with different environments
  • Promote applications across clusters, from staging to production

For an overview of Red Hat OpenShift GitOps, see Understanding OpenShift GitOps.

3.1.1. Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see Red Hat CTO Chris Wright’s message.

3.1.2. Release notes for Red Hat OpenShift GitOps 1.2.1

Red Hat OpenShift GitOps 1.2.1 is now available on OpenShift Container Platform 4.7 and 4.8.

3.1.2.1. Support matrix

Some features in this release are currently in Technology Preview. These experimental features are not intended for production use.

Technology Preview Features Support Scope

In the table below, features are marked with the following statuses:

  • TP: Technology Preview
  • GA: General Availability

Note the following scope of support on the Red Hat Customer Portal for these features:

Table 3.1. Support matrix

FeatureRed Hat OpenShift GitOps 1.2.1

Argo CD

GA

Argo CD ApplicationSet

TP

Red Hat OpenShift GitOps Application Manager (kam)

TP

3.1.2.2. Fixed issues

The following issues were resolved in the current release:

  • Previously, huge memory spikes were observed on the application controller on startup. The flag --kubectl-parallelism-limit for the application controller is now set to 10 by default, however this value can be overridden by specifying a number for .spec.controller.kubeParallelismLimit in the Argo CD CR specification. GITOPS-1255
  • The latest Triggers APIs caused Kubernetes build failure due to duplicate entries in the kustomization.yaml when using the kam bootstrap command. The Pipelines and Tekton triggers components have now been updated to v0.24.2 and v0.14.2, respectively, to address this issue. GITOPS-1273
  • Persisting RBAC roles and bindings are now automatically removed from the target namespace when the Argo CD instance from the source namespace is deleted. GITOPS-1228
  • Previously, when deploying an Argo CD instance into a namespace, the Argo CD instance would change the "managed-by" label to be its own namespace. This fix would make namespaces unlabelled while also making sure the required RBAC roles and bindings are created and deleted for the namespace. GITOPS-1247
  • Previously, the default resource request limits on Argo CD workloads, specifically for the repo-server and application controller, were found to be very restrictive. The existing resource quota has now been removed and the default memory limit has been increased to 1024M in the repo server. Please note that this change will only affect new installations; existing Argo CD instance workloads will not be affected. GITOPS-1274

3.1.3. Release notes for Red Hat OpenShift GitOps 1.2

Red Hat OpenShift GitOps 1.2 is now available on OpenShift Container Platform 4.7 and 4.8.

3.1.3.1. Support matrix

Some features in this release are currently in Technology Preview. These experimental features are not intended for production use.

Technology Preview Features Support Scope

In the table below, features are marked with the following statuses:

  • TP: Technology Preview
  • GA: General Availability

Note the following scope of support on the Red Hat Customer Portal for these features:

Table 3.2. Support matrix

FeatureRed Hat OpenShift GitOps 1.2

Argo CD

GA

Argo CD ApplicationSet

TP

Red Hat OpenShift GitOps Application Manager (kam)

TP

3.1.3.2. New features

In addition to the fixes and stability improvements, the following sections highlight what is new in Red Hat OpenShift GitOps 1.2:

  • If you do not have read or write access to the openshift-gitops namespace, you can now use the DISABLE_DEFAULT_ARGOCD_INSTANCE environment variable in the GitOps Operator and set the value to TRUE to prevent the default Argo CD instance from starting in the openshift-gitops namespace.
  • Resource requests and limits are now configured in Argo CD workloads. Resource quota is enabled in the openshift-gitops namespace. As a result, out-of-band workloads deployed manually in the openshift-gitops namespace must be configured with resource requests and limits and the resource quota may need to be increased.
  • Argo CD authentication is now integrated with Red Hat SSO and it is automatically configured with OpenShift 4 Identity Provider on the cluster. This feature is disabled by default. To enable Red Hat SSO, add SSO configuration in ArgoCD CR as shown below. Currently,keycloak is the only supported provider.

    apiVersion: argoproj.io/v1alpha1
    kind: ArgoCD
    metadata:
      name: example-argocd
      labels:
        example: basic
    spec:
      sso:
        provider: keycloak
      server:
        route:
         enabled: true
  • You can now define hostnames using route labels to support router sharding. Support for setting labels on the server (argocd server), grafana, and prometheus routes is now available. To set labels on a route, add labels under the route configuration for a server in the ArgoCD CR.

    Example ArgoCD CR YAML to set labels on argocd server

    apiVersion: argoproj.io/v1alpha1
    kind: ArgoCD
    metadata:
      name: example-argocd
      labels:
        example: basic
    spec:
      server:
        route:
         enabled: true
         labels:
           key1: value1
           key2: value2

  • The GitOps Operator now automatically grants permissions to Argo CD instances to manage resources in target namespaces by applying labels. Users can label the target namespace with the label argocd.argoproj.io/managed-by: <source-namespace>, where the source-namespace is the namespace where the argocd instance is deployed.

3.1.3.3. Fixed issues

The following issues were resolved in the current release:

  • Previously, if a user created additional instances of Argo CD managed by the default cluster instance in the openshift-gitops namespace, the application responsible for the new Argo CD instance would get stuck in an OutOfSync status. This issue has now been resolved by adding an owner reference to the cluster secret. GITOPS-1025

3.1.3.4. Known issues

These are the known issues in Red Hat OpenShift GitOps 1.2:

  • When an Argo CD instance is deleted from the source namespace, the argocd.argoproj.io/managed-by labels in the target namespaces are not removed. GITOPS-1228
  • Resource quota has been enabled in the openshift-gitops namespace in Red Hat OpenShift GitOps 1.2. This can affect out-of-band workloads deployed manually and workloads deployed by the default Argo CD instance in the openshift-gitops namespace. When you upgrade from Red Hat OpenShift GitOps v1.1.2 to v1.2 such workloads must be configured with resource requests and limits. If there are any additional workloads, the resource quota in the openshift-gitops namespace must be increased.

    Current Resource Quota for openshift-gitops namespace.

    ResourceRequestsLimits

    CPU

    6688m

    13750m

    Memory

    4544Mi

    9070Mi

    You can use the below command to update the CPU limits.

    $ oc patch resourcequota openshift-gitops-compute-resources -n openshift-gitops --type='json' -p='[{"op": "replace", "path": "/spec/hard/limits.cpu", "value":"9000m"}]'

    You can use the below command to update the CPU requests.

    $ oc patch resourcequota openshift-gitops-compute-resources -n openshift-gitops --type='json' -p='[{"op": "replace", "path": "/spec/hard/cpu", "value":"7000m"}]

    You can replace the path in the above commands from cpu to memory to update the memory.

3.1.4. Release notes for Red Hat OpenShift GitOps 1.1

Red Hat OpenShift GitOps 1.1 is now available on OpenShift Container Platform 4.7.

3.1.4.1. Support matrix

Some features in this release are currently in Technology Preview. These experimental features are not intended for production use.

Technology Preview Features Support Scope

In the table below, features are marked with the following statuses:

  • TP: Technology Preview
  • GA: General Availability

Note the following scope of support on the Red Hat Customer Portal for these features:

Table 3.3. Support matrix

FeatureRed Hat OpenShift GitOps 1.1

Argo CD

GA

Argo CD ApplicationSet

TP

Red Hat OpenShift GitOps Application Manager (kam)

TP

3.1.4.2. New features

In addition to the fixes and stability improvements, the following sections highlight what is new in Red Hat OpenShift GitOps 1.1:

  • The ApplicationSet feature is now added (Technology Preview). The ApplicationSet feature enables both automation and greater flexibility when managing Argo CD applications across a large number of clusters and within monorepos. It also makes self-service usage possible on multitenant Kubernetes clusters.
  • Argo CD is now integrated with cluster logging stack and with the OpenShift Container Platform Monitoring and Alerting features.
  • Argo CD auth is now integrated with OpenShift Container Platform.
  • Argo CD applications controller now supports horizontal scaling.
  • Argo CD Redis servers now support high availability (HA).

3.1.4.3. Fixed issues

The following issues were resolved in the current release:

  • Previously, Red Hat OpenShift GitOps did not work as expected in a proxy server setup with active global proxy settings. This issue is fixed and now Argo CD is configured by the Red Hat OpenShift GitOps Operator using fully qualified domain names (FQDN) for the pods to enable communication between components. GITOPS-703
  • The Red Hat OpenShift GitOps backend relies on the ?ref= query parameter in the Red Hat OpenShift GitOps URL to make API calls. Previously, this parameter was not read from the URL, causing the backend to always consider the default reference. This issue is fixed and the Red Hat OpenShift GitOps backend now extracts the reference query parameter from the Red Hat OpenShift GitOps URL and only uses the default reference when there is no input reference provided. GITOPS-817
  • Previously, the Red Hat OpenShift GitOps backend failed to find the valid GitLab repository. This was because the Red Hat OpenShift GitOps backend checked for main as the branch reference, instead of master in the GitLab repository. This issue is fixed now. GITOPS-768
  • The Environments page in the Developer perspective of the OpenShift Container Platform web console now shows the list of applications and the number of environments. This page also displays an Argo CD link that directs you to the Argo CD Applications page that lists all the applications. The Argo CD Applications page has LABELS (for example, app.kubernetes.io/name=appName) that help you filter only the applications of your choice. GITOPS-544

3.1.4.4. Known issues

These are the known issues in Red Hat OpenShift GitOps 1.1:

  • Red Hat OpenShift GitOps does not support Helm v2 and ksonnet.
  • The Red Hat SSO (RH SSO) Operator is not supported in disconnected clusters. As a result, the Red Hat OpenShift GitOps Operator and RH SSO integration is not supported in disconnected clusters.
  • When you delete an Argo CD application from the OpenShift Container Platform web console, the Argo CD application gets deleted in the user interface, but the deployments are still present in the cluster. As a workaround, delete the Argo CD application from the Argo CD console. GITOPS-830

3.1.4.5. Breaking Change

3.1.4.5.1. Upgrading from Red Hat OpenShift GitOps v1.0.1

When you upgrade from Red Hat OpenShift GitOps v1.0.1 to v1.1, the Red Hat OpenShift GitOps Operator renames the default Argo CD instance created in the openshift-gitops namespace from argocd-cluster to openshift-gitops.

This is a breaking change and needs the following steps to be performed manually, before the upgrade:

  1. Go to the OpenShift Container Platform web console and copy the content of the argocd-cm.yml config map file in the openshift-gitops namespace to a local file. The content may look like the following example:

    Example argocd config map YAML

    kind: ConfigMap
    apiVersion: v1
    metadata:
    selfLink: /api/v1/namespaces/openshift-gitops/configmaps/argocd-cm
    resourceVersion: '112532'
    name: argocd-cm
    uid: f5226fbc-883d-47db-8b53-b5e363f007af
    creationTimestamp: '2021-04-16T19:24:08Z'
    managedFields:
    ...
    namespace: openshift-gitops
    labels:
      app.kubernetes.io/managed-by: argocd-cluster
      app.kubernetes.io/name: argocd-cm
      app.kubernetes.io/part-of: argocd
    data: "" 1
    admin.enabled: 'true'
    statusbadge.enabled: 'false'
    resource.exclusions: |
      - apiGroups:
        - tekton.dev
        clusters:
        - '*'
        kinds:
        - TaskRun
        - PipelineRun
    ga.trackingid: ''
    repositories: |
      - type: git
        url: https://github.com/user-name/argocd-example-apps
    ga.anonymizeusers: 'false'
    help.chatUrl: ''
    url: >-
      https://argocd-cluster-server-openshift-gitops.apps.dev-svc-4.7-041614.devcluster.openshift.com   "" 2
    help.chatText: ''
    kustomize.buildOptions: ''
    resource.inclusions: ''
    repository.credentials: ''
    users.anonymous.enabled: 'false'
    configManagementPlugins: ''
    application.instanceLabelKey: ''

    1
    Restore only the data section of the content in the argocd-cm.yml config map file manually.
    2
    Replace the URL value in the config map entry with the new instance name openshift-gitops.
  2. Delete the default argocd-cluster instance.
  3. Edit the new argocd-cm.yml config map file to restore the entire data section manually.
  4. Replace the URL value in the config map entry with the new instance name openshift-gitops. For example, in the preceding example, replace the URL value with the following URL value:

    url: >-
      https://openshift-gitops-server-openshift-gitops.apps.dev-svc-4.7-041614.devcluster.openshift.com
  5. Login to the Argo CD cluster and verify that the previous configurations are present.

3.2. Understanding OpenShift GitOps

3.2.1. About GitOps

GitOps is a declarative way to implement continuous deployment for cloud native applications. You can use GitOps to create repeatable processes for managing OpenShift Container Platform clusters and applications across multi-cluster Kubernetes environments. GitOps handles and automates complex deployments at a fast pace, saving time during deployment and release cycles.

The GitOps workflow pushes an application through development, testing, staging, and production. GitOps either deploys a new application or updates an existing one, so you only need to update the repository; GitOps automates everything else.

GitOps is a set of practices that use Git pull requests to manage infrastructure and application configurations. In GitOps, the Git repository is the only source of truth for system and application configuration. This Git repository contains a declarative description of the infrastructure you need in your specified environment and contains an automated process to make your environment match the described state. Also, it contains the entire state of the system so that the trail of changes to the system state are visible and auditable. By using GitOps, you resolve the issues of infrastructure and application configuration sprawl.

GitOps defines infrastructure and application definitions as code. Then, it uses this code to manage multiple workspaces and clusters to simplify the creation of infrastructure and application configurations. By following the principles of the code, you can store the configuration of clusters and applications in Git repositories, and then follow the Git workflow to apply these repositories to your chosen clusters. You can apply the core principles of developing and maintaining software in a Git repository to the creation and management of your cluster and application configuration files.

3.2.2. About Red Hat OpenShift GitOps

Red Hat OpenShift GitOps ensures consistency in applications when you deploy them to different clusters in different environments, such as: development, staging, and production. Red Hat OpenShift GitOps organizes the deployment process around the configuration repositories and makes them the central element. It always has at least two repositories:

  1. Application repository with the source code
  2. Environment configuration repository that defines the desired state of the application

These repositories contain a declarative description of the infrastructure you need in your specified environment. They also contain an automated process to make your environment match the described state.

Red Hat OpenShift GitOps uses Argo CD to maintain cluster resources. Argo CD is an open-source declarative tool for the continuous integration and continuous deployment (CI/CD) of applications. Red Hat OpenShift GitOps implements Argo CD as a controller so that it continuously monitors application definitions and configurations defined in a Git repository. Then, Argo CD compares the specified state of these configurations with their live state on the cluster.

Argo CD reports any configurations that deviate from their specified state. These reports allow administrators to automatically or manually resync configurations to the defined state. Therefore, Argo CD enables you to deliver global custom resources, like the resources that are used to configure OpenShift Container Platform clusters.

3.2.2.1. Key features

Red Hat OpenShift GitOps helps you automate the following tasks:

  • Ensure that the clusters have similar states for configuration, monitoring, and storage
  • Recover or recreate clusters from a known state
  • Apply or revert configuration changes to multiple OpenShift Container Platform clusters
  • Associate templated configuration with different environments
  • Promote applications across clusters, from staging to production

3.3. Getting started with OpenShift GitOps

Red Hat OpenShift GitOps uses Argo CD to manage specific cluster-scoped resources, including platform operators, optional Operator Lifecycle Manager (OLM) operators, and user management.

This guide explains how to install the Red Hat OpenShift GitOps Operator to an OpenShift Container Platform cluster and logging in to the Argo CD instance.

3.3.1. Installing GitOps Operator in web console

Prerequisites

  • Access to the OpenShift Container Platform web console.
  • An account with the cluster-admin role.
  • You are logged in to the OpenShift cluster as an administrator.
Warning

If you have already installed the Community version of the Argo CD Operator, remove the Argo CD Community Operator before you install the Red Hat OpenShift GitOps Operator.

Procedure

  1. Open the Administrator perspective of the web console and navigate to OperatorsOperatorHub in the menu on the left.
  2. Search for OpenShift GitOps, click the Red Hat OpenShift GitOps tile, and then click Install.

    Red Hat OpenShift GitOps will be installed in all namespaces of the cluster.

After the Red Hat OpenShift GitOps Operator is installed, it automatically sets up a ready-to-use Argo CD instance that is available in the openshift-gitops namespace, and an Argo CD icon is displayed in the console toolbar. You can create subsequent Argo CD instances for your applications under your projects.

3.4. Configuring Argo CD to recursively sync a Git repository with your application

3.4.1. Configuring an OpenShift cluster by deploying an application with cluster configurations

With Red Hat OpenShift GitOps, you can configure Argo CD to recursively sync the content of a Git directory with an application that contains custom configurations for your cluster.

Prerequisites

  • Red Hat OpenShift GitOps is installed in your cluster.

3.4.1.1. Logging in to the Argo CD instance by using your OpenShift credentials

Red Hat OpenShift GitOps Operator automatically creates a ready-to-use Argo CD instance that is available in the openshift-gitops namespace.

Prerequisites

  • You have installed the Red Hat OpenShift GitOps Operator in your cluster.

Procedure

  1. In the Administrator perspective of the web console, navigate to OperatorsInstalled Operators to verify that the Red Hat OpenShift GitOps Operator is installed.
  2. Navigate to the red hat applications menu icon menu → OpenShift GitOpsCluster Argo CD. The login page of the Argo CD UI is displayed in a new window.
  3. Obtain the password for the Argo CD instance:

    1. Navigate to the Developer perspective of the web console. A list of available projects is displayed.
    2. Navigate to the openshift-gitops project.
    3. Use the left navigation panel to navigate to the Secrets page.
    4. Select the openshift-gitops-cluster instance to display the password.
    5. Copy the password.
  4. Use this password and admin as the username to log in to the Argo CD UI in the new window.

3.4.1.2. Creating an application by using the Argo CD dashboard

Argo CD provides a dashboard which allows you to create applications.

This sample workflow walks you through the process of configuring Argo CD to recursively sync the content of the cluster directory to the cluster-configs application. The directory defines the OpenShift Container Platform web console cluster configurations that add a link to the Red Hat Developer Blog - Kubernetes under the red hat applications menu icon menu in the web console, and defines a namespace spring-petclinic on the cluster.

Procedure

  1. In the Argo CD dashboard, click NEW APP to add a new Argo CD application.
  2. For this workflow, create a cluster-configs application with the following configurations:

    Application Name
    cluster-configs
    Project
    default
    Sync Policy
    Manual
    Repository URL
    https://github.com/redhat-developer/openshift-gitops-getting-started
    Revision
    HEAD
    Path
    cluster
    Destination
    https://kubernetes.default.svc
    Namespace
    spring-petclinic
    Directory Recurse
    checked
  3. Click CREATE to create your application.
  4. Open the Administrator perspective of the web console and navigate to AdministrationNamespaces in the menu on the left.
  5. Search for and select the namespace, then enter argocd.argoproj.io/managed-by=openshift-gitops in the Label field so that the Argo CD instance in the openshift-gitops namespace can manage your namespace.

3.4.1.3. Creating an application by using the oc tool

You can create Argo CD applications in your terminal by using the oc tool.

Procedure

  1. Download the sample application:

    $ git clone git@github.com:redhat-developer/openshift-gitops-getting-started.git
  2. Create the application:

    $ oc create -f openshift-gitops-getting-started/argo/cluster.yaml
  3. Run the oc get command to review the created application:

    $ oc get application -n openshift-gitops
  4. Add a label to the namespace your application is deployed in so that the Argo CD instance in the openshift-gitops namespace can manage it:

    $ oc label namespace spring-petclinic argocd.argoproj.io/managed-by=openshift-gitops

3.4.1.4. Synchronizing your application with your Git repository

Procedure

  1. In the Argo CD dashboard, notice that the cluster-configs Argo CD application has the statuses Missing and OutOfSync. Because the application was configured with a manual sync policy, Argo CD does not sync it automatically.
  2. Click SYNC on the cluster-configs tile, review the changes, and then click SYNCHRONIZE. Argo CD will detect any changes in the Git repository automatically. If the configurations are changed, Argo CD will change the status of the cluster-configs to OutOfSync. You can modify the synchronization policy for Argo CD to automatically apply changes from your Git repository to the cluster.
  3. Notice that the cluster-configs Argo CD application now has the statuses Healthy and Synced. Click the cluster-configs tile to check the details of the synchronized resources and their status on the cluster.
  4. Navigate to the OpenShift Container Platform web console and click red hat applications menu icon to verify that a link to the Red Hat Developer Blog - Kubernetes is now present there.
  5. Navigate to the Project page and search for the spring-petclinic namespace to verify that it has been added to the cluster.

    Your cluster configurations have been successfully synchronized to the cluster.

3.4.2. Deploying a Spring Boot application with Argo CD

With Argo CD, you can deploy your applications to the OpenShift cluster either by using the Argo CD dashboard or by using the oc tool.

Prerequisites

  • Red Hat OpenShift GitOps is installed in your cluster.

3.4.2.1. Logging in to the Argo CD instance by using your OpenShift credentials

Red Hat OpenShift GitOps Operator automatically creates a ready-to-use Argo CD instance that is available in the openshift-gitops namespace.

Prerequisites

  • You have installed the Red Hat OpenShift GitOps Operator in your cluster.

Procedure

  1. In the Administrator perspective of the web console, navigate to OperatorsInstalled Operators to verify that the Red Hat OpenShift GitOps Operator is installed.
  2. Navigate to the red hat applications menu icon menu → OpenShift GitOpsCluster Argo CD. The login page of the Argo CD UI is displayed in a new window.
  3. Obtain the password for the Argo CD instance:

    1. Navigate to the Developer perspective of the web console. A list of available projects is displayed.
    2. Navigate to the openshift-gitops project.
    3. Use the left navigation panel to navigate to the Secrets page.
    4. Select the openshift-gitops-cluster instance to display the password.
    5. Copy the password.
  4. Use this password and admin as the username to log in to the Argo CD UI in the new window.

3.4.2.2. Creating an application by using the Argo CD dashboard

Argo CD provides a dashboard which allows you to create applications.

This sample workflow walks you through the process of configuring Argo CD to recursively sync the content of the cluster directory to the cluster-configs application. The directory defines the OpenShift Container Platform web console cluster configurations that add a link to the Red Hat Developer Blog - Kubernetes under the red hat applications menu icon menu in the web console, and defines a namespace spring-petclinic on the cluster.

Procedure

  1. In the Argo CD dashboard, click NEW APP to add a new Argo CD application.
  2. For this workflow, create a cluster-configs application with the following configurations:

    Application Name
    cluster-configs
    Project
    default
    Sync Policy
    Manual
    Repository URL
    https://github.com/redhat-developer/openshift-gitops-getting-started
    Revision
    HEAD
    Path
    cluster
    Destination
    https://kubernetes.default.svc
    Namespace
    spring-petclinic
    Directory Recurse
    checked
  3. For this workflow, create a spring-petclinic application with the following configurations:

    Application Name
    spring-petclinic
    Project
    default
    Sync Policy
    Automatic
    Repository URL
    https://github.com/redhat-developer/openshift-gitops-getting-started
    Revision
    HEAD
    Path
    app
    Destination
    https://kubernetes.default.svc
    Namespace
    spring-petclinic
  4. Click CREATE to create your application.
  5. Open the Administrator perspective of the web console and navigate to AdministrationNamespaces in the menu on the left.
  6. Search for and select the namespace, then enter argocd.argoproj.io/managed-by=openshift-gitops in the Label field so that the Argo CD instance in the openshift-gitops namespace can manage your namespace.

3.4.2.3. Creating an application by using the oc tool

You can create Argo CD applications in your terminal by using the oc tool.

Procedure

  1. Download the sample application:

    $ git clone git@github.com:redhat-developer/openshift-gitops-getting-started.git
  2. Create the application:

    $ oc create -f openshift-gitops-getting-started/argo/app.yaml
    $ oc create -f openshift-gitops-getting-started/argo/cluster.yaml
  3. Run the oc get command to review the created application:

    $ oc get application -n openshift-gitops
  4. Add a label to the namespace your application is deployed in so that the Argo CD instance in the openshift-gitops namespace can manage it:

    $ oc label namespace spring-petclinic argocd.argoproj.io/managed-by=openshift-gitops
    $ oc label namespace spring-petclinic argocd.argoproj.io/managed-by=openshift-gitops

3.4.2.4. Verifying Argo CD self-healing behavior

Argo CD constantly monitors the state of deployed applications, detects differences between the specified manifests in Git and live changes in the cluster, and then automatically corrects them. This behavior is referred to as self-healing.

You can test and observe the self-healing behavior in Argo CD.

Prerequisites

  • The sample app-spring-petclinic application is deployed and configured.

Procedure

  1. In the Argo CD dashboard, verify that your application has the Synced status.
  2. Click the app-spring-petclinic tile in the Argo CD dashboard to view the application resources that are deployed to the cluster.
  3. In the OpenShift web console, navigate to the Developer perspective.
  4. Modify the Spring PetClinic deployment and commit the changes to the app/ directory of the Git repository. Argo CD will automatically deploy the changes to the cluster.
  5. Test the self-healing behavior by modifying the deployment on the cluster and scaling it up to two pods while watching the application in the OpenShift web console.

    1. Run the following command to modify the deployment:

      $ oc scale deployment spring-petclinic --replicas 2  -n spring-petclinic
    2. In the OpenShift web console, notice that the deployment scales up to two pods and immediately scales down again to one pod. Argo CD detected a difference from the Git repository and auto-healed the application on the OpenShift cluster.
  6. In the Argo CD dashboard, click the app-spring-petclinic tile → APP DETAILSEVENTS. The EVENTS tab displays the following events: Argo CD detecting out of sync deployment resources on the cluster and then resyncing the Git repository to correct it.

3.5. Configuring SSO for Argo CD on OpenShift

After the Red Hat OpenShift GitOps Operator is installed, Argo CD automatically creates a user with admin permissions. To manage multiple users, Argo CD allows cluster administrators to configure SSO.

Note

Bundled Dex OIDC provider is not supported.

Prerequisites

  • Red Hat SSO is installed on the cluster.

3.5.1. Creating a new client in Keycloak

Procedure

  1. Log in to your Keycloak server, select the realm you want to use, navigate to the Clients page, and then click Create in the upper-right section of the screen.
  2. Specify the following values:

    Client ID
    argocd
    Client Protocol
    openid-connect
    Route URL
    <your-argo-cd-route-url>
    Access Type
    confidential
    Valid Redirect URIs
    <your-argo-cd-route-url>/auth/callback
    Base URL
    /applications
  3. Click Save to see the Credentials tab added to the Client page.
  4. Copy the secret from the Credentials tab for further configuration.

3.5.2. Configuring the groups claim

To manage users in Argo CD, you must configure a groups claim that can be included in the authentication token.

Procedure

  1. In the Keycloak dashboard, navigate to Client Scope and add a new client with the following values:

    Name
    groups
    Protocol
    openid-connect
    Display On Content Scope
    On
    Include to Token Scope
    On
  2. Click Save and navigate to groupsMappers.
  3. Add a new token mapper with the following values:

    Name
    groups
    Mapper Type
    Group Membership
    Token Claim Name

    groups

    The token mapper adds the groups claim to the token when the client requests groups.

  4. Navigate to ClientsClient Scopes and configure the client to provide the groups scope. Select groups in the Assigned Default Client Scopes table and click Add selected. The groups scope must be in the Available Client Scopes table.
  5. Navigate to UsersAdminGroups and create a group ArgoCDAdmins.

3.5.3. Configuring Argo CD OIDC

To configure Argo CD OpenID Connect (OIDC), you must generate your client secret, encode it, and add it to your custom resource.

Prerequisites

  • You have obtained your client secret.

Procedure

  1. Store the client secret you generated.

    1. Encode the client secret in base64:

      $ echo -n '83083958-8ec6-47b0-a411-a8c55381fbd2' | base64
    2. Edit the secret and add the base64 value to an oidc.keycloak.clientSecret key:

      $ oc edit secret argocd-secret -n <namespace>

      Example YAML of the secret

      apiVersion: v1
      kind: Secret
      metadata:
        name: argocd-secret
      data:
        oidc.keycloak.clientSecret: ODMwODM5NTgtOGVjNi00N2IwLWE0MTEtYThjNTUzODFmYmQy

  2. Edit the argocd custom resource and add the OIDC configuration to enable the Keycloak authentication:

    $ oc edit argocd -n <your_namespace>

    Example of argocd custom resource

    apiVersion: argoproj.io/v1alpha1
    kind: ArgoCD
    metadata:
      creationTimestamp: null
      name: argocd
      namespace: argocd
    spec:
      resourceExclusions: |
        - apiGroups:
          - tekton.dev
          clusters:
          - '*'
          kinds:
          - TaskRun
          - PipelineRun
      oidcConfig: |
        name: OpenShift Single Sign-On
        issuer: https://keycloak.example.com/auth/realms/myrealm 1
        clientID: argocd 2
        clientSecret: $oidc.keycloak.clientSecret 3
        requestedScopes: ["openid", "profile", "email", "groups"] 4
      server:
        route:
          enabled: true

    1
    issuer must end with the correct realm name (in this example myrealm).
    2
    clientID is the Client ID you configured in your Keycloak account.
    3
    clientSecret points to the right key you created in the argocd-secret secret.
    4
    requestedScopes contains the groups claim if you did not add it to the Default scope.

3.5.4. Keycloak Identity Brokering with OpenShift

You can configure a Keycloak instance to use OpenShift for authentication through Identity Brokering. This allows for Single Sign-On (SSO) between the OpenShift cluster and the Keycloak instance.

Prerequisites

  • jq CLI tool is installed.

Procedure

  1. Obtain the OpenShift Container Platform API URL:

    $ curl -s -k -H "Authorization: Bearer $(oc whoami -t)" https://<openshift-user-facing-api-url>/apis/config.openshift.io/v1/infrastructures/cluster | jq ".status.apiServerURL".
    Note

    The address of the OpenShift Container Platform API is often protected by HTTPS. Therefore, you must configure X509_CA_BUNDLE in the container and set it to /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. Otherwise, Keycloak cannot communicate with the API Server.

  2. In the Keycloak server dashboard, navigate to Identity Providers and select Openshift v4. Specify the following values:

    Base Url
    OpenShift 4 API URL
    Client ID
    keycloak-broker
    Client Secret

    A secret that you want define

    Now you can log in to Argo CD with your OpenShift credentials through Keycloak as an Identity Broker.

3.5.5. Registering an additional an OAuth client

If you need an additional OAuth client to manage authentication for your OpenShift Container Platform cluster, you can register one.

Procedure

  • To register your client:

    $ oc create -f <(echo '
    kind: OAuthClient
    apiVersion: oauth.openshift.io/v1
    metadata:
     name: keycloak-broker 1
    secret: "..." 2
    redirectURIs:
    - "https://keycloak-keycloak.apps.dev-svc-4.7-020201.devcluster.openshift.com/auth/realms/myrealm/broker/openshift-v4/endpoint" 3
    grantMethod: prompt 4
    ')
1
The name of the OAuth client is used as the client_id parameter when making requests to <namespace_route>/oauth/authorize and <namespace_route>/oauth/token.
2
The secret is used as the client_secret parameter when making requests to <namespace_route>/oauth/token.
3
The redirect_uri parameter specified in requests to <namespace_route>/oauth/authorize and <namespace_route>/oauth/token must be equal to or prefixed by one of the URIs listed in the redirectURIs parameter value.
4
If the user has not granted access to this client, the grantMethod determines which action to take when this client requests tokens. Specify auto to automatically approve the grant and retry the request, or prompt to prompt the user to approve or deny the grant.

3.5.6. Configure groups and Argo CD RBAC

Role-based access control (RBAC) allows you to provide relevant permissions to users.

Prerequisites

  • You have created the ArgoCDAdmins group in Keycloak.
  • The user you want to give permissions to has logged in to Argo CD.

Procedure

  1. In the Keycloak dashboard navigate to UsersGroups. Add the user to the Keycloak group ArgoCDAdmins.
  2. Ensure that ArgoCDAdmins group has the required permissions in the argocd-rbac config map.

    • Edit the config map:

      $ oc edit configmap argocd-rbac-cm -n <namespace>

      Example of a config map that defines admin permissions.

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: argocd-rbac-cm
      data:
        policy.csv: |
          g, /ArgoCDAdmins, role:admin

3.5.7. In-built permissions for Argo CD

This section lists the permissions that are granted to ArgoCD to manage specific cluster-scoped resources which include platform operators, optional OLM operators, and user management. Note that ArgoCD is not granted cluster-admin permissions.

Table 3.4. Permissions granted to Argo CD

Resource group

What it configures for a user or an administrator

operators.coreos.com

Optional operators managed by OLM

user.openshift.io, rbac.authorization.k8s.io

Groups, Users, and their permissions

config.openshift.io

Control plane operators managed by CVO used to configure cluster-wide build configuration, registry configuration, and scheduler policies

storage.k8s.io

Storage

console.openshift.io

Console customization