Adding a ZTP cluster back to ACM without reinstalling it using a baremetal host override.

Issue

Adding a ZTP cluster imported into ACM into git without a redeployment
With just a few steps in ArgoCD, a cluster that has gone missing can be easily added back into git. This process can be completed without deleting the installation of OpenShift on the baremetal host.

Downloading the Argo binary and logging in via the OpenShift route

First, download and install the ArgoCD binary and login via the route that's been exposed in the openshift-gitops-server namespace. This process will help complete the rest of the steps in this article.

# Pull down the argo CLI client
curl -sSL -o argocd-linux-amd64 \ 
https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64

sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd

rm argocd-linux-amd64

argoPass=$(oc get secret/openshift-gitops-cluster -n openshift-gitops -o \ 
jsonpath='{.data.admin\.password}' | base64 -d)

argoURL=$(oc get route openshift-gitops-server -n openshift-gitops -o \ 
jsonpath='{.spec.host}{"\n"}')

argocd login --insecure --grpc-web $argoURL  --username admin --password \ 
$argoPass

Disable Pruning and AutoSyncing in the Argo Application using the ArgoCD binary

Next, configure the application to not prune the existing CRs before adding them back to the kustomization.yaml file that Argo uses as a source of truth. It is important to make sure that the CRs that have been added back have all of the information that the site-config will generate, this allows Argo to compare the manifests and find a match.

This method uses the ArgoCD binary to set the syncing policy to none application wide. OpenShift does not include the Argo client by default, so it will be downloaded and used to control Argo from the command line.

oc patch -n openshift-gitops  applications.argoproj.io clusters  \
--type='json' -p='[{"op": "replace", "path": "/spec/syncPolicy/automated/prune" \
, "value":false}]'
application.argoproj.io/clusters patched

Alternatively, you can disable auto-syncing for the clusters application

argocd app set clusters --sync-policy none

oc get applications -n openshift-gitops clusters -o json | jq .spec
{
  "destination": {
    "namespace": "clusters-sub",
    "server": "https://kubernetes.default.svc"
  },
  "project": "ztp-app-project",
  "source": {
    "path": "site-configs",
    "repoURL": "http://jumphost.inbound.example.bos2.lab:3000/kni/faredge-ztp.git",
    "targetRevision": "master"
  },
  "syncPolicy": {
    "syncOptions": [
    "CreateNamespace=true"
    ]
  }
}

Prerequisites: existing cluster kubeconfig and custom resource overrides

Two copies of the imported cluster's kubeconfig will be required in the target namespace. One will be used by the hub cluster to issue commands on the spoke, the second will be referenced in the clusterdeployment override in an upcoming step.

NOTE: A copy of the cluster-admin kubeconfig can be obtained in the static pod manifests on the node or can be obtained via a secret in the openshift-kube-apiserver namespace. Also, ensure that there is not a preexisting kubeconfig in the managedcluster namespace before creating the kubeconfig that will be used to import the cluster.

ssh core@zt-sno4 'sudo cp /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/lb-ext.kubeconfig /home/core/ && sudo chmod o+r /home/core/lb-ext.kubeconfig'
scp core@zt-sno4:/home/core/lb-ext.kubeconfig kubeconfig-zt-sno4.yaml

# or 

oc get secret -n openshift-kube-apiserver node-kubeconfigs -ojson | jq '.data["lb-ext.kubeconfig"]' --raw-output | base64 -d > kubeconfig-zt-sno4.yaml

First, create the YAML for the cluster import.

cat hub-import-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: auto-import-secret
  namespace: zt-sno4
type: Opaque
stringData:
  autoImportRetry: "5"
  kubeconfig: |
    apiVersion: v1
    kind: Config
    ..raw kubeconfig here..

Next, create the target namespace and both secrets. If the namespace already exists, clear out any existing kubeconfigs that could cause issues.

NOTE: The content of the secret must be "kubeconfig".

oc -n zt-sno4 delete secret zt-sno4-admin-kubeconfig 
oc create ns zt-sno4 
oc create -f hub-import-secret.yaml
oc create secret generic auto-import-secret --from-file=kubeconfig=kubeconfig-zt-sno4.yaml

Now, with pruning and automated synchronization disabled we can verify the existing CRs will match the values supplied in the imported cluster’s site-config. You can view the assisted service CRs in the cluster’s namespace to accomplish this. Create both a baremetalhost and a clusterdeployment override CR to pass into the site-config note the externallyProvisioned section in the baremetalhost and the installed: true in the clusterdeployment. This will prevent it from being reinstalled.

---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
    name: "{{ .Node.HostName }}"
    namespace: "{{ .Cluster.ClusterName }}"
    annotations:
      argocd.argoproj.io/sync-wave: "1"
      inspect.metal3.io: "{{ .Node.IronicInspect }}"
      bmac.agent-install.openshift.io/hostname: "{{ .Node.HostName }}"
      bmac.agent-install.openshift.io/installer-args: "{{ .Node.InstallerArgs }}"
      bmac.agent-install.openshift.io/ignition-config-overrides: "{{ .Node.IgnitionConfigOverride }}"
      bmac.agent-install.openshift.io/role: "{{ .Node.Role }}"
      baremetalhost.metal3.io/detached: "assisted-service-controller"
    labels:
      infraenvs.agent-install.openshift.io: "{{ .Cluster.ClusterName }}"
spec:
   bootMode: "{{ .Node.BootMode }}"
   externallyProvisioned: true
   bmc:
      address: "{{ .Node.BmcAddress }}"
      disableCertificateVerification: true
      credentialsName: "{{ .Node.BmcCredentialsName.Name }}"
   bootMACAddress: "{{ .Node.BootMACAddress }}"
   automatedCleaningMode: disabled
   online: true
   rootDeviceHints: "{{ .Node.RootDeviceHints }}"
   userData:  "{{ .Node.UserData }}"

---
apiVersion: hive.openshift.io/v1
kind: ClusterDeployment
metadata:
  name: "{{ .Cluster.ClusterName }}"
  namespace: "{{ .Cluster.ClusterName }}"
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  baseDomain: "{{ .Site.BaseDomain }}"
  installed: true
  clusterInstallRef:
    group: extensions.hive.openshift.io
    kind: AgentClusterInstall
    name: "{{ .Cluster.ClusterName }}"
    version: v1beta1
  clusterName: "{{ .Cluster.ClusterName }}"
  # Override: clusterMetadata
  clusterMetadata:
    adminKubeconfigSecretRef:
      name: zt-sno4-admin-kubeconfig
    clusterID: ""
    infraID: ""
  platform:
    agentBareMetal:
      agentSelector:
        matchLabels:
          cluster-name: "{{ .Cluster.ClusterName }}"
  pullSecretRef:
    name: "{{ .Site.PullSecretRef.Name }}"

Now, add the crTemplates section to the site-config so that the new files are honored. Also verify the tree structure is as follows:

clusterNetwork:
  - cidr: "fd01::/48"
    hostPrefix: 64
machineNetwork:
  - cidr: "2600:52:7:59::/64"
serviceNetwork:
  - "fd02::/112"
networkType: "OVNKubernetes"
additionalNTPSources:
  - "2600:52:7:59::11"
crTemplates:
  BareMetalHost: "BareMetalHostOverride.yaml"
  ClusterDeployment: "ClusterDeploymentOverride.yaml"

# tree
├── BareMetalHostOverride.yaml
├── ClusterDeploymentOverride.yaml
├── hub-import-secret.yaml
├── inbound-example-bos2-lab.yaml
├── kustomization.yaml
└── zt-sno4.yaml

Now, kick off a sync between ArgoCD and what’s being added to git. Use the Argo client we downloaded earlier and be sure to commit the overrides and the site-config additions for “zt-sno4”.

oc get applications.argoproj.io clusters -n openshift-gitops -o json | jq .spec.source
{
  "path": "site-configs",
  "repoURL": "http://jumphost.inbound.example.bos2.lab:3000/kni/faredge-ztp.git",
  "targetRevision": "master"
}

cat ~/git/faredge/site-configs/kustomization.yaml

generators:
# list of all the siteConfig*.yaml exist in the repository
  - zt-sno1-ipv6.yaml
  - zt-sno4.yaml


# Login to OpenShift argo via route
argoPass=$(oc get secret/openshift-gitops-cluster -n openshift-gitops -o \
 jsonpath='{.data.admin\.password}' | base64 -d)

argoURL=$(oc get route openshift-gitops-server -n openshift-gitops -o \
 jsonpath='{.spec.host}{"\n"}')

argocd --insecure --grpc-web $argoURL  --username \ 
admin --password $argoPass

# Sync both policies and clusters
argocd-linux-amd64 app sync policies
argocd-linux-amd64 app sync clusters

Verify the cluster has been added to Argo

Let’s verify the cluster has been added to Argo and is in a healthy state. If for any reason you are missing annotations or labels you can enable Argo’s self healing setting for the application and it will match the existing CRs with what’s in git.

Note: Many of the other fields in the assisted service CRs are immutable and can cause an error in Argo if they are not present.

# Collect a diff between the manifests stored in git and the ones present on the cluster
argocd app diff clusters

..omitted..

# Add any writable settings such as annotations/labels to the CRs in OpenShift

argocd app set clusters --self-heal

# Rerun a sync and verify the application is healthy
argocd app sync clusters

…omitted..

agent.open-cluster-management.io    KlusterletAddonConfig  zt-sno4      zt-sno4                     Synced                  klusterletaddonconfig.agent.open-cluster-management.io/zt-sno4 configured. Warning: resource klusterletaddonconfigs/zt-sno4 is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by  apply.  apply should only be used on resources created declaratively by either  create --save-config or  apply. The missing annotation will be patched automatically.
                                    Namespace                           zt-sno1                     Synced                   
                                    Namespace                           zt-sno4                     Synced                   
cluster.open-cluster-management.io  ManagedCluster                      zt-sno1                     Synced                   
cluster.open-cluster-management.io  ManagedCluster                      zt-sno4                     Synced  

Now, the cluster should be added back to Argo without having a redeployment kicked off. Last, enable automated syncing as the final step.

oc get applications.argoproj.io -A

NAMESPACE          NAME       SYNC STATUS   HEALTH STATUS
openshift-gitops   clusters   Synced        Healthy
openshift-gitops   policies   Synced        Healthy

argocd app set clusters --sync-policy automated

Now, confirm the cluster has been added back and the policies have been processed.

oc get managedcluster zt-sno4
NAME      HUB ACCEPTED   MANAGED CLUSTER URLS                           JOINED   AVAILABLE   AGE
zt-sno4   true           https://api.zt-sno4.inbound.example.bos2.lab:6443   True     True        3d5h

oc --kubeconfig=kubeconfig-zt-sno4.yaml get policies -A

NAMESPACE   NAME                                                           REMEDIATION ACTION   COMPLIANCE STATE   AGE
zt-sno3     ztp-vdu-mb-policies.vdu-mb-22a-config-policy         inform               Compliant          5h42m
zt-sno3     ztp-vdu-mb-policies.vdu-mb-22a-sriov-policy          inform               Compliant          5h42m
zt-sno3     ztp-vdu-mb-policies.vdu-mb-valid-22a                 inform               Compliant          5h42m
zt-sno3     ztp-group-du-sno.group-du-sno-414-prega-config-policy          inform               Compliant          5h42m
zt-sno3     ztp-group-du-sno.group-du-sno-414-prega-cs-config-policy       inform               Compliant          5h42m
zt-sno3     ztp-group-du-sno.group-du-sno-414-prega-subscriptions-policy   inform               Compliant          5h42m

Resources
https://github.com/noseka1/multicluster-management-rhacm-argocd
https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/
https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.5/html-single/clusters/index#importing-the-cluster-manual
https://github.com/openshift/assisted-service/blob/386410998c9bc84a5f7463bd3298884c4aa923be/docs/hive-integration/import-installed-cluster.md#4-copy-the-kubeconfig-from-the-ocp-cluster-into-the-hub