Guardrails Sampling Rollout in Dryrun Mode on ARO Clusters

Azure Red Hat OpenShift (ARO) is introducing a new feature called "Guardrails" to enhance the safety measures during daily operations. As a precursor to the full deployment, a sampling rollout is scheduled in the ARO clusters.

This KCS outlines the potential symptoms and observations you might encounter during this rollout.

During the rollout, you may notice one or more of the following symptoms in your ARO clusters:

1. Presence of Gatekeeper Binaries in the openshift-azure-guardrails Namespace

   You may observe the presence of gatekeeper binaries in the openshift-azure-guardrails namespace. To verify, use the following command:

$ oc get all -n openshift-azure-guardrails

The output should display the following:

NAME                                                 READY   STATUS    RESTARTS      AGE
pod/gatekeeper-audit-86c44946ff-k8bm6                1/1     Running   1 (14m ago)   14m
pod/gatekeeper-controller-manager-5cdfdcc845-2r7b5   1/1     Running   0             14m
pod/gatekeeper-controller-manager-5cdfdcc845-5c4bx   1/1     Running   0             14m
pod/gatekeeper-controller-manager-5cdfdcc845-f5jmd   1/1     Running   0             14m

NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/gatekeeper-webhook-service   ClusterIP   172.30.53.244   <none>        443/TCP   14m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/gatekeeper-audit                1/1     1            1           14m
deployment.apps/gatekeeper-controller-manager   3/3     3            3           14m

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/gatekeeper-audit-86c44946ff                1         1         1       14m
replicaset.apps/gatekeeper-controller-manager-5cdfdcc845   3         3         3       14m

2. Presence of Gatekeeper CRDs in the Default Namespace

   Gatekeeper Custom Resource Definitions (CRDs) might be observed in the default namespace. To check, use the following command:

$ oc get crd |grep gatekeeper.sh

The output should display the following:

arodenyhostmount.constraints.gatekeeper.sh                        2023-09-11T00:23:37Z
arodenylabels.constraints.gatekeeper.sh                           2023-09-11T00:23:37Z
arodenymachineconfig.constraints.gatekeeper.sh                    2023-09-11T00:23:37Z
arodenymastertolerationtaints.constraints.gatekeeper.sh           2023-09-11T00:23:41Z
arodenyprivilegednamespace.constraints.gatekeeper.sh              2023-09-11T00:23:41Z
assign.mutations.gatekeeper.sh                                    2023-09-11T00:23:09Z
assignmetadata.mutations.gatekeeper.sh                            2023-09-11T00:23:09Z
configs.config.gatekeeper.sh                                      2023-09-11T00:23:09Z
constraintpodstatuses.status.gatekeeper.sh                        2023-09-11T00:23:09Z
constrainttemplatepodstatuses.status.gatekeeper.sh                2023-09-11T00:23:09Z
constrainttemplates.templates.gatekeeper.sh                       2023-09-11T00:23:10Z
expansiontemplate.expansion.gatekeeper.sh                         2023-09-11T00:23:10Z
modifyset.mutations.gatekeeper.sh                                 2023-09-11T00:23:09Z
mutatorpodstatuses.status.gatekeeper.sh                           2023-09-11T00:23:09Z
providers.externaldata.gatekeeper.sh                              2023-09-11T00:23:10Z

3. Observation of Gatekeeper Constraints and ConstraintTemplates in Dryrun Mode

   The Gatekeeper Constraints and ConstraintTemplates may be observed in dryrun mode in the default namespace. To verify, use the following commands:

$ oc get constraint

The expected output will be:

$ oc get constrainttemplate

The expected output will be:

The output should indicate that the constraints are deployed in dryrun mode, with no violations reported at this stage.

Conclusion

Currently, the constraints are deployed in dry-run mode, meaning no operations will be blocked at this stage. The only drawback is the extra resources consumed by the gatekeeper deployment.

Please don't worry if you come across these modifications. Additional updates will be provided once the decision is made to activate the Guardrails feature, which aims to improve safety measures in ARO clusters. We value your understanding and cooperation.