Failed to configure audit log forwarding in ROSA HCP clusters

Solution Unverified - Updated -

Environment

  • Red Hat OpenShift Service on AWS Hosted Control Planes (ROSA HCP)
    • 4
  • AWS

Issue

  • Following the solution Enabling CloudWatch Audit Log Forwarding for ROSA HCP cluster, the ocm patch command throws an error:

    Failed to configure audit log forwarding with role: arn:aws:iam::[AWS_account]:role/[IAM_ROLE_NAME]: Please check if the OIDC provider or the role's trust policy is correctly configured

    Failed to configure audit log forwarding with role: arn:aws:iam::[AWS_account]:role/[IAM_ROLE_NAME]: Failed to find provided audit log forwarding role

Resolution

It is needed to ensure that all the steps in Enabling CloudWatch Audit Log Forwarding for ROSA HCP cluster were followed, as they get the data from the cluster information.

Check the OIDC provider

Ensure that the OIDC provider configured is valid. To do so, check the OIDC Endpoint URL:

  1. Get the OIDC Endpoint URL from the rosa describe cluster command:

    $ rosa describe cluster --cluster [cluster_ID_or_cluster_name]
[...]
OIDC Endpoint URL:          https://rh-oidc.s3.us-east-1.amazonaws.com/xxxxxxxxxxxxxxxxxxxxxxxx (Managed)

  2. Run the rosa create oidc-provider command and select mode auto and the same OIDC provider shown by above command:

    $ rosa create oidc-provider
? OIDC provider creation mode: auto
? OIDC Configuration ID: xxxxxxxxxxxxxxxxxxxxxxxx | https://rh-oidc.s3.us-east-1.amazonaws.com/xxxxxxxxxxxxxxxxxxxxxxxx
I: OIDC provider already exists.

    If the OIDC already configured is OK, the message OIDC provider already exists will be shown.

  3. Ensure that the OIDC provider is the same in the trust relationship configured when following Enabling CloudWatch Audit Log Forwarding for ROSA HCP cluster.

Note: If the OIDC provider is not managed by the ROSA installation, but a custom one, it's possible that the rosa describe cluster command shows an OIDC Endpoint URL without the ID.

Check the role

  1. Check that the role exists and the trust relationship is correct (in some cases, it could be needed to add the --profile parameter with a valid [PROFILE_NAME] from the credential file to the command):

    $ aws iam get-role --role-name [IAM_ROLE_NAME]

  2. Ensure that the [AWS_account] in the role_arn is the same than the one shown in the rosa describe cluster -c $CLUSTER_ID | grep "STS Role ARN:" command.

Root Cause

It is needed to check if the OIDC provider configured in the cluster is valid, and if the trust relationship is using it. Also, ensure that the role_arn is correct.

Diagnostic Steps

Run the ocm patch command for configuring the role_arn for the audit_log with --debug parameter to try to find additional information in the debug logs:

$ ocm patch /api/clusters_mgmt/v1/clusters/$CLUSTER_ID --debug << EOM
{
    "aws": {"audit_log": {"role_arn":"$IAM_ROLE_ARN"}}
}
EOM

[...]
{
  "kind": "Error",
  "id": "400",
  "href": "/api/clusters_mgmt/v1/errors/400",
  "code": "CLUSTERS-MGMT-400",
  "reason": "Failed to configure audit log forwarding with role: arn:aws:iam::[AWS_account]:role/[IAM_ROLE_NAME]: Please check if the OIDC provider or the role's trust policy is correctly configured",
  "operation_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

[...]
{
  "kind": "Error",
  "id": "400",
  "href": "/api/clusters_mgmt/v1/errors/400",
  "code": "CLUSTERS-MGMT-400",
  "reason": "Failed to configure audit log forwarding with role: arn:aws:iam::[AWS_account]:role/[IAM_ROLE_NAME]: Failed to find provided audit log forwarding role",
  "operation_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments