SRE Splunk can not get audit log from Private Link ROSA cluster

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
    • 4

Issue

  • When customer contact SRE to get audit log from ROSA private link cluster ,there are no audit log gathered , even customer already opened the firewall and checked the connection from cluster to splunk through Proxy returns 200 Connection established.

Resolution

  • Exclude splunk related URL in below document from proxy re-encryption. Hence setup the URLs only to bypass the proxy as per the document.

  • Before above setting got changed , Audit log still can be get using below solution as a workaround.
    KCS.

Root Cause

  • The splunk-forwarder-operator as a log forwarding endpoint to be used by Red Hat SRE for log-based alerting Not bypass the proxy in private link ROSA

Diagnostic Steps

  • Check the connection using below command through proxy returns 200
$ oc exec -n openshift-monitoring -it alertmanager-main-1 -- /bin/bash
bash4.4$ curl -v --proxy http://vpce-xxxxxxxx-xxxx.vpce-svc-xxxxxx.ap-northeast-1.vpce.amazonaws.com:80 telnet://inputs1.osdsecuritylogs.splunkcloud.com:9997 -m 20
* Rebuilt URL to: telnet://inputs1.osdsecuritylogs.splunkcloud.com:9997/
*   Trying 10.xxx.xxx.xxx...
* TCP_NODELAY set
* Connected to vpce-xxxxxxxx-xxxx.vpce-svc-xxxxxx.ap-northeast-1.vpce.amazonaws.com (10.xxx.xxx.xxx) port 80 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to inputs1.osdsecuritylogs.splunkcloud.com:9997
> CONNECT inputs1.osdsecuritylogs.splunkcloud.com:9997 HTTP/1.1
> Host: inputs1.osdsecuritylogs.splunkcloud.com:9997
> User-Agent: curl/7.61.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
  • However , Check the connection using below command without using proxy returns Timeout
$ oc exec alertmanager-main-1 -n openshift-monitoring -- curl -v inputs1.osdsecuritylogs.splunkcloud.com:9997 -m 20
......
* Failed to connect to inputs1.osdsecuritylogs.splunkcloud.com port 9997: Connection timed out
* Closing connection 0

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments