How to configure and capture HAproxy logs for the OpenShift Default Router

Resolution

In Openshift the Ingress is deployed as an Operator, and it is available in the openshift-ingress-operator namespace and implemented using HAProxy in the backgroung. By default, there is one instance of the Ingress Controller named default. This is the one that includes the configuration of the ingress that is being deployed.

In order to enable debugging and capture logs we need to add an additional configuration to the Ingress Controller resource. This can be done using ether OpenShift Management Console or CLI.

The method below describes CLI way of accomplishing this task.
1. Log into the cluster as user with cluster-admin role
2. Change to openshift-ingress-operator namespace

$ oc project openshift-ingress-operator

3. Verify default ingress is available

$ oc get ingresscontroller

NAME      AGE
default   69m

4. Edit the default IngressController resource

oc edit ingresscontrollers/default

and add the snippet under the spec parameter that starts the definition of the specification of the IngressController itself:

   logging:
    access:
      destination:
        type: Container
      httpLogFormat: >-
        log_source="haproxy-default" log_type="http" c_ip="%ci" c_port="%cp"
        req_date="%tr" fe_name_transport="%ft" be_name="%b" server_name="%s"
        res_time="%TR" tot_wait_q="%Tw" Tc="%Tc" Tr="%Tr" Ta="%Ta"
        status_code="%ST" bytes_read="%B" bytes_uploaded="%U"
        captrd_req_cookie="%CC" captrd_res_cookie="%CS" term_state="%tsc"
        actconn="%ac" feconn="%fc" beconn="%bc" srv_conn="%sc" retries="%rc"
        srv_queue="%sq" backend_queue="%bq" captrd_req_headers="%hr"
        captrd_res_headers="%hs" http_request="%r"

Example:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  creationTimestamp: "2023-04-22T08:38:18Z"
  finalizers:
  - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
  generation: 4
  name: default
  namespace: openshift-ingress-operator
  resourceVersion: "98353"
  uid: 88a9834c-05b6-43fc-b6d0-85a284482318
spec:
  clientTLS:
    clientCA:
      name: ""
    clientCertificatePolicy: ""
  defaultCertificate:
    name: xxx-primary-cert-bundle-secret
  domain: xxx.xxxx.com
  httpCompression: {}
  httpEmptyRequestsPolicy: Respond
  httpErrorCodePages:
    name: ""
  logging:
    access:
      destination:
        type: Container
      httpLogFormat: log_source="haproxy-default" log_type="http" c_ip="%ci" c_port="%cp"
        req_date="%tr" fe_name_transport="%ft" be_name="%b" server_name="%s" res_time="%TR"
        tot_wait_q="%Tw" Tc="%Tc" Tr="%Tr" Ta="%Ta" status_code="%ST" bytes_read="%B"
        bytes_uploaded="%U" captrd_req_cookie="%CC" captrd_res_cookie="%CS" term_state="%tsc"
        actconn="%ac" feconn="%fc" beconn="%bc" srv_conn="%sc" retries="%rc" srv_queue="%sq"
        backend_queue="%bq" captrd_req_headers="%hr" captrd_res_headers="%hs" http_request="%r"
      logEmptyRequests: Log
  nodePlacement:
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/infra: ""
    tolerations:

This will deploy another container on the router pods in the openshift-ingress namespace following the sidecar pattern named logs. Also this container will print the logs from the requests reaching the ingress component, so next time the consumer is not able to call a service, you will be able to see the incoming requests with all their metadata.

5. By default Openshift deploys two replicas of router-default pod in the openshift-ingress namespace. We need to check and capture logs from both pods to make sure we capture traffic we need as the communication will happen though one of the routers.

oc get po -n openshift-ingress 

NAME                             READY   STATUS    RESTARTS   AGE
router-default-7566d94ff-j2kj5   2/2     Running   0          3m23s
router-default-7566d94ff-sbsw4   2/2     Running   0          3m22s

Example to capture logs and pipe it into a file.

$ oc logs -c logs router-default-7566d94ff-j2kj5 -n openshift-ingress | tee -a router-default.log

The log file can be attached to the support request if required.

6. Once the debugging is completed reverse the configuration the same way it was added above.