How to configure and capture HAproxy logs for the OpenShift Default Router

Resolution

In Openshift, the Ingress is deployed as an operator, and it is available in the openshift-ingress-operator namespace and implemented using HAProxy in the background. By default, there is one instance of the Ingress Controller named default. This is the one that includes the configuration of the Ingress that is being deployed.

In order to enable debugging and capture logs, we need to add an additional configuration to the Ingress Controller resource. This can be done by following the procedure below:

1. Log into the cluster as user with cluster-admin role.

2. Change to openshift-ingress-operator namespace:

   $ oc project openshift-ingress-operator

3. Check the default IngressController:

   $ oc get ingresscontroller
   NAME      AGE
   default   69m
   

4. Create a backup of the IngressController resource:

   $ oc get ingresscontroller/default -o yaml > ingresscontroller_default_bkp.yaml

5. Edit the default IngressController resource:

   $ oc edit ingresscontroller/default

6. Add the snippet under the spec parameter that starts the definition of the specification of the IngressController:

   logging:
   access:
     destination:
       type: Container
     httpLogFormat: >-
       log_source="haproxy-default" log_type="http" c_ip="%ci" c_port="%cp"
       req_date="%tr" fe_name_transport="%ft" be_name="%b" server_name="%s"
       res_time="%TR" tot_wait_q="%Tw" Tc="%Tc" Tr="%Tr" Ta="%Ta"
       status_code="%ST" bytes_read="%B" bytes_uploaded="%U"
       captrd_req_cookie="%CC" captrd_res_cookie="%CS" term_state="%tsc"
       actconn="%ac" feconn="%fc" beconn="%bc" srv_conn="%sc" retries="%rc"
       srv_queue="%sq" backend_queue="%bq" captrd_req_headers="%hr"
       captrd_res_headers="%hs" http_request="%r"
   

   Example:

   ...
   spec:
     clientTLS:
       clientCA:
         name: ""
       clientCertificatePolicy: ""
     defaultCertificate:
       name: xxx-primary-cert-bundle-secret
     domain: xxx.xxxx.com
     httpCompression: {}
     httpEmptyRequestsPolicy: Respond
     httpErrorCodePages:
       name: ""
     logging:
       access:
         destination:
           type: Container
         httpLogFormat: log_source="haproxy-default" log_type="http" c_ip="%ci" c_port="%cp"
           req_date="%tr" fe_name_transport="%ft" be_name="%b" server_name="%s" res_time="%TR"
           tot_wait_q="%Tw" Tc="%Tc" Tr="%Tr" Ta="%Ta" status_code="%ST" bytes_read="%B"
           bytes_uploaded="%U" captrd_req_cookie="%CC" captrd_res_cookie="%CS" term_state="%tsc"
           actconn="%ac" feconn="%fc" beconn="%bc" srv_conn="%sc" retries="%rc" srv_queue="%sq"
           backend_queue="%bq" captrd_req_headers="%hr" captrd_res_headers="%hs" http_request="%r"
         logEmptyRequests: Log
     nodePlacement:
       nodeSelector:
         matchLabels:
           node-role.kubernetes.io/infra: ""
   

   This will deploy another container on the router pods in the openshift-ingress namespace following the sidecar pattern named logs. Also this container will print the logs from the requests reaching the ingress component, so next time the consumer is not able to call a service, you will be able to see the incoming requests with all their metadata.

7. By default, OpenShift deploys two replicas of router-default pod in the openshift-ingress namespace. We need to check and capture logs from both pods to make sure we capture all the traffic which can happen through one of the two routers:

   oc get pod -n openshift-ingress 
   
   NAME                             READY   STATUS    RESTARTS   AGE
   router-default-7566d94ff-j2kj5   2/2     Running   0          3m23s
   router-default-7566d94ff-sbsw4   2/2     Running   0          3m22s
   

8. Capturing the logs and sending it to a text file:

   $ oc logs -c logs router-default-7566d94ff-j2kj5 -n openshift-ingress | tee -a router-default.log
   

9. Once the debugging is completed, reverse the configuration as backed up before.

Reference: Configuring Ingress access logging.