How to view logs about changes made to Deployments from Kibana ?

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Logging subsystem for Red Hat OpenShift
    • 5

Issue

  • How to identify the user who deleted a given Deployment ?
  • Why are audit logs not forwarded to Kibana?
  • Why is it not possible to create an audit index pattern in Kibana even after configuring ClusterLogging CRD ?

  • Kubeapiserver audit logs shows logs related to all changes made to a Deployment resource by running:

    $ ssh master-0.cluster.example.com sudo cat /var/log/kube-apiserver/audit.log >         /tmp/clusteraudit.log && \
ssh master-1.cluster.example.com sudo cat /var/log/kube-apiserver/audit.log >> /tmp/clusteraudit.log && \
ssh master-2.cluster.example.com sudo cat /var/log/kube-apiserver/audit.log >> /tmp/clusteraudit.log && \
cat /tmp/clusteraudit.log | jq '. | select((.objectRef.resource=="deployments") \
and (.objectRef.namespace=="sbelmasg")) | \
.objectRef.namespace + " " + .objectRef.name + " " + .verb + " " + .user.username + " " + .stageTimestamp'

How can the same logs viewed from Kibana?

Resolution

  • Configure ClusterLogForwarder to forward audit logs to internal or external log collectors. To be able to view any changes made to Deployment workloads, application logs for the concerned namespaces should also be forwarded in ClusterLogForwarder specs. Below is an example that forwards application & audit logs for all workloads in test1 & test2 namespaces to the default(i.e. internal Elasticsearch instance)

    apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  inputs:
  - application:
      namespaces:
      - test1
      - test2
    name: application-logs
  pipelines:
  - inputRefs:
    - application-logs
    name: application-logs
    outputRefs:
    - default
  - inputRefs:
    - infrastructure
    - audit
    name: infra_and_audit_logs
    outputRefs:
    - default

  • Events are not forwarded by default. Event Router pods forward events related to audit index pattern. Follow Event Router pods to deploy Event Router Pods.

  • Create audit index pattern in Kibana

  • Login to Kibana to view Deployment logs in audit index using filters.
    Example filters can be: objectRef.resource:"Deployments", objectRef.namespace:"test1,test2", objectRef.name:"", verb:"create,get,update,patch", user.username:""

Root Cause

  • The audit logs are not stored in the internal OpenShift Container Platform Elasticsearch instance by default. To view the audit logs in Kibana, create ClusterLogForwarder CRD to configure a pipeline that uses either default output or external log collector for audit logs.

  • The audit logs are also not collected by default by the collector component. Event Router pods must be deployed for this purpose. Event Router pod watch Kubernetes events and log them for collection by the logging subsystem.

  • audit index patterns can only be created in Kibana if audit logs are forwarded in ClusterLogForwarder specs.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments