6.3. Forwarding logs using the Log Forwarding API

The Log Forwarding API enables you to configure custom pipelines to send container and node logs to specific endpoints within or outside of your cluster. You can send logs by type to the internal OpenShift Container Platform Elasticsearch instance and to remote destinations not managed by OpenShift Container Platform cluster logging, such as an existing logging service, an external Elasticsearch cluster, external log aggregation solutions, or a Security Information and Event Management (SIEM) system.

重要

The Log Fowarding API is currently a Technology Preview feature. Technology Preview features are not supported with Red Hat production service level agreements (SLAs), might not be functionally complete, and Red Hat does not recommend to use them for production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

See the Red Hat Technology Preview features support scope for more information.

You can send different types of logs to different systems allowing you to control who in your organization can access each type. Optional TLS support ensures that you can send logs using secure communication as required by your organization.

Using the Log Forwarding API is optional. If you want to forward logs to only the internal OpenShift Container Platform Elasticsearch instance, do not configure the Log Forwarding API.

6.3.1. Understanding the Log Forwarding API

Forwarding cluster logs by using the Log Forwarding API requires a combination of outputs and pipelines. These resources send logs to specific endpoints inside and outside of your OpenShift Container Platform cluster.

注意

If you want to use only the default internal OpenShift Container Platform Elasticsearch logstore, do not configure any outputs and pipelines.

An output is the destination for log data and a pipeline defines simple routing for one source to one or more outputs.

An output can be either:

  • elasticsearch to forward logs to an external Elasticsearch 6 (all releases) cluster, specified by server name or FQDN, and/or the internal OpenShift Container Platform Elasticsearch logstore.
  • forward to forward logs to an external log aggregation solution. This option uses the Fluentd forward protocols.

A pipeline associates the type of data to an output. A type of data you can forward is one of the following:

  • logs.app - Container logs generated by user applications running in the cluster, except infrastructure container applications.
  • logs.infra - Logs generated by both infrastructure components running in the cluster and OpenShift Container Platform nodes, such as journal logs. Infrastructure components are pods that run in the openshift*, kube*, or default projects.
  • logs.audit - Logs generated by the node audit system (auditd), which are stored in the /var/log/audit/audit.log file, and the audit logs from the Kubernetes apiserver and the OpenShift apiserver.

To use the Log Forwarding API, you create a custom logforwarding configuration file with outputs and pipelines to send logs to destinations you specify.

Note the following:

  • The internal OpenShift Container Platform Elasticsearch logstore does not provide secure storage for audit logs. We recommend you ensure that the system to which you forward audit logs is compliant with your organizational and governmental regulations and is properly secured. OpenShift Container Platform cluster logging does not comply with those regulations.
  • An output supports TLS communication using a secret. Secrets must have keys of: tls.crt, tls.key, and ca-bundle.crt which point to the respective certificates for which they represent. Secrets must have the key shared_key for use when using forward in a secure manner.
  • You are responsible for creating and maintaining any additional configurations that external destinations might require, such as keys and secrets, service accounts, port opening, or global proxy configuration.

The following example creates three outputs:

  • the internal OpenShift Container Platform Elasticsearch logstore,
  • an unsecured externally-managed Elasticsearch logstore,
  • a secured external log aggregator using the forward protocol.

Three pipelines send:

  • the application logs to the internal OpenShift Container Platform Elasticsearch logstore,
  • the infrastructure logs to an external Elasticsearch logstore,
  • the audit logs to the secured device over the forward protocol.

Sample log forwarding outputs and pipelines

apiVersion: "logging.openshift.io/v1alpha1"
kind: "LogForwarding"
metadata:
  name: instance 1
  namespace: openshift-logging
spec:
  disableDefaultForwarding: true 2
  outputs: 3
   - name: elasticsearch 4
     type: "elasticsearch"  5
     endpoint: elasticsearch.openshift-logging.svc:9200 6
     secret: 7
        name: fluentd
   - name: elasticsearch-insecure
     type: "elasticsearch"
     endpoint: elasticsearch-insecure.messaging.svc.cluster.local
     insecure: true 8
   - name: secureforward-offcluster
     type: "forward"
     endpoint: https://secureforward.offcluster.com:24224
     secret:
        name: secureforward
  pipelines: 9
   - name: container-logs 10
     inputSource: logs.app 11
     outputRefs: 12
     - elasticsearch
     - secureforward-offcluster
   - name: infra-logs
     inputSource: logs.infra
     outputRefs:
     - elasticsearch-insecure
   - name: audit-logs
     inputSource: logs.audit
     outputRefs:
     - secureforward-offcluster

1
The name of the log forwarding CR must be instance.
2
Parameter to enable log forwarding. Set to true to enable log forwarding.
3
Configuration for the outputs.
4
A name to describe the output.
5
The type of output, either elasticsearch or forward.
6
The log forwarding endpoint, either the server name or FQDN. For the internal OpenShift Container Platform Elasticsearch logstore, specify elasticsearch.openshift-logging.svc:9200.
7
Optional name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project.
8
Optional setting if the endpoint does not use a secret, resulting in insecure communication.
9
Configuration for the pipelines.
10
A name to describe the pipeline.
11
The source type, logs.app, logs.infra, or logs.audit.
12
The name of one or more outputs configured in the CR.
Fluentd log handling when the external log aggregator is unavailable

If your external logging aggregator becomes unavailable and cannot receive logs, Fluentd continues to collect logs and stores them in a buffer. When the log aggregator becomes available, log forwarding resumes, including the buffered logs. If the buffer fills completely, Fluentd stops collecting logs. OpenShift Container Platform rotates the logs and deletes them. You cannot adjust the buffer size or add a persistent volume claim (PVC) to the Fluentd daemon set or pods.

注意

Because the internal OpenShift Container Platform Elasticsearch log store does not provide secure storage for audit logs, audit logs are not stored in the internal Elasticsearch instance by default. If you want to send the audit logs to the internal log store, for example to view the audit logs in Kibana, you must use the Log Forwarding API as described in Forward audit logs to the log store.

6.3.2. Enabling the Log Forwarding API

You must enable the Log Forwarding API before you can forward logs using the API.

Procedure

To enable the Log Forwarding API:

  1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project: 

    $ oc edit ClusterLogging instance

  2. Add the clusterlogging.openshift.io/logforwardingtechpreview annotation and set to enabled: 

    apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  annotations:
    clusterlogging.openshift.io/logforwardingtechpreview: enabled 1
  name: "instance"
  namespace: "openshift-logging"
spec:

...

  collection: 2
    logs:
      type: "fluentd"
      fluentd: {}
    1
    Enables and disables the Log Forwarding API. Set to enabled to use log forwarding. To use the only the OpenShift Container Platform Elasticsearch instance, set to disabled or do not add the annotation.
    2
    The spec.collection block must be defined to use Fluentd in the ClusterLogging CR.

6.3.3. Configuring log forwarding using the Log Forwarding API

To configure the Log Forwarding, edit the ClusterLogging custom resource (CR) to add the clusterlogging.openshift.io/logforwardingtechpreview: enabled annotation and create a LogForwarding custom resource to specify the outputs, pipelines, and enable log forwarding.

If you enable Log Forwarding, you should define a pipeline all for three source types: logs.app, logs.infra, and logs.audit. The logs from any undefined source type are dropped. For example, if you specify a pipeline for the logs.app and log-audit types, but do not specify a pipeline for the logs.infra type, logs.infra logs are dropped.

Procedure

To configure log forwarding using the API:

  1. Create a LogForwarding CR YAML file similar to the following: 

    apiVersion: "logging.openshift.io/v1alpha1"
kind: "LogForwarding"
metadata:
  name: instance 1
  namespace: openshift-logging 2
spec:
  disableDefaultForwarding: true 3
  outputs: 4
   - name: elasticsearch
     type: "elasticsearch"
     endpoint: elasticsearch.openshift-logging.svc:9200
     secret:
        name: fluentd
   - name: elasticsearch-insecure
     type: "elasticsearch"
     endpoint: elasticsearch-insecure.messaging.svc.cluster.local
     insecure: true
   - name: secureforward-offcluster
     type: "forward"
     endpoint: https://secureforward.offcluster.com:24224
     secret:
        name: secureforward
  pipelines: 5
   - name: container-logs
     inputSource: logs.app
     outputRefs:
     - elasticsearch
     - secureforward-offcluster
   - name: infra-logs
     inputSource: logs.infra
     outputRefs:
     - elasticsearch-insecure
   - name: audit-logs
     inputSource: logs.audit
     outputRefs:
     - secureforward-offcluster
    1
    The name of the log forwarding CR must be instance.
    2
    The namespace for the log forwarding CR must be openshift-logging.
    3
    Set to true to disable the default log forwarding behavior.
    4
    Add one or more endpoints:
    • Specify the type of output, either elasticsearch or forward.
    • Enter a name for the output.
    • Enter the endpoint, either the server name, FQDN, or IP address. If the cluster-wide proxy using the CIDR annotation is enabled, the endpoint must be a server name or FQDN, not an IP Address. For the internal OpenShift Container Platform Elasticsearch instance, specify elasticsearch.openshift-logging.svc:9200.
    • Optional: Enter the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project.
    • Specify insecure: true if the endpoint does not use a secret, resulting in insecure communication.
    5
    Add one or more pipelines:
    • Enter a name for the pipeline
    • Specify the source type: logs.app, logs.infra, or logs.audit.

    • Specify the name of one or more outputs configured in the CR.

      注意

      If you set disableDefaultForwarding: true you must configure a pipeline and output for all three types of logs, application, infrastructure, and audit. If you do not specify a pipeline and output for a log type, those logs are not stored and will be lost.

  2. Create the CR object: 

    $ oc create -f <file-name>.yaml

6.3.3.1. Example log forwarding custom resources

A typical Log Forwarding configuration would be similar to the following examples.

The following Log Forwarding custom resource sends all logs to a secured external Elasticsearch log store:

Sample custom resource to forward to an Elasticsearch log store

apiVersion: logging.openshift.io/v1alpha1
kind: LogForwarding
metadata:
  name: instance
  namespace: openshift-logging
spec:
  disableDefaultForwarding: true
  outputs:
    - name: user-created-es
      type: elasticsearch
      endpoint: 'elasticsearch-server.openshift-logging.svc:9200'
      secret:
        name: piplinesecret
  pipelines:
    - name: app-pipeline
      inputSource: logs.app
      outputRefs:
        - user-created-es
    - name: infra-pipeline
      inputSource: logs.infra
      outputRefs:
        - user-created-es
    - name: audit-pipeline
      inputSource: logs.audit
      outputRefs:
        - user-created-es

The following Log Forwarding custom resource sends all logs to a secured Fluentd instance using the Fluentd forward protocol.

Sample custom resource to use the forward protocol

apiVersion: logging.openshift.io/v1alpha1
kind: LogForwarding
metadata:
  name: instance
  namespace: openshift-logging
spec:
  disableDefaultForwarding: true
  outputs:
    - name: fluentd-created-by-user
      type: forward
      endpoint: 'fluentdserver.openshift-logging.svc:24224'
      secret:
        name: fluentdserver
  pipelines:
    - name: app-pipeline
      inputSource: logs.app
      outputRefs:
        - fluentd-created-by-user
    - name: infra-pipeline
      inputSource: logs.infra
      outputRefs:
        - fluentd-created-by-user
    - name: clo-default-audit-pipeline
      inputSource: logs.audit
      outputRefs:
        - fluentd-created-by-user

6.3.4. Disabling the Log Forwarding API

To disable the Log Forwarding API and to stop forwarding logs to the speified endpoints, remove the metadata.annotations.clusterlogging.openshift.io/logforwardingtechpreview:enabled parameter from the ClusterLogging CR and delete the LogForwarding CR. The container and node logs will be forwarded to the internal OpenShift Container Platform Elasticsearch instance.

注意

Setting disableDefaultForwarding=false prevents cluster logging from sending logs to the specified endpoints and to default internal OpenShift Container Platform Elasticsearch instance.

Procedure

To disable the Log Forwarding API:

  1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project: 

    $ oc edit ClusterLogging instance

  2. Remove the clusterlogging.openshift.io/logforwardingtechpreview annotation: 

    apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  annotations:
    clusterlogging.openshift.io/logforwardingtechpreview: enabled 1
  name: "instance"
  namespace: "openshift-logging"
....
    1
    Remove this annotation.

  3. Delete the Log Forwarding custom resource: 

    $ oc delete LogForwarding instance -n openshift-logging