第 6 章 Forwarding logs to third party systems

By default, OpenShift Container Platform cluster logging sends logs to the default internal Elasticsearch log store, defined in the ClusterLogging custom resource (CR).

You can configure cluster logging to send logs to destinations outside of your OpenShift Container Platform cluster instead of the default Elasticsearch log store using the following methods:

  • Sending logs using the Fluentd forward protocol. You can create a config map to use the Fluentd forward protocol to securely send logs to an external logging aggregator that accepts the Fluent forward protocol.
  • Sending logs using syslog. You can create a config map to use the syslog protocol to send logs to an external syslog (RFC 3164) server.

Alternatively, you can use the Log Forwarding API, currently in Technology Preview. The Log Forwarding API, which is easier to configure than the Fluentd protocol and syslog, exposes configuration for sending logs to the internal Elasticsearch log store and to external Fluentd log aggregation solutions.

You cannot use the config map methods and the Log Forwarding API in the same cluster.

重要

The Log Forwarding API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

The methods for forwarding logs using a config map are deprecated and will be replaced by the Log Forwarding API in a future release.

6.1. Forwarding logs using the Fluentd forward protocol

You can use the Fluentd forward protocol to send a copy of your logs to an external log aggregator, instead of the default Elasticsearch log store. On the OpenShift Container Platform cluster, you use the Fluentd forward protocol to send logs to a server configured to accept the protocol. You are responsible to configure the external log aggregator to receive the logs from OpenShift Container Platform.

注意

This method for forwarding logs is deprecated in OpenShift Container Platform and will be replaced by the Log Forwarding API in a future release.

To configure OpenShift Container Platform to send logs using the Fluentd forward protocol, create a ConfigMap called secure-forward in the openshift-logging namespace that points to an external log aggregator.

重要

Starting with the OpenShift Container Platform 4.3, the process for using the Fluentd forward protocol has changed. You now need to create a ConfigMap, as described below.

Additionally, you can add any certificates required by your configuration to a secret named secure-forward that will be mounted to the Fluentd Pods.

Sample secure-forward.conf

<store>
  @type forward
  <security>
    self_hostname ${hostname} # ${hostname} is a placeholder.
    shared_key "fluent-receiver"
  </security>
  transport tls
  tls_verify_hostname false           # Set false to ignore server cert hostname.

  tls_cert_path '/etc/ocp-forward/ca-bundle.crt'
  <buffer>
    @type file
    path '/var/lib/fluentd/secureforwardlegacy'
    queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '1024' }"
    chunk_limit_size "#{ENV['BUFFER_SIZE_LIMIT'] || '1m' }"
    flush_interval "#{ENV['FORWARD_FLUSH_INTERVAL'] || '5s'}"
    flush_at_shutdown "#{ENV['FLUSH_AT_SHUTDOWN'] || 'false'}"
    flush_thread_count "#{ENV['FLUSH_THREAD_COUNT'] || 2}"
    retry_max_interval "#{ENV['FORWARD_RETRY_WAIT'] || '300'}"
    retry_forever true
    # the systemd journald 0.0.8 input plugin will just throw away records if the buffer
    # queue limit is hit - 'block' will halt further reads and keep retrying to flush the
    # buffer to the remote - default is 'exception' because in_tail handles that case
    overflow_action "#{ENV['BUFFER_QUEUE_FULL_ACTION'] || 'exception'}"
  </buffer>
  <server>
    host fluent-receiver.openshift-logging.svc  # or IP
    port 24224
  </server>
</store>

Sample secure-forward ConfigMap based on the configuration

apiVersion: v1
data:
 secure-forward.conf: "<store>
     \ @type forward
     \ <security>
     \   self_hostname ${hostname} # ${hostname} is a placeholder.
     \   shared_key \"fluent-receiver\"
     \ </security>
     \ transport tls
     \ tls_verify_hostname false           # Set false to ignore server cert hostname.
     \ tls_cert_path '/etc/ocp-forward/ca-bundle.crt'
     \ <buffer>
     \   @type file
     \   path '/var/lib/fluentd/secureforwardlegacy'
     \   queued_chunks_limit_size \"#{ENV['BUFFER_QUEUE_LIMIT'] || '1024' }\"
     \   chunk_limit_size \"#{ENV['BUFFER_SIZE_LIMIT'] || '1m' }\"
     \   flush_interval \"#{ENV['FORWARD_FLUSH_INTERVAL'] || '5s'}\"
     \   flush_at_shutdown \"#{ENV['FLUSH_AT_SHUTDOWN'] || 'false'}\"
     \   flush_thread_count \"#{ENV['FLUSH_THREAD_COUNT'] || 2}\"
     \   retry_max_interval \"#{ENV['FORWARD_RETRY_WAIT'] || '300'}\"
     \   retry_forever true
     \   # the systemd journald 0.0.8 input plugin will just throw away records if the buffer
     \   # queue limit is hit - 'block' will halt further reads and keep retrying to flush the
     \   # buffer to the remote - default is 'exception' because in_tail handles that case
     \   overflow_action \"#{ENV['BUFFER_QUEUE_FULL_ACTION'] || 'exception'}\"
     \ </buffer>
     \ <server>
     \   host fluent-receiver.openshift-logging.svc  # or IP
     \   port 24224
     \ </server>
     </store>"
kind: ConfigMap
metadata:
  creationTimestamp: "2020-01-15T18:56:04Z"
  name: secure-forward
  namespace: openshift-logging
  resourceVersion: "19148"
  selfLink: /api/v1/namespaces/openshift-logging/configmaps/secure-forward
  uid: 6fd83202-93ab-d851b1d0f3e8

Procedure

To configure OpenShift Container Platform to forward logs using the Fluentd forward protocol:

  1. Create a configuration file named secure-forward.conf for the forward parameters:

    1. Configure the secrets and TLS information: 

       <store>
  @type forward

  self_hostname ${hostname} 1
  shared_key <SECRET_STRING> 2

  transport tls 3

  tls_verify_hostname true 4
  tls_cert_path <path_to_file> 5
      1
      Specify the default value of the auto-generated certificate common name (CN).
      2
      Enter the Shared key between nodes
      3
      Specify tls to enable TLS validation.
      4
      Set to true to verify the server cert hostname. Set to false to ignore server cert hostname.
      5
      Specify the path to private CA certificate file as /etc/ocp-forward/ca_cert.pem.

      To use mTLS, see the Fluentd documentation for information about client certificate, key parameters, and other settings.

    2. Configure the name, host, and port for your external Fluentd server: 

        <server>
    name 1
    host 2
    hostlabel 3
    port 4
  </server>
  <server> 5
    name
    host
  </server>
      1
      Optionally, enter a name for this server.
      2
      Specify the host name or IP of the server.
      3
      Specify the host label of the server.
      4
      Specify the port of the server.
      5
      Optionally, add additional servers. If you specify two or more servers, forward uses these server nodes in a round-robin order.

      For example: 

        <server>
    name externalserver1
    host 192.168.1.1
    hostlabel externalserver1.example.com
    port 24224
  </server>
  <server>
    name externalserver2
    host externalserver2.example.com
    port 24224
  </server>
  </store>

  2. Create a ConfigMap named secure-forward in the openshift-logging namespace from the configuration file: 

    $ oc create configmap secure-forward --from-file=secure-forward.conf -n openshift-logging

  3. Optional: Import any secrets required for the receiver: 

    $ oc create secret generic secure-forward --from-file=<arbitrary-name-of-key1>=cert_file_from_fluentd_receiver --from-literal=shared_key=value_from_fluentd_receiver

    For example: 

    $ oc create secret generic secure-forward --from-file=ca-bundle.crt=ca-for-fluentd-receiver/ca.crt --from-literal=shared_key=fluentd-receiver

  4. Refresh the fluentd Pods to apply the secure-forward secret and secure-forward ConfigMap: 

    $ oc delete pod --selector logging-infra=fluentd
  5. Configure the external log aggregator to accept messages securely from OpenShift Container Platform.