Centralized logging allows you to have one central place to view logs across your entire OpenStack environment. These logs come from the operating system, such as syslog and audit log files, infrastructure components such as RabbitMQ and MariaDB, and OpenStack services such as Identity, Compute, and others.

The centralized logging toolchain consists of a number of components, including:

The centralized logging components and their interactions are laid out in the following diagrams:

Figure 2.1. Centralized logging architecture at a high level

Figure 2.2. Single-node deployment for Red Hat OpenStack Platform

Figure 2.3. HA deployment for Red Hat OpenStack Platform

Availability monitoring allows you to have one central place to monitor the high-level functionality of all components across your entire OpenStack environment.

The availability monitoring toolchain consists of a number of components, including:

The availability monitoring components and their interactions are laid out in the following diagrams:

Figure 2.4. Availability monitoring architecture at a high level

Figure 2.5. Single-node deployment for Red Hat OpenStack Platform

Figure 2.6. HA deployment for Red Hat OpenStack Platform

For Fluentd configuration settings, copy /usr/share/openstack-tripleo-heat-templates/environments/logging-environment.yaml and modify the file to suit your environment. For example:

resource_registry:
  OS::TripleO::Services::Fluentd: ../deployment/deprecated/logging/fluentd-container-puppet.yaml

parameter_defaults:
  LoggingServers:
    - host: log0.example.com
      port: 24224
    - host: log1.example.com
      port: 24224

## (note the use of port 24284 for ssl connections)

resource_registry:
  OS::TripleO::Services::Fluentd: ../deployment/deprecated/logging/fluentd-container-puppet.yaml

parameter_defaults:
  LoggingServers:
    - host: 192.0.2.11
      port: 24284
  LoggingUsesSSL: true
  LoggingSharedKey: secret
  LoggingSSLCertificate: |
    -----BEGIN CERTIFICATE-----
    ...certificate data here...
    -----END CERTIFICATE-----

For the Sensu client configuration settings, copy /usr/share/openstack-tripleo-heat-templates/environments/monitoring-environment.yaml and modify the file to suit your environment. For example:

resource_registry:
  OS::TripleO::Services::SensuClient: ../deployment/deprecated/monitoring/sensu-client-container-puppet.yaml

parameter_defaults:
  MonitoringRabbitHost: 10.10.10.10
  MonitoringRabbitPort: 5672
  MonitoringRabbitUserName: sensu
  MonitoringRabbitPassword: sensu
  MonitoringRabbitUseSSL: true
  MonitoringRabbitVhost: "/sensu"
  SensuClientCustomConfig:
    api:
      warning: 10
      critical: 20

Include the modified YAML files with your openstack overcloud deploy command to install the Sensu client and Fluentd tools on all overcloud nodes. For example:

$ openstack overcloud deploy --templates  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml -e ~/templates/monitoring-environment.yaml -e ~/templates/logging-environment.yaml --control-scale 3 --compute-scale 1 --ntp-server 192.168.122.10

You can filter and transform events sent to Fluentd by setting the LoggingDefaultFilters parameter in your environment file. For example, the record_transformer type can modify incoming events:

parameter_defaults:

  LoggingDefaultFilters:
      - tag_pattern: '**'
        type: record_transformer
        enable_ruby: true
        record:
          openstack: '{"hostname": "${hostname}","tag": "${tag}","region": "regionOne","inputname":"fluent-plugin-in_tail","name":"fluentd openstack","fluentd version":"0.12.26", "pipeline_metadata": {"collector":{"ipaddr4":"#{ begin Socket.ip_address_list.select { |a| a.ipv4? && !a.ipv4_loopback? }.map { |a| a.ip_address } end}","ipaddr6":"#{ begin Socket.ip_address_list.select { |a| a.ipv6? && !a.ipv6_loopback? }.map { |a| a.ip_address } end}"}}}'
        remove_keys: host

      - tag_pattern: 'openstack.**'
        type: record_transformer
        record:
          service: '${tag_parts[1]}'

      - tag_pattern: 'system.messages**'
        type: record_transformer
        enable_ruby: true
        record:
          openstack: '{"info": "${record[''message'']}", "systemd":{"t":{"PID":"${record[''pid'']}"},"u":{"SYSLOG_IDENTIFIER":"${record[''ident'']}"}}}'
        remove_keys: 'ident,message,pid'

As a result, the data received by Kibana has been transformed accordingly:

{
  "_index": "logstash-2017.06.29",
  "_type": "fluentd",
  "_id": "AVz132QmRtyd8nnlv_11",
  "_score": null,
  "_source": {
    "pid": "22691",
    "priority": "INFO",
    "message": "cinder.api.openstack.wsgi [req-04bc2808-f86f-4443-86e6-bfc596969937 - - - - -] OPTIONS http://overcloud-controller-0.lab.local/",
    "openstack": {
      "hostname": "overcloud-controller-0",
      "tag": "openstack.cinder.api",
      "region": "regionOne",
      "inputname": "fluent-plugin-in_tail",
      "name": "fluentd openstack",
      "fluentd version": "0.12.26",
      "pipeline_metadata": {
        "collector": {
          "ipaddr4": "[\"192.168.24.14\", \"192.168.24.8\", \"10.0.0.4\", \"10.0.0.5\", \"172.16.2.8\", \"172.16.2.4\", \"172.16.2.14\", \"172.16.1.7\", \"172.16.1.9\", \"172.16.3.10\", \"172.16.3.11\", \"172.16.0.14\"]",
          "ipaddr6": "[\"fe80::293:33ff:fed8:2228%eth0\", \"fe80::293:33ff:fed8:2228%br-ex\", \"fe80::b86c:79ff:fe8f:9fb8%vlan10\", \"fe80::4c78:6fff:feff:14fc%vlan20\", \"fe80::9ced:dfff:fe8c:2d62%vlan30\", \"fe80::ecde:1bff:fe5d:e362%vlan40\", \"fe80::549c:51ff:feea:dfa8%vlan50\", \"fe80::e093:8fff:fef9:69b6%vxlan_sys_4789\"]"
        }
      }
    },
    "service": "cinder",
    "@timestamp": "2017-06-29T21:59:38+00:00"
  },
  "fields": {
    "@timestamp": [
      1498773578000
    ]
  },
  "sort": [
    1498773578000
  ]
}