Chapter 2. Logging

Red Hat OpenStack Platform writes informational messages to specific log files; you can use these messages for troubleshooting and monitoring system events.

Note

You need not attach the individual log files to your support cases manually. All the required information will be gathered automatically by the sosreport utility, which is described in Chapter 5, Troubleshooting.

2.1. Log Files for OpenStack Services

Each OpenStack component has a separate logging directory containing files specific to a running service.

2.1.1. Bare Metal Provisioning (ironic) Log Files

ServiceService NameLog Path

OpenStack Ironic API

openstack-ironic-api.service

/var/log/containers/ironic/ironic-api.log

OpenStack Ironic Conductor

openstack-ironic-conductor.service

/var/log/containers/ironic/ironic-conductor.log

2.1.2. Block Storage (cinder) Log Files

ServiceService NameLog Path

Block Storage API

openstack-cinder-api.service

/var/log/containers/cinder/api.log

Block Storage Backup

openstack-cinder-backup.service

/var/log/containers/cinder/backup.log

Informational messages

The cinder-manage command

/var/log/containers/cinder/cinder-manage.log

Block Storage Scheduler

openstack-cinder-scheduler.service

/var/log/containers/cinder/scheduler.log

Block Storage Volume

openstack-cinder-volume.service

/var/log/containers/cinder/volume.log

2.1.3. Compute (nova) Log Files

ServiceService NameLog Path

OpenStack Compute API service

openstack-nova-api.service

/var/log/containers/nova/nova-api.log

OpenStack Compute certificate server

openstack-nova-cert.service

/var/log/containers/nova/nova-cert.log

OpenStack Compute service

openstack-nova-compute.service

/var/log/containers/nova/nova-compute.log

OpenStack Compute Conductor service

openstack-nova-conductor.service

/var/log/containers/nova/nova-conductor.log

OpenStack Compute VNC console authentication server

openstack-nova-consoleauth.service

/var/log/containers/nova/nova-consoleauth.log

Informational messages

nova-manage command

/var/log/containers/nova/nova-manage.log

OpenStack Compute NoVNC Proxy service

openstack-nova-novncproxy.service

/var/log/containers/nova/nova-novncproxy.log

OpenStack Compute Scheduler service

openstack-nova-scheduler.service

/var/log/containers/nova/nova-scheduler.log

2.1.4. Dashboard (horizon) Log Files

ServiceService NameLog Path

Log of certain user interactions

Dashboard interface

/var/log/containers/horizon/horizon.log

The Apache HTTP server uses several additional log files for the Dashboard web interface, which can be accessed using a web browser or command-line clients (keystone, nova). The following log files can be helpful in tracking the usage of the Dashboard and diagnosing faults:

PurposeLog Path

All processed HTTP requests

/var/log/containers/httpd/horizon_access.log

HTTP errors

/var/log/containers/httpd/horizon_error.log

Admin-role API requests

/var/log/containers/httpd/keystone_wsgi_admin_access.log

Admin-role API errors

/var/log/containers/httpd/keystone_wsgi_admin_error.log

Member-role API requests

/var/log/containers/httpd/keystone_wsgi_main_access.log

Member-role API errors

/var/log/containers/httpd/keystone_wsgi_main_error.log

Note

There is also /var/log/containers/httpd/default_error.log, which stores errors reported by other web services running on the same host.

2.1.5. Data Processing (sahara) Log Files

ServiceService NameLog Path

Sahara API Server

openstack-sahara-all.service
openstack-sahara-api.service

/var/log/containers/sahara/sahara-all.log
/var/log/containers/messages

Sahara Engine Server

openstack-sahara-engine.service

/var/log/containers/messages

2.1.6. Database as a Service (trove) Log Files

ServiceService NameLog Path

OpenStack Trove API Service

openstack-trove-api.service

/var/log/containers/trove/trove-api.log

OpenStack Trove Conductor Service

openstack-trove-conductor.service

/var/log/containers/trove/trove-conductor.log

OpenStack Trove guestagent Service

openstack-trove-guestagent.service

/var/log/containers/trove/logfile.txt

OpenStack Trove taskmanager Service

openstack-trove-taskmanager.service

/var/log/containers/trove/trove-taskmanager.log

2.1.7. Identity Service (keystone) Log Files

ServiceService NameLog Path

OpenStack Identity Service

openstack-keystone.service

/var/log/containers/keystone/keystone.log

2.1.8. Image Service (glance) Log Files

ServiceService NameLog Path

OpenStack Image Service API server

openstack-glance-api.service

/var/log/containers/glance/api.log

OpenStack Image Service Registry server

openstack-glance-registry.service

/var/log/containers/glance/registry.log

2.1.9. Networking (neutron) Log Files

ServiceService NameLog Path

OpenStack Neutron DHCP Agent

neutron-dhcp-agent.service

/var/log/containers/neutron/dhcp-agent.log

OpenStack Networking Layer 3 Agent

neutron-l3-agent.service

/var/log/containers/neutron/l3-agent.log

Metadata agent service

neutron-metadata-agent.service

/var/log/containers/neutron/metadata-agent.log

Metadata namespace proxy

n/a

/var/log/containers/neutron/neutron-ns-metadata-proxy-UUID.log

Open vSwitch agent

neutron-openvswitch-agent.service

/var/log/containers/neutron/openvswitch-agent.log

OpenStack Networking service

neutron-server.service

/var/log/containers/neutron/server.log

2.1.10. Object Storage (swift) Log Files

OpenStack Object Storage sends logs to the system logging facility only.

Note

By default, all Object Storage log files to /var/log/containers/swift/swift.log, using the local0, local1, and local2 syslog facilities.

The log messages of Object Storage are classified into two broad categories: those by REST API services and those by background daemons. The API service messages contain one line per API request, in a manner similar to popular HTTP servers; both the frontend (Proxy) and backend (Account, Container, Object) services post such messages. The daemon messages are less structured and typically contain human-readable information about daemons performing their periodic tasks. However, regardless of which part of Object Storage produces the message, the source identity is always at the beginning of the line.

An example of a proxy message:

Apr 20 15:20:34 rhev-a24c-01 proxy-server: 127.0.0.1 127.0.0.1 20/Apr/2015/19/20/34 GET /v1/AUTH_zaitcev%3Fformat%3Djson%26marker%3Dtestcont HTTP/1.0 200 - python-swiftclient-2.1.0 AUTH_tk737d6... - 2 - txc454fa8ea4844d909820a-0055355182 - 0.0162 - - 1429557634.806570053 1429557634.822791100

An example of ad-hoc messages from background daemons:

Apr 27 17:08:15 rhev-a24c-02 object-auditor: Object audit (ZBF). Since Mon Apr 27 21:08:15 2015: Locally: 1 passed, 0 quarantined, 0 errors files/sec: 4.34 , bytes/sec: 0.00, Total time: 0.23, Auditing time: 0.00, Rate: 0.00
Apr 27 17:08:16 rhev-a24c-02 object-auditor: Object audit (ZBF) "forever" mode completed: 0.56s. Total quarantined: 0, Total errors: 0, Total files/sec: 14.31, Total bytes/sec: 0.00, Auditing time: 0.02, Rate: 0.04
Apr 27 17:08:16 rhev-a24c-02 account-replicator: Beginning replication run
Apr 27 17:08:16 rhev-a24c-02 account-replicator: Replication run OVER
Apr 27 17:08:16 rhev-a24c-02 account-replicator: Attempted to replicate 5 dbs in 0.12589 seconds (39.71876/s)
Apr 27 17:08:16 rhev-a24c-02 account-replicator: Removed 0 dbs
Apr 27 17:08:16 rhev-a24c-02 account-replicator: 10 successes, 0 failures

2.1.11. Orchestration (heat) Log Files

ServiceService NameLog Path

OpenStack Heat API Service

openstack-heat-api.service

/var/log/containers/heat/heat-api.log

OpenStack Heat Engine Service

openstack-heat-engine.service

/var/log/containers/heat/heat-engine.log

Orchestration service events

n/a

/var/log/containers/heat/heat-manage.log

2.1.12. Shared Filesystem Service (manila) Log Files

ServiceService NameLog Path

OpenStack Manila API Server

openstack-manila-api.service

/var/log/containers/manila/api.log

OpenStack Manila Scheduler

openstack-manila-scheduler.service

/var/log/containers/manila/scheduler.log

OpenStack Manila Share Service

openstack-manila-share.service

/var/log/containers/manila/share.log

Note

Some information from the Manila Python library can also be logged in /var/log/containers/manila/manila-manage.log.

2.1.13. Telemetry (ceilometer) Log Files

ServiceService NameLog Path

OpenStack ceilometer notification agent

openstack-ceilometer-notification.service

/var/log/containers/ceilometer/agent-notification.log

OpenStack ceilometer alarm evaluation

openstack-ceilometer-alarm-evaluator.service

/var/log/containers/ceilometer/alarm-evaluator.log

OpenStack ceilometer alarm notification

openstack-ceilometer-alarm-notifier.service

/var/log/containers/ceilometer/alarm-notifier.log

OpenStack ceilometer API

httpd.service

/var/log/containers/ceilometer/api.log

Informational messages

MongoDB integration

/var/log/containers/ceilometer/ceilometer-dbsync.log

OpenStack ceilometer central agent

openstack-ceilometer-central.service

/var/log/containers/ceilometer/central.log

OpenStack ceilometer collection

openstack-ceilometer-collector.service

/var/log/containers/ceilometer/collector.log

OpenStack ceilometer compute agent

openstack-ceilometer-compute.service

/var/log/containers/ceilometer/compute.log

2.1.14. Log Files for Supporting Services

The following services are used by the core OpenStack components and have their own log directories and files.

ServiceService NameLog Path

Message broker (RabbitMQ)

rabbitmq-server.service

/var/log/rabbitmq/rabbit@short_hostname.log
/var/log/rabbitmq/rabbit@short_hostname-sasl.log (for Simple Authentication and Security Layer related log messages)

Database server (MariaDB)

mariadb.service

/var/log/mariadb/mariadb.log

Document-oriented database (MongoDB)

mongod.service

/var/log/mongodb/mongodb.log

Virtual network switch (Open vSwitch)

openvswitch-nonetwork.service

/var/log/openvswitch/ovsdb-server.log
/var/log/openvswitch/ovs-vswitchd.log

2.2. Configure Logging Options

Each component maintains its own separate logging configuration in its respective configuration file. For example, in Compute, these options are set in /etc/nova/nova.conf:

  • Increase the level of informational logging by enabling debugging. This option greatly increases the amount of information captured, so you may want to consider using it only temporarily, or first reviewing your log rotation settings.

    debug=True
  • Enable verbose logging:

    verbose=True
  • Change the log file path:

    log_dir=/var/log/containers/nova
  • Send your logs to a central syslog server:

    use_syslog=True
    syslog_log_facility=LOG_USER
Note

Options are also available for timestamp configuration and log formatting, among others. Review the component’s configuration file for additional logging options.

2.3. Remote Logging Installation and Configuration

All OpenStack services generate and update log files. These log files record actions, errors, warnings, and other events. In a distributed environment like OpenStack, collecting these logs in a central location simplifies debugging and administration.

For more information about centralized logging, see the Monitoring Tools Configuration guide.