Red Hat Training
A Red Hat training course is available for Red Hat OpenStack Platform
Chapter 2. Logging
Red Hat OpenStack Platform writes informational messages to specific log files; you can use these messages for troubleshooting and monitoring system events.
You need not attach the individual log files to your support cases manually. All the required information will be gathered automatically by the sosreport utility, which is described in Chapter 4, Troubleshooting.
2.1. Log Files for OpenStack Services
Each OpenStack component has a separate logging directory containing files specific to a running service.
2.1.1. Bare Metal Provisioning (ironic) Log Files
| Service | Service Name | Log Path |
|---|---|---|
| OpenStack Ironic API | openstack-ironic-api.service | /var/log/ironic/ironic-api.log |
| OpenStack Ironic Conductor | openstack-ironic-conductor.service | /var/log/ironic/ironic-conductor.log |
2.1.2. Block Storage (cinder) Log Files
| Service | Service Name | Log Path |
|---|---|---|
| Block Storage API | openstack-cinder-api.service | /var/log/cinder/api.log |
| Block Storage Backup | openstack-cinder-backup.service | /var/log/cinder/backup.log |
| Informational messages | The cinder-manage command | /var/log/cinder/cinder-manage.log |
| Block Storage Scheduler | openstack-cinder-scheduler.service | /var/log/cinder/scheduler.log |
| Block Storage Volume | openstack-cinder-volume.service | /var/log/cinder/volume.log |
2.1.3. Compute (nova) Log Files
| Service | Service Name | Log Path |
|---|---|---|
| OpenStack Compute API service | openstack-nova-api.service | /var/log/nova/nova-api.log |
| OpenStack Compute certificate server | openstack-nova-cert.service | /var/log/nova/nova-cert.log |
| OpenStack Compute service | openstack-nova-compute.service | /var/log/nova/nova-compute.log |
| OpenStack Compute Conductor service | openstack-nova-conductor.service | /var/log/nova/nova-conductor.log |
| OpenStack Compute VNC console authentication server | openstack-nova-consoleauth.service | /var/log/nova/nova-consoleauth.log |
| Informational messages | nova-manage command | /var/log/nova/nova-manage.log |
| OpenStack Compute NoVNC Proxy service | openstack-nova-novncproxy.service | /var/log/nova/nova-novncproxy.log |
| OpenStack Compute Scheduler service | openstack-nova-scheduler.service | /var/log/nova/nova-scheduler.log |
2.1.4. Dashboard (horizon) Log Files
| Service | Service Name | Log Path |
|---|---|---|
| Log of certain user interactions | Dashboard interface | /var/log/horizon/horizon.log |
The Apache HTTP server uses several additional log files for the Dashboard web interface, which can be accessed using a web browser or command-line clients (keystone, nova). The following log files can be helpful in tracking the usage of the Dashboard and diagnosing faults:
| Purpose | Log Path |
|---|---|
| All processed HTTP requests | /var/log/httpd/horizon_access.log |
| HTTP errors | /var/log/httpd/horizon_error.log |
| Admin-role API requests | /var/log/httpd/keystone_wsgi_admin_access.log |
| Admin-role API errors | /var/log/httpd/keystone_wsgi_admin_error.log |
| Member-role API requests | /var/log/httpd/keystone_wsgi_main_access.log |
| Member-role API errors | /var/log/httpd/keystone_wsgi_main_error.log |
There is also /var/log/httpd/default_error.log, which stores errors reported by other web services running on the same host.
2.1.5. Data Processing (sahara) Log Files
| Service | Service Name | Log Path |
|---|---|---|
| Sahara API Server |
openstack-sahara-all.service |
/var/log/sahara/sahara-all.log |
| Sahara Engine Server | openstack-sahara-engine.service | /var/log/messages |
2.1.6. Database as a Service (trove) Log Files
| Service | Service Name | Log Path |
|---|---|---|
| OpenStack Trove API Service | openstack-trove-api.service | /var/log/trove/trove-api.log |
| OpenStack Trove Conductor Service | openstack-trove-conductor.service | /var/log/trove/trove-conductor.log |
| OpenStack Trove guestagent Service | openstack-trove-guestagent.service | /var/log/trove/logfile.txt |
| OpenStack Trove taskmanager Service | openstack-trove-taskmanager.service | /var/log/trove/trove-taskmanager.log |
2.1.7. Identity Service (keystone) Log Files
| Service | Service Name | Log Path |
|---|---|---|
| OpenStack Identity Service | openstack-keystone.service | /var/log/keystone/keystone.log |
2.1.8. Image Service (glance) Log Files
| Service | Service Name | Log Path |
|---|---|---|
| OpenStack Image Service API server | openstack-glance-api.service | /var/log/glance/api.log |
| OpenStack Image Service Registry server | openstack-glance-registry.service | /var/log/glance/registry.log |
2.1.9. Networking (neutron) Log Files
| Service | Service Name | Log Path |
|---|---|---|
| OpenStack Neutron DHCP Agent | neutron-dhcp-agent.service | /var/log/neutron/dhcp-agent.log |
| OpenStack Networking Layer 3 Agent | neutron-l3-agent.service | /var/log/neutron/l3-agent.log |
| Metadata agent service | neutron-metadata-agent.service | /var/log/neutron/metadata-agent.log |
| Metadata namespace proxy | n/a | /var/log/neutron/neutron-ns-metadata-proxy-UUID.log |
| Open vSwitch agent | neutron-openvswitch-agent.service | /var/log/neutron/openvswitch-agent.log |
| OpenStack Networking service | neutron-server.service | /var/log/neutron/server.log |
2.1.10. Object Storage (swift) Log Files
OpenStack Object Storage sends logs to the system logging facility only.
By default, all Object Storage log files to /var/log/swift/swift.log, using the local0, local1, and local2 syslog facilities.
The log messages of Object Storage are classified into two broad categories: those by REST API services and those by background daemons. The API service messages contain one line per API request, in a manner similar to popular HTTP servers; both the frontend (Proxy) and backend (Account, Container, Object) services post such messages. The daemon messages are less structured and typically contain human-readable information about daemons performing their periodic tasks. However, regardless of which part of Object Storage produces the message, the source identity is always at the beginning of the line.
An example of a proxy message:
Apr 20 15:20:34 rhev-a24c-01 proxy-server: 127.0.0.1 127.0.0.1 20/Apr/2015/19/20/34 GET /v1/AUTH_zaitcev%3Fformat%3Djson%26marker%3Dtestcont HTTP/1.0 200 - python-swiftclient-2.1.0 AUTH_tk737d6... - 2 - txc454fa8ea4844d909820a-0055355182 - 0.0162 - - 1429557634.806570053 1429557634.822791100
An example of ad-hoc messages from background daemons:
Apr 27 17:08:15 rhev-a24c-02 object-auditor: Object audit (ZBF). Since Mon Apr 27 21:08:15 2015: Locally: 1 passed, 0 quarantined, 0 errors files/sec: 4.34 , bytes/sec: 0.00, Total time: 0.23, Auditing time: 0.00, Rate: 0.00 Apr 27 17:08:16 rhev-a24c-02 object-auditor: Object audit (ZBF) "forever" mode completed: 0.56s. Total quarantined: 0, Total errors: 0, Total files/sec: 14.31, Total bytes/sec: 0.00, Auditing time: 0.02, Rate: 0.04 Apr 27 17:08:16 rhev-a24c-02 account-replicator: Beginning replication run Apr 27 17:08:16 rhev-a24c-02 account-replicator: Replication run OVER Apr 27 17:08:16 rhev-a24c-02 account-replicator: Attempted to replicate 5 dbs in 0.12589 seconds (39.71876/s) Apr 27 17:08:16 rhev-a24c-02 account-replicator: Removed 0 dbs Apr 27 17:08:16 rhev-a24c-02 account-replicator: 10 successes, 0 failures
2.1.11. Orchestration (heat) Log Files
| Service | Service Name | Log Path |
|---|---|---|
| OpenStack Heat API Service | openstack-heat-api.service | /var/log/heat/heat-api.log |
| Openstack Heat Engine Service | openstack-heat-engine.service | /var/log/heat/heat-engine.log |
| Orchestration service events | n/a | /var/log/heat/heat-manage.log |
2.1.12. Shared Filesystem Service (manila) Log Files
| Service | Service Name | Log Path |
|---|---|---|
| OpenStack Manila API Server | openstack-manila-api.service | /var/log/manila/api.log |
| OpenStack Manila Scheduler | openstack-manila-scheduler.service | /var/log/manila/scheduler.log |
| OpenStack Manila Share Service | openstack-manila-share.service | /var/log/manila/share.log |
Some information from the Manila Python library can also be logged in /var/log/manila/manila-manage.log.
2.1.13. Telemetry (ceilometer) Log Files
| Service | Service Name | Log Path |
|---|---|---|
| OpenStack ceilometer notification agent | openstack-ceilometer-notification.service | /var/log/ceilometer/agent-notification.log |
| OpenStack ceilometer alarm evaluation | openstack-ceilometer-alarm-evaluator.service | /var/log/ceilometer/alarm-evaluator.log |
| OpenStack ceilometer alarm notification | openstack-ceilometer-alarm-notifier.service | /var/log/ceilometer/alarm-notifier.log |
| OpenStack ceilometer API | httpd.service | /var/log/ceilometer/api.log |
| Informational messages | MongoDB integration | /var/log/ceilometer/ceilometer-dbsync.log |
| OpenStack ceilometer central agent | openstack-ceilometer-central.service | /var/log/ceilometer/central.log |
| OpenStack ceilometer collection | openstack-ceilometer-collector.service | /var/log/ceilometer/collector.log |
| OpenStack ceilometer compute agent | openstack-ceilometer-compute.service | /var/log/ceilometer/compute.log |
2.1.14. Log Files for Supporting Services
The following services are used by the core OpenStack components and have their own log directories and files.
| Service | Service Name | Log Path |
|---|---|---|
| Message broker (RabbitMQ) | rabbitmq-server.service |
/var/log/rabbitmq/rabbit@short_hostname.log |
| Database server (MariaDB) | mariadb.service | /var/log/mariadb/mariadb.log |
| Document-oriented database (MongoDB) | mongod.service | /var/log/mongodb/mongodb.log |
| Virtual network switch (Open vSwitch) | openvswitch-nonetwork.service |
/var/log/openvswitch/ovsdb-server.log |
2.2. Configure Logging Options
Each component maintains its own separate logging configuration in its respective configuration file. For example, in Compute, these options are set in /etc/nova/nova.conf:
Increase the level of informational logging by enabling debugging. This option greatly increases the amount of information captured, so you may want to consider using it only temporarily, or first reviewing your log rotation settings.
debug=True
Enable verbose logging:
verbose=True
Change the log file path:
log_dir=/var/log/nova
Send your logs to a central syslog server:
use_syslog=True syslog_log_facility=LOG_USER
Options are also available for timestamp configuration and log formatting, among others. Review the component’s configuration file for additional logging options.
2.3. Remote Logging Installation and Configuration
All OpenStack services generate and update log files. These log files record actions, errors, warnings, and other events. In a distributed environment like OpenStack, collecting these logs in a central location simplifies debugging and administration.
For more information about centralized logging, see the Monitoring Tools Configuration guide.
