Chapter 5. Managing Load-balancing service instance logs

You can enable tenant flow logging or suppress logging to the amphora local file system. You can also forward administrative or tenant flow logs to syslog receivers at a set of containers or to other syslog receivers at endpoints of your choosing.

In addition, you can control a range of other logging features such as, setting the syslog facility value, changing the tenant flow log format, and widening the scope of administrative logging to include logs from sources like the kernel and from cron.

5.1. Basics of offloading Load-balancing service instance (amphora) logs

By default, Load-balancing service instances (amphorae) store logs on the local machine in the systemd journal. However, you can specify that amphorae offload logs to syslog receivers to aggregate both administrative and tenant traffic flow logs. Log offloading enables administrators to go to one location for logs, and retain logs when amphorae are rotated.

Additional resources

5.2. Enabling Load-balancing service instance administrative log offloading

By default, Load-balancing service instances (amphorae) store logs on the local machine in the systemd journal. However, you can specify that the amphorae offload logs to syslog receivers to aggregate administrative logs. Log offloading enables administrators to go to one location for logs, and retain logs when the amphorae are rotated.

Procedure

  1. Log in to the undercloud host as the stack user.

  2. Source the undercloud credentials file: 

    $ source ~/stackrc

  3. Create a custom YAML environment file.

    Example

    $ vi /home/stack/templates/my-octavia-environment.yaml

  4. In the YAML environment file under parameter_defaults, set OctaviaLogOffload to true. 

    parameter_defaults:
    OctaviaLogOffload: true
    ...
    Note

    The amphorae offload administrative logs use the syslog facility value of local1, by default, unless you specify another value with the OctaviaAdminLogFacility parameter.

    Example

    parameter_defaults:
    OctaviaLogOffload: true
    OctaviaAdminLogFacility: 2
    ...

  5. The amphorae forward only load balancer-related administrative logs, such as the haproxy admin logs, keepalived, and amphora agent logs. If you want to configure the amphorae to send all of the administrative logs from the amphorae, such as the kernel, system, and security logs, set OctaviaForwardAllLogs to true.

    Example

    parameter_defaults:
    OctaviaLogOffload: true
    OctaviaForwardAllLogs: true
    ...

  6. The amphorae use a set of default containers defined by the Orchestration service (heat) that contain syslog receivers listening for log messages. If you want to use a different set of endpoints, you can specify those with the OctaviaAdminLogTargets parameter: 

    OctaviaAdminLogTargets: <ip_address>:<port>[, <ip_address>:<port>]

    Example

    parameter_defaults:
    OctaviaLogOffload: true
    OctaviaAdminLogTargets: 192.0.2.1:10514, 2001:db8:1::10:10514
    ...

  7. By default, when you enable log offloading, tenant flow logs are also offloaded.

    If you want to disable tenant flow log offloading, set the OctaviaConnectionLogging to false.

    Example

    parameter_defaults:
    OctaviaLogOffload: true
    OctaviaConnectionLogging: false
    ...

  8. Run the deployment command and include the core heat templates, environment files, and this new custom environment file.

    Important

    The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence.

    Example

    $ openstack overcloud deploy --templates \
-e [your-environment-files] \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
-e /home/stack/templates/my-octavia-environment.yaml

Verification

  • Unless you specified specific endpoints with the OctaviaAdminLogTargets or OctaviaTenantLogTargets, the amphorae offload logs to the RHOSP Controller in the same location as the other RHOSP logs (/var/log/containers/octavia/).

  • Check the appropriate location for the presence of the following log files:

    • octavia-amphora.log-- Log file for the administrative log.
    • (if enabled) octavia-tenant-traffic.log-- Log file for the tenant traffic flow log.

Additional resources

5.3. Enabling tenant flow log offloading for Load-balancing service instances

By default, Load-balancing service instances (amphorae) store logs on the local machine in the systemd journal. You can specify that the amphorae offload logs to syslog receivers on endpoints that contain disk space sufficient for tenant flow logs that can grow in size depending on the number of tenant connections.

Tenant flow log offloading for Load-balancing service instances is automatically enabled when administrative log offloading is enabled. The only case when administrative log offloading is on and tenant flow log offloading is off, is when the OctaviaConnectionLogging parameter is set to false.

Important

Tenant flow logging can produce a large number of syslog messages depending on how many connections the load balancers are receiving. Tenant flow logging produces one log entry for each connection to the load balancer. Monitor log volume and configure your syslog receivers appropriately based on the expected number of connections that your load balancers manage.

Procedure

  1. Log in to the undercloud host as the stack user.

  2. Source the undercloud credentials file: 

    $ source ~/stackrc

  3. Locate the environment file in which the OctaviaConnectionLogging parameter is set: 

    $ grep -rl OctaviaConnectionLogging /home/stack/templates/

  4. If you do not find the file, create an environment file: 

    $ vi /home/stack/templates/my-octavia-environment.yaml

  5. Add the OctaviaLogOffload and OctaviaConnectionLogging parameters to the parameter_defaults section of the environment file and set the values to true: 

    parameter_defaults:
    OctaviaLogOffload: true
    OctaviaConnectionLogging: true
    ...
    Note

    The amphorae use the syslog facility default value of local0 to offload tenant flow logs unless you use the OctaviaTenantLogFacility parameter to specify another value.

  6. Optional: The amphorae use a set of default containers that contain syslog receivers that listen for log messages. You can change the admin and tenant log endpoints using the parameters OctaviaAdminLogTargets and OctaviaTenantLogTargets. 

    OctaviaAdminLogTargets: <ip-address>:<port>[, <ip-address>:<port>]
OctaviaTenantLogTargets: <ip-address>:<port>[, <ip-address>:<port>]

  7. Run the deployment command and include the core heat templates, environment files, and the custom environment file you modified.

    Important

    The order of the environment files is important because the parameters and resources defined in subsequent environment files take precedence. 

    $ openstack overcloud deploy --templates \
-e <your_environment_files> \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
-e /home/stack/templates/my-octavia-environment.yaml

Verification

  • Unless you specified specific endpoints with the OctaviaAdminLogTargets or OctaviaTenantLogTargets, the amphorae offload logs to the RHOSP Controller in the same location as the other RHOSP logs (/var/log/containers/octavia/).

  • Check the appropriate location for the presence of the following log files:

    • octavia-amphora.log-- Log file for the administrative log.
    • octavia-tenant-traffic.log-- Log file for the tenant traffic flow log.

Additional resources

5.4. Disabling Load-balancing service instance tenant flow logging

Tenant flow log offloading for Load-balancing service instances (amphorae) is automatically enabled when you enable administrative log offloading.

To keep administrative log offloading enabled and to disable tenant flow logging, you must set the OctaviaConnectionLogging parameter to false.

When the OctaviaConnectionLogging parameter is false, the amphorae do not write tenant flow logs to the disk inside the amphorae, nor offload any logs to syslog receivers listening elsewhere.

Procedure

  1. Log in to the undercloud host as the stack user.

  2. Source the undercloud credentials file: 

    $ source ~/stackrc

  3. Locate the YAML custom environment file in which amphora logging is configured.

    Example

    $ grep -rl OctaviaLogOffload /home/stack/templates/

  4. In the custom environment file, under parameter_defaults, set OctaviaConnectionLogging to false.

    Example

    parameter_defaults:
    OctaviaLogOffload: true
    OctaviaConnectionLogging: false
    ...

  5. Run the deployment command and include the core heat templates, environment files, and the custom environment file in which you set OctaviaConnectionLogging to true.

    Important

    The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence.

    Example

    $ openstack overcloud deploy --templates \
-e [your-environment-files] \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
-e /home/stack/templates/my-octavia-environment.yaml

Verification

  • Unless you specified specific endpoints with the OctaviaAdminLogTargets or OctaviaTenantLogTargets, the amphorae offload logs to the RHOSP Controller in the same location as the other RHOSP logs (/var/log/containers/octavia/).
  • Check the appropriate location for the absence of octavia-tenant-traffic.log.

Additional resources

5.5. Disabling Load-balancing service instance local log storage

Even when you configure Load-balancing service instances (amphorae) to offload administrative and tenant flow logs, the amphorae continue to write these logs to the disk inside the amphorae. To improve the performance of the load balancer, you can stop logging locally.

Important

If you disable logging locally, you also disable all log storage in the amphora, including kernel, system, and security logging.

Note

If you disable local log storage and the OctaviaLogOffload parameter is set to false, ensure that you set OctaviaConnectionLogging to false for improved load balancing performance.

Procedure

  1. Log in to the undercloud host as the stack user.

  2. Source the undercloud credentials file: 

    $ source ~/stackrc

  3. Create a custom YAML environment file.

    Example

    $ vi /home/stack/templates/my-octavia-environment.yaml

  4. In the environment file under parameter_defaults, set OctaviaDisableLocalLogStorage to true. 

    parameter_defaults:
    OctaviaDisableLocalLogStorage: true
    ...

  5. Run the deployment command and include the core heat templates, environment files, and this new custom environment file.

    Important

    The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence.

    Example

    $ openstack overcloud deploy --templates \
-e <your_environment_files> \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
-e /home/stack/templates/my-octavia-environment.yaml

Verification

  • On the amphora instance, check the location where log files are written, and verify that no new log files are being written.

Additional resources

5.6. Heat parameters for Load-balancing service instance logging

When you want to configure Load-balancing service instance (amphora) logging, you set values for one or more Orchestration service (heat) parameters that control logging and run the openstack overcloud deploy command.

These heat parameters for amphora logging enable you to control features such as turning on log offloading, defining custom endpoints to offload logs to, setting the syslog facility value for logs, and so on.

Table 5.1. Heat parameters for all logs

ParameterDefaultDescription

OctaviaLogOffload

false

When true, instances offload their logs. If no endpoints are specified, then, by default, the instance offloads its log to the same location as the other RHOSP logs (/var/log/containers/octavia/).

OctaviaDisableLocalLogStorage

false

When true, instances do not store logs on the instance host filesystem. This includes all kernel, system, and security logs.

OctaviaForwardAllLogs

false

When true, instances forward all log messages to the administrative log endpoints, including non-load balancing related logs such as the cron and kernel logs.

For instances to recognize OctaviaForwardAllLogs, you must also enable OctaviaLogOffload.

Table 5.2. Heat parameters for admin logging

ParameterDefaultDescription

OctaviaAdminLogTargets

No value.

A comma-delimited list of syslog endpoints (<host>:<port>) to receive administrative log messages.

These endpoints can be a container, VM, or physical host that is running a process that is listening for the log messages on the specified port.

When OctaviaAdminLogTargets is not present, the instance offloads its log to the same location as the other RHOSP logs (/var/log/containers/octavia/) on a set of containers that is defined by RHOSP director.

OctaviaAdminLogFacility

1

A number between 0 and 7 that is the syslog "LOG_LOCAL" facility to use for the administrative log messages.

Table 5.3. Heat parameters for tenant flow logging

ParameterDefaultDescription

OctaviaConnectionLogging

true

When true, tenant connection flows are logged.

When OctaviaConnectionLogging is false, the amphorae stop logging tenant connections regardless of the OctaviaLogOffload setting. OctaviaConnectionLogging disables local tenant flow log storage and, if log offloading is enabled, it does not forward tenant flow logs.

OctaviaTenantLogTargets

No value.

A comma-delimited list of syslog endpoints (<host>:<port>) to receive tenant traffic flow log messages.

These endpoints can be a container, VM, or physical host that is running a process that is listening for the log messages on the specified port.

When OctaviaTenantLogTargets is not present, the instance offloads its log to the same location as the other RHOSP logs (/var/log/containers/octavia/) on a set of containers that is defined by RHOSP director.

OctaviaTenantLogFacility

0

A number between 0 and 7 that is the syslog "LOG_LOCAL" facility to use for the tenant traffic flow log messages.

OctaviaUserLogFormat

"{{ '{{' }} project_id {{ '}}' }} {{ '{{' }} lb_id {{ '}}' }} %f %ci %cp %t %{+Q}r %ST %B %U %[ssl_c_verify] %{+Q}[ssl_c_s_dn] %b %s %Tt %tsc"

The format for the tenant traffic flow log.

The alphanumerics represent specific octavia fields, and the curly braces ({}) are substitution variables.

Additional resources

5.7. Load-balancing service instance tenant flow log format

The log format that the tenant flow logs for Load-balancing service instances (amphorae) follows is the HAProxy log format. The two exceptions are the project_id and lb_id variables whose values are provided by the amphora provider driver.

Sample

Here is a sample log entry with rsyslog as the syslog receiver: 

Jun 12 00:44:13 amphora-3e0239c3-5496-4215-b76c-6abbe18de573 haproxy[1644]: 5408b89aa45b48c69a53dca1aaec58db fd8f23df-960b-4b12-ba62-2b1dff661ee7 261ecfc2-9e8e-4bba-9ec2-3c903459a895 172.24.4.1 41152 12/Jun/2019:00:44:13.030 "GET / HTTP/1.1" 200 76 73 - "" e37e0e04-68a3-435b-876c-cffe4f2138a4 6f2720b3-27dc-4496-9039-1aafe2fee105 4 --

Notes

  • A hyphen (-) indicates any field that is unknown or not applicable to the connection.

  • The prefix in the earlier sample log entry originates from the rsyslog receiver, and is not part of the syslog message from the amphora: 

    Jun 12 00:44:13 amphora-3e0239c3-5496-4215-b76c-6abbe18de573 haproxy[1644]:”

Default

The default amphora tenant flow log format is: 

`"{{ '{{' }} project_id {{ '}}' }} {{ '{{' }} lb_id {{ '}}' }} %f %ci %cp %t %{+Q}r %ST %B %U %[ssl_c_verify] %{+Q}[ssl_c_s_dn] %b %s %Tt %tsc"`

Refer to the table that follows for a description of the format.

Table 5.4. Data variables for tenant flow logs format variable definitions.

VariableTypeField name

{{project_id}}

UUID

Project ID (substitution variable from the amphora provider driver)

{{lb_id}}

UUID

Load balancer ID (substitution variable from the amphora provider driver)

%f

string

frontend_name

%ci

IP address

client_ip

%cp

numeric

client_port

%t

date

date_time

%ST

numeric

status_code

%B

numeric

bytes_read

%U

numeric

bytes_uploaded

%ssl_c_verify

Boolean

client_certificate_verify (0 or 1)

%ssl_c_s_dn

string

client_certificate_distinguised_name

%b

string

pool_id

%s

string

member_id

%Tt

numeric

processing_time (milliseconds)

%tsc

string

termination_state (with cookie status)

Additional resources