Chapter 9. Exposing 3scale APIcast Metrics to Prometheus

Important

For this release of 3scale, Prometheus installation and configuration are not supported. Optionally, you can use the community version of Prometheus to visualize metrics and alerts for APIcast-managed API services.

9.1. About Prometheus

Prometheus is an open-source systems monitoring toolkit that you can use to monitor 3scale APIcast services deployed in the Red Hat OpenShift environment.

If you want to monitor your services with Prometheus, your services must expose a Prometheus endpoint. This endpoint is an HTTP interface that exposes a list of metrics and the current value of the metrics. Prometheus periodically scrapes these target-defined endpoints and writes the collected data into its database.

9.1.1. Prometheus queries

In the Prometheus UI, you can write queries in Prometheus Query Language (PromQL) to extract metric information. With PromQL, you can select and aggregate time series data in real time.

For example, you can use the following query to select all the values that Prometheus has recorded within the last 5 minutes for all time series that have the metric name http_requests_total:

http_requests_total[5m]

You can further define or filter the results of a query by specifying a label (a key:value pair) for the metric. For example, you can use the following query to select all the values that Prometheus has recorded within the last 5 minutes for all time series that have the metric name http_requests_total and a job label set to integration:

http_requests_total{job="integration"}[5m]

The result of a query can either be shown as a graph, viewed as tabular data in Prometheus’s expression browser, or consumed by external systems by using the Prometheus HTTP API. Prometheus provides a graphical view of the data. For a more robust graphical dashboard to view Prometheus metrics, Grafana is a popular choice.

You can also use the the PromQL language to configure alerts in the Prometheus alertmanager tool.

Note

Grafana is a community-supported feature. Deploying Grafana to monitor Red Hat 3scale products is not supported with Red Hat production service level agreements (SLAs).

9.2. APIcast integration with Prometheus

APIcast integration with Prometheus is available for the following deployment options:

  • Self-managed APIcast (both with hosted or on-premises API manager)
  • Built-in APIcast on-premise
Note

APIcast integration with Prometheus is not available in hosted API manager and hosted APIcast.

By default, Prometheus can monitor the APIcast metrics listed in Table 9.2, “Prometheus Default Metrics for 3scale APIcast”.

9.2.1. Additional options

Optionally, if you have cluster admin access to the OpenShift cluster, you can extend the total_response_time_seconds, upstream_response_time_seconds, and upstream_status metrics to include service_id and service_system_name labels. To extend these metrics, set the APICAST_EXTENDED_METRICS OpenShift environment variable to true with this command:

oc set env dc/apicast APICAST_EXTENDED_METRICS=true

If you use the APIcast Batch policy (described in Section 4.1.2, “3scale Batcher”), Prometheus can also monitor the metrics listed in Table 9.3, “Prometheus Metrics for 3scale APIcast Batch Policy”.

Note

If a metric has no value, Prometheus hides the metric. For example, if nginx_error_log has no errors to report, Prometheus does not display the nginx_error_log metric. The nginx_error_log metric is only visible if it has a value.

Additional resources

For information about Prometheus, refer to Prometheus: Getting Started.

9.3. OpenShift environment variables for 3scale APIcast

To configure your Prometheus instance, you can set the OpenShift environment variable described in Table 9.1, “Prometheus Environment Variables for 3scale APIcast”.

Table 9.1. Prometheus Environment Variables for 3scale APIcast

Environment VariableDescriptionDefault

APICAST_EXTENDED_METRICS

A boolean value that enables additional information on Prometheus metrics. The following metrics have the service_id and service_system_name labels which provide more in-depth details about APIcast:

  • total_response_time_seconds
  • upstream_response_time_seconds
  • upstream_status

false

Additional resources

For information on setting environment variables, see the relevant OpenShift guides:

For information about supported configurations, refer to the Red Hat 3scale API Management Supported Configurations page.

9.4. 3scale APIcast metrics exposed to Prometheus

After you set up Prometheus to monitor 3scale APIcast, by default it can monitor the metrics listed in in Table 9.2, “Prometheus Default Metrics for 3scale APIcast”.

The metrics listed in Table 9.3, “Prometheus Metrics for 3scale APIcast Batch Policy” are only available when you use the Section 4.1.2, “3scale Batcher”.

Table 9.2. Prometheus Default Metrics for 3scale APIcast

MetricDescriptionTypeLabels

nginx_http_connections

Number of HTTP connections

gauge

state(accepted,active,handled,reading,total,waiting,writing)

nginx_error_log

APIcast errors

counter

level(debug,info,notice,warn,error,crit,alert,emerg)

openresty_shdict_capacity

Capacity of the dictionaries shared between workers

gauge

dict(one for every dictionary)

openresty_shdict_free_space

Free space of the dictionaries shared between workers

gauge

dict(one for every dictionary)

nginx_metric_errors_total

Number of errors of the Lua library that manages the metrics

counter

none

total_response_time_seconds

Time needed to send a response to the client (in seconds)

Note: To access the service_id and service_system_name labels, you must set the APICAST_EXTENDED_METRICS environment variable to true as described in Section 9.2, “APIcast integration with Prometheus”.

histogram

service_id, service_system_name

upstream_response_time_seconds

Response times from upstream servers (in seconds)

Note: To access the service_id and service_system_name labels, you must set the APICAST_EXTENDED_METRICS environment variable to true as described in Section 9.2, “APIcast integration with Prometheus”.

histogram

service_id, service_system_name

upstream_status

HTTP status from upstream servers

Note: To access the service_id and service_system_name labels, you must set the APICAST_EXTENDED_METRICS environment variable to true as described in Section 9.2, “APIcast integration with Prometheus”.

counter

status, service_id, service_system_name

threescale_backend_calls

Authorize and report requests to the 3scale backend (Apisonator)

counter

endpoint(authrep, auth, report), status(2xx, 4xx, 5xx)

Table 9.3. Prometheus Metrics for 3scale APIcast Batch Policy

MetricDescriptionTypeLabels

batching_policy_auths_cache_hits

Hits in the auths cache of the 3scale batching policy

counter

none

batching_policy_auths_cache_misses

Misses in the auths cache of the 3scale batching policy

counter

none