Chapter 4. Troubleshooting

The following sections explain how to resolve issues that may occur in Metrics Store.

4.1. Information Is Missing from Kibana

If Kibana is not displaying metric or log information as expected, use journalctl to investigate the collectd and rsyslog log files as follows:

  • If only metrics information is missing, check the collectd log files.
  • If only log information is missing, check the rsyslog log files.

  • If both metrics and logs information are missing, check both log files.

    1. To investigate collectd log files, log in as root and run the following command: 

      # journalctl -u collectd

    2. To investigate rsyslog log files, run the following command: 

      # journalctl -u rsyslog

To learn about additional journalctl options, see journalctl in Linux man pages .

4.2. Extracting Elasticsearch logs

To extract Elasticsearch logs

  1. Log in to the Metrics Store virtual machine as root, and run the following command. Where openshift-logging is the namespace for the Elasticsearch pod: 

    for espod in $(oc -n openshift-logging get pods -l component=es -o jsonpath='{.items[*].metadata.name}') ; do
    oc -n openshift-logging exec -c elasticsearch $espod -- logs > $espod.log 2>&1
done
  2. Optionally, you can use the use the OpenShift logging-dump tool located at logging dump tool script

4.3. Searching Elasticsearch Logs

  1. To search for all RHV/oVirt logs, ordered by timestamp from newest to oldest.Log in to the Metrics Store virtual machine as root, and run the following command, where openshift-logging is the namespace for the Elasticsearch pod: 

    oc -n opesnshift-logging exec -c elasticsearch $(oc -n openshift-logging get pods -l component=es -o jsonpath='{.items[0].metadata.name}') -- es_util --query=project.ovirt-logs*/_search?sort=@timestamp:desc​\&pretty | more
  2. The output is presented in the human readable JSON format.

4.4. Searching log record results

To search the log report results, you can use the get last rec from host search tool.

The search tool shows how long it has been since the Metrics Store virtual machine received a record from each host that it knows about during a given time interval (last 3 hours by default).

For each host that the Metrics Store virtual machine receives logs from over the defined duration (default 3h) duration, it prints out "green", "yellow", or "red", depending on whether the Metrics Store virtual machine received a record from that host recently or not.

This is followed by the number of seconds it has been since the last record was received from that host, and the number of records received.

  1. Log in to the Metrics Store virtual machine as root, and clone the script repository: 

    # git clone https://github.com/jcantrill/cluster-logging-tools -b release-3.11

  2. Run the script: 

    # cd cluster-logging-tools/scripts
[OLDEST=3h] ./get-last-rec-from-host

  3. To change the interval, change the OLDEST value to a longer or shorter interval in hours.

    For example, to go back 1 day (24 hours), use: 

    OLDEST=24h ./get-last-rec-from-host
Note

Some hosts may not be listed, if no records were received from that host during the given time interval.