Metrics Store Installation Guide

Red Hat Virtualization 4.3

Installing Metrics Store for Red Hat Virtualization

Red Hat Virtualization Documentation Team

Red Hat Customer Content Services

Abstract

A comprehensive guide to installing and configuring Metrics Store for Red Hat Virtualization. Metrics Store collects logs and metrics for Red Hat Virtualization 4.2 and later.

Preface

Metrics Store collects logs and metrics from Red Hat Virtualization. The data is transferred from Red Hat Virtualization to Red Hat OpenShift where it is stored and aggregated in Elasticsearch and saved in indexes. The data can then be analyzed and visualized in Kibana.

  • Elasticsearch is a distributed, RESTful search and analytics engine that lets you perform and combine many types of searches.
  • Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. You can easily perform advanced data analysis and visualize your data in a variety of charts and tables.

Chapter 1. Workflow and architecture

To install Metrics Store, complete the following major tasks:

Metrics Store architecture

The Metrics Store architecture is based on the Red Hat OpenShift EFK logging stack, running on Red Hat OpenShift Container Platform 3.11.

Metrics Store uses the following services:

  • collectd (hosts) collects metrics from hosts, virtual machines, and databases.
  • rsyslog (hosts) collects metrics, adds log data, enriches the data with metadata, and sends the enriched data to Elasticsearch.
  • Elasticsearch (Metrics Store virtual machine) stores and indexes the data.
  • Kibana (Metrics Store virtual machine) analyzes and presents the data as dashboards and charts.

Figure 1.1. Metrics Store architecture

Metrics Store architecture

Chapter 2. Installing Metrics Store

Prerequisites

2.1. Create the Metrics Store virtual machines

To create the Metrics Store virtual machines, perform the following tasks:

  1. Configure the Metrics Store installation.
  2. Create the following Metrics Store virtual machines:

    • The Metrics Store Installer virtual machine - a temporary virtual machine for deploying Red Hat OpenShift and services on the Metrics Store virtual machines.
    • One or more Metrics Store virtual machines.
  3. Verify the Metrics Store virtual machines.

Configure the Metrics Store installation

  1. Log in to the Manager machine using SSH.
  2. Update the packages:

    # yum update
  3. Copy metrics-store-config.yml.example to create metrics-store-config.yml:

    # cp /etc/ovirt-engine-metrics/metrics-store-config.yml.example /etc/ovirt-engine-metrics/config.yml.d/metrics-store-config.yml
  4. Edit the parameters in metrics-store-config.yml and save the file. The parameters are documented in the file.
  5. On the Manager machine, copy /etc/ovirt-engine-metrics/secure_vars.yaml.example to /etc/ovirt-engine-metrics/secure_vars.yaml:

    # cp /etc/ovirt-engine-metrics/secure_vars.yaml.example /etc/ovirt-engine-metrics/secure_vars.yaml
  6. Edit the parameters in /etc/ovirt-engine-metrics/secure_vars.yaml to match the details of your specific environment.
  7. Encrypt the secure_vars.yaml file:

    # ansible-vault encrypt /etc/ovirt-engine-metrics/secure_vars.yaml

Create Metrics Store virtual machines

  1. Go to the ovirt-engine-metrics directory:

    # cd /usr/share/ovirt-engine-metrics
  2. Run the ovirt-metrics-store-installation playbook to create the virtual machines:

    # ANSIBLE_JINJA2_EXTENSIONS="jinja2.ext.do" ./configure_ovirt_machines_for_metrics.sh --playbook=ovirt-metrics-store-installation.yml --ask-vault-pass

Verify the creation of the virtual machines

  1. Log in to the Administration Portal.
  2. Click ComputeVirtual Machines to verify that the metrics-store-installer virtual machine and the Metrics Store virtual machines are running.

2.2. Deploy Metrics Store services on Red Hat OpenShift

Deploy and verify Red Hat OpenShift, Elasticsearch, Curator (for managing Elasticsearch indices and snapshots), and Kibana on the Metrics Store virtual machines.

Procedure

  1. Log in to the metrics-store-installer virtual machine.
  2. Run the install_okd playbook to deploy Red Hat OpenShift and Metrics Store services to the Metrics Store virtual machines:

    # ANSIBLE_CONFIG="/usr/share/ansible/openshift-ansible/ansible.cfg" \
      ANSIBLE_ROLES_PATH="/usr/share/ansible/roles/:/usr/share/ansible/openshift-ansible/roles" \
      ansible-playbook -i integ.ini install_okd.yaml -e @vars.yaml -e @secure_vars.yaml --ask-vault-pass
  3. Verify the deployment by logging in to each Metrics Store virtual machine:

    1. Log in to the openshift-logging project:

      # oc project openshift-logging
    2. Check that the Elasticsearch, Curator, and Kibana pods are running:

      # oc get pods

      If Elasticsearch is not running, see Troubleshooting related to ElasticSearch in the OpenShift Container Platform 3.11 documentation.

  4. Check the Kibana host name and record it so you can access the Kibana console in Chapter 4, Verify the Metrics Store installation:

    # oc get routes

Cleanup

  1. Log in to the Administration Portal.
  2. Click ComputeVirtual Machines and delete the metrics-store-installer virtual machine.

2.3. Configure networking for Metrics Store virtual machines

Network configuration prerequisites

  • Create a wildcard DNS record (*.example.com) for the DNS zone of the Metrics Store virtual machines.
  • Add the hostnames of the Metrics Store virtual machines to your DNS server.

Set a static MAC address for a Metrics Store virtual machine (optional)

  1. Log in to the Administration Portal.
  2. Click ComputeVirtual Machines and select a Metrics Store virtual machine.
  3. In the Network Interfaces tab, select a NIC and click Edit.
  4. Select Custom MAC Address, enter the MAC address, and click OK.
  5. Reboot the virtual machine.

Configure firewall ports

The following table describes the firewall settings needed for communication between the ports used by Metrics Store.

Table 2.1. Configure the firewall to allow connections to specific ports

IDPort(s)ProtocolSourcesDestinationsPurpose

MS1

9200

TCP

RHV Red Hat Virtualization Hosts
RHV Manager

Metrics Store VM

Transfer data to ElasticSearch.

MS2

5601

TCP

Kibana user

Metrics Store VM

Give users access to the Kibana web interface.

Note

Whether a connection is encrypted depends on how you deployed the software.

Chapter 3. Deploy collectd and rsyslog

Deploy collectd and rsyslog on the hosts to collect logs and metrics.

Note

You do not need to repeat this procedure for new hosts. The Manager configures the hosts automatically.

Procedure

  1. Log in to the Manager machine using SSH.
  2. Copy /etc/ovirt-engine-metrics/config.yml.example to create /etc/ovirt-engine-metrics/config.yml.d/config.yml:

    # cp /etc/ovirt-engine-metrics/config.yml.example /etc/ovirt-engine-metrics/config.yml.d/config.yml
  3. Edit the ovirt_env_name and elasticsearch_host parameters in config.yml and save the file. These parameters are mandatory and are documented in the file.

    Note

    If you add a Manager or an Elasticsearch installation, copy the Manager’s public key to your Metrics Store virtual machine using the following commands:

    # mytemp=$(mktemp -d)
    # cp /etc/pki/ovirt-engine/keys/engine_id_rsa $mytemp
    # ssh-keygen -y -f $mytemp/engine_id_rsa > $mytemp/engine_id_rsa.pub
    # ssh-copy-id -i $mytemp/engine_id_rsa.pub root@{elasticsearch_host}
    # rm -rf $mytemp
  4. Deploy collectd and rsyslog on the hosts:

    # /usr/share/ovirt-engine-metrics/setup/ansible/configure_ovirt_machines_for_metrics.sh

Chapter 4. Verify the Metrics Store installation

Verify the Metrics Store installation using the Kibana console. You can view the collected logs and create data visualizations.

Procedure

  1. Log in to the Kibana console using the URL (https://kibana.example.com) that you recorded in Section 2.2, “Deploy Metrics Store services on Red Hat OpenShift”. Use the default admin user, and the password you defined during the metrics store installation.

    Optionally, you can access the Red Hat OpenShift Container Platform portal at https://example.com:8443 (using the same admin user credentials).

  2. In the Discover tab, check that you can view the project.ovirt-logs-ovirt_env_name-uuid index.

    See the Discover section in the Kibana User Guide for information about working with logs.

  3. In the Visualization tab, you can create data visualizations for the project.ovirt-metrics-ovirt_env_name-uuid and the project.ovirt-logs-ovirt_env_name-uuid indexes.

    The Metrics Store User Guide describes the available parameters. See the Visualize section of the Kibana User Guide for information about visualizing logs and metrics.

Appendix A. Sharing reports with non-administrators

Users without administrator privileges can view collected logs and metrics as read-only users. The following example creates a user named user name with view (read-only) permissions.

  1. Log in to the Metrics Store virtual machine.
  2. Create a new user:

      # oc create user username
      # oc create identity allow_all: username
      # oc create useridentitymapping allow_all: username username
  3. Log in to the openshift-logging project:

      # oc project openshift-logging
  4. Assign a view role to the user:

      # oc adm policy add-role-to-user view user name
  5. Create a password for the user:

      # oc login --username=user name --password=password