How to use metricbeat to collect OCP metrics

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
  • Azure Red Hat OpenShift (ARO)

Issue

Some customers may have their ELK clusters and would like to monitor their OCP cluster within a single portal by injecting cluster monitoring metrics into elasticsearch. Rather than deploying a daemonset of metricbeat to collect metrics one by one, there is another way to use metricbeat to scrape metrics from prometheus.

Resolution

Disclaimer: The following information has been provided by Red Hat, but is outside the scope of the posted Service Level Agreements and support procedures. The information is provided as-is and any configuration settings or installed applications made from the information in this article could make the Operating System unsupported by Red Hat Global Support Services. The intent of this article is to provide information to accomplish the system's needs. Use of the information in this article at the user's own risk.

  1. Create a new namespace, for example metricbeat
  2. Create configmap for metricbeat config, please refer to metricbeat doc, update here if any change to the metricbeat config, and the output target, for example to console, ES or others
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-config
  namespace: metricbeat
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false

    processors:
      - add_cloud_metadata: ~

    output.console:
      enabled: true
      pretty: false
  1. Create configmap for metricbeat module, this module control how metricbeat collect metrics, please refer to prometheus /federate api for mode detail
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-modules
  namespace: metricbeat
  labels:
    k8s-app: metricbeat
data:
  remotewrite.yml: |-
    - module: prometheus
      period: 10s
      hosts: ["https://prometheus-k8s.openshift-monitoring.svc.cluster.local:9091"]
      metrics_path: '/federate'
      query:
        'match[]': '{__name__!=""}'
      # This can be used for service account based authorization:
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      #ssl.verification_mode: "none"
      ssl.certificate_authorities:
        - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
  1. Create serviceAccount for metricbeat
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metricbeat-sa
  namespace: metricbeat
  labels:
    k8s-app: metricbeat
  1. Create clusterrolebinding for metricbeat
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metricbeat-sa-binding
subjects:
- kind: ServiceAccount
  name: metricbeat-sa
  namespace: metricbeat
roleRef:
  kind: ClusterRole
  name: prometheus-k8s
  apiGroup: rbac.authorization.k8s.io
  1. Create deployment for metricbeat
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metricbeat
  namespace: metricbeat
  labels:
    k8s-app: metricbeat
spec:
  selector:
    matchLabels:
      k8s-app: metricbeat
  template:
    metadata:
      labels:
        k8s-app: metricbeat
    spec:
      serviceAccount: metricbeat-sa
      containers:
      - name: metricbeat
        image: docker.elastic.co/beats/metricbeat:7.16.0
        args: [
          "-c", "/etc/metricbeat.yml",
          "-e",
        ]
        resources:
          limits:
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 9201
        volumeMounts:
        - name: config
          mountPath: /etc/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        - name: modules
          mountPath: /usr/share/metricbeat/modules.d
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: metricbeat-daemonset-config
      - name: modules
        configMap:
          defaultMode: 0640
          name: metricbeat-daemonset-modules

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments