After the installation of openshift-logging all the node label changed

Solution Verified - Updated -

Environment

  • OpenShift Container Platform 3.x

Issue

  • Node labels changed to infra after the Installation of openshift-logging using ansible-playbook.
  • The logging playbook overwrites the label based on the inventory variable openshift_logging_fluentd_nodeselector on all the nodes.

Resolution

  • It is better not to use the node selector which is overwriting the node label.
  • Update the inventory and rerun the logging playbook.
openshift_logging_fluentd_nodeselector={'logging-infra-fluentd': 'true'}
openshift_logging_fluentd_hosts=['--all']

OR

  • Manually add the label logging-infra-fluentd=true to the node, Change node selector to the fluentd and remove unwanted label infra.
$ oc label  node <node_name> logging-infra-fluentd=true --overwrite=true

Root Cause

  • The fluentd pods are running as daemonset and it will run on all the nodes of the cluster. Adding openshift_logging_fluentd_nodeselector={'node-role.kubernetes.io/infra': 'true'} node selector will overwrite the node label.
  • The following is the task playbook would run while adding the required labels during the logging installation.
# We need to pause between nodes while labeling to avoid overloading the scheduler.
- name: Label OCP nodes for Fluentd
  shell: >
    {{ openshift_client_binary }} --config=/etc/origin/master/admin.kubeconfig \
    label --overwrite node {{ fluentd_host }} \
    {% for k, v in openshift_logging_fluentd_nodeselector.items() %} {{ k }}={{ v }} {% endfor %} \
    && \
    sleep {{ openshift_logging_fluentd_label_delay }}
  with_items: "{{ openshift_logging_fluentd_hosts }}"
  loop_control:
    loop_var: fluentd_host

Diagnostic Steps

  • Check the current labels on the node with the help of the following commands,
$ oc get nodes --show-labels
  • Also check the label fluentd daemonset is looking to get a fluentd pod running on the node,
$ oc get ds logging-fluentd -n openshift-logging

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments