Error fetching openstack meta_data.json file in RHOCP 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
    • 4.6
  • Red Hat OpenStack Platform
    • 16.1.3
    • Cisco ACI CNI Plugin enabled

Issue

  • Node is inNotReady state .
  • Kubelet service unable to start with below error:
failed to run Kubelet: error fetching current node name from cloud provider: error fetching http://XXX.XXX.XXX.XXX/openstack/2012-08-10/meta_data.json: Get "http://XXX.XXX.XXX.XXX/openstack/2012-08-10/meta_data.json": dial tcp XXX.XXX.XXX.XXX:80: i/o timeout

Resolution

  • This issue generally happens if metadata service malfunctioned on Openstack compute hosts.
  • Restarting ciscoaci_neutron_opflex_agent container might fix the issue and Openshift nodes start query http://XXX.XXX.XXX.XXX/openstack/2012-08-10/meta_data.json metadata file.

Note : If such issue faced in the cluster, make sure to open a case with CISCO too along with Red Hat for a faster troubleshooting process.

Diagnostic Steps

  • Check for the below error in the kubelet service logs using command:
$ journalctl -u kubelet

Error:

Apr 09 13:26:12 a0410ocpinfralog3 hyperkube[152550]: I0409 13:26:12.303364  152550 metadata.go:166] Attempting to fetch metadata from http://XXX.XXX.XXX.XXX/openstack/2012-08-10/meta_data.json
Apr 09 13:26:42 a0410ocpinfralog3 hyperkube[152550]: F0409 13:26:42.303830  152550 server.go:265] failed to run Kubelet: error fetching current node name from cloud provider: error fetching http://XXX.XXX.XXX.XXX/openstack/2012-08-10/meta_data.json: Get "http://XXX.XXX.XXX.XXX/openstack/2012-08-10/meta_data.json": dial tcp XXX.XXX.XXX.XXX:80: i/o timeout
  • Check if the kubelet service is running or not.
$ sudo systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-mco-default-env.conf, 10-mco-default-madv.conf, 20-nodenet.conf
   Active: activating (start) since Mon 2022-04-18 06:47:42 UTC; 4s ago
  Process: 769879 ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state (code=exited, status=0/SUCCESS)
  Process: 769876 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
 Main PID: 769881 (kubelet)
    Tasks: 19 (limit: 411715)
   Memory: 30.7M
      CPU: 137ms
   CGroup: /system.slice/kubelet.service
           └─769881 kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --containe>

Apr 18 06:47:43 a0410ocpappworker hyperkube[769881]: I0418 06:47:43.091725  769881 mount_linux.go:178] Detected OS with systemd
Apr 18 06:47:43 a0410ocpappworker hyperkube[769881]: I0418 06:47:43.101531  769881 subpath_mount_linux.go:157] Detected OS with systemd
Apr 18 06:47:43 a0410ocpappworker hyperkube[769881]: I0418 06:47:43.101742  769881 server.go:412] Version: v1.19.14+fcff70a
Apr 18 06:47:43 a0410ocpappworker hyperkube[769881]: I0418 06:47:43.101894  769881 feature_gate.go:244] feature gates: &{map[APIPriorityAndFairness:true LegacyNodeRol>
Apr 18 06:47:43 a0410ocpappworker hyperkube[769881]: I0418 06:47:43.101990  769881 feature_gate.go:244] feature gates: &{map[APIPriorityAndFairness:true LegacyNodeRol>
Apr 18 06:47:43 a0410ocpappworker hyperkube[769881]: W0418 06:47:43.102118  769881 plugins.go:105] WARNING: openstack built-in cloud provider is now deprecated. Pleas>
Apr 18 06:47:43 a0410ocpappworker hyperkube[769881]: I0418 06:47:43.102253  769881 server.go:533] Successfully initialized cloud provider: "openstack" from the config>
Apr 18 06:47:43 a0410ocpappworker hyperkube[769881]: I0418 06:47:43.102280  769881 openstack_instances.go:49] openstack.Instances() called
Apr 18 06:47:43 a0410ocpappworkerhyperkube[769881]: E0418 06:47:43.102288  769881 openstack_instances.go:55] cannot initialize cloud provider, only limited functiona>
Apr 18 06:47:43 a0410ocpappworker hyperkube[769881]: I0418 06:47:43.110816  769881 metadata.go:166] Attempting to fetch metadata from http://XXX.XXX.XXX.XXX/openstack>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments