Adding numeric value tags to machinesets in OpenShift 4 on AWS
Issue
When creating a machineset definition yaml file it is possible to add a tags section containing a list of tags to apply to the host instances created by the cloud provider (AWS):
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
...
spec:
providerSpec:
value:
...
tags:
- name: kubernetes.io/cluster/koshka-j9f8v
value: owned
- name: Environment
value: Test
- name: POBox
value: 09080
If any of the tags has a numeric value, as shown in the example above, when the machineset is created the corresponding machines never get created:
$ oc create -f machineset1.yaml
machineset.machine.openshift.io/koshka-82qmp-worker-eu-west-3a2 created
$ oc get machinesets -n openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
...
koshka-82qmp-worker-eu-west-3a2 1 1 50m
$ oc get machines
NAME PHASE TYPE REGION ZONE AGE
putri-qnc9k-worker-eu-west-3a-mzt72 65m
The logs from the machine controller pod contains error messages like the following:
$ oc logs machine-api-controllers-7f744c5666-k4k9k -c machine-controller -n openshift-machine-api
...
E0806 12:24:32.307766 1 actuator.go:512] koshka-82qmp-worker-eu-west-3a2-8j6j5: Error decoding MachineProviderConfig: decoding failure: v1beta1.AWSMachineProviderConfig.Tags: []v1beta1.TagSpecification: v1beta1.TagSpecification.Value: ReadString: expects " or n, but found 4, error found in #10 byte of ...|,"value":41805}],"us|..., bigger context ...|Planet","value":"Earth"},{"name":"POBox","value":41805}],"userDataSecret":{"name":"worker-user-data"|...
E0806 12:24:32.307778 1 actuator.go:493] koshka-82qmp-worker-eu-west-3a2-8j6j5: Error getting existing instances: decoding failure: v1beta1.AWSMachineProviderConfig.Tags: []v1beta1.TagSpecification: v1beta1.TagSpecification.Value: ReadString: expects " or n, but found 4, error found in #10 byte of ...|,"value":41805}],"us|..., bigger context ...|Planet","value":"Earth"},{"name":"POBox","value":41805}],"userDataSecret":{"name":"worker-user-data"|...
E0806 12:24:32.307791 1 controller.go:266] koshka-82qmp-worker-eu-west-3a2-8j6j5: failed to check if machine exists: decoding failure: v1beta1.AWSMachineProviderConfig.Tags: []v1beta1.TagSpecification: v1beta1.TagSpecification.Value: ReadString: expects " or n, but found 4, error found in #10 byte of ...|,"value":41805}],"us|..., bigger context ...|Planet","value":"Earth"},{"name":"POBox","value":41805}],"userDataSecret":{"name":"worker-user-data"|...
If the stuck machineset is deleted, the corresponding machine stays in Deleting state for ever and the following event appears in the machine-api namespace:
$ oc delete machineset koshka-82qmp-worker-eu-west-3a2 -n openshift-machine-api
machineset.machine.openshift.io "koshka-82qmp-worker-eu-west-3a2" deleted
$ oc get machines -n openshift-machine-api
NAME PHASE TYPE REGION ZONE AGE
...
koshka-82qmp-worker-eu-west-3a2-8j6j5 Deleting 12m
$ oc get events -n openshift-machine-api
100s Warning FailedDelete machine/koshka-82qmp-worker-eu-west-3a2-8j6j5 error decoding MachineProviderConfig: decoding failure: v1beta1.AWSMachineProviderConfig.Tags: []v1beta1.TagSpecification: v1beta1.TagSpecification.Value: ReadString: expects " or n, but found 4, error found in #10 byte of ...|,"value":41805}],"us|..., bigger context ...|Planet","value":"Earth"},{"name":"POBox","value":41805}],"userDataSecret":{"name":"worker-user-data"|...
Deleting the machine with the following command will cause the terminal to block until canceled with CTRL-C, and the machine is not deleted:
$ oc delete machine koshka-82qmp-worker-eu-west-3a2-8j6j5 -n openshift-machine-api
machine.machine.openshift.io "koshka-82qmp-worker-eu-west-3a2-6t2vx" deleted
^C
$ $ oc get machines -n openshift-machine-api
NAME PHASE TYPE REGION ZONE AGE
...
koshka-82qmp-worker-eu-west-3a2-8j6j5 Deleting 87m
To delete the machine, edit the yaml definition and remove the finalizers section
$ oc edit machine koshka-82qmp-worker-eu-west-3a2-8j6j5 -n openshift-machine-api
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
creationTimestamp: "2020-08-06T12:52:11Z"
deletionGracePeriodSeconds: 0
deletionTimestamp: "2020-08-06T14:18:06Z"
finalizers: <-------------- Delete this line
- machine.machine.openshift.io <--------------------- Delete this line
This issue is not present when adding user defined tags with numeric values to the install-config.yaml file used by the Openshift installer, as can be seen in this example in the Openshift documentation. In this case the list of user defined tags is properly added to the machinesets for the worker nodes.
Environment
- Red Hat OpenShift Container Platform
- 4.2
- 4.3
- 4.4
The issue is fixed in OCP 4.5
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.