Deployment fails with Error: Error from DataBinding 'hiera' while looking up 'tripleo::trusted_cas::ca_map': (<unknown>): did not find expected key while parsing a block mapping in Red Hat OpenStack Platform 10

Issue

The deployment of a new role fails with the following error message:

Debug: hiera(): Hiera JSON backend starting
Debug: hiera(): Looking up step in JSON backend
Debug: hiera(): Looking for data source 31393638-3132-4753-4839-313258595943
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/31393638-3132-4753-4839-313258595943.json, skipping
Debug: hiera(): Looking for data source heat_config_ComputeRoleNameDeployment_Step1
Debug: hiera(): Looking up step in JSON backend
Debug: hiera(): Looking for data source 31393638-3132-4753-4839-313258595943
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/31393638-3132-4753-4839-313258595943.json, skipping
Debug: hiera(): Looking for data source heat_config_ComputeRoleNameDeployment_Step1
Debug: importing '/etc/puppet/modules/tripleo/manifests/init.pp' in environment production
Debug: importing '/etc/puppet/modules/tripleo/manifests/trusted_cas.pp' in environment production
Debug: Automatically imported tripleo::trusted_cas from tripleo/trusted_cas into production
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: hiera(): Looking up tripleo::trusted_cas::ca_map in JSON backend
Debug: hiera(): Looking for data source 31393638-3132-4753-4839-313258595943
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/31393638-3132-4753-4839-313258595943.json, skipping
Debug: hiera(): Looking for data source heat_config_ComputeRoleNameDeployment_Step1
Debug: hiera(): Looking for data source ComputeRoleName_extraconfig
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/ComputeRoleName_extraconfig.json, skipping
Debug: hiera(): Looking for data source extraconfig
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/extraconfig.json, skipping
Debug: hiera(): Looking for data source service_names
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/service_names.json, skipping
Debug: hiera(): Looking for data source service_configs
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/service_configs.json, skipping
Debug: hiera(): Looking for data source ComputeRoleName
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/ComputeRoleName.json, skipping
Debug: hiera(): Looking for data source bootstrap_node
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/bootstrap_node.json, skipping
Debug: hiera(): Looking for data source all_nodes
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/all_nodes.json, skipping
Debug: hiera(): Looking for data source vip_data
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/vip_data.json, skipping
Debug: hiera(): Looking for data source RedHat
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/RedHat.json, skipping
Debug: hiera(): Hiera YAML backend starting
Debug: hiera(): Looking up tripleo::trusted_cas::ca_map in YAML backend
Debug: hiera(): Looking for data source 31393638-3132-4753-4839-313258595943
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/31393638-3132-4753-4839-313258595943.yaml, skipping
Debug: hiera(): Looking for data source heat_config_ComputeRoleNameDeployment_Step1
Debug: hiera(): Cannot find datafile /etc/puppet/hieradata/heat_config_ComputeRoleNameDeployment_Step1.yaml, skipping
Debug: hiera(): Looking for data source ComputeRoleName_extraconfig
Error: Error from DataBinding 'hiera' while looking up 'tripleo::trusted_cas::ca_map': (<unknown>): did not find expected key while parsing a block mapping at line 2 column 1 on node compute-1.example.com
Error: Error from DataBinding 'hiera' while looking up 'tripleo::trusted_cas::ca_map': (<unknown>): did not find expected key while parsing a block mapping at line 2 column 1 on node compute-1.example.com

From the above, it's obvious that puppet tries to go through every .json file according to the order that should be in /etc/puppet/hiera.yaml. However, puppet aborts when it parses file ComputeRoleName_extraconfig.

We don't find anything but heat_config_ComputeRoleNameDeployment_Step1.json, which is o.k.:

[user@host hieradata]$ cat heat_config_ComputeRoleNameDeployment_Step1.json
{"update_identifier": "1537124308", "deploy_resource_name": "2", "deploy_signal_id": "http://192.168.40.11:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3Ac64cb418f5bd47069559dfa34937cef9%3Astacks%2Fovercloud-AllNodesDeploySteps-sygifyfscxlx-ComputeRoleNameDeployment_Step1-y4bymtvtlko3%2F7f2a4ad4-ebaf-417b-95cc-de5d898d26cd%2Fresources%2F2?Timestamp=2019-07-04T19%3A53%3A29Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=28dc69bfe20b40c0aedbbd916ad55925&SignatureVersion=2&Signature=YcPa54wb56dWNXOfRoHZ6mFHMOU6QZyKZfjcxVbQHhI%3D", "deploy_signal_transport": "CFN_SIGNAL", "deploy_signal_verb": "POST", "step": 1, "deploy_server_id": "bc2a0202-6aac-4728-8068-1b7a79602f95", "deploy_stack_id": "overcloud-AllNodesDeploySteps-sygifyfscxlx-ComputeRoleNameDeployment_Step1-y4bymtvtlko3/7f2a4ad4-ebaf-417b-95cc-de5d898d26cd", "deploy_action": "CREATE"}
[user@host hieradata]$ cat heat_config_ComputeRoleNameDeployment_Step1.json | python -m json.tool
{
    "deploy_action": "CREATE",
    "deploy_resource_name": "2",
    "deploy_server_id": "bc2a0202-6aac-4728-8068-1b7a79602f95",
    "deploy_signal_id": "http://192.168.40.11:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3Ac64cb418f5bd47069559dfa34937cef9%3Astacks%2Fovercloud-AllNodesDeploySteps-sygifyfscxlx-ComputeRoleNameDeployment_Step1-y4bymtvtlko3%2F7f2a4ad4-ebaf-417b-95cc-de5d898d26cd%2Fresources%2F2?Timestamp=2019-07-04T19%3A53%3A29Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=28dc69bfe20b40c0aedbbd916ad55925&SignatureVersion=2&Signature=YcPa54wb56dWNXOfRoHZ6mFHMOU6QZyKZfjcxVbQHhI%3D",
    "deploy_signal_transport": "CFN_SIGNAL",
    "deploy_signal_verb": "POST",
    "deploy_stack_id": "overcloud-AllNodesDeploySteps-sygifyfscxlx-ComputeRoleNameDeployment_Step1-y4bymtvtlko3/7f2a4ad4-ebaf-417b-95cc-de5d898d26cd",
    "step": 1,
    "update_identifier": "1537124308"
}

Why do we stop at {role name}_extraconfig?

According to /etc/puppet/hiera.yaml, this should continue and pick up service_configs, e.g.:

:backends:
  - json
  - yaml
:json:
  :datadir: /etc/puppet/hieradata
:yaml:
  :datadir: /etc/puppet/hieradata
:hierarchy:
  - "%{::uuid}"
  - heat_config_%{::deploy_config_name}
  - ComputeRoleName_extraconfig
  - extraconfig
  - service_names
  - service_configs
  - ComputeRoleName
  - bootstrap_node
  - all_nodes
  - vip_data
  - "%{::osfamily}"
[user@host ~]$ cat _additional_files/text/hiera.yaml 
---
:backends:
  - json
  - yaml
:json:
  :datadir: /etc/puppet/hieradata
:yaml:
  :datadir: /etc/puppet/hieradata
:hierarchy:
  - "%{::uuid}"
  - heat_config_%{::deploy_config_name}
  - computesriov40dpdk_extraconfig
  - extraconfig
  - service_names
  - service_configs
  - computesriov40dpdk
  - bootstrap_node
  - all_nodes
  - vip_data
  - "%{::osfamily}"

file service_configs.yaml holds the key/value pair:

[user@host hieradata]$ grep cas -R
service_configs.yaml:tripleo::trusted_cas::ca_map: {}
[user@host hieradata]$