Error: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-activate-/srv/data]/returns: change from notrun to 0 failed: Command exceeded timeout (truncated, view all with --long)

Latest response

I was trying to deploy the RHOSP 11 with Ceph storage,

I have successfull heat templates which was successfully deploying cloud with ceph integration ( Hyper-Converged ) still last week. Deployment step got failed in starting of mistral and found Redhat have updated the code, found it through the following link and updated the code.
https://access.redhat.com/solutions/3459241

Followed by, I get error in step 3.1 Ceph Disk activation .
I found the exact error through the command.
"openstack stack failures list overcloud"

overcloud.AllNodesDeploySteps.ComputeDeployment_Step3.1:
resource_type: OS::Heat::StructuredDeployment
physical_resource_id: 12bfa168-8779-431e-a807-3ab4619ea169
status: CREATE_FAILED
status_reason: |
Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
deploy_stdout: |
Error: Command exceeded timeout
Error: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-activate-/srv/data]/returns: change from notrun to 0 failed: Command exceeded timeout
(truncated, view all with --long)

Is there anything I have to change in the new code or any configuration mismatches.

Any help would be greatly appreciated.

Thanks in advance!!

Responses

Might need to include --long to see what caused the error:

$ openstack stack failures list overcloud --long

Please find the error log, I was not able to see anything that indicated the error or problem directly.

[stack@dir]$ openstack stack failures list overcloud --long overcloud.AllNodesDeploySteps.ComputeDeployment_Step3.1: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 844fa8a7-6550-459e-bdaa-05a5b6a25745 status: CREATE_FAILED status_reason: | Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 deploy_stdout: | Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked. Notice: Compiled catalog for overcloud-compute-1.localdomain in environment production in 3.98 seconds Notice: /Stage[main]/Main/Package_manifest[/var/lib/tripleo/installed-packages/overcloud_compute3]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/auth_strategy]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/core_plugin]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/host]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/dhcp_agents_per_network]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/allow_overlapping_ips]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/global_physnet_mtu]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[agent/root_helper]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/service_plugins]/ensure: created Notice: /Stage[main]/Ceilometer/Ceilometer_config[DEFAULT/http_timeout]/ensure: created Notice: /Stage[main]/Ceilometer/Ceilometer_config[DEFAULT/host]/ensure: created Notice: /Stage[main]/Ceilometer/Ceilometer_config[publisher/telemetry_secret]/ensure: created Notice: /Stage[main]/Ceilometer/Ceilometer_config[database/event_time_to_live]/ensure: created Notice: /Stage[main]/Ceilometer/Ceilometer_config[database/metering_time_to_live]/ensure: created Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Kmod::Load[nf_conntrack_proto_sctp]/Exec[modprobe nf_conntrack_proto_sctp]/returns: executed successfully Notice: /Stage[main]/Neutron::Logging/Oslo::Log[neutron_config]/Neutron_config[DEFAULT/log_dir]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Default[neutron_config]/Neutron_config[DEFAULT/control_exchange]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Concurrency[neutron_config]/Neutron_config[oslo_concurrency/lock_path]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/heartbeat_timeout_threshold]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_password]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_use_ssl]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_userid]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_hosts]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_ha_queues]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[DEFAULT/rpc_backend]/ensure: created Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::config::end]: Triggered 'refresh' from 18 events Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::service::begin]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Ceilometer::Logging/Oslo::Log[ceilometer_config]/Ceilometer_config[DEFAULT/log_dir]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[oslo_messaging_rabbit/heartbeat_timeout_threshold]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[oslo_messaging_rabbit/rabbit_password]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[oslo_messaging_rabbit/rabbit_use_ssl]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[oslo_messaging_rabbit/rabbit_userid]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[oslo_messaging_rabbit/rabbit_hosts]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[oslo_messaging_rabbit/rabbit_ha_queues]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[DEFAULT/rpc_backend]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Notifications[ceilometer_config]/Ceilometer_config[oslo_messaging_notifications/topics]/ensure: created Notice: /Stage[main]/Ceilometer::Deps/Anchor[ceilometer::config::end]: Triggered 'refresh' from 14 events Notice: /Stage[main]/Ceilometer::Deps/Anchor[ceilometer::service::begin]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: ++ readlink -f /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + disk=/srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + test -b /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + echo /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + egrep -e '^/dev' -q -v Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + mkdir -p /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + getent passwd ceph Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + chown -h ceph:ceph /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: ++ readlink -f /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: ++ readlink -f '' Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + ceph-disk prepare --cluster ceph --cluster-uuid eb2bb192-b1c9-11e6-9205-525400330666 /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + udevadm settle Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: executed successfully Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[fcontext_/srv/data]/returns: executed successfully Notice: /Stage[main]/Tripleo::Firewall/Exec[nonpersistent_v4_rules_cleanup]/returns: executed successfully Notice: /Stage[main]/Tripleo::Firewall/Exec[reload_iptables]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Tripleo::Firewall/Exec[nonpersistent_v6_rules_cleanup]/returns: executed successfully Notice: /Stage[main]/Tripleo::Firewall/Exec[reload_ip6tables]: Triggered 'refresh' from 1 events Notice: Applied catalog in 306.67 seconds deploy_stderr: | exception: connect failed Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/var/lib/heat-config/heat-config-puppet/3f729bd7-2e89-46dd-8ffd-9db4ad9b08dc.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in deprecation') Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/var/lib/heat-config/heat-config-puppet/3f729bd7-2e89-46dd-8ffd-9db4ad9b08dc.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:indeprecation') Warning: ModuleLoader: module 'ceph' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules (file & line not available) Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/3f729bd7-2e89-46dd-8ffd-9db4ad9b08dc.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in deprecation') Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules (file & line not available) Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/3f729bd7-2e89-46dd-8ffd-9db4ad9b08dc.pp", 34]: (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:indeprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/3f729bd7-2e89-46dd-8ffd-9db4ad9b08dc.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/3f729bd7-2e89-46dd-8ffd-9db4ad9b08dc.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:indeprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/3f729bd7-2e89-46dd-8ffd-9db4ad9b08dc.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/3f729bd7-2e89-46dd-8ffd-9db4ad9b08dc.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:indeprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/3f729bd7-2e89-46dd-8ffd-9db4ad9b08dc.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in deprecation') Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules (file & line not available) Warning: Scope(Class[Neutron]): neutron::rabbit_host, neutron::rabbit_hosts, neutron::rabbit_password, neutron::rabbit_port, neutron::rabbit_user and neutron::rabbit_virtual_host are deprecated. Please use neutron::default_transport_url instead. Warning: Scope(Class[Ceilometer]): ceilometer::rabbit_host, ceilometer::rabbit_hosts, ceilometer::rabbit_password, ceilometer::rabbit_port, ceilometer::rabbit_userid and ceilometer::rabbit_virtual_host are deprecated. Please use ceilometer::default_transport_url instead. Warning: This method is deprecated, please use match expressions with Stdlib::Compat::String instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at : (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:indeprecation') Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Bool instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at : (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in deprecation') Warning: Scope(Oslo::Messaging::Rabbit[neutron_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Warning: Scope(Oslo::Messaging::Rabbit[ceilometer_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Error: Command exceeded timeout Error: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-activate-/srv/data]/returns: change from notrun to 0 failed: Command exceeded timeout overcloud.AllNodesDeploySteps.ComputeDeployment_Step3.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: f7e0be0e-021c-43c9-be02-89cceaabd28e status: CREATE_FAILED status_reason: | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 deploy_stdout: | Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked. Notice: Compiled catalog for overcloud-compute-0.localdomain in environment production in 4.17 seconds Notice: /Stage[main]/Main/Package_manifest[/var/lib/tripleo/installed-packages/overcloud_compute3]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/auth_strategy]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/core_plugin]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/host]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/dhcp_agents_per_network]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/allow_overlapping_ips]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/global_physnet_mtu]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[agent/root_helper]/ensure: created Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/service_plugins]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/File[/etc/neutron/plugin.ini]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/File[/etc/default/neutron-server]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[ml2/type_drivers]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[ml2/tenant_network_types]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[ml2/mechanism_drivers]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[ml2/path_mtu]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[ml2/extension_drivers]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[securitygroup/firewall_driver]/ensure: created Notice: /Stage[main]/Ceilometer/Ceilometer_config[DEFAULT/http_timeout]/ensure: created Notice: /Stage[main]/Ceilometer/Ceilometer_config[DEFAULT/host]/ensure: created Notice: /Stage[main]/Ceilometer/Ceilometer_config[publisher/telemetry_secret]/ensure: created Notice: /Stage[main]/Ceilometer/Ceilometer_config[database/event_time_to_live]/ensure: created Notice: /Stage[main]/Ceilometer/Ceilometer_config[database/metering_time_to_live]/ensure: created Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Kmod::Load[nf_conntrack_proto_sctp]/Exec[modprobe nf_conntrack_proto_sctp]/returns: executed successfully Notice: /Stage[main]/Neutron::Logging/Oslo::Log[neutron_config]/Neutron_config[DEFAULT/log_dir]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Default[neutron_config]/Neutron_config[DEFAULT/control_exchange]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Concurrency[neutron_config]/Neutron_config[oslo_concurrency/lock_path]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/heartbeat_timeout_threshold]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_password]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_use_ssl]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_userid]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_hosts]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_ha_queues]/ensure: created Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[DEFAULT/rpc_backend]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Type_driver[vxlan]/Neutron_plugin_ml2[ml2_type_vxlan/vxlan_group]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Type_driver[vxlan]/Neutron_plugin_ml2[ml2_type_vxlan/vni_ranges]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Type_driver[vlan]/Neutron_plugin_ml2[ml2_type_vlan/network_vlan_ranges]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Type_driver[flat]/Neutron_plugin_ml2[ml2_type_flat/flat_networks]/ensure: created Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Type_driver[gre]/Neutron_plugin_ml2[ml2_type_gre/tunnel_id_ranges]/ensure: created Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::config::end]: Triggered 'refresh' from 31 events Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::service::begin]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Ceilometer::Logging/Oslo::Log[ceilometer_config]/Ceilometer_config[DEFAULT/log_dir]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[oslo_messaging_rabbit/heartbeat_timeout_threshold]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[oslo_messaging_rabbit/rabbit_password]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[oslo_messaging_rabbit/rabbit_use_ssl]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[oslo_messaging_rabbit/rabbit_userid]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[oslo_messaging_rabbit/rabbit_hosts]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[oslo_messaging_rabbit/rabbit_ha_queues]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Rabbit[ceilometer_config]/Ceilometer_config[DEFAULT/rpc_backend]/ensure: created Notice: /Stage[main]/Ceilometer/Oslo::Messaging::Notifications[ceilometer_config]/Ceilometer_config[oslo_messaging_notifications/topics]/ensure: created Notice: /Stage[main]/Ceilometer::Deps/Anchor[ceilometer::config::end]: Triggered 'refresh' from 14 events Notice: /Stage[main]/Ceilometer::Deps/Anchor[ceilometer::service::begin]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: ++ readlink -f /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + disk=/srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + test -b /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + echo /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + egrep -e '^/dev' -q -v Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + mkdir -p /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + getent passwd ceph Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + chown -h ceph:ceph /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: ++ readlink -f /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: ++ readlink -f '' Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + ceph-disk prepare --cluster ceph --cluster-uuid eb2bb192-b1c9-11e6-9205-525400330666 /srv/data Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: + udevadm settle Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-prepare-/srv/data]/returns: executed successfully Notice: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[fcontext_/srv/data]/returns: executed successfully Notice: /Stage[main]/Tripleo::Firewall/Exec[nonpersistent_v4_rules_cleanup]/returns: executed successfully Notice: /Stage[main]/Tripleo::Firewall/Exec[reload_iptables]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Tripleo::Firewall/Exec[nonpersistent_v6_rules_cleanup]/returns: executed successfully Notice: /Stage[main]/Tripleo::Firewall/Exec[reload_ip6tables]: Triggered 'refresh' from 1 events Notice: Applied catalog in 306.83 seconds deploy_stderr: | exception: connect failed Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/var/lib/heat-config/heat-config-puppet/944cf9fa-229d-4b1f-8d75-49ea31a4de30.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:indeprecation') Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/var/lib/heat-config/heat-config-puppet/944cf9fa-229d-4b1f-8d75-49ea31a4de30.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in deprecation') Warning: ModuleLoader: module 'ceph' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules (file & line not available) Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/944cf9fa-229d-4b1f-8d75-49ea31a4de30.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:indeprecation') Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules (file & line not available) Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/944cf9fa-229d-4b1f-8d75-49ea31a4de30.pp", 34]: (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/944cf9fa-229d-4b1f-8d75-49ea31a4de30.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:indeprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/944cf9fa-229d-4b1f-8d75-49ea31a4de30.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/944cf9fa-229d-4b1f-8d75-49ea31a4de30.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:indeprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/944cf9fa-229d-4b1f-8d75-49ea31a4de30.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README. at ["/var/lib/heat-config/heat-config-puppet/944cf9fa-229d-4b1f-8d75-49ea31a4de30.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:indeprecation') Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules (file & line not available) Warning: Scope(Class[Neutron]): neutron::rabbit_host, neutron::rabbit_hosts, neutron::rabbit_password, neutron::rabbit_port, neutron::rabbit_user and neutron::rabbit_virtual_host are deprecated. Please use neutron::default_transport_url instead. Warning: Scope(Class[Ceilometer]): ceilometer::rabbit_host, ceilometer::rabbit_hosts, ceilometer::rabbit_password, ceilometer::rabbit_port, ceilometer::rabbit_userid and ceilometer::rabbit_virtual_host are deprecated. Please use ceilometer::default_transport_url instead. Warning: This method is deprecated, please use match expressions with Stdlib::Compat::String instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at : (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in deprecation') Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Bool instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at : (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:indeprecation') Warning: Scope(Oslo::Messaging::Rabbit[neutron_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Warning: Scope(Oslo::Messaging::Rabbit[ceilometer_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Error: Command exceeded timeout Error: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-activate-/srv/data]/returns: change from notrun to 0 failed: Command exceeded timeout

So the problem is most likely that puppet is trying to run ceph-disk activate on the OSD and hanging. Two ways to diagnose what's going wrong:

  1. Check the ceph logs, which I think are in /var/logs/ceph/.
  2. Check journalctl for ceph-disk activate error messages.
  3. Run ceph-disk activate /srv/data on the node as the root user and see what happens.
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.