[RHOSP14] SkyDive integration has failed while pulling skydive-analyzer & skydive-agent
Environment
- Red Hat OpenStack Platform release 14.0.0 RC (Rocky)
- SkyDive 14.0
Issue
-
The default Skydive integration steps in RHOSP14 get failed with below ansible error while ansible trying to pull the skydive image with latest non-uniform latest tag.
Pulling repository 192.168.24.1:8787/rhosp14/openstack-skydive-agent\", \"stdout_lines\": [\"Trying to pull repository 192.168.24.1:8787/rhosp14/openstack-skydive-agent ... \", \"Pulling repository 192.168.24.1:8787/rhosp14/openstack-skydive-agent\"]}", "", "PLAY RECAP *********************************************************************", "compute-0 : ok=18 changed=10 unreachable=0 failed=1 ", "compute-1 : ok=18 changed=10 unreachable=0 failed=1 ", "controller-0 : ok=18 changed=10 unreachable=0 failed=1 ", "controller-1 : ok=18 changed=10 unreachable=0 failed=1 ", "controller-2 : ok=18 changed=10 unreachable=0 failed=1 ", "localhost : ok=1 changed=0 unreachable=0 failed=0 ", "", "Thursday 21 February 2019 16:02:42 -0500 (0:00:00.632) 0:00:58.435 ***** ", "=============================================================================== "]} NO MORE HOSTS LEFT ************************************************************* PLAY RECAP ********************************************************************* compute-0 : ok=165 changed=70 unreachable=0 failed=0 compute-1 : ok=166 changed=70 unreachable=0 failed=0 controller-0 : ok=209 changed=92 unreachable=0 failed=0 controller-1 : ok=203 changed=91 unreachable=0 failed=0 controller-2 : ok=203 changed=91 unreachable=0 failed=0 undercloud : ok=27 changed=17 unreachable=0 failed=1
Disclaimer
- In RedHat OpenStack14, Skydive is included with this release as a Technology Preview. For more information on the support scope for features marked as technology previews.
SkyDive High-Level Architecture
-
Refer offical docuemntation, to get more details on Skydive usecase & architecture in Redhat OpenStack14.
Resolution
Workaround steps to fix the Skydive integration issue
-
Step 1: Using
skopeo copy
modify Skydive non-uniform image tag to uniform image tag.Example:
$ skopeo copy --dest-tls-verify=false --src-tls-verify=false docker://192.168.24.1:8787/rhosp14/openstack-skydive-analyzer:14.0-46 docker://192.168.24.1:8787/rhosp14/openstack-skydive-analyzer:14.0 Getting image source signatures Skipping fetch of repeat blob sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82 Skipping fetch of repeat blob sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8 Skipping fetch of repeat blob sha256:ac004ffada07835518e8a3bb147adf7bfc9f66beb6466538a0b44faf49ee152f Skipping fetch of repeat blob sha256:406ac8e6c2630f56682d5d3acd480553440810eef7ab446072f53d932f29e62a Skipping fetch of repeat blob sha256:6fd299b58f2e1c8fe863a1f33b4f2bbabce6877a33bcf27fdc9e3e59986eeaf2 Copying config sha256:7fa5bddbf06f827c49b1564126809dbb477296dcdd843a5da52c03e2cbda3bd6 0 B / 3.94 KB [------------------------------------------------------------] 0s Writing manifest to image destination Storing signatures
$ skopeo copy --dest-tls-verify=false --src-tls-verify=false docker://192.168.24.1:8787/rhosp14/openstack-skydive-agent:14.0-48 docker://192.168.24.1:8787/rhosp14/openstack-skydive-agent:14.0 Getting image source signatures Skipping fetch of repeat blob sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82 Skipping fetch of repeat blob sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8 Skipping fetch of repeat blob sha256:ac004ffada07835518e8a3bb147adf7bfc9f66beb6466538a0b44faf49ee152f Skipping fetch of repeat blob sha256:406ac8e6c2630f56682d5d3acd480553440810eef7ab446072f53d932f29e62a Skipping fetch of repeat blob sha256:2d20b82152a9926bbf6578ab483bb877d9c7b06842caacf3e830f9ddc71f375f Copying config sha256:ff6fc1b63da709f9b373ff3dc5c19ad674ee7547201ad3c3cdbf5c5400ff25b0 0 B / 4.01 KB [------------------------------------------------------------] 0s Writing manifest to image destination Storing signatures
-
Step 2: Review the modified image tag.
Example:
$ skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhosp14/openstack-skydive-analyzer:14.0 { "Name": "192.168.24.1:8787/rhosp14/openstack-skydive-analyzer", "Digest": "sha256:f5fd35df2f302e9103e4e5e50165065eb11d83647dd3c8c4a2d0b07e52a50cb9", "RepoTags": [ "14.0-46", "14.0" ], "Created": "2019-01-16T17:36:53.504128Z", "DockerVersion": "1.13.1", "Labels": { "architecture": "x86_64", "authoritative-source-url": "registry.access.redhat.com", "batch": "20190116.1", "build-date": "2019-01-16T17:35:35.311086", "com.redhat.build-host": "cpt-0001.osbs.prod.upshift.rdu2.redhat.com", "com.redhat.component": "openstack-skydive-analyzer-container", "description": "Red Hat OpenStack Platform 14.0 skydive-analyzer", "distribution-scope": "public", "io.k8s.description": "Red Hat OpenStack Platform 14.0 skydive-analyzer", "io.k8s.display-name": "Red Hat OpenStack Platform 14.0 skydive-analyzer", "io.openshift.tags": "rhosp osp openstack osp-14.0", "name": "rhosp14/openstack-skydive-analyzer", "release": "46", "summary": "Red Hat OpenStack Platform 14.0 skydive-analyzer", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp14/openstack-skydive-analyzer/images/14.0-46", "vcs-ref": "0aa9572f23d4eb372e0ceb8f9a3d899e29fad5c8", "vcs-type": "git", "vendor": "Red Hat, Inc.", "version": "14.0" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82", "sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8", "sha256:ac004ffada07835518e8a3bb147adf7bfc9f66beb6466538a0b44faf49ee152f", "sha256:406ac8e6c2630f56682d5d3acd480553440810eef7ab446072f53d932f29e62a", "sha256:6fd299b58f2e1c8fe863a1f33b4f2bbabce6877a33bcf27fdc9e3e59986eeaf2" ] }
(undercloud) [stack@undercloud-0 ~]$ skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhosp14/openstack-skydive-agent:14.0 { "Name": "192.168.24.1:8787/rhosp14/openstack-skydive-agent", "Digest": "sha256:6c4d3ec7bae6ca358664ced64b1aab1d30a9b87107a571e60fa4d60873c00065", "RepoTags": [ "14.0-48", "14.0" ], "Created": "2019-01-16T17:34:16.475746Z", "DockerVersion": "1.13.1", "Labels": { "architecture": "x86_64", "authoritative-source-url": "registry.access.redhat.com", "batch": "20190116.1", "build-date": "2019-01-16T17:31:36.559087", "com.redhat.build-host": "cpt-0005.osbs.prod.upshift.rdu2.redhat.com", "com.redhat.component": "openstack-skydive-agent-container", "description": "Red Hat OpenStack Platform 14.0 skydive-agent", "distribution-scope": "public", "io.k8s.description": "Red Hat OpenStack Platform 14.0 skydive-agent", "io.k8s.display-name": "Red Hat OpenStack Platform 14.0 skydive-agent", "io.openshift.tags": "rhosp osp openstack osp-14.0", "name": "rhosp14/openstack-skydive-agent", "release": "48", "summary": "Red Hat OpenStack Platform 14.0 skydive-agent", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp14/openstack-skydive-agent/images/14.0-48", "vcs-ref": "939331b940a647450555db8adef41597d9e7c045", "vcs-type": "git", "vendor": "Red Hat, Inc.", "version": "14.0" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82", "sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8", "sha256:ac004ffada07835518e8a3bb147adf7bfc9f66beb6466538a0b44faf49ee152f", "sha256:406ac8e6c2630f56682d5d3acd480553440810eef7ab446072f53d932f29e62a", "sha256:2d20b82152a9926bbf6578ab483bb877d9c7b06842caacf3e830f9ddc71f375f" ] }
-
Step 3: In tripleo environment tmplate, include uniform image tag in Skydive global vairiables.
Example:
parameter_defaults: SkydiveVars: globals: skydive_docker_image_tag: 14.0
-
Step 4: Ensure the overcloud deployment script have included below skydive-environment.yaml service template.
-e /usr/share/openstack-tripleo-heat-templates/environments/services/skydive-environment.yaml
-
Step 5: After successfully overcloud deployment, ensure the controller and compute node run skydive container service.
From Controller:
# docker ps | grep -i skydive 3e45f21f3a56 192.168.24.1:8787/rhosp14/openstack-skydive-analyzer:14.0 "/usr/bin/skydive ..." About an hour ago Up About an hour 0.0.0.0:8082->8082/tcp, 0.0.0.0:12379-12380->12379-12380/tcp, 0.0.0.0:8082->8082/udp skydive-analyzer-controller-0 9c4e6faca91d 10.74.167.35:8787/red_hat_ecs-osp14_containers-skydive-agent:14.0 "/usr/bin/skydive ..." About an hour ago Up About an hour skydive-agent-controller-0
From Compute:
aeec76ba2480 192.168.24.1:8787/rhosp14/openstack-skydive-agent:14.0 "/usr/bin/skydive ..." About an hour ago Up About an hour skydive-agent-compute-1
-
Step 6: From overcloud controller nodes, refer template
/etc/skydive/skydive.yml
to get the default credential# egrep "username|password" /etc/skydive/skydive.yml | head -2
-
The GUI portal can access via
8082
port.http://<Dashboard-IP>:8082
Root Cause
-
During integration, the skydive image doesn't have a uniform image tag which gets conflicted to pull the images from the registry.
-
Example:
@undercloud-0 ~]$ sudo docker images | grep -i skydive 192.168.24.1:8787/rhosp14/openstack-skydive-analyzer 14.0-46 e79783d298bf 5 weeks ago 588 MB 192.168.24.1:8787/rhosp14/openstack-skydive-agent 14.0-48 b5c4d6b712df 5 weeks ago 642 MB
Diagnostic Steps
-
Use
skopeo inspect
commands to verify the current image tag of Skydive analyzer & agent images.Example:
(undercloud) [stack@undercloud-0 ~]$ skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhosp14/openstack-skydive-analyzer:14.0-46 { "Name": "192.168.24.1:8787/rhosp14/openstack-skydive-analyzer", "Digest": "sha256:f5fd35df2f302e9103e4e5e50165065eb11d83647dd3c8c4a2d0b07e52a50cb9", "RepoTags": [ "14.0-46" ], "Created": "2019-01-16T17:36:53.504128Z", "DockerVersion": "1.13.1", "Labels": { "architecture": "x86_64", "authoritative-source-url": "registry.access.redhat.com", "batch": "20190116.1", "build-date": "2019-01-16T17:35:35.311086", "com.redhat.build-host": "cpt-0001.osbs.prod.upshift.rdu2.redhat.com", "com.redhat.component": "openstack-skydive-analyzer-container", "description": "Red Hat OpenStack Platform 14.0 skydive-analyzer", "distribution-scope": "public", "io.k8s.description": "Red Hat OpenStack Platform 14.0 skydive-analyzer", "io.k8s.display-name": "Red Hat OpenStack Platform 14.0 skydive-analyzer", "io.openshift.tags": "rhosp osp openstack osp-14.0", "name": "rhosp14/openstack-skydive-analyzer", "release": "46", "summary": "Red Hat OpenStack Platform 14.0 skydive-analyzer", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp14/openstack-skydive-analyzer/images/14.0-46", "vcs-ref": "0aa9572f23d4eb372e0ceb8f9a3d899e29fad5c8", "vcs-type": "git", "vendor": "Red Hat, Inc.", "version": "14.0" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82", "sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8", "sha256:ac004ffada07835518e8a3bb147adf7bfc9f66beb6466538a0b44faf49ee152f", "sha256:406ac8e6c2630f56682d5d3acd480553440810eef7ab446072f53d932f29e62a", "sha256:6fd299b58f2e1c8fe863a1f33b4f2bbabce6877a33bcf27fdc9e3e59986eeaf2" ] }
(undercloud) [stack@undercloud-0 ~]$ skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhosp14/openstack-skydive-agent:14.0-48 { "Name": "192.168.24.1:8787/rhosp14/openstack-skydive-agent", "Digest": "sha256:6c4d3ec7bae6ca358664ced64b1aab1d30a9b87107a571e60fa4d60873c00065", "RepoTags": [ "14.0-48" ], "Created": "2019-01-16T17:34:16.475746Z", "DockerVersion": "1.13.1", "Labels": { "architecture": "x86_64", "authoritative-source-url": "registry.access.redhat.com", "batch": "20190116.1", "build-date": "2019-01-16T17:31:36.559087", "com.redhat.build-host": "cpt-0005.osbs.prod.upshift.rdu2.redhat.com", "com.redhat.component": "openstack-skydive-agent-container", "description": "Red Hat OpenStack Platform 14.0 skydive-agent", "distribution-scope": "public", "io.k8s.description": "Red Hat OpenStack Platform 14.0 skydive-agent", "io.k8s.display-name": "Red Hat OpenStack Platform 14.0 skydive-agent", "io.openshift.tags": "rhosp osp openstack osp-14.0", "name": "rhosp14/openstack-skydive-agent", "release": "48", "summary": "Red Hat OpenStack Platform 14.0 skydive-agent", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp14/openstack-skydive-agent/images/14.0-48", "vcs-ref": "939331b940a647450555db8adef41597d9e7c045", "vcs-type": "git", "vendor": "Red Hat, Inc.", "version": "14.0" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82", "sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8", "sha256:ac004ffada07835518e8a3bb147adf7bfc9f66beb6466538a0b44faf49ee152f", "sha256:406ac8e6c2630f56682d5d3acd480553440810eef7ab446072f53d932f29e62a", "sha256:2d20b82152a9926bbf6578ab483bb877d9c7b06842caacf3e830f9ddc71f375f" ] }
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments