[RHOSP14] SkyDive integration has failed while pulling skydive-analyzer & skydive-agent

Solution In Progress - Updated -

Environment

  • Red Hat OpenStack Platform release 14.0.0 RC (Rocky)
  • SkyDive 14.0

Issue

  • The default Skydive integration steps in RHOSP14 get failed with below ansible error while ansible trying to pull the skydive image with latest non-uniform latest tag.

    Pulling repository 192.168.24.1:8787/rhosp14/openstack-skydive-agent\", \"stdout_lines\": [\"Trying to pull repository 192.168.24.1:8787/rhosp14/openstack-skydive-agent ... \", \"Pulling repository 192.168.24.1:8787/rhosp14/openstack-skydive-agent\"]}", "", "PLAY RECAP *********************************************************************", "compute-0                  : ok=18   changed=10   unreachable=0    failed=1   ", "compute-1                  : ok=18   changed=10   unreachable=0    failed=1   ", "controller-0               : ok=18   changed=10   unreachable=0    failed=1   ", "controller-1               : ok=18   changed=10   unreachable=0    failed=1   ", "controller-2               : ok=18   changed=10   unreachable=0    failed=1   ", "localhost                  : ok=1    changed=0    unreachable=0    failed=0   ", "", "Thursday 21 February 2019  16:02:42 -0500 (0:00:00.632)       0:00:58.435 ***** ", "=============================================================================== "]}


NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
compute-0                  : ok=165  changed=70   unreachable=0    failed=0   
compute-1                  : ok=166  changed=70   unreachable=0    failed=0   
controller-0               : ok=209  changed=92   unreachable=0    failed=0   
controller-1               : ok=203  changed=91   unreachable=0    failed=0   
controller-2               : ok=203  changed=91   unreachable=0    failed=0   
undercloud                 : ok=27   changed=17   unreachable=0    failed=1

Disclaimer

  • In RedHat OpenStack14, Skydive is included with this release as a Technology Preview. For more information on the support scope for features marked as technology previews.

SkyDive High-Level Architecture

  • Refer offical docuemntation, to get more details on Skydive usecase & architecture in Redhat OpenStack14.

    alt text

Resolution

Workaround steps to fix the Skydive integration issue

  • Step 1: Using skopeo copy modify Skydive non-uniform image tag to uniform image tag.

    Example:

    $ skopeo copy --dest-tls-verify=false --src-tls-verify=false docker://192.168.24.1:8787/rhosp14/openstack-skydive-analyzer:14.0-46 docker://192.168.24.1:8787/rhosp14/openstack-skydive-analyzer:14.0
Getting image source signatures
Skipping fetch of repeat blob sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82
Skipping fetch of repeat blob sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8
Skipping fetch of repeat blob sha256:ac004ffada07835518e8a3bb147adf7bfc9f66beb6466538a0b44faf49ee152f
Skipping fetch of repeat blob sha256:406ac8e6c2630f56682d5d3acd480553440810eef7ab446072f53d932f29e62a
Skipping fetch of repeat blob sha256:6fd299b58f2e1c8fe863a1f33b4f2bbabce6877a33bcf27fdc9e3e59986eeaf2
Copying config sha256:7fa5bddbf06f827c49b1564126809dbb477296dcdd843a5da52c03e2cbda3bd6
 0 B / 3.94 KB [------------------------------------------------------------] 0s
Writing manifest to image destination
Storing signatures

    $ skopeo copy --dest-tls-verify=false --src-tls-verify=false docker://192.168.24.1:8787/rhosp14/openstack-skydive-agent:14.0-48 docker://192.168.24.1:8787/rhosp14/openstack-skydive-agent:14.0
Getting image source signatures
Skipping fetch of repeat blob sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82
Skipping fetch of repeat blob sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8
Skipping fetch of repeat blob sha256:ac004ffada07835518e8a3bb147adf7bfc9f66beb6466538a0b44faf49ee152f
Skipping fetch of repeat blob sha256:406ac8e6c2630f56682d5d3acd480553440810eef7ab446072f53d932f29e62a
Skipping fetch of repeat blob sha256:2d20b82152a9926bbf6578ab483bb877d9c7b06842caacf3e830f9ddc71f375f
Copying config sha256:ff6fc1b63da709f9b373ff3dc5c19ad674ee7547201ad3c3cdbf5c5400ff25b0
 0 B / 4.01 KB [------------------------------------------------------------] 0s
Writing manifest to image destination
Storing signatures

  • Step 2: Review the modified image tag.

    Example:

    $ skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhosp14/openstack-skydive-analyzer:14.0
{
    "Name": "192.168.24.1:8787/rhosp14/openstack-skydive-analyzer",
    "Digest": "sha256:f5fd35df2f302e9103e4e5e50165065eb11d83647dd3c8c4a2d0b07e52a50cb9",
    "RepoTags": [
        "14.0-46",
        "14.0"
    ],
    "Created": "2019-01-16T17:36:53.504128Z",
    "DockerVersion": "1.13.1",
    "Labels": {
        "architecture": "x86_64",
        "authoritative-source-url": "registry.access.redhat.com",
        "batch": "20190116.1",
        "build-date": "2019-01-16T17:35:35.311086",
        "com.redhat.build-host": "cpt-0001.osbs.prod.upshift.rdu2.redhat.com",
        "com.redhat.component": "openstack-skydive-analyzer-container",
        "description": "Red Hat OpenStack Platform 14.0 skydive-analyzer",
        "distribution-scope": "public",
        "io.k8s.description": "Red Hat OpenStack Platform 14.0 skydive-analyzer",
        "io.k8s.display-name": "Red Hat OpenStack Platform 14.0 skydive-analyzer",
        "io.openshift.tags": "rhosp osp openstack osp-14.0",
        "name": "rhosp14/openstack-skydive-analyzer",
        "release": "46",
        "summary": "Red Hat OpenStack Platform 14.0 skydive-analyzer",
        "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp14/openstack-skydive-analyzer/images/14.0-46",
        "vcs-ref": "0aa9572f23d4eb372e0ceb8f9a3d899e29fad5c8",
        "vcs-type": "git",
        "vendor": "Red Hat, Inc.",
        "version": "14.0"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82",
        "sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8",
        "sha256:ac004ffada07835518e8a3bb147adf7bfc9f66beb6466538a0b44faf49ee152f",
        "sha256:406ac8e6c2630f56682d5d3acd480553440810eef7ab446072f53d932f29e62a",
        "sha256:6fd299b58f2e1c8fe863a1f33b4f2bbabce6877a33bcf27fdc9e3e59986eeaf2"
    ]
}

    (undercloud) [stack@undercloud-0 ~]$ skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhosp14/openstack-skydive-agent:14.0
{
    "Name": "192.168.24.1:8787/rhosp14/openstack-skydive-agent",
    "Digest": "sha256:6c4d3ec7bae6ca358664ced64b1aab1d30a9b87107a571e60fa4d60873c00065",
    "RepoTags": [
        "14.0-48",
        "14.0"
    ],
    "Created": "2019-01-16T17:34:16.475746Z",
    "DockerVersion": "1.13.1",
    "Labels": {
        "architecture": "x86_64",
        "authoritative-source-url": "registry.access.redhat.com",
        "batch": "20190116.1",
        "build-date": "2019-01-16T17:31:36.559087",
        "com.redhat.build-host": "cpt-0005.osbs.prod.upshift.rdu2.redhat.com",
        "com.redhat.component": "openstack-skydive-agent-container",
        "description": "Red Hat OpenStack Platform 14.0 skydive-agent",
        "distribution-scope": "public",
        "io.k8s.description": "Red Hat OpenStack Platform 14.0 skydive-agent",
        "io.k8s.display-name": "Red Hat OpenStack Platform 14.0 skydive-agent",
        "io.openshift.tags": "rhosp osp openstack osp-14.0",
        "name": "rhosp14/openstack-skydive-agent",
        "release": "48",
        "summary": "Red Hat OpenStack Platform 14.0 skydive-agent",
        "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp14/openstack-skydive-agent/images/14.0-48",
        "vcs-ref": "939331b940a647450555db8adef41597d9e7c045",
        "vcs-type": "git",
        "vendor": "Red Hat, Inc.",
        "version": "14.0"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82",
        "sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8",
        "sha256:ac004ffada07835518e8a3bb147adf7bfc9f66beb6466538a0b44faf49ee152f",
        "sha256:406ac8e6c2630f56682d5d3acd480553440810eef7ab446072f53d932f29e62a",
        "sha256:2d20b82152a9926bbf6578ab483bb877d9c7b06842caacf3e830f9ddc71f375f"
    ]
}

  • Step 3: In tripleo environment tmplate, include uniform image tag in Skydive global vairiables.

    Example:

    parameter_defaults:
  SkydiveVars:
    globals:
      skydive_docker_image_tag: 14.0

  • Step 4: Ensure the overcloud deployment script have included below skydive-environment.yaml service template.

    -e /usr/share/openstack-tripleo-heat-templates/environments/services/skydive-environment.yaml

  • Step 5: After successfully overcloud deployment, ensure the controller and compute node run skydive container service.

    From Controller:

    # docker ps | grep -i skydive
3e45f21f3a56        192.168.24.1:8787/rhosp14/openstack-skydive-analyzer:14.0                   "/usr/bin/skydive ..."   About an hour ago   Up About an hour             0.0.0.0:8082->8082/tcp, 0.0.0.0:12379-12380->12379-12380/tcp, 0.0.0.0:8082->8082/udp   skydive-analyzer-controller-0
9c4e6faca91d        10.74.167.35:8787/red_hat_ecs-osp14_containers-skydive-agent:14.0                      "/usr/bin/skydive ..."   About an hour ago   Up About an hour                                                                                                    skydive-agent-controller-0

    From Compute:

    aeec76ba2480        192.168.24.1:8787/rhosp14/openstack-skydive-agent:14.0                  "/usr/bin/skydive ..."   About an hour ago   Up About an hour                          skydive-agent-compute-1

  • Step 6: From overcloud controller nodes, refer template /etc/skydive/skydive.yml to get the default credential

    # egrep "username|password" /etc/skydive/skydive.yml | head -2

  • The GUI portal can access via 8082 port.

    http://<Dashboard-IP>:8082

Root Cause

  • During integration, the skydive image doesn't have a uniform image tag which gets conflicted to pull the images from the registry.

  • Example:

    @undercloud-0 ~]$ sudo docker images | grep -i skydive
192.168.24.1:8787/rhosp14/openstack-skydive-analyzer             14.0-46             e79783d298bf        5 weeks ago         588 MB
192.168.24.1:8787/rhosp14/openstack-skydive-agent                14.0-48             b5c4d6b712df        5 weeks ago         642 MB

Diagnostic Steps

  • Use skopeo inspect commands to verify the current image tag of Skydive analyzer & agent images.

    Example:

    (undercloud) [stack@undercloud-0 ~]$ skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhosp14/openstack-skydive-analyzer:14.0-46
{
    "Name": "192.168.24.1:8787/rhosp14/openstack-skydive-analyzer",
    "Digest": "sha256:f5fd35df2f302e9103e4e5e50165065eb11d83647dd3c8c4a2d0b07e52a50cb9",
    "RepoTags": [
        "14.0-46"
    ],
    "Created": "2019-01-16T17:36:53.504128Z",
    "DockerVersion": "1.13.1",
    "Labels": {
        "architecture": "x86_64",
        "authoritative-source-url": "registry.access.redhat.com",
        "batch": "20190116.1",
        "build-date": "2019-01-16T17:35:35.311086",
        "com.redhat.build-host": "cpt-0001.osbs.prod.upshift.rdu2.redhat.com",
        "com.redhat.component": "openstack-skydive-analyzer-container",
        "description": "Red Hat OpenStack Platform 14.0 skydive-analyzer",
        "distribution-scope": "public",
        "io.k8s.description": "Red Hat OpenStack Platform 14.0 skydive-analyzer",
        "io.k8s.display-name": "Red Hat OpenStack Platform 14.0 skydive-analyzer",
        "io.openshift.tags": "rhosp osp openstack osp-14.0",
        "name": "rhosp14/openstack-skydive-analyzer",
        "release": "46",
        "summary": "Red Hat OpenStack Platform 14.0 skydive-analyzer",
        "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp14/openstack-skydive-analyzer/images/14.0-46",
        "vcs-ref": "0aa9572f23d4eb372e0ceb8f9a3d899e29fad5c8",
        "vcs-type": "git",
        "vendor": "Red Hat, Inc.",
        "version": "14.0"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82",
        "sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8",
        "sha256:ac004ffada07835518e8a3bb147adf7bfc9f66beb6466538a0b44faf49ee152f",
        "sha256:406ac8e6c2630f56682d5d3acd480553440810eef7ab446072f53d932f29e62a",
        "sha256:6fd299b58f2e1c8fe863a1f33b4f2bbabce6877a33bcf27fdc9e3e59986eeaf2"
    ]
}

    (undercloud) [stack@undercloud-0 ~]$ skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhosp14/openstack-skydive-agent:14.0-48
{
    "Name": "192.168.24.1:8787/rhosp14/openstack-skydive-agent",
    "Digest": "sha256:6c4d3ec7bae6ca358664ced64b1aab1d30a9b87107a571e60fa4d60873c00065",
    "RepoTags": [
        "14.0-48"
    ],
    "Created": "2019-01-16T17:34:16.475746Z",
    "DockerVersion": "1.13.1",
    "Labels": {
        "architecture": "x86_64",
        "authoritative-source-url": "registry.access.redhat.com",
        "batch": "20190116.1",
        "build-date": "2019-01-16T17:31:36.559087",
        "com.redhat.build-host": "cpt-0005.osbs.prod.upshift.rdu2.redhat.com",
        "com.redhat.component": "openstack-skydive-agent-container",
        "description": "Red Hat OpenStack Platform 14.0 skydive-agent",
        "distribution-scope": "public",
        "io.k8s.description": "Red Hat OpenStack Platform 14.0 skydive-agent",
        "io.k8s.display-name": "Red Hat OpenStack Platform 14.0 skydive-agent",
        "io.openshift.tags": "rhosp osp openstack osp-14.0",
        "name": "rhosp14/openstack-skydive-agent",
        "release": "48",
        "summary": "Red Hat OpenStack Platform 14.0 skydive-agent",
        "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp14/openstack-skydive-agent/images/14.0-48",
        "vcs-ref": "939331b940a647450555db8adef41597d9e7c045",
        "vcs-type": "git",
        "vendor": "Red Hat, Inc.",
        "version": "14.0"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82",
        "sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8",
        "sha256:ac004ffada07835518e8a3bb147adf7bfc9f66beb6466538a0b44faf49ee152f",
        "sha256:406ac8e6c2630f56682d5d3acd480553440810eef7ab446072f53d932f29e62a",
        "sha256:2d20b82152a9926bbf6578ab483bb877d9c7b06842caacf3e830f9ddc71f375f"
    ]
}

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments