Steps related to the changes of key signing and distribution of Red Hat Ceph Storage for Ubuntu and CentOS

Resolution

Step 1: Validate your existing ISO or Tarball checksum

On the Calamari admin node, download and install the Red Hat Release Key 2:

curl https://www.redhat.com/security/fd431d51.txt | gpg --import

Download the file called inktank-sha256sums.txt that is attached to this page containing the known-good checksums. On the command line it can be fetched it with:

curl -OL https://access.redhat.com/sites/default/files/attachments/inktank-sha256sums.txt

Validate the inktank-sha256sums.txt file you downloaded:

gpg --verify inktank-sha256sums.txt

You should see the text:

“Good signature from "Red Hat, Inc. (release key 2) security@redhat.com"

If you also see text with the following warning, you can safely ignore it:

"WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner"

Verify your existing ISO image (or tarball), using the downloaded inktank-sha256sums.txt file to check the checksum is valid. In the directory that contains both your existing ISO image (or tarball) and the downloaded inktank-sha256sums.txt file:

• Run the following command to check the checksum is valid:
  • sha256sum -c inktank-sha256sums.txt
  • You should see the text “OK” next to your ISO or tarball's filename.

Step 2: See if you have the Inktank key installed

All customers should see if the Inktank key is present on your systems:

• On Ubuntu, “sudo apt-key list” will list all your keys. Look for the Inktank key, like so:

$ sudo apt-key list
...
pub   4096R/9DCEEEAD 2013-10-30
uid                  Inktank <release@inktank.com>
sub   4096R/3C574FB5 2013-10-30

• On CentOS, “rpm -qa gpg-pubkey” will list all your keys. Look for the Inktank key entry, “gpg-pubkey-9dceeead-52719d72”, like so:

$ rpm -q gpg-pubkey-9dceeead-52719d72
gpg-pubkey-9dceeead-52719d72

Step 3: Remove the Inktank Key

All customers are advised to remove the old Inktank key on each node:

• On CentOS:

  • sudo rpm -e --allmatches gpg-pubkey-9dceeead-52719d72

• On Ubuntu:

  • sudo apt-key del 9DCEEEAD

Step 4: Update your package repositories

Customers are advised to download the new software builds signed by the Red Hat release key to ensure they meet any key integrity checks.

The URL to download this software signed with the Red Hat release key is listed in an email sent to customers who are registered as being an Ubuntu or CentOS customer. If you have not received this email or don’t have the download URL, please contact your account manager or support.

Customers running v1.2.2 or earlier should download the latest version on the Red Hat download site (v1.2.3 or higher) and follow the upgrade steps to upgrade to v1.2.3:

• For Ubuntu
• For CentOS

If upgrading, all customers (including those using CentOS) can skip Step 5 below.

If already running v1.2.3, to update the packages your cluster uses to the new ones signed by Red Hat, run the following commands after you have downloaded the ISO:

• Unmount any current image (if any is mounted):
  • sudo umount /mnt
• Mount the new ISO image:
  • sudo mount -r -o loop <ISO> /mnt
• Install updated ice-setup:
  • Ubuntu: sudo dpkg -i /mnt/ice-setup*
  • Centos: sudo yum -y remove ice_setup && sudo yum -y localinstall /mnt/ice_setup*
• Rerun ice_setup (perform this command in the directory where you originally ran ice_setup; the example path in the installation instructions was ~/ceph-config):
  • sudo ice_setup -d /mnt
  • This will run an interactive ice_setup session, which will perform the following steps, with verbose logging:
  • Step 1: Replace and refresh calamari-server and ceph-deploy repos
  • Step 2: Attempt to reinstall calamari (which will not install anything, as the new packages are not changed binaries)
  • Step 3: Attempt to install ceph-deploy (same no-op as above). ice-setup will prompt for a FQDN for the Calamari host, and a URL scheme (http or https); select those as appropriate for your installation
  • Step 4: Replace/refresh the ceph repository for cluster hosts
  • Step 5: Replace/refresh the calamari-minions repository for cluster hosts
• If release.asc exists in your current working directory, delete it.
  • rm release.asc
• On Ubuntu only: Re-run ceph-deploy on each cluster node:
  • ceph-deploy install <node1> <node2>

Step 5 Reinstall packages (CentOS only)

As all the individual RPM packages have been signed by a new key, customers on CentOS are advised to do a re-installation of the new packages over their currently installed packages on each Ceph node. We recommend using the same version of software you are currently running to mitigate any potential issues when restarting software on each node. Restarting the software is not a necessary step and can be performed when is convenient for the customer.

NOTE: If you upgraded to a later version of the software (as noted in step 4) you do not need to perform this step.

To re-install the software:

1. Run the following commands on the Calamari admin node (ignoring any warnings about packages that are not already installed):
   • sudo yum clean metadata
   • sudo yum reinstall -y $(repoquery --disablerepo=* --enablerepo=calamari --queryformat='%{NAME}' list '*')
   • sudo yum reinstall -y $(repoquery --disablerepo=* --enablerepo=ceph_deploy --queryformat='%{NAME}' list '*')
2. On any Ceph Monitor, OSD, or RGW nodes:
   • sudo yum clean metadata
   • sudo yum reinstall -y $(repoquery --disablerepo=* --enablerepo=ceph --queryformat='%{NAME}' list '*')
3. On any Ceph Monitor or OSD nodes, also run:
   • sudo yum reinstall -y $(repoquery --disablerepo=* --enablerepo=calamari-minion --queryformat='%{NAME}' list '*')