Chapter 6. Managing RHEL for Edge images

To manage the RHEL for Edge images, you can perform any of the following administrative tasks:

  • Edit the RHEL for Edge image blueprint using Image Builder in RHEL web console
  • Edit the RHEL for Edge image blueprint using Image Builder command-line
  • Update the RHEL for Edge images
  • Configure rpm-ostree remotes on nodes/Update node policy
  • Restore RHEL for Edge images manually or automatically using a Greenboot

6.1. Editing a RHEL for Edge image blueprint using Image Builder in RHEL web console

You can edit the RHEL for Edge image blueprint to:

  • add additional components that you may require
  • modify the version of any existing component
  • remove any existing component

6.1.1. Adding a component to RHEL for Edge image blueprint using Image Builder in RHEL web console

To add a component to a RHEL for Edge image blueprint, ensure that you have met the following prerequisites and then follow the procedure to edit the corresponding blueprint.

Prerequisites

  • On a RHEL system, you have accessed the Image Builder dashboard.
  • You have created a blueprint for RHEL for Edge image.

Procedure

  1. On the Image Builder dashboard, click the RHEL for Edge image blueprint that you want to edit.

    To search for a specific blueprint, enter the blueprint name in the Filter By Name text box, and then press Enter.

  2. On the upper right side of the blueprint, click Edit Packages.

    The view changes to the Edit Packages mode.

  3. Enter the component name that you want to add in the Filter By Name text box, and then press Enter.

    A list with the component name appears.

  4. Click the btn[+] sign adjacent to the component.

    The component is added to the Blueprint.

  5. Click Commit.

    The blueprint updates are saved, and a message with the pending commit appears.

  6. On the summary dialogue box, review the changes and then click Commit.

    A message confirming the successful commit appears.

    As a result, a new version of the blueprint is created and the right pane lists the latest components.

6.1.2. Changing the version of an existing component in a RHEL for Edge image blueprint using the RHEL web console

You had selected a default (latest) version or had chosen a version for the components that you included in the blueprint. If required, you can now change the version for any component that you might want to.

To do so, ensure that you have met the following prerequisites and then follow the procedure to change the version of the component in the corresponding blueprint.

Prerequisites

  • On a RHEL system, you have accessed the Image Builder dashboard.
  • You have created a blueprint for RHEL for Edge image.
  • You have added at least one component to the RHEL for Edge blueprint.

Procedure

  1. On the Image Builder dashboard, click the blueprint that you want to edit.

    To search for a specific blueprint, enter the blueprint name in the Filter By Name text box, and then press Enter.

  2. On the upper right side of the blueprint, click Edit Packages.

    The view changes to the Edit Packages mode, and the right panel lists the component names that are currently committed to the blueprint.

  3. Click the component name.
  4. Select the desired version from the Component Options Version dropdown list.
  5. Click Apply Changes.

    The change is saved and the right pane lists the latest changes.

  6. Click Commit.

    The new version is saved in the blueprint. A message with pending commits appears.

  7. On the summary dialogue box, review the changes and then click Commit.

    A message confirming the successful commit appears.

    As a result a new version of the blueprint is created and the right pane lists the latest components.

6.1.3. Removing a component from RHEL for Edge image blueprint using Image Builder in RHEL web console

To remove one or more unwanted components from a RHEL for Edge image blueprint that you created, ensure that you have met the following prerequisites and then follow the procedure.

Prerequisites

  • On a RHEL system, you have accessed the Image Builder dashboard.
  • You have created a blueprint for RHEL for Edge image.
  • You have added at least one component to the RHEL for Edge blueprint.

Procedure

  1. On the Image Builder dashboard, click the blueprint that you want to edit.

    To search for a specific blueprint, enter the blueprint name in the Filter By Name text box, and then press Enter.

  2. On the upper right side of the blueprint, click Edit Packages.

    The view changes to the Edit Packages mode. The right panel lists the component names that are currently committed to the blueprint.

  3. From the More Options menu, click Remove.

    Optionally, click the component name and then click Remove.

  4. Click Commit.

    A message with pending commits appears.

  5. Review your changes and then click Commit.

    A message confirming the successful commit appears.

    As a result, a new version of the blueprint is created and the right pane lists the latest components.

6.2. Editing a RHEL for Edge image blueprint using command-line interface

You can change the specifications for your RHEL for Edge image blueprint using Image Builder command-line. To do so, ensure that you have met the following prerequisites and then follow the procedure to edit the corresponding blueprint.

Prerequisites

  • You have access to the Image Builder command-line.
  • You have created a RHEL for Edge image blueprint.

Procedure

  1. Save (export) the blueprint to a local text file:

    # composer-cli blueprints save BLUEPRINT-NAME
  2. Edit the BLUEPRINT-NAME.toml file with a text editor of your choice and make your changes.

    Before finishing with the edits, make sure the file is a valid blueprint:

  3. Increase the version number.

    Ensure that you use a Semantic Versioning scheme.

    Note

    if you do not change the version, the patch component of the version is increased automatically.

  4. Check if the contents are valid TOML specifications. See the TOML documentation for more information.

    Note

    TOML documentation is a community product and is not supported by Red Hat. You can report any issues with the tool at https://github.com/toml-lang/toml/issues.

  5. Save the file and close the editor.
  6. Push (import) the blueprint back into Image Builder command-line:

    # composer-cli blueprints push BLUEPRINT-NAME.toml
    Note

    Wwhen pushing the blueprint back into the Image Builder command-line, provide the file name including the .toml extension.

  7. Verify that the contents uploaded to Image Builder match your edits:

    # composer-cli blueprints show BLUEPRINT-NAME
  8. Check whether the components and versions listed in the blueprint and their dependencies are valid:

    # composer-cli blueprints depsolve BLUEPRINT-NAME

6.3. Updating RHEL for Edge images

6.3.1. How are RHEL for Edge image updates deployed

With RHEL for Edge images, you can either deploy the updates manually or can automate the deployment process. The updates are applied in an atomic manner, where the state of each update is known, and the updates are staged and applied only upon reboot. Because no changes are seen until you reboot the device, you can schedule a reboot to ensure the highest possible uptime.

During the image update, because only the updated OS content is transferred over the network, the deployment process is more efficient compared to transferring the entire image. The OS binaries and libraries (/usr) are read-only, and the read-write state is maintained in /var and /etc directories.

With the delta transfer, you can deploy the updates even in case of intermittent and low-bandwidth connections, or to disconnected devices by using local media. Additionally, you can also create static-deltas to further reduce the network usage. The static-deltas pull all individual updates into a single file archive, and significantly reduces the TCP network overhead of transferring the OS updates over a single TCP connection as opposed to transferring each update individually over multiple connections.

The following diagram illustrates the RHEL for Edge image update deployment process:

Image Deployment

6.3.2. Deploying RHEL for Edge image updates manually

After you have edited a RHEL for Edge blueprint, you can update the image commit. Image Builder generates a new commit for the updated RHEL for Edge image. Use this new commit to deploy the image with latest package versions or with additional packages.

To deploy RHEL for Edge images updates, ensure that you meet the prerequisites and then follow the procedure.

Prerequisites

Procedure

  1. On the Image Builder dashboard, for the blueprint that you have edited, click Create Image.
  2. On the Create Image window, perform the following steps:

    1. From the Type dropdown list, select “RHEL for Edge Commit (.tar)”.
    2. In the Parent commit textbox, specify the parent commit ID that was previously generated. See Section 5.1, “Extracting RHEL for Edge image commit”.
    3. In the Ref textbox, you can either specify a name for your commit or leave it empty. By default, the web console specifies the Ref as rhel/8/arch_name/edge.
    4. Click Create. Image Builder creates a RHEL for Edge image for the updated blueprint.

      To view the RHEL for Edge image creation progress, click the blueprint name from the breadcrumbs, and then click the Images tab.

      Note

      The image creation process takes a few minutes to complete.

      The resulting image includes the latest packages that you have added, if any, and have the original commit ID as a parent.

  3. Download the resulting RHEL for Edge image. For more information about downloading a RHEL for Edge image, see Section 3.4, “Downloading a RHEL for Edge image”.
  4. Extract the OSTree commit. For more information about extracting an OSTree commit, see Section 5.1, “Extracting RHEL for Edge image commit”.
  5. Build a docker container, serving the child commit ID this time.

    #  podman build -t <_name-of-server_> --build-arg commit=<uuid>-child_commit.tar .
  6. Run the container.

    #  podman run --rm -p 8000:80 <_name-of-server_>
  7. On the RHEL system provisioned, from the original edge image, verify the current status.

    $ rpm-ostree status

    If there is no new commit ID, run the following command to verify if there is any upgrade available:

    $ rpm-ostree upgrade --check

    The command output provides the current active OSTree commit ID.

  8. Update OSTree to make the new OSTree commit ID available.

    $ rpm-ostree update

    OSTree verifies if there is an update on the repository. If yes, it fetches the update and requests you to reboot your system so that you can activate the deployment of this new commit update.

  9. Check the current status again:

    $ rpm-ostree status

    You can now see that there are 2 commits available:

    • The active parent commit.
    • A new commit that is not active and contains 1 added difference.
  10. To activate the new deployment and to make the new commit active, reboot your system.

    $ systemctl reboot

    The Anaconda Installer reboots into the new deployment. On the login screen, you can see a new deployment available for you to boot.

  11. Use the arrow key on your keyboard to select the GRUB menu entry and click Enter.
  12. Provide your login user account credentials.
  13. Verify the OSTree status:

    $ rpm-ostree status

    The command output provides the active commit ID.

  14. You can verify the latest packages are now available. For example:

    $ package_name

    The update shows that the package you have installed is available and ready for use.

6.3.3. Deploying RHEL for Edge automatic image updates

After you install a RHEL for Edge image on an Edge device, you can check for image updates available, if any, and can auto-apply them.

The rpm-ostreed-automatic.service (systemd service) and rpm-ostreed-automatic.timer (systemd timer) control the frequency of checks and upgrades. The available updates, if any, appear as staged deployments.

Deploying automatic image updates involves the following high-level steps:

  • Update the image update policy
  • Enable automatic download and staging of updates

6.3.4. Updating the RHEL for Edge image update policy

To update the image update policy, use the AutomaticUpdatePolicy and an IdleExitTimeout setting from the rpm-ostreed.conf file at /etc/rpm-ostreed.conf location on an Edge device.

The AutomaticUpdatePolicy setting has the following options:

  • none: Disables automatic updates. By default, the AutomaticUpdatePolicy setting is set to none.
  • check: Downloads enough metadata to display available updates with rpm-ostree status.
  • stage: Downloads and unpacks the updates that are applied on a reboot.

To enable automatic updates, perform the following steps:

Procedure

  1. In the /etc/rpm-ostreed.conf file, update the following:

    • Change the value of AutomaticUpdatePolicy to check.
    • To run the update checks, specify a value in seconds for IdleExitTimeout.
  2. Reload the rpm-ostreed service and enable the systemd timer.

    # systemctl reload rpm-ostreed
    # systemctl enable rpm-ostreed-automatic.timer --now
  3. Verify the rpm-ostree status to ensure the automatic update policy is configured and time is active.

    # rpm-ostree status

    The command output shows the following:

    State: idle; auto updates enabled (check; last run <minutes> ago)

    Additionally, the output also displays information about the available updates.

6.3.5. Enabling RHEL for Edge automatic download and staging of updates

After you update the image update policy to check for image updates, the updates if any are displayed along with the update details. If you decide to apply the updates, enable the policy to automatically download and stage the updates. The available image updates are then downloaded and staged for deployment. The updates are applied and take effect when you reboot the Edge device.

To enable the policy for automatic download and staging of updates, perform the following updates:

Procedure

  1. In the /etc/rpm-ostreed.conf file, update ‘AutomaticUpdatePolicy’ to stage.
  2. Reload the rpm-ostreed service.

    # systemctl reload rpm-ostreed
  3. Verify the rpm-ostree status

    # rpm-ostree status

    The command output shows the following:

    State: idle
    AutomaticUpdates: stage; rpm-ostreed-automatic.timer: last run <time> ago
  4. To initiate the updates, you can either wait for the timer to initiate the updates, or can manually start the service.

    # systemctl start rpm-ostreed-automatic.service

    After the updates are initiated, the rpm-ostree status shows the following:

    # rpm-ostree status
    State: busy; auto updates enabled (stage; running)
    Transaction: upgrade

    When the update is complete, a new deployment is staged in the list of deployments, and the original booted deployment is left untouched. You can decide if you want to boot the system using the new deployment or can wait for the next update.

    To view the list of deployments, run the rpm-ostree status command.

    Following is a sample output.

    # rpm-ostree status
    State: idle; auto updates enabled (stage; last run <time> ago)
    Deployments:

    To view the list of deployments with the updated package details, run the rpm-ostree status -v command.

6.4. Rolling back RHEL for Edge images

You can verify if the updated image is successfully deployed or not. If the deployment is unsuccessful, you can roll back to a previous version (commit). To roll back to a previous functional state, you can either perform the steps manually or can use an automated process.

6.4.1. How are RHEL for Edge images rolled back

With RHEL for Edge images, only transactional updates are applied to the operating system. With the transactional updates, you can easily rollback the unsuccessful updates to the last known good state, preventing system failure during updates.

You can use intelligent rollbacks with Greenboot to eliminate the issues of choosing between application stability and application of security updates.

Greenboot leverages rpm-ostree and runs custom health checks that run on system startup. In the event of an issue, the system rolls back the changes and preserves the last working state.

The following diagram illustrates the RHEL for Edge image roll back process.

Image restoration process

6.4.2. Rolling back RHEL for Edge images manually

If the deployment for RHEL for Edge image update fails or if the update fails to work successfully, then you can manually roll back to a previous deployment version.

To roll back to a previous version, perform the following steps:

Procedure

  1. Run the rollback command:

    $ rpm-ostree rollback

    The command output provides details about the commit ID that is being moved and indicates a completed transaction with the details of the package being removed.

  2. Reboot the system.

    $ systemctl reboot

    The command activates the previous commit with the stable content. The changes are applied and the previous version is restored.

6.4.3. Rolling back RHEL for Edge images using an automated process

Greenboot provides a framework that is integrated into the boot process and can trigger rpm-ostree rollbacks when a health check fails. For the health checks, you can create a custom script that indicates whether a health check is passed or failed. Based on the result, you can decide when a rollback should be triggered.

To create a health check script, perform the following steps:

Procedure

  1. Create a script that returns a standard exit code 0.

    For example, the following script ensures that the configured DNS server is available:

    #!/bin/bash
    
    DNS_SERVER=$(grep nameserver /etc/resolv.conf | cut -f2 -d" ")
    COUNT=0
    # check DNS server is available
    ping -c1 $DNS_SERVER
    while [ $? != '0' ] && [ $COUNT -lt 10 ]; do
    ((COUNT++))
    echo "Checking for DNS: Attempt $COUNT ."
    sleep 10
    ping -c 1 $DNS_SERVER
    done
  2. Include an executable file for the health checks at /etc/greenboot/check/required.d/.

    chmod +x check-dns.sh

    During the next reboot, the script is executed as part of the boot process, before the system enters the boot-complete.target. If the health checks are successful, no action is taken. If the health checks fail, the system is rebooted several times, before marking the update as failed and rolling back to the previous update.