Chapter 11. Managing RHEL for Edge images

To manage the RHEL for Edge images, you can perform any of the following administrative tasks:

  • Edit the RHEL for Edge image blueprint using image builder in RHEL web console
  • Edit the RHEL for Edge image blueprint using image builder command-line
  • Update the RHEL for Edge images
  • Configure rpm-ostree remotes on nodes, to update node policy
  • Restore RHEL for Edge images manually or automatically using Greenboot

11.1. Editing a RHEL for Edge image blueprint using image builder in RHEL web console

You can edit the RHEL for Edge image blueprint to:

  • Add additional components that you may require
  • Modify the version of any existing component
  • Remove any existing component

11.1.1. Adding a component to RHEL for Edge image blueprint using image builder in RHEL web console

To add a component to a RHEL for Edge image blueprint, ensure that you have met the following prerequisites and then follow the procedure to edit the corresponding blueprint.

Prerequisites

  • On a RHEL system, you have accessed the image builder dashboard.
  • You have created a blueprint for RHEL for Edge image.

Procedure

  1. On the image builder dashboard, click the blueprint that you want to edit.

    To search for a specific blueprint, enter the blueprint name in the filter text box, and click Enter.

  2. On the upper right side of the blueprint, click Edit Packages.

    The Edit blueprints wizard opens.

  3. On the Details page, update the blueprint name and click Next.

  4. On the Packages page, follow the steps:

    1. In the Available Packages, enter the package name that you want to add in the filter text box, and click Enter.

      A list with the component name appears.

    2. Click > to add the component to the blueprint.

  5. On the Review page, click Save.

    The blueprint is now updated with the new package.

11.1.2. Removing a component from RHEL for Edge image blueprint using image builder in RHEL web console

To remove one or more unwanted components from a RHEL for Edge image blueprint that you created, ensure that you have met the following prerequisites and then follow the procedure.

Prerequisites

  • On a RHEL system, you have accessed the image builder dashboard.
  • You have created a blueprint for RHEL for Edge image.
  • You have added at least one component to the RHEL for Edge blueprint.

Procedure

  1. On the image builder dashboard, click the blueprint that you want to edit.

    To search for a specific blueprint, enter the blueprint name in the filter text box, and click Enter.

  2. On the upper right side of the blueprint, click Edit Packages.

    The Edit blueprints wizard opens.

  3. On the Details page, update the blueprint name and click Next.

  4. On the Packages page, follow the steps:

    1. From the Chosen packages, click < to remove the chosen component. You can also click << to remove all the packages at once.

  5. On the Review page, click Save.

    The blueprint is now updated.

11.1.3. Editing a RHEL for Edge image blueprint using command-line interface

You can change the specifications for your RHEL for Edge image blueprint using image builder command-line. To do so, ensure that you have met the following prerequisites and then follow the procedure to edit the corresponding blueprint.

Prerequisites

  • You have access to the image builder command-line.
  • You have created a RHEL for Edge image blueprint.

Procedure

  1. Save (export) the blueprint to a local text file: 

    # composer-cli blueprints save BLUEPRINT-NAME

  2. Edit the BLUEPRINT-NAME.toml file with a text editor of your choice and make your changes.

    Before finishing with the edits, verify that the file is a valid blueprint:

  3. Increase the version number.

    Ensure that you use a Semantic Versioning scheme.

    Note

    if you do not change the version, the patch component of the version is increased automatically.

  4. Check if the contents are valid TOML specifications. See the TOML documentation for more information.

    Note

    TOML documentation is a community product and is not supported by Red Hat. You can report any issues with the tool at https://github.com/toml-lang/toml/issues.

  5. Save the file and close the editor.

  6. Push (import) the blueprint back into image builder command-line: 

    # composer-cli blueprints push BLUEPRINT-NAME.toml
    Note

    When pushing the blueprint back into the image builder command-line, provide the file name including the .toml extension.

  7. Verify that the contents uploaded to image builder match your edits: 

    # composer-cli blueprints show BLUEPRINT-NAME

  8. Check whether the components and versions listed in the blueprint and their dependencies are valid: 

    # composer-cli blueprints depsolve BLUEPRINT-NAME

11.2. Updating RHEL for Edge images

11.2.1. How are RHEL for Edge image updates deployed

With RHEL for Edge images, you can either deploy the updates manually or can automate the deployment process. The updates are applied in an atomic manner, where the state of each update is known, and the updates are staged and applied only upon reboot. Because no changes are seen until you reboot the device, you can schedule a reboot to ensure the highest possible uptime.

During the image update, only the updated operating system content is transferred over the network. This makes the deployment process more efficient compared to transferring the entire image. The operating system binaries and libraries in /usr are read-only, and the read and write state is maintained in /var and /etc directories.

When moving to a new deployment, the /etc and the /var directories are copied to the new deployment with read and write permissions. The /usr directory is copied as a soft link to the new deployment directory, with read-only permissions.

The following diagram illustrates the RHEL for Edge image update deployment process:

Image Deployment

By default, the new system is booted using a procedure similar to a chroot operation. The new /sysroot directory mainly has the following parts:

  • Repository database at the /sysroot/ostree/repo directory.
  • File system revisions at the /sysroot/ostree/deploy/rhel/deploy directory, which are created by each operation in the system update.
  • The /sysroot/ostree/boot directory, which links to deployments on the previous point. Note that /ostree is a soft link to /sysroot/ostree. The files from the /sysroot/ostree/boot directory are not duplicated. The same file is used if it is not changed during the deployment. The files are hard-links to another file stored in the /sysroot/ostree/repo/objects directory.

The operating system selects the deployment in the following way:

  1. The dracut tool parses the ostree kernel argument in the initramfs root file system and sets up the /usr directory as a read-only bind mount.
  2. Bind the deployment directory in /sysroot to / directory.
  3. Re-mount the operating system already mounted dirs using the MS_MOVE mount flag

If anything goes wrong, you can perform a deployment rollback by removing the old deployments with the rpm-ostree cleanup command. Each client machine contains an OSTree repository stored in /ostree/repo, and a set of deployments stored in /ostree/deploy/$STATEROOT/$CHECKSUM.

With the deployment updates in RHEL for Edge image, you can benefit from a better system consistency across multiple devices, easier reproducibility, and better isolation between the pre and post system states change.

11.2.2. Deploying RHEL for Edge image updates manually

After you have edited a RHEL for Edge blueprint, you can update the image commit. Image builder generates a new commit for the updated RHEL for Edge image. Use this new commit to deploy the image with latest package versions or with additional packages.

To deploy RHEL for Edge images updates, ensure that you meet the prerequisites and then follow the procedure.

Prerequisites

Procedure

  1. On the image builder dashboard click Create Image.

  2. On the Create Image window, perform the following steps:

    1. In the Image output page:

      1. From the Select a blueprint dropdown list, select the blueprint that you edited.
      2. From the Image output type dropdown list, select RHEL for Edge Commit (.tar). Click Next.

    2. In the OSTree settings page, enter:

      1. In the Repository URL field, enter the URL to the OSTree repository of the commit to embed in the image. For example, http://10.0.2.2:8080/repo/. See Setting up a web server to install RHEL for Edge image.
      2. In the Parent commit field, specify the parent commit ID that was previously generated. See Extracting RHEL for Edge image commit.
      3. In the Ref field, you can either specify a name for your commit or leave it empty. By default, the web console specifies the Ref as rhel/9/arch_name/edge. Click Next.

    3. In the Review page, check the customizations and click Create image. Image builder starts to create a RHEL for Edge image for the updated blueprint. The image creation process takes a few minutes to complete.

      To view the RHEL for Edge image creation progress, click the blueprint name from the breadcrumbs, and then click the Images tab.

      The resulting image includes the latest packages that you have added, if any, and have the original commit ID as a parent.

  3. Download the resulting RHEL for Edge Commit (.tar) image.

    1. From the Images tab, click Download to save the RHEL for Edge Commit (.tar) image to your system.

  4. Extract the OSTree commit (.tar) file. 

    # tar -xf UUID-commit.tar -C UPGRADE_FOLDER

  5. Upgrade the OSTree repo: 

    # ostree pull-local --repo http://10.0.2.2:8080/repo UPGRADE_FOLDER/repo OSTREE_REF
# ostree summary --update --repo http://10.0.2.2:8080/repo

  6. Build a docker container, serving the child commit ID this time. 

    # podman build -t name-of-server --build-arg commit=UUID-child_commit.tar .

  7. Run the container. 

    # podman run --rm -p 8000:80 name-of-server

  8. On the RHEL system provisioned, from the original edge image, verify the current status. 

    $ rpm-ostree status

    If there is no new commit ID, run the following command to verify if there is any upgrade available: 

    $ rpm-ostree upgrade --check

    The command output provides the current active OSTree commit ID.

  9. Update OSTree to make the new OSTree commit ID available. 

    $ rpm-ostree upgrade

    OSTree verifies if there is an update on the repository. If yes, it fetches the update and requests you to reboot your system so that you can activate the deployment of this new commit update.

  10. Check the current status again: 

    $ rpm-ostree status

    You can now see that there are 2 commits available:

    • The active parent commit.
    • A new commit that is not active and contains 1 added difference.

  11. To activate the new deployment and to make the new commit active, reboot your system. 

    # systemctl reboot

    The Anaconda Installer reboots into the new deployment. On the login screen, you can see a new deployment available for you to boot.

  12. If you want to boot into the newest deployment (commit), the rpm-ostree upgrade command automatically orders the boot entries so that the new deployment is first in the list. Optionally, you can use the arrow key on your keyboard to select the GRUB menu entry and press Enter.
  13. Provide your login user account credentials.

  14. Verify the OSTree status: 

    $ rpm-ostree status

    The command output provides the active commit ID.

  15. To view the changed packages, if any, run a diff between the parent commit and the new commit: 

    $ rpm-ostree db diff parent_commit new_commit

    The update shows that the package you have installed is available and ready for use.

11.2.3. Deploying RHEL for Edge image updates manually using the command-line

After you have edited a RHEL for Edge blueprint, you can update the image commit. Image builder generates a new commit for the updated RHEL for Edge image. Use the new commit to deploy the image with latest package versions or with additional packages using the CLI.

To deploy RHEL for Edge image updates using the CLI, ensure that you meet the prerequisites, and then follow the procedure.

Prerequisites

Procedure

  1. Create the RHEL for Edge Commit (.tar) image with the following arguments: 

    # composer-cli compose start-ostree --ref ostree_ref --url URL-OSTree-repository -blueprint_name_ image-type

    where

    • ref is the reference you provided during the creation of the RHEL for Edge Container commit. For example, rhel/9/x86_64/edge.
    • URL-OSTree-repository is the URL to the OSTree repository of the commit to embed in the image. For example, http://10.0.2.2:8080/repo/. See Setting up a web server to install RHEL for Edge image.

    • image-type is edge-commit.

      Image builder creates a RHEL for Edge image for the updated blueprint.

  2. Check the RHEL for Edge image creation progress: 

    # composer-cli compose status
    Note

    The image creation processes can take up to ten to thirty minutes to complete.

    The resulting image includes the latest packages that you have added, if any, and has the original commit ID as a parent.

  3. Download the resulting RHEL for Edge image. For more information, see Downloading a RHEL for Edge image using the image builder command-line interface.

  4. Extract the OSTree commit. 

    # tar -xf UUID-commit.tar -C UPGRADE_FOLDER
  5. Serve the OSTree commit by using httpd. See Setting up a web server to install RHEL for Edge image.

  6. Upgrade the OSTree repo: 

    # ostree pull-local --repo http://10.0.2.2:8080/repo UPGRADE_FOLDER/repo OSTREE_REF
# ostree summary --update --repo http://10.0.2.2:8080/repo

  7. On the RHEL system provisioned from the original edge image, verify the current status: 

    $ rpm-ostree status

    If there is no new commit ID, run the following command to verify if there is any upgrade available: 

    $ rpm-ostree upgrade --check

    The command output provides the current active OSTree commit ID.

  8. Update OSTree to make the new OSTree commit ID available: 

    $ rpm-ostree upgrade

    OSTree verifies if there is an update on the repository. If yes, it fetches the update and requests you to reboot your system so that you can activate the deployment of the new commit update.

  9. Check the current status again: 

    $ rpm-ostree status

    You should now see that there are 2 commits available:

    • The active parent commit
    • A new commit that is not active and contains 1 added difference

  10. To activate the new deployment and make the new commit active, reboot your system: 

    # systemctl reboot

    The Anaconda Installer reboots into the new deployment. On the login screen, you can see a new deployment available for you to boot.

  11. If you want to boot into the newest deployment, the rpm-ostree upgrade command automatically orders the boot entries so that the new deployment is first in the list. Optionally, you can use the arrow key on your keyboard to select the GRUB menu entry and press Enter.
  12. Log in using your account credentials.

  13. Verify the OSTree status: 

    $ rpm-ostree status

    The command output provides the active commit ID.

  14. To view the changed packages, if any, run a diff between the parent commit and the new commit: 

    $ rpm-ostree db diff parent_commit new_commit

    The update shows that the package you have installed is available and ready for use.

11.2.4. Deploying RHEL for Edge image updates manually for non-network-base deployments

After you have edited a RHEL for Edge blueprint, you can update the image commit. Image builder generates a new commit for the updated RHEL for Edge image. Use this new commit to deploy the image with the latest package versions or with additional packages.

To deploy RHEL for Edge images updates, ensure that you meet the prerequisites and then follow the procedure.

Prerequisites

Procedure

  1. On the image builder dashboard click Create Image.

  2. On the Create Image window, perform the following steps:

    1. In the Image output page:

      1. From the Select a blueprint dropdown list, select the blueprint that you edited.
      2. From the Image output type dropdown list, select RHEL for Edge Container (.tar). Click Next.

    2. In the OSTree settings page, enter:

      1. In the Repository URL field, enter the URL to the OSTree repository of the commit to embed in the image. For example, http://10.0.2.2:8080/repo/. See Setting up a web server to install RHEL for Edge image.
      2. In the Parent commit field, specify the parent commit ID that was previously generated. See Extracting RHEL for Edge image commit.
      3. In the Ref field, you can either specify a name for your commit or leave it empty. By default, the web console specifies the Ref as rhel/9/arch_name/edge. Click Next.

    3. In the Review page, check the customizations and click Create image.

      Image builder creates a RHEL for Edge image for the updated blueprint.

      To view the progress of RHEL for Edge image creation, click the Images tab.

      Note

      The image creation process takes a few minutes to complete.

      The resulting image includes the latest packages that you have added, if any, and has the original commit ID as a parent.

  3. Download the resulting RHEL for Edge image

    1. From the Images tab, click Download to save the RHEL for Edge Container (.tar) image to your system.

  4. Load the RHEL for Edge Container image into Podman, serving the child commit ID this time. 

    $ cat ./child-commit_ID-container.tar | sudo podman load

  5. Run Podman. 

    #  sudo podman run -p 8080:8080 localhost/edge-test

  6. Upgrade the OSTree repo: 

    # ostree pull-local --repo http://10.0.2.2:8080/repo UPGRADE_FOLDER/repo OSTREE_REF
# ostree summary --update --repo http://10.0.2.2:8080/repo

  7. On the RHEL system provisioned, from the original edge image, verify the current status. 

    $ rpm-ostree status

    If there is no new commit ID, run the following command to verify if there is any upgrade available: 

    $ rpm-ostree upgrade --check

    If there are updates available, the command output provides information about the available updates in the OSTree repository, such as the current active OSTree commit ID. Else, it prompts a message informing that there are no updates available.

  8. Update OSTree to make the new OSTree commit ID available. 

    $ rpm-ostree upgrade

    OSTree verifies if there is an update on the repository. If yes, it fetches the update and requests you to reboot your system so that you can activate the deployment of this new commit update.

  9. Check the current status: 

    $ rpm-ostree status

    You can now see that there are 2 commits available:

    • The active parent commit.
    • A new commit that is not active and contains 1 added difference.

  10. To activate the new deployment and to make the new commit active, reboot your system. 

    # systemctl reboot

    The Anaconda Installer reboots into the new deployment. On the login screen, you can see a new deployment available for you to boot.

  11. If you want to boot into the newest commit/deployment, the rpm-ostree upgrade command automatically orders the boot entries so that the new deployment is first in the list. Optionally, you can use the arrow key on your keyboard to select the GRUB menu entry and press Enter.
  12. Provide your login user account credentials.

  13. Verify the OSTree status: 

    $ rpm-ostree status

    The command output provides the active commit ID.

  14. To view the changed packages, if any, run a diff between the parent commit and the new commit: 

    $ rpm-ostree db diff parent_commit new_commit

    The update shows that the package you have installed is available and ready for use.

11.3. Upgrading RHEL for Edge systems

11.3.1. Upgrading your RHEL 8 system to RHEL 9

You can upgrade your RHEL 8 system to RHEL 9 using the rpm-ostree rebase command. It fully supports the default package set of RHEL for Edge upgrades from the most recent updates of RHEL 8 to the most recent updates of RHEL 9. The upgrade downloads and installs the RHEL 9 image in the background. After the upgrade finishes, you must reboot your system to use the new RHEL 9 image.

Note

The upgrade does not support every possible rpm package version and inclusions. You must test your package additions to ensure that it works as expected.

Prerequisites

  • You have a running RHEL for Edge 8 system
  • You have an OSTree repository server by HTTP
  • You created a blueprint for RHEL for Edge 9 image that you will upgrade

Procedure

  1. On the system where image builder runs, create a RHEL for Edge 9 image:

    1. Start the image compose: 

      $ sudo composer-cli compose start blueprint-name edge-commit
    2. After the compose finishes, download the image.

    3. Extract the downloaded image to /var/www/html/ folder: 

      $ sudo tar -xf image_file -C /var/www/html

    4. Start the httpd service: 

      $ systemctl start httpd.service

  2. On the RHEL for Edge device, check the current remote repository configuration: 

    $ sudo cat /etc/ostree/remotes.d/edge.conf
    Note

    Depending on how your Kickstart file is configured, the /etc/ostree/remotes.d repository can be empty. If you configured your remote repository, you can see it’s configuration. For the edge-installer, raw-image, and simplified-installer images, the remote is configured by default.

  3. Check the current URL repository: 

    $ sudo ostree remote show-url edge

    edge is the of the Ostree repository.

  4. List the remote reference branches: 

    $ ostree remote refs edge

    You can see the following output: 

    Error: Remote refs not available; server has no summary file

  5. To add the new repository:

    1. Configure the URL key to add a remote repository. For example: 

      $ sudo ostree remote add \ --no-gpg-verify rhel9 http://192.168.122.1/repo/

    2. Configure the URL key to point to the RHEL 9 commit for the upgrade. For example: 

      $ sudo cat /etc/ostree/remotes.d/edge.conf [remote "edge"] url=http://192.168.122.1/ostree/repo/ gpg-verify=false

    3. Confirm if the URL has been set to the new remote repository: 

      $ sudo cat /etc/ostree/remotes.d/rhel9.conf [remote "edge"] url=http://192.168.122.1/repo/ gpg-verify=false

    4. See the new URL repository: 

      $ sudo ostree remote show-url rhel9 http://192.168.122.1/ostree-rhel9/repo/

    5. List the current remote list options: 

      $ sudo ostree remote list

output:
edge
rhel9

  6. Rebase your system to the RHEL version, providing the reference path for the RHEL 9 version: 

    $ rpm-ostree rebase rhel9:rhel/9/x86_64/edge

  7. Reboot your system. 

    $ systemctl reboot
  8. Enter your username and password.

  9. Check the current system status: 

    $ rpm-ostree status

Verification

  1. Check the current status of the currently running deployment: 

    $ rpm-ostree status

  2. Optional: List the processor and tasks managed by the kernel in real-time. 

    $ top

  3. If the upgrade does not support your requirements, you have the option to manually rollback to the previous stable deployment RHEL 8 version: 

    $ sudo rpm-ostree rollback

  4. Reboot your system. Enter your username and password: 

    $ systemctl reboot

    After rebooting, your system runs RHEL 9 successfully.

    Note

    If your upgrade succeeds and you do not want to use the previous deployment RHEL 8 version, you can delete the old repository: 

    $ sudo ostree remote delete edge

11.4. Deploying RHEL for Edge automatic image updates

After you install a RHEL for Edge image on an Edge device, you can check for image updates available, if any, and can auto-apply them.

The rpm-ostreed-automatic.service (systemd service) and rpm-ostreed-automatic.timer (systemd timer) control the frequency of checks and upgrades. The available updates, if any, appear as staged deployments.

Deploying automatic image updates involves the following high-level steps:

  • Update the image update policy
  • Enable automatic download and staging of updates

11.4.1. Updating the RHEL for Edge image update policy

To update the image update policy, use the AutomaticUpdatePolicy and an IdleExitTimeout setting from the rpm-ostreed.conf file at /etc/rpm-ostreed.conf location on an Edge device.

The AutomaticUpdatePolicy settings controls the automatic update policy and has the following update checks options:

  • none: Disables automatic updates. By default, the AutomaticUpdatePolicy setting is set to none.
  • check: Downloads enough metadata to display available updates with rpm-ostree status.
  • stage: Downloads and unpacks the updates that are applied on a reboot.

The IdleExitTimeout setting controls the time in seconds of inactivity before the daemon exit and has the following options:

  • 0: Disables auto-exit.
  • 60: By default, the IdleExitTimeout setting is set to 60.

To enable automatic updates, perform the following steps:

Procedure

  1. In the /etc/rpm-ostreed.conf file, update the following:

    • Change the value of AutomaticUpdatePolicy to check.
    • To run the update checks, specify a value in seconds for IdleExitTimeout.

  2. Reload the rpm-ostreed service and enable the systemd timer. 

    # systemctl reload rpm-ostreed
# systemctl enable rpm-ostreed-automatic.timer --now

  3. Verify the rpm-ostree status to ensure the automatic update policy is configured and time is active. 

    # rpm-ostree status

    The command output shows the following: 

    State: idle; auto updates enabled (check; last run <minutes> ago)

    Additionally, the output also displays information about the available updates.

11.4.2. Enabling RHEL for Edge automatic download and staging of updates

After you update the image update policy to check for image updates, the updates if any are displayed along with the update details. If you decide to apply the updates, enable the policy to automatically download and stage the updates. The available image updates are then downloaded and staged for deployment. The updates are applied and take effect when you reboot the Edge device.

To enable the policy for automatic download and staging of updates, perform the following updates:

Procedure

  1. In the /etc/rpm-ostreed.conf file, update ‘AutomaticUpdatePolicy’ to stage.

  2. Reload the rpm-ostreed service. 

    # systemctl enable rpm-ostreed-automatic.timer --now

  3. Verify the rpm-ostree status 

    # rpm-ostree status

    The command output shows the following: 

    State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: last run <time> ago

  4. To initiate the updates, you can either wait for the timer to initiate the updates, or can manually start the service. 

    # systemctl start rpm-ostreed-automatic.service

    After the updates are initiated, the rpm-ostree status shows the following: 

    # rpm-ostree status
State: busy
AutomaticUpdates: stage; rpm-ostreed-automatic.service: running
Transaction: automatic (stage)

    When the update is complete, a new deployment is staged in the list of deployments, and the original booted deployment is left untouched. You can decide if you want to boot the system using the new deployment or can wait for the next update.

    To view the list of deployments, run the rpm-ostree status command.

    Following is a sample output. 

    # rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: last run <time> ago
Deployments:

    To view the list of deployments with the updated package details, run the rpm-ostree status -v command.

11.5. Rolling back RHEL for Edge images

You can verify if the updated image is successfully deployed or not. If the deployment is unsuccessful, you can roll back to a previous version (commit). To roll back to a previous functional state, you can either perform the steps manually or can use an automated process.

11.5.1. How are RHEL for Edge images rolled back

With RHEL for Edge images, only transactional updates are applied to the operating system. With the transactional updates, you can easily rollback the unsuccessful updates to the last known good state, preventing system failure during updates.

You can use intelligent rollbacks with Greenboot to eliminate the issues of choosing between application stability and application of security updates.

Greenboot leverages rpm-ostree and runs custom health checks that run on system startup. In case of an issue, the system rolls back the changes and preserves the last working state. When you deploy an rpm-ostree update, it runs scripts to check that critical services can still work after the update. If the system does not work, the update rolls back to the last known working version of the system. This process ensures that your RHEL for Edge device is in an operational state.

The following is the Greenboot directory structure:

Example 11.1. Greenboot directory structure

etc
└─ greenboot
   ├─ check
   |   └─ required.d
   |   └─ init.py
   └─ green.d
   └─ red.d
/etc/greenboot/check/required.d
Contains the health checks that must not fail.
/etc/greenboot/check/wanted.d
Contains the health checks that might fail.
/etc/greenboot/green.d
Contains the scripts to run after a successful boot.
/etc/greenboot/red.d
Contains the scripts to run after a failed boot. The system attempts to boot three times and in case of failure, it executes the scripts.

The following diagram illustrates the RHEL for Edge image roll back process.

Image restoration process

11.5.2. Rolling back RHEL for Edge images manually

If the deployment for RHEL for Edge image update fails or if the update fails to work successfully, then you can manually roll back to a previous deployment version.

To roll back to a previous version, perform the following steps:

Procedure

  1. Run the rollback command: 

    # rpm-ostree rollback

    The command output provides details about the commit ID that is being moved and indicates a completed transaction with the details of the package being removed.

  2. Reboot the system. 

    # systemctl reboot

    The command activates the previous commit with the stable content. The changes are applied and the previous version is restored.

11.5.3. Rolling back RHEL for Edge images using an automated process

Greenboot checks provides a framework that is integrated into the boot process and can trigger rpm-ostree rollbacks when a health check fails. For the health checks, you can create a custom script that indicates whether a health check passed or failed. Based on the result, you can decide when a rollback should be triggered.

To create a health check script, perform the following steps:

Procedure

  1. Create a script that returns a standard exit code 0.

    For example, the following script ensures that the configured DNS server is available: 

    #!/bin/bash

DNS_SERVER=$(grep ^nameserver /etc/resolv.conf | head -n 1 | cut -f2 -d" ")
COUNT=0
# check DNS server is available
ping -c1 $DNS_SERVER
while [ $? != '0' ] && [ $COUNT -lt 10 ]; do
((COUNT++))
echo "Checking for DNS: Attempt $COUNT ."
sleep 10
ping -c 1 $DNS_SERVER
done

  2. Include an executable file for the health checks at /etc/greenboot/check/required.d/. 

    chmod +x check-dns.sh

    During the next reboot, the script is executed as part of the boot process, before the system enters the boot-complete.target. If the health checks are successful, no action is taken. If the health checks fail, the system is rebooted several times, before marking the update as failed and rolling back to the previous update.

Verification steps

To check if the default gateway is reachable, run the following health check script:

  1. Create a script that returns a standard exit code 0. 

    #!/bin/bash

DEF_GW=$(ip r | awk '/^default/ {print $3}')
SCRIPT=$(basename $0)

count=10
connected=0
ping_timeout=5
interval=5

while [ $count -gt 0 -a $connected -eq 0 ]; do
  echo "$SCRIPT: Pinging default gateway $DEF_GW"
  ping -c 1 -q -W $ping_timeout $DEF_GW > /dev/null 2>&1 && connected=1 || sleep $interval
  ((--count))
done

if [ $connected -eq 1 ]; then
  echo "$SCRIPT: Default gateway $DEF_GW is reachable."
  exit 0
else
  echo "$SCRIPT: Failed to ping default gateway $DEF_GW!" 1>&2
  exit 1
fi

  2. Include an executable file for the health checks at /etc/greenboot/check/required.d/ directory. 

    chmod +x check-gw.sh