Chapter 3. Administering the Environment

3.1. Administering the Self-Hosted Engine

3.1.1. Maintaining the Self-hosted engine

3.1.1.1. Self-hosted engine maintenance modes explained

The maintenance modes enable you to start, stop, and modify the Manager virtual machine without interference from the high-availability agents, and to restart and modify the self-hosted engine nodes in the environment without interfering with the Manager.

There are three maintenance modes:

  • global - All high-availability agents in the cluster are disabled from monitoring the state of the Manager virtual machine. The global maintenance mode must be applied for any setup or upgrade operations that require the ovirt-engine service to be stopped, such as upgrading to a later version of Red Hat Virtualization.
  • local - The high-availability agent on the node issuing the command is disabled from monitoring the state of the Manager virtual machine. The node is exempt from hosting the Manager virtual machine while in local maintenance mode; if hosting the Manager virtual machine when placed into this mode, the Manager will migrate to another node, provided there is one available. The local maintenance mode is recommended when applying system changes or updates to a self-hosted engine node.
  • none - Disables maintenance mode, ensuring that the high-availability agents are operating.

3.1.1.2. Setting local maintenance mode

Enabling local maintenance mode stops the high-availability agent on a single self-hosted engine node.

Setting the local maintenance mode from the Administration Portal

  1. Put a self-hosted engine node into local maintenance mode:

    1. In the Administration Portal, click ComputeHosts and select a self-hosted engine node.
    2. Click ManagementMaintenance and OK. Local maintenance mode is automatically triggered for that node.
  2. After you have completed any maintenance tasks, disable the maintenance mode:

    1. In the Administration Portal, click ComputeHosts and select the self-hosted engine node.
    2. Click ManagementActivate.

Setting the local maintenance mode from the command line

  1. Log in to a self-hosted engine node and put it into local maintenance mode:

    # hosted-engine --set-maintenance --mode=local
  2. After you have completed any maintenance tasks, disable the maintenance mode:

    # hosted-engine --set-maintenance --mode=none

3.1.1.3. Setting global maintenance mode

Enabling global maintenance mode stops the high-availability agents on all self-hosted engine nodes in the cluster.

Setting the global maintenance mode from the Administration Portal

  1. Put all of the self-hosted engine nodes into global maintenance mode:

    1. In the Administration Portal, click ComputeHosts and select any self-hosted engine node.
    2. Click More Actions ( moreactions ), then click Enable Global HA Maintenance.
  2. After you have completed any maintenance tasks, disable the maintenance mode:

    1. In the Administration Portal, click ComputeHosts and select any self-hosted engine node.
    2. Click More Actions ( moreactions ), then click Disable Global HA Maintenance.

Setting the global maintenance mode from the command line

  1. Log in to any self-hosted engine node and put it into global maintenance mode:

    # hosted-engine --set-maintenance --mode=global
  2. After you have completed any maintenance tasks, disable the maintenance mode:

    # hosted-engine --set-maintenance --mode=none

3.1.2. Administering the Manager Virtual Machine

The hosted-engine utility provides many commands to help administer the Manager virtual machine. You can run hosted-engine on any self-hosted engine node. To see all available commands, run hosted-engine --help. For additional information on a specific command, run hosted-engine --command --help.

3.1.2.1. Updating the Self-Hosted Engine Configuration

To update the self-hosted engine configuration, use the hosted-engine --set-shared-config command. This command updates the self-hosted engine configuration on the shared storage domain after the initial deployment.

To see the current configuration values, use the hosted-engine --get-shared-config command.

To see a list of all available configuration keys and their corresponding types, enter the following command:

# hosted-engine --set-shared-config key --type=type --help

Where type is one of the following:

he_local

Sets values in the local instance of /etc/ovirt-hosted-engine/hosted-engine.conf on the local host, so only that host uses the new values. To enable the new value, restart the ovirt-ha-agent and ovirt-ha-broker services.

he_shared

Sets values in /etc/ovirt-hosted-engine/hosted-engine.conf on shared storage, so all hosts that are deployed after a configuration change use these values. To enable the new value on a host, redeploy that host.

ha

Sets values in /var/lib/ovirt-hosted-engine-ha/ha.conf on local storage. New settings take effect immediately.

broker

Sets values in /var/lib/ovirt-hosted-engine-ha/broker.conf on local storage. Restart the ovirt-ha-broker service to enable new settings.

3.1.2.2. Configuring Email Notifications

You can configure email notifications using SMTP for any HA state transitions on the self-hosted engine nodes. The keys that can be updated include: smtp-server, smtp-port, source-email, destination-emails, and state_transition.

To configure email notifications:

  1. On a self-hosted engine node, set the smtp-server key to the desired SMTP server address:

    # hosted-engine --set-shared-config smtp-server smtp.example.com --type=broker
    Note

    To verify that the self-hosted engine configuration file has been updated, run:

    # hosted-engine --get-shared-config smtp-server --type=broker
    broker : smtp.example.com, type : broker
  2. Check that the default SMTP port (port 25) has been configured:

    # hosted-engine --get-shared-config smtp-port --type=broker
    broker : 25, type : broker
  3. Specify an email address you want the SMTP server to use to send out email notifications. Only one address can be specified.

    # hosted-engine --set-shared-config source-email source@example.com --type=broker
  4. Specify the destination email address to receive email notifications. To specify multiple email addresses, separate each address by a comma.

    # hosted-engine --set-shared-config destination-emails destination1@example.com,destination2@example.com --type=broker

To verify that SMTP has been properly configured for your self-hosted engine environment, change the HA state on a self-hosted engine node and check if email notifications were sent. For example, you can change the HA state by placing HA agents into maintenance mode. See Maintaining the Self-Hosted Engine for more information.

3.1.3. Configuring Memory Slots Reserved for the Self-Hosted Engine on Additional Hosts

If the Manager virtual machine shuts down or needs to be migrated, there must be enough memory on a self-hosted engine node for the Manager virtual machine to restart on or migrate to it. This memory can be reserved on multiple self-hosted engine nodes by using a scheduling policy. The scheduling policy checks if enough memory to start the Manager virtual machine will remain on the specified number of additional self-hosted engine nodes before starting or migrating any virtual machines. See Creating a Scheduling Policy in the Administration Guide for more information about scheduling policies.

To add more self-hosted engine nodes to the Red Hat Virtualization Manager, see Adding self-hosted engine nodes to the Manager.

Configuring Memory Slots Reserved for the Self-Hosted Engine on Additional Hosts

  1. Click ComputeClusters and select the cluster containing the self-hosted engine nodes.
  2. Click Edit.
  3. Click the Scheduling Policy tab.
  4. Click + and select HeSparesCount.
  5. Enter the number of additional self-hosted engine nodes that will reserve enough free memory to start the Manager virtual machine.
  6. Click OK.

3.1.4. Adding Self-Hosted Engine Nodes to the Red Hat Virtualization Manager

Add self-hosted engine nodes in the same way as a standard host, with an additional step to deploy the host as a self-hosted engine node. The shared storage domain is automatically detected and the node can be used as a failover host to host the Manager virtual machine when required. You can also attach standard hosts to a self-hosted engine environment, but they cannot host the Manager virtual machine. Have at least two self-hosted engine nodes to ensure the Manager virtual machine is highly available. You can also add additional hosts using the REST API. See Hosts in the REST API Guide.

Prerequisites

Procedure

  1. In the Administration Portal, click ComputeHosts.
  2. Click New.

    For information on additional host settings, see Explanation of Settings and Controls in the New Host and Edit Host Windows in the Administration Guide.

  3. Use the drop-down list to select the Data Center and Host Cluster for the new host.
  4. Enter the Name and the Address of the new host. The standard SSH port, port 22, is auto-filled in the SSH Port field.
  5. Select an authentication method to use for the Manager to access the host.

    • Enter the root user’s password to use password authentication.
    • Alternatively, copy the key displayed in the SSH PublicKey field to /root/.ssh/authorized_keys on the host to use public key authentication.
  6. Optionally, configure power management, where the host has a supported power management card. For information on power management configuration, see Host Power Management Settings Explained in the Administration Guide.
  7. Click the Hosted Engine tab.
  8. Select Deploy.
  9. Click OK.

3.1.5. Reinstalling an Existing Host as a Self-Hosted Engine Node

You can convert an existing, standard host in a self-hosted engine environment to a self-hosted engine node capable of hosting the Manager virtual machine.

Warning

When installing or reinstalling the host’s operating system, Red Hat strongly recommends that you first detach any existing non-OS storage that is attached to the host to avoid accidental initialization of these disks, and with that, potential data loss.

Procedure

  1. Click ComputeHosts and select the host.
  2. Click ManagementMaintenance and OK.
  3. Click InstallationReinstall.
  4. Click the Hosted Engine tab and select DEPLOY from the drop-down list.
  5. Click OK.

The host is reinstalled with self-hosted engine configuration, and is flagged with a crown icon in the Administration Portal.

3.1.6. Booting the Manager Virtual Machine in Rescue Mode

This topic describes how to boot the Manager virtual machine into rescue mode when it does not start. For more information, see Booting to Rescue Mode in the Red Hat Enterprise Linux System Administrator’s Guide.

  1. Connect to one of the hosted-engine nodes:

    $ ssh root@host_address
  2. Put the self-hosted engine in global maintenance mode:

    # hosted-engine --set-maintenance --mode=global
  3. Check if there is already a running instance of the Manager virtual machine:

    # hosted-engine --vm-status

    If a Manager virtual machine instance is running, connect to its host:

    # ssh root@host_address
  4. Shut down the virtual machine:

    # hosted-engine --vm-shutdown
    Note

    If the virtual machine does not shut down, execute the following command:

    # hosted-engine --vm-poweroff
  5. Start the Manager virtual machine in pause mode:

    hosted-engine --vm-start-paused
  6. Set a temporary VNC password:

    hosted-engine --add-console-password

    The command outputs the necessary information you need to log in to the Manger virtual machine with VNC.

  7. Log in to the Manager virtual machine with VNC. The Manager virtual machine is still paused, so it appears to be frozen.
  8. Resume the Manager virtual machine with the following command on its host:

    Warning

    After running the following command, the boot loader menu appears. You need to enter into rescue mode before the boot loader proceeds with the normal boot process. Read the next step about entering into rescue mode before proceeding with this command.

    # /usr/bin/virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf resume HostedEngine
  9. Boot the Manager virtual machine in rescue mode.
  10. Disable global maintenance mode

    # hosted-engine --set-maintenance --mode=none

You can now run rescue tasks on the Manager virtual machine.

3.1.7. Removing a Host from a Self-Hosted Engine Environment

To remove a self-hosted engine node from your environment, place the node into maintenance mode, undeploy the node, and optionally remove it. The node can be managed as a regular host after the HA services have been stopped, and the self-hosted engine configuration files have been removed.

Procedure

  1. In the Administration Portal, click ComputeHosts and select the self-hosted engine node.
  2. Click ManagementMaintenance and OK.
  3. Click InstallationReinstall.
  4. Click the Hosted Engine tab and select UNDEPLOY from the drop-down list. This action stops the ovirt-ha-agent and ovirt-ha-broker services and removes the self-hosted engine configuration file.
  5. Click OK.
  6. Optionally, click Remove. This opens the Remove Host(s) confirmation window.
  7. Click OK.

3.1.8. Updating a Self-Hosted Engine

To update a self-hosted engine from your current version to the latest version, you must place the environment in global maintenance mode and then follow the standard procedure for updating between minor versions.

Enabling global maintenance mode

You must place the self-hosted engine environment in global maintenance mode before performing any setup or upgrade tasks on the Manager virtual machine.

Procedure

  1. Log in to one of the self-hosted engine nodes and enable global maintenance mode:

    # hosted-engine --set-maintenance --mode=global
  2. Confirm that the environment is in global maintenance mode before proceeding:

    # hosted-engine --vm-status

    You should see a message indicating that the cluster is in global maintenance mode.

Updating the Red Hat Virtualization Manager

Procedure

  1. On the Manager machine, check if updated packages are available:

    # engine-upgrade-check
  2. Update the setup packages:

    # yum update ovirt\*setup\* rh\*vm-setup-plugins
  3. Update the Red Hat Virtualization Manager with the engine-setup script. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service.

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully
    Note

    The engine-setup script is also used during the Red Hat Virtualization Manager installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and might not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup.

    Important

    The update process might take some time. Do not stop the process before it completes.

  4. Update the base operating system and any optional packages installed on the Manager:

    # yum update --nobest
    Important

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    Important

    If any kernel packages were updated:

    1. Disable global maintenance mode
    2. Reboot the machine to complete the update.
Disabling global maintenance mode

Procedure

  1. Log in to the Manager virtual machine and shut it down.
  2. Log in to one of the self-hosted engine nodes and disable global maintenance mode:

    # hosted-engine --set-maintenance --mode=none

    When you exit global maintenance mode, ovirt-ha-agent starts the Manager virtual machine, and then the Manager automatically starts. It can take up to ten minutes for the Manager to start.

  3. Confirm that the environment is running:

    # hosted-engine --vm-status

    The listed information includes Engine Status. The value for Engine status should be:

    {"health": "good", "vm": "up", "detail": "Up"}
    Note

    When the virtual machine is still booting and the Manager hasn’t started yet, the Engine status is:

    {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}

    If this happens, wait a few minutes and try again.

3.1.9. Changing the FQDN of the Manager in a Self-Hosted Engine

You can use the ovirt-engine-rename command to update records of the fully qualified domain name (FQDN) of the Manager.

For details see Renaming the Manager with the Ovirt Engine Rename Tool.