Red Hat Training

A Red Hat training course is available for Red Hat JBoss Operations Network

4. Configuring High Availability

High availability with JBoss ON servers means that all JBoss ON servers which use the same, central database interact together in a cloud. This allows seamless failover between servers when a server has to be taken offline for maintenance, and it provides a natural method for load balancing agent and resource operations.
Agents are assigned, or partitioned, among servers in the high availability cloud. Agents aren't strictly assigned to a server to be managed. Agents are handled by servers first based on a preference, or affinity, for a server because they belong to to the same affinity group. After affinity, or if an affinity group isn't configured, then agents are managed by the server in a round-robin style for availability.

Important

Only agents have an affinity preference in high availability. This means that agents have a preference in which server they attempt to contact. JBoss ON uses two-way communication, however, so servers also contact agents. Servers — regardless of the partition or the agent affinity configuration — can contact any agent in JBoss ON even if the server is not in that agent's affinity group or if the server does not manage the agent.

4.1. Prepping High Availability

High availability means that JBoss ON servers have natural and transparent failover and redundancy: if one server fails, both agents and UI access can be seamless switched to another server. A multi-server high availability configuration provides fault tolerance and improved scalability.
When multiple JBoss ON servers are installed, they use the same backend database to store and retrieve data. This approach distributes the load over multiple servers and provides agent failover.
If an agent's server becomes unavailable, the agent to switch to a different server. Through the use of affinity groups, agents switch to a preferred alternative server.

4.1.1. Deciding When to Use High Availability

In many circumstances, it may be satisfactory to run a single-server configuration. However, some environments require a multi-server approach because of the demands on the system:
  • Agent report processing is not meeting requirements, for collecting metrics, generating alerts or events, or reporting resource availability.
  • You have a geographically distributed environment with multiple data centers or logical grouping of agents to servers.
  • The agent load is too high for the server, meaning the server is having trouble processing the agent load. The optimal load per server will depend on the server, rather than the number of agents the server is managing.
There are three things to note about high availability server-agent connections:
  • The JBoss ON server defined in the agent's initial configuration is used only for the agent registration. It is not necessarily the one that the agent is told to connect to.
  • High availability servers communicate solely through the database. Therefore, server endpoints do not need to be visible to each other. No direct server to server communication is ever made.
  • Only agents have an affinity preference in high availability. This means that agents have a preference in which server they attempt to contact. JBoss ON uses two-way communication, however, so servers also contact agents. Servers — regardless of the partition or the agent affinity configuration — can contact any agent in JBoss ON even if the server is not in that agent's affinity group or if the server does not manage the agent.

4.1.2. High Availability Infrastructure Impact

In general, JBoss ON servers can be added or removed from the high availability cloud at any time. A single-server environment can be turned into a multi-server environment by installing a new JBoss ON server on a second machine and configuring it to use.
There are two things to consider when implementing a high availability cloud:
  • The potential demands on the shared backend database.
  • The IP addresses or names of the servers.
4.1.2.1. Database Impact
Although JBoss ON servers can be added to the high availability server cloud with relative ease, it should be done cautiously due to the potential impact on the back-end database. Each JBoss ON server limits its concurrent database connections, but there is no restriction on the cloud itself. Adding a second server can double the potential database connections, even if the number of agents remains the same. The increase is linear as servers are added.
Each JBoss ON server instance has built-in mechanisms for limiting the load it will put on the database. Each JBoss ON server can use less connections than the maximum limit, but the limits guarantee that they will each never use more than the maximum allowed connections to the database.

Note

A high availability configuration does not necessarily imply a large number of JBoss ON agents. It may be the case that a relatively small JBoss ON implementation may be in place, with only a few JBoss ON agents. Those agents, however, may need high availability, and, therefore, failover servers are required. In this case, the backend database will still have a high number of potential connections but, in reality, will not reach that limit.
4.1.2.2. Server and Agent Endpoints
In a multi-server high availability configuration, it is possible for any agent to try to connect to any server. It is critical that every JBoss ON agent be able to resolve the endpoint address set for every server in the high availability server cloud. When defining the server in the installer, it is important that the endpoint address be public to the degree that the agent population can reach the server via the defined address and port
An agent connecting to a server must provide an endpoint reachable by the server to allow for the necessary two-way communication.
4.1.2.3. Summary of Server Requirements
  1. All high availability servers must be running the same version of JBoss ON.
  2. All high availability servers must be uniquely named. This string is defined during server installation.
  3. Each high availability server must define a unique endpoint that is resolvable by all JBoss ON agents running against the high availability server cloud. This address/port is defined during installation. It can be an alias as long as that alias is resolvable by all JBoss ON agents. Any given JBoss ON agent may be running against any JBoss ON server at a given time.

4.2. Putting Servers in Maintenance Mode

Putting a JBoss ON server in maintenance mode temporarily removes it from the high availability cloud so it no longer processes agent operations. This is useful when the server is offline for upgrades or because of a service interruption.
  1. Click the Administration tab in the top menu.
  2. In the Topology menu table, select the Servers item.
  3. Select the check box next to the name of the JBoss ON server to put into maintenance mode.
  4. Click the SET MAINTENANCE button.
The JBoss ON server can be added back to the high availability cloud by clicking SET NORMAL button. Agents migrate back to the server incrementally.

4.3. Removing Servers from the High Availability Cloud

A JBoss ON server that is in maintenance mode can be permanently removed from the high availability cloud.
  1. Click the Administration tab in the top menu.
  2. In the Topology menu table, select the Servers item.
  3. Select the check box next to the name of the JBoss ON server to remove from the cloud, and click SET MAINTENANCE.
  4. When the screen refreshes, select the check box next to the name of the JBoss ON server again, and click the REMOVE SELECTED button.

4.4. Defining Affinity for Agents

By default, agent load is distributed evenly amongst the servers in the cloud. Balance can change in failover situations, but agent load is evenly distributed when all agents and all servers are running.

4.4.1. About Affinity

Affinity groups link agents to servers so that agents in an affinity group first try connecting to a server in the same affinity group before connecting to a server outside the affinity group.
Affinity groups provide three distinct advantages:
  • Physical efficiency. Generally, if certain agent-server connections clearly run more efficiently than others, then defining affinity to prefer those connections makes sense. This could include servers and agents co-located in the same data center, geographic grouping, or network topology.
  • Logical Efficiency. There may be organizational reasons, apart from operating efficiency, to group specific agents and servers together, such as administrative responsibilities or business unit assignments.
  • Warm backup. It may be the case that certain machines should not be assigned agent load unless specifically needed for failover purposes. In this case, all agents should be assigned affinity to a subset of available servers, leaving some servers without any associated agents in normal operation.
Section 4.1, “Prepping High Availability” covers the common considerations and restrictions when planning availability for servers and this also applies to planning affinity for agents.
Affinity assignments can be added or removed at any time, but it is useful to consider your initial approach, even if it confirms that affinity assignments are unnecessary.

Important

Only agents have an affinity preference in high availability. This means that agents have a preference in which server they attempt to contact. JBoss ON uses two-way communication, however, so servers also contact agents. Servers — regardless of the partition or the agent affinity configuration — can contact any agent in JBoss ON even if the server is not in that agent's affinity group or if the server does not manage the agent.

Note

Since high availability environments typically involve many agents, it may be useful to perform pre-configured agent installations to avoid having to answer initial setup questions interactively.

4.4.2. Creating Affinity Groups

An affinity group sets a preference for which JBoss ON servers manage which JBoss ON agents. An affinity group only sets a preference or hint for which server will manage the agent, not an absolute requirement. All agents are still managed within the JBoss ON server cloud, so any JBoss ON server can, theoretically, manage any JBoss ON agent based on the current load and performance.

Important

Only agents have an affinity preference in high availability. This means that agents have a preference in which server they attempt to contact. JBoss ON uses two-way communication, however, so servers also contact agents. Servers — regardless of the partition or the agent affinity configuration — can contact any agent in JBoss ON even if the server is not in that agent's affinity group or if the server does not manage the agent.
The affinity groups page shows the number of agents and servers assigned to each affinity group.
Listing Affinity Groups

Figure 2. Listing Affinity Groups

Note

An agent and a server can only belong to a single affinity group.
To create a new affinity group:

Note

To edit an affinity group, click its name, then manage it the same as creating a new affinity group.
  1. Click the Administration tab in the top menu.
  2. In the Topology menu table on the left, select the Affinity Groups item.
  3. Click the CREATE NEW button.
  4. Enter a name for the new affinity group, and click OK.
  5. In the new affinity group's details page, click the EDIT GROUP AGENTS button.
  6. In the lower section, Agents not part of an affinity group, click the checkboxes by the agent names to add to the group, and click ADD TO GROUP.
  7. Click the Return to Affinity Group Link.
  8. As with the agents, click the EDIT GROUP SERVERS button to open the server lists and look at the list in the lower section of servers which do not currently belong to the affinity group. Click the checkboxes by the server names to add to the group, and click ADD TO GROUP.
Once both servers and agents have been added to the affinity group, the group is fully configured.

4.5. Managing Partition Events

When an agent is assigned to be managed by a server, that is a partition. Partition events are almost like log messages that occur whenever a change in the partition configuration occurs.

4.5.1. Viewing Partition Events

The partition events log is accessed in the high availability configuration.
  1. Click the Administration tab in the top menu.
  2. In the Topology menu table on the left, select the Partition Events item.
  3. The partition events page lists all of the events that have been recorded. (Table 7, “Partition Events Entries” describes the different fields.) Click the type name of any partition event opens up that record with more information about how the partition assignments were affected.
    The partition events log can be filtered to display entries which match the type, status, or details in the event record.
There are basically four categories of partition events that are recorded.
  • Affinity group changes
  • Agent events
  • Server events
  • Partition changes
All of the recorded events are listed in Table 6, “Types of Partition Events”.

Table 6. Types of Partition Events

Partition Event Description
Affinity Group Changes
AFFINITY_GROUP_CHANGE Registers a change in the agent or server assignments for an affinity group or that a group was added.
AFFINITY_GROUP_DELETE Registers an affinity group was deleted from the JBoss ON configuration.
AGENT_AFFINITY_GROUP_ASSIGN Registers that an agent was added to an affinity group.
AGENT_AFFINITY_GROUP_REMOVE Registers that an agent was removed from an affinity group.
SERVER_AFFINITY_GROUP_ASSIGN Registers that a server was added to an affinity group.
SERVER_AFFINITY_GROUP_REMOVE Registers that a server was removed from an affinity group.
Agent Events
AGENT_CONNECT Shows that a JBoss ON agent was started.
AGENT_SHUTDOWN Shows that a JBoss ON agent was stopped.
AGENT_LEAVE Shows that a JBoss ON agent was permanently removed from the JBoss ON configuration.
AGENT_REGISTRATION Shows that a new JBoss ON agent was added to the JBoss ON configuration.
Server Events
SERVER_DELETION Shows that a JBoss ON server was permanently removed from the JBoss ON configuration.
SERVER_COMPUTE_POWER_CHANGE
OPERATION_MODE_CHANGE Shows that a server went stopped, was started, or was newly installed. The type also shows how the mode transitioned (such as server.example.com: DOWN --> NORMAL).
Partition Changes
SYSTEM_INITIATED_PARTITION Shows that JBoss ON initiated a repartition of the servers.
ADMIN_INITIATED_PARTITION Shows that a JBoss ON user initiated a repartition of the servers.

Table 7. Partition Events Entries

Field Description
Execution Time The time of the partition event.
Type Shows the partition event type. This indicates what happened in the system affecting agent partition.
Details Gives detailed information about the partition event; the type of information given varies based on the partition event type.
Initiated By Shows the name of the user who initiated the action generating the partition event. Events initiated by the JBoss ON server itself show they were initiated by admin.
Execution Status Shows low for status descriptions. There are four different status settings:
  • Audit shows that a change was made, but not an event that affects the partition topology.
  • Immediate shows that a partition change was made at the time of the event.
  • Requested shows that a partition change was requested and deferred until the next execution of the JBoss ON server cloud job (usually once a minute). Repartition requests usually have a requested status.
  • Completed shows that a change has been completed.

4.5.2. Removing Partition Events

There are two ways to remove partition event records:
  • By selecting individual records and click REMOVE SELECTED
  • By clicking the PURGE ALL to remove all events
Removing Partition Events

Figure 3. Removing Partition Events