Chapter 23. Enabling High Availability in Fuse Fabric

Abstract

When all of your servers and clients are deployed within the same fabric, you can use an alternative mechanism for implementing high availability cluster, which works by exploiting the fabric registry. Because all the parts of the application must be deployed on the same fabric, this mechanism is suitable for deployment on a LAN.

23.1. Load Balancing Cluster

23.1.1. Introduction to Load Balancing

Overview

The fabric load balancing mechanism exploits the fact that fabric provides a distributed fabric registry, which is accessible to all of the container in the fabric. This makes it possible to use the fabric registry as a discovery mechanism for locating WS endpoints in the fabric. By storing all of the endpoint addresses belonging to a particular cluster under the same registry node, any WS clients in the fabric can easily discover the location of the endpoints in the cluster.

Fuse Fabric

A fabric is a distributed collection of containers that share a common database of configuration settings (the fabric registry). Every container in the fabric has a fabric agent deployed in it, which manages the container and redeploys applications to the container whenever a new profile is assigned to the container (a profile is the basic deployment unit in a fabric).

Load-balancing cluster

Figure 23.1, “Fabric Load Balancing for Apache CXF” gives an overview of the fabric load balancing mechanism for Apache CXF endpoints.

Figure 23.1. Fabric Load Balancing for Apache CXF

fabric ha 01

In this example, two WS servers are created, with the URIs, http://localhost:8185/Foo and http://localhost:8186/Foo. For both of these servers, the load balancer feature is configured to store the cluster endpoints under the path, demo/lb, in the fabric registry.

Now, when the WS client starts, it is configured to look up the cluster path, demo/lb, in the fabric registry. Because the demo/lb path is associated with multiple endpoint addresses, fabric implements a random load balancing algorithm to choose one of the available URIs to connect to.

FabricLoadBalancerFeature

The fabric load balancer feature is implemented by the following class:

io.fabric8.cxf.FabricLoadBalancerFeature

The FabricLoadBalancerFeature class exposes the following bean properties:

fabricPath
This property specifies a node in the fabric registry (specified relative to the base node, /fabric/cxf/endpoints) that is used to store the data for a particular endpoint cluster.
curator
A proxy reference to the OSGi service exposed by the fabric agent (of type, org.apache.curator.framework.CuratorFramework).
maximumConnectionTimeout
The maximum length of time to attempt to connect to the fabric agent, specified in milliseconds. The default is 10000 (10 seconds).
connectionRetryTime
How long to wait between connection attempts, specified in milliseconds. The default is 100.
loadBalanceStrategy
By implementing a bean of type io.fabric8.cxf.LoadBalanceStrategy and setting this property, you can customise the load balancing algorithm used by the load balancing feature.

Prerequisites

To use the fabric load balancer feature in your application, your project must satisfy the following prerequisites:

Maven dependency

The fabric load balancer feature requires the fabric-cxf Maven artifact. Add the following dependency to your project’s POM file:

<dependency>
    <groupId>io.fabric8</groupId>
    <artifactId>fabric-cxf</artifactId>
    <version>7.1.0.fuse-710023-redhat-00001</version>
</dependency>

OSGi package import

If you are packaging your project as an OSGi bundle, you must add io.fabric8.cxf to the list of imported packages. For example, using the Maven bundle plug-in, you can specify this package import by adding io.fabric8.cxf to the comma-separated list in the Import-Package element, as follows:

<plugin>
    <groupId>org.apache.felix</groupId>
    <artifactId>maven-bundle-plugin</artifactId>
    <version>2.2.0</version>
    <extensions>true</extensions>
    <configuration>
        <instructions>
            <Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName>
            <Import-Package>
                ...
                io.fabric8.cxf,
                *
            </Import-Package>
            ...
        </instructions>
    </configuration>
</plugin>

Fabric deployment

When you come to deploy your application into a Red Hat Fuse container, you must deploy it into a fabric. The fabric load balancer feature is not supported in a standalone container.

Required feature

The fabric load balancer requires the fabric-cxf Apache Karaf feature to be installed in the container. In the context of a fabric, this means you must add the fabric-cxf feature to the relevant deployment profile. For example, if you are using the cxf-lb-server profile to deploy a load-balancing WS server, you can add the fabric-cxf feature by entering the following console command:

JBossFuse:karaf@root> profile-edit -f fabric-cxf cxf-lb-server

23.1.2. Configure the Server

Overview

To configure a WS server to use fabric load balancing, you must configure a fabric load balancer feature and install it in the default Apache CXF bus instance. This section describes how to configure the load balancer feature in Spring XML and in blueprint XML.

Prerequisites

For the basic prerequisites to build a fabric load-balancing WS server, see the section called “Prerequisites”.

Blueprint XML

The following fragment from a blueprint XML file shows how to add the fabric load balancer feature, FabricLoadBalancerFeature, to an Apache CXF bus. Any Apache CXF endpoints subsequently created on this bus will automatically have the load-balancer feature enabled.

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
           ...
           xmlns:cxf="http://cxf.apache.org/blueprint/core"
           ...
>
    ...
    <reference id="curator"
               interface="org.apache.curator.framework.CuratorFramework" />

    <!-- The FabricFailOverFeature will try to access other service endpoint with round rad -->
    <bean id="fabricLoadBalancerFeature" class="io.fabric8.cxf.FabricLoadBalancerFeature">
        <property name="curator" ref="curator" />
        <property name="fabricPath" value="cxf/demo" />
    </bean>

    <!-- setup the feature on the bus to help publish the services to the fabric-->
    <cxf:bus bus="cxf">
        <cxf:features>
            <ref component-id="fabricLoadBalancerFeature"/>
        </cxf:features>
    </cxf:bus>
    ...
</blueprint>

The following beans are used to install the fabric load-balancer feature:

curator reference
The curator reference is a proxy of the local fabric agent, which it accesses through the org.apache.curator.framework.CuratorFramework interface. This reference is needed in order to integrate the load balancer feature with the underlying fabric.
FabricLoadBalancerFeature bean

The FabricLoadBalancerFeature bean is initialized with the following properties:

curator
A reference to the Apache Curator client, CuratorFramework.
fabricPath
The path of a node in the fabric registry, where the cluster data is stored (for example, the addresses of the endpoints in the load-balancing cluster). The node path is specified relative to the base node, /fabric/cxf/endpoints.
Apache CXF bus
The cxf:bus element installs the fabric load balancer feature in the default bus instance.

Example using Blueprint XML

Example 23.1, “WS Server with Fabric Load Balancer Feature” shows a complete Blueprint XML example of a WS endpoint configured to use the fabric load balancing feature.

Example 23.1. WS Server with Fabric Load Balancer Feature

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xmlns:cxf="http://cxf.apache.org/blueprint/core"
           xmlns:jaxws="http://cxf.apache.org/blueprint/jaxws"
           xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.0.0"
           xsi:schemaLocation="
           http://www.osgi.org/xmlns/blueprint/v1.0.0
           https://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
           http://cxf.apache.org/schemas/blueprint/core
           http://cxf.apache.org/schemas/blueprint/core.xsd
           http://cxf.apache.org/blueprint/jaxws
           http://cxf.apache.org/blueprint/jaxws.xsd
           ">

    <reference id="curator" interface="org.apache.curator.framework.CuratorFramework" />

    <!-- The FabricFailOverFeature will try to access other service endpoint with round rad -->
    <bean id="fabicLoadBalancerFeature" class="io.fabric8.cxf.FabricLoadBalancerFeature">
        <property name="curator" ref="curator" />
        <property name="fabricPath" value="cxf/demo" />
    </bean>

    <!-- setup the feature on the bus to help publish the services to the fabric-->
    <cxf:bus bus="cxf">
        <cxf:features>
            <ref component-id="fabicLoadBalancerFeature"/>
        </cxf:features>
    </cxf:bus>

    <bean id="hello1" class="io.fabric8.demo.cxf.server.HelloImpl">
        <property name="hello" value="Hi"/>
    </bean>

    <bean id="hello2" class="io.fabric8.demo.cxf.server.HelloImpl">
        <property name="hello" value="Hello"/>
    </bean>

    <!--
    TODO: We should use address in the form of bind.address:$[app1.port]/server/server1, but currently only fuseenterprise
    has appX.port system properties defined
    -->

    <!-- publish the service with the address of fail, cxf client will get the simulated IOException -->
    <jaxws:server id="service1" serviceClass="io.fabric8.demo.cxf.Hello" address="http://localhost:9000/server/server1">
        <jaxws:serviceBean>
             <ref component-id="hello1" />
        </jaxws:serviceBean>
    </jaxws:server>

    <jaxws:server id="service2" serviceClass="io.fabric8.demo.cxf.Hello" address="http://localhost:9000/server/server2">
        <jaxws:serviceBean>
            <ref component-id="hello2" />
        </jaxws:serviceBean>
    </jaxws:server>

</blueprint>

The preceding Spring XML configuration consists of the following main sections:

  • Enabling the fabric load balancing feature—the fabric load balancing feature is installed in the default bus instance, as previously described. In this example, the fabricPath property is set to the value, cxf/demo.
  • Creating the WS endpoints—create the WS endpoints in the usual way, using the jaxws:server element (this can be used as an alternative to the jaxws:endpoint element). By default, this endpoint is automatically associated with the default bus instance, which has load balancing enabled.

23.1.3. Configure the Client

Overview

To configure a WS client to use fabric load balancing, you must install the fabric load balancer feature directly in the client proxy instance. This section describes how to configure the load balancer feature in Blueprint XML, and by programming in Java.

Prerequisites

For the basic prerequisites to build a fabric load-balancing WS client, see the section called “Prerequisites”.

Blueprint XML

The following fragment from a blueprint XML file shows how to add the fabric load balancer feature, FabricLoadBalancerFeature, directly into a WS client proxy instance.

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
           ...
           xmlns:jaxws="http://cxf.apache.org/blueprint/jaxws"
           xmlns:cxf="http://cxf.apache.org/blueprint/core"
           ...
>
    <!-- Create a client proxy, with load balancing enabled -->
    <jaxws:client id="ClientProxyBeanID"
        address="http://dummyaddress"
        serviceClass="SEI">
        <jaxws:features>
            <ref component-id="fabricLoadBalancerFeature" />
        </jaxws:features>
    </jaxws:client>
    ...
    <reference id="curator"
               interface="org.apache.curator.framework.CuratorFramework" />

    <!-- The FabricFailOverFeature will try to access other service endpoint with round rad -->
    <bean id="fabricLoadBalancerFeature" class="io.fabric8.cxf.FabricLoadBalancerFeature">
        <property name="curator" ref="curator" />
        <property name="fabricPath" value="ZKPath" />
    </bean>
    ...
</blueprint>

The fabric load balancer feature is installed directly into the WS client proxy by inserting it as a child of the jaxws:features element (or, as in this case, by inserting a bean reference to the actual instance). The following beans are used to initialise the fabric load-balancer feature:

curator reference
The curator reference is a proxy of the local fabric agent, which it accesses through the org.apache.curator.framework.CuratorFramework interface. This reference is needed in order to integrate the load balancer feature with the underlying fabric.
FabricLoadBalancerFeature bean

The FabricLoadBalancerFeature bean is initialized with the following properties:

curator
A reference to the Apache Curator client, curator.
fabricPath
The path of a node in the fabric registry, where the cluster data is stored (for example, the addresses of the endpoints in the load-balancing cluster). The node path is specified relative to the base node, /fabric/cxf/endpoints.

Java

As an alternative to using XML configuration, you can enable the fabric load balancing feature on the client side by programming directly in Java. The following example shows how to enable fabric load balancing on a proxy for the Hello Web service.

// Java
package io.fabric8.demo.cxf.client;

import org.apache.cxf.feature.AbstractFeature;
import org.apache.cxf.frontend.ClientProxyFactoryBean;
import org.apache.cxf.jaxws.JaxWsProxyFactoryBean;
import io.fabric8.cxf.FabricLoadBalancerFeature;
import io.fabric8.demo.cxf.Hello;

import java.util.ArrayList;
import java.util.List;

public class Client {

    private Hello hello;

    public void initializeHelloProxy() {
        // The feature will try to create a zookeeper client itself
        // by checking the system property of zookeeper.url
        FabricLoadBalancerFeature feature = new FabricLoadBalancerFeature();
        // Feature will use this path to locate the service
        feature.setFabricPath("demo/lb");

        ClientProxyFactoryBean clientFactory = new JaxWsProxyFactoryBean();
        clientFactory.setServiceClass(ClientProxyFactoryBean.class);
        // The address is not the actual address that the client will access
        clientFactory.setAddress("http://dummyaddress");

        List<AbstractFeature> features = new ArrayList<AbstractFeature>();
        features.add(feature);
        // we need to setup the feature on the client factory
        clientFactory.setFeatures(features);

        // Create the proxy of Hello
        hello = clientFactory.create(Hello.class);
    }

    public static void main(String args[]) {
        initializeHelloProxy();
        ...
    }
}

In this example, the fabricPath property is set to the value, demo/lb (which matches the example value used by the server in Example 23.1, “WS Server with Fabric Load Balancer Feature”).

The address that the client proxy accesses is set to a dummy value, http://dummyaddress, because this value is not used. When the client is initialized, the load balancer feature substitutes the address value retrieved from the demo/lb node of the fabric registry.

23.2. Failover Cluster

Overview

A failover cluster in Fuse Fabric is based on an ordered list of WS endpoints that are registered under a particular node in the fabric registry. A client detects the failure of a master endpoint by catching the exception that occurs when it tries to make an invocation. When that happens, the client automatically moves to the next available endpoint in the cluster.

Failover cluster

Figure 23.2, “Fabric Failover for Apache CXF” gives an overview of the fabric failover mechanism for Apache CXF endpoints.

Figure 23.2. Fabric Failover for Apache CXF

fabric ha 02

In this example, two WS servers are created, with the URIs, http://localhost:8185/Foo and http://localhost:8186/Foo. In both servers, the failover feature is configured to store the cluster endpoints under the path, demo/fo, in the fabric registry. The cluster endpoints stored under demo/fo are ordered. The first endpoint in the cluster is the master and all of the other endpoints are slaves.

The failover algorithm works as follows:

  1. When the WS client starts, it is configured to look up the cluster path, demo/fo, in the fabric registry. The failover feature initially returns the first address registered under demo/fo (the master).
  2. At some point, the master server could fail. The client determines whether the master has failed by catching the exception that occurs when it tries to make an invocation: if the caught exception matches one of the exceptions in a specified list (by default, just the java.io.IOException), the master is deemed to have failed and the client now ignores the corresponding address entry under demo/fo.
  3. The client selects the next address entry under demo/fo and attempts to connect to that server. Assuming that this server is healthy, it is effectively the new master.
  4. At some point in the future, if the failed old master is restarted successfully, it creates a new address entry under demo/foafter the existing entries, and is then available to clients, in case the other server (or servers) fail.

FabricFailOverFeature

The fabric failover feature is implemented by the following class:

io.fabric8.cxf.FabricFailOverFeature

The FabricFailOverFeature class exposes the following bean properties:

fabricPath
This property specifies a node in the fabric registry (specified relative to the base node, /fabric/cxf/endpoints) that is used to store the data for a particular endpoint cluster.
curator
A proxy reference to the OSGi service (of type, org.apache.curator.framework.CuratorFramework) for the Apache Curator client, which is exposed by the fabric agent.
maximumConnectionTimeout
The maximum length of time to attempt to connect to the fabric agent, specified in milliseconds. The default is 10000 (10 seconds).
connectionRetryTime
How long to wait between connection attempts, specified in milliseconds. The default is 100.
exceptions

A semicolon-separated list of exceptions that signal to the client that a server has failed. If not set, this property defaults to java.io.IOException.

For example, you could set the exceptions property to a value like the following:

java.io.IOException;javax.xml.ws.soap.SOAPFaultException

Blueprint XML

The configuration of WS servers and WS clients in the failover case is similar to the load balancing case (see Section 23.1.2, “Configure the Server” and Section 23.1.3, “Configure the Client”), except that instead of instantiating and referencing a FabricLoadBalancerFeature bean, you must instantiate and reference a FabricFailOverFeature bean.

In blueprint XML you can create a FabricFailOverFeature bean instance as follows:

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
           ...
           xmlns:cxf="http://cxf.apache.org/blueprint/core"
           ...
>
    ...
    <!-- Reference the fabric agent -->
    <reference id="curator"
               interface="org.apache.curator.framework.CuratorFramework" />

    <!-- Create the Fabric load balancer feature -->
    <bean id="failoverFeature"
          class="io.fabric8.cxf.FabricFailOverFeature">
        <property name="curator" ref="curator" />
        <property name="fabricPath" value="ZKPath" />
    </bean>
    ...
</blueprint>

Remember to customise the value of the fabricPath property and to reference the appropriate bean ID (failoverFeature in the preceding example).