Chapter 5. Design and Development

5.1. Overview

The source code for the Lambda Air application is made available in a public github repository. This chapter briefly covers each microservice and its functionality while reviewing the pieces of the software stack used in the reference architecture.

5.2. Resource Limits

OpenShift allows administrators to set constraints to limit the number of objects or amount of compute resources that are used in each project. While these constraints apply to projects in the aggregate, each pod may also request minimum resources and/or be constrained with limits on its memory and CPU use.

The OpenShift template provided in the project repository uses this capability to request that at least 20% of a CPU core and 200 megabytes of memory be made available to its container. Twice the processing power and four times the memory may be provided to the container, if necessary and available, but no more than that will be assigned.

resources:
  limits:
    cpu: "400m"
    memory: "800Mi"
  requests:
    cpu: "200m"
    memory: "200Mi"

When the fabric8 Maven plugin is used to create the image and direct edits to the deployment configuration are not convenient, resource fragments can be used to provide the desired snippets. This application provides deployment.yml files to leverage this capability and set resource requests and limits on Spring Boot projects:

spec:
   replicas: 1
   template:
     spec:
       containers:
         - resources:
             requests:
               cpu: '200m'
               memory: '400Mi'
             limits:
               cpu: '400m'
               memory: '800Mi'

Control over the memory and processing use of individual services is often critical. Proper configuration of these values, as specified above, is seamless to the deployment and administration process. However, it can be helpful to set up resource quotas in projects for the purpose of enforcing their inclusion in pod deployment configurations.

5.3. Spring Boot REST Service

5.3.1. Overview

The Airports service is the simplest microservice of the application, which makes it a good point of reference for building a basic Spring Boot REST service.

5.3.2. Spring Boot Application Class

To designate a Java project as a Spring Boot application, include a Java class that is annotated with SpringBootApplication and implements the default Java main method:

package com.redhat.refarch.obsidian.brownfield.lambdaair.airports;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class RestApplication
{
	public static void main(String[] args)
	{
		SpringApplication.run( RestApplication.class, args );
	}
}

It is also good practice to declare the application name, which can be done as part of the common application properties. This application uses application.yml files that begin with each project’s name:

spring:
  application:
    name: airports

5.3.3. Maven Project File

Each microservice project includes a Maven POM file, which in addition to declaring the module properties and dependencies, also includes a profile definition to use fabric8-maven-plugin to create and deploy an OpenShift image.

The POM file uses a property to declare the base image containing the operating system and Java Development Kit (JDK). All the services in this application build on top of a Red Hat Enterprise Linux (RHEL) base image, containing a supported version of OpenJDK:

<properties>
...
    <fabric8.generator.from>registry.access.redhat.com/redhat-openjdk-18/openjdk18-openshift</fabric8.generator.from>
</properties>

To easily include the dependencies for a simple Spring Boot application that provides a REST service, declare the following two artifacts:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-tomcat</artifactId>
</dependency>

Every service in this application also declares a dependency on the Spring Boot Actuator component, which includes a number of additional features to help monitor and manage your application.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

When a dependency on the Actuator is declared, fabric8 generates default OpenShift health probes that communicate with Actuator services to determine whether a service is running and ready to service requests:

    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /health
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 180
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /health
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1

5.3.4. Spring Boot REST Controller

To receive and process REST requests, include a Java class annotated with RestController:

...
import org.springframework.web.bind.annotation.RestController;

@RestController
public class Controller

Specify the listening port for this service in the application properties:

server:
  port: 8080

Each REST operation is implemented by a Java method. Business operations typically require specifying request arguments:

@RequestMapping( value = "/airports", method = RequestMethod.GET )
public Collection<Airport> airports(
		@RequestParam( value = "filter", required = false ) String filter)
{
	...

5.3.5. Startup Initialization

The Airports service uses eager initialization to load airport data into memory at the time of startup. This is implemented through an ApplicationListener that listens for a specific type of event:

import org.springframework.context.ApplicationListener;
import org.springframework.context.event.ContextRefreshedEvent;
import org.springframework.stereotype.Component;

@Component
public class ApplicationInitialization implements ApplicationListener<ContextRefreshedEvent>
{
	@Override
	public void onApplicationEvent(ContextRefreshedEvent contextRefreshedEvent)

5.4. Ribbon and Load Balancing

5.4.1. Overview

The Flights service has a similar structure to that of the Airports service, but relies on, and calls the Airports service. As such, it makes use of Ribbon and the generated OpenShift service for high availability.

5.4.2. RestTemplate and Ribbon

To quickly and easily declare the required dependencies to use Ribbon, add the following artifact as a Maven dependency:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-ribbon</artifactId>
</dependency>

This application also makes use of the Jackson JSR 310 libraries to correctly serialize and deserialize Java 8 date objects:

<dependency>
    <groupId>com.fasterxml.jackson.datatype</groupId>
    <artifactId>jackson-datatype-jsr310</artifactId>
    <version>2.8.8</version>
</dependency>

Declare a load balanced RestTemplate and use injection to assign it to a field:

@LoadBalanced
@Bean
RestTemplate restTemplate()
{
	return new RestTemplate();
}

@Autowired
private RestTemplate restTemplate;

For outgoing calls, simply use the restTemplate field:

Airport[] airportArray = restTemplate.getForObject( "http://zuul/airports/airports", Airport[].class );

The service address provided as the host part of the URL is resolved through Ribbon, based on values provided in application properties:

zuul:
  ribbon:
    listOfServers: zuul:8080

In this case, Ribbon expects a list of statically defined service addresses, but a single one is provided with the hostname of zuul with port 8080. Zuul uses the second part of the address, the root web context, to redirect the request through statically or dynamic routing, as explained later in this document.

The provided hostname of zuul is the OpenShift service name, and is resolved to the cluster IP address of the service, then routed to an internal OpenShift load balancer. The OpenShift service name is determined when a service is created using the oc tool, or when deploying an image using the fabric8 Maven plugin, it is declared in the service yaml file.

Ribbon is effectively not load balancing requests, but rather sending them to an OpenShift internal load balancer, which is aware of replication and failure of service instances, and can redirect the request properly.

5.5. Spring Boot MVC

5.5.1. Overview

The Presentation service makes minimal use of Spring MVC to serve the client-side HTML application to calling browsers.

5.5.2. ModelAndView Mapping

After including the required Maven dependencies, the Presentation service provides a simple Controller class to serve HTML content:

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.servlet.ModelAndView;

@Controller
@RequestMapping( "/" )
public class WebRequestController
{
	@GetMapping
	public ModelAndView list()
	{
		return new ModelAndView( "index" );
	}
}

This class declares that the service will be listening to HTTP GET requests on the root context of the application. It serves the index file, provided as an index.html in the templates directory, back to the browser.

Templates typically allow parameter substitution, but as previously mentioned, this service makes very minimal use of Spring MVC functionality.

5.5.3. Bower Package Manager

The Presentation service uses Bower package manager to declare, download and update JavaScript libraries. Libraries, versions and components to download (or rather, those to ignore) are specified in a bower JSON file. Running bower install downloads the declared libraries to the bower_components directory, which can in turn be imported in the HTML application.

5.5.4. PatternFly

The HTML application developed for this reference architecture uses PatternFly to provide consistent visual design and improved user experience.

PatternFly stylesheets are imported in the main html:

    <!-- PatternFly Styles -→
    <link href="bower_components/patternfly/dist/css/patternfly.min.css" rel="stylesheet"
        media="screen, print"/>
    <link href="bower_components/patternfly/dist/css/patternfly-additions.min.css" rel="stylesheet"
        media="screen, print"/>

The associated JavaScript is also included in the header:

    <!-- PatternFly -→
    <script src="bower_components/patternfly/dist/js/patternfly.min.js"></script>

5.5.5. JavaScript

The presentation tier of this application is built in HTML5 and relies heavily on JavaScript. This includes using ajax calls to the API gateway, as well as minor changes to HTML elements that visible and displayed to the user.

5.5.5.1. jQuery UI

Some features of the jQuery UI library, including autocomplete for airport fields, are utilized in the presentation layer.

5.5.5.2. jQuery Bootstrap Table

To display flight search results in a dynamic table with pagination, and the ability to expand each row to reveal more data, a jQuery Bootstrap Table library is included and utilized.

5.6. Hystrix

5.6.1. Overview

The Presentation service includes a second listening controller, this time a REST controller, that acts as an API gateway. The API gateway makes simple REST calls to the Airports service, similar to the previously discussed Flights service, but also calls the Sales service to get pricing information and uses a different pattern for this call. Hystrix is used to avoid a large number of hung threads and lengthy timeouts when the Sales service is down. Instead, flight information can be returned without providing a ticket price. The reactive interface of Hystrix is also leveraged to implement parallel processing.

5.6.2. Circuit Breaker

Hystrix provides multiple patterns for the use of its API. The Presentation service uses Hystrix commands for its outgoing calls to Sales. This is implemented as a Hystrix command:

private class PricingCall extends HystrixCommand<Itinerary>
{
	private Flight flight;

	PricingCall(Flight flight)
	{
		super( HystrixCommandGroupKey.Factory.asKey( "Sales" ), HystrixThreadPoolKey.Factory.asKey( "SalesThreads" ) );
		this.flight = flight;
	}

	@Override
	protected Itinerary run() throws Exception
	{
		try
		{
			return restTemplate.postForObject( "http://zuul/sales/price", flight, Itinerary.class );
		}
		catch( Exception e )
		{
			logger.log( Level.SEVERE, "Failed!", e );
			throw e;
		}
	}

	@Override
	protected Itinerary getFallback()
	{
		logger.warning( "Failed to obtain price, " + getFailedExecutionException().getMessage() + " for " + flight );
		return new Itinerary( flight );
	}
}

After being instantiated and provided a flight for pricing, the command takes one of two routes. When successful and able to reach the service being called, the run method is executed which uses the now-familiar pattern of calling the service through Ribbon and the OpenShift service abstraction. However, if an error prevents us from reaching the Sales service, getFallback() provides a chance to recover from the error, which in this case involves returning the itinerary without a price.

The fallback scenario can happen simply because the call has failed, but also in cases when the circuit is open (tripped). Configure Hystrix as part of the service properties to specify when a thread should time out and fail, as well as the queue used for concurrent processing of outgoing calls.

To configure the command timeout for a specific command (and not globally), the HystrixCommandKey is required. This defaults to the command class name, which is PricingCall in this implementation.

Configure thread pool properties for this specific thread pool by using the specified thread pool key of SalesThreads.

hystrix.command.PricingCall.execution.isolation.thread.timeoutInMilliseconds: 2000

hystrix:
  threadpool:
    SalesThreads:
      coreSize: 20
      maxQueueSize: 200
      queueSizeRejectionThreshold: 200

5.6.3. Concurrent Reactive Execution

We assume technical considerations have led to the Sales service accepting a single flight object in its API. To reduce lag time and take advantage of horizontal scaling, the service uses Reactive Commands for batch processing of pricing calls.

The API gateway service queries and stores the configured thread pool size as a field:

@Value("${hystrix.threadpool.SalesThreads.coreSize}")
private int threadSize;

The thread size is later used as the batch size for the concurrent calls to calculate the price of a flight:

int batchLimit = Math.min( index + threadSize, itineraries.length );
for( int batchIndex = index; batchIndex < batchLimit; batchIndex++ )
{
	observables.add( new PricingCall( itineraries[batchIndex] ).toObservable() );
}

The Reactive zip operator is used to process the calls for each batch concurrently and store results in a collection. The number of batches depends on the ratio of total flights found to the batch size, which is set to 20 in this service configuration.

5.7. OpenShift ConfigMap

5.7.1. Overview

While considering the concurrent execution of pricing calls, it should be noted that the API gateway is itself multi-threaded, so the batch size is not the final determinant of the thread count. In this example of a batch size of 20, with a maximum queue size of 200 and the same threshold leading to rejection, receiving more than 10 concurrent query calls can lead to errors. These values should be fine-tuned based on realistic expectations of load as well as the horizontal scaling of the environment.

This configuration can be externalized by creating a ConfigMap for each OpenShift environment, with overriding values provided in a properties file that is then provided to all future pods.

5.7.2. Property File Mount

Refer to the steps in creating the environment for detailed instructions on how to create an external application properties and mounting it in the pod. The property file is placed in the application class path and the provided values supersede those of the application.

5.8. Zipkin

5.8.1. Overview

This reference architecture uses Spring Sleuth to collect and broadcast tracing data to OpenZipkin, which is deployed as an OpenShift service and backed by a persistent MySQL database image. The tracing data can be queried from the Zipkin console, which is exposed through an OpenShift route. Logging integration is also possible, although not demonstrated, to use trace IDs to tie in together the distributed execution of the same business request.

5.8.2. MySQL Database

This reference architecture uses the provided and support OpenShift MySQL image to store persistent zipkin data.

5.8.2.1. Persistent Volume

To enable persistent storage for the MySQL database image, this reference architecture creates and mounts a logical volume that is expose through NFS. An OpenShift persistent volume exposes the storage to the image. Once the storage is set up and shared by the NFS server:

$ oc create -f zipkin-mysql-pv.json
persistentvolume "zipkin-mysql-data" created

5.8.2.2. MySQL Image

This reference architecture provides a single OpenShift template to create a database image, the database schema required for OpenZipkin, and the OpenZipkin image itself. This template relies on the MySQL image definition that is available by default in the openshift project.

5.8.2.3. Database Initialization

This reference architecture demonstrates the use of lifecycle hooks to initialize a database after the pod has been created. Specifically, a post hook is used as follows:

       recreateParams:
         post:
           failurePolicy: Abort
           execNewPod:
             containerName: mysql
             command:
             - /bin/sh
             - -c
             - hostname && sleep 10 && /opt/rh/rh-mysql57/root/usr/bin/mysql -h $DATABASE_SERVICE_NAME -u $MYSQL_USER -D $MYSQL_DATABASE -p$MYSQL_PASSWORD -P 3306 < /docker-entrypoint-initdb.d/init.sql && echo Initialized database
             env:
             - name: DATABASE_SERVICE_NAME
               value: ${DATABASE_SERVICE_NAME}
             volumes:
             - mysql-init-script

Notice that the hook uses the command line mysql utility to run the SQL script located /docker-entrypoint-initdb.d/init.sql. Some database images standardize on this location for initialization, in which case a lifecycle hook is not required.

The SQL script to create the schema is embedded in the template as a config map. It is then declared as a volume and mounted at its final path under /docker-entrypoint-initdb.d/.

5.8.3. OpenZipkin Image

The template uses the image provided by OpenZipkin:

image: openzipkin/zipkin:1.19.2

Required parameters for OpenZipkin to access the associated MySQL database are either configured or generated as part of the same template. Database passwords are randomly generated by OpenShift as part of the template and stored in a secret, which makes them inaccessible to users and administrators in the future. That is why a template message is printed to allow a one-time access to the database password for monitoring and troubleshooting purposes.

To create the Zipkin service:

$ oc new-app -f LambdaAir/Zipkin/zipkin-mysql.yml
--> Deploying template "lambdaair/" for "zipkin-mysql.yml" to project lambdaair


     ---------
     MySQL database service, with persistent storage. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/mysql-container/blob/master/5.7/README.md.

     NOTE: Scaling to more than one replica is not supported. You must have persistent volumes available in your cluster to use this template.

     The following service(s) have been created in your project: zipkin-mysql.

            Username: zipkin
            Password: Y4hScBSPH5bAhDL2
       Database Name: zipkin
      Connection URL: mysql://zipkin-mysql:3306/

     For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/mysql-container/blob/master/5.7/README.md.


     * With parameters:
        * Memory Limit=512Mi
        * Namespace=openshift
        * Database Service Name=zipkin-mysql
        * MySQL Connection Username=zipkin
        * MySQL Connection Password=Y4hScBSPH5bAhDL2 # generated
        * MySQL root user Password=xYVNsuRXRV5xqu4A # generated
        * MySQL Database Name=zipkin
        * Volume Capacity=1Gi
        * Version of MySQL Image=5.7

--> Creating resources ...
    secret "zipkin-mysql" created
    service "zipkin" created
    service "zipkin-mysql" created
    persistentvolumeclaim "zipkin-mysql" created
    configmap "zipkin-mysql-cnf" created
    configmap "zipkin-mysql-initdb" created
    deploymentconfig "zipkin" created
    deploymentconfig "zipkin-mysql" created
    route "zipkin" created
--> Success
    Run 'oc status' to view your app.

5.8.4. Spring Sleuth

While the Zipkin service allows distributed tracing data to be aggregated, persisted and used for reporting, this application relies on Spring Sleuth to correlate calls and send data to Zipkin.

Integration with Ribbon and other framework libraries make it very easy to use Spring Sleuth in the application. Include the libraries by declaring a dependency in the project Maven file:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>

Also specify in the application properties the percentage of requests that should be traced, as well as the address to the zipkin server. Once again, we rely on the OpenShift service abstract to reach zipkin.

spring:
  sleuth:
    sampler:
      percentage: 1.0
  zipkin:
    baseUrl: http://zipkin/

The percentage value of 1.0 means 100% of requests will be captured.

These two steps are enough to collect tracing data, but a Tracer object can also be injected into the code for extended functionality. While every remote call can produce and store a trace by default, adding a tag can help to better understand zipkin reports. The service also creates and demarcates tracing spans of interest to collect more meaningful tracing data.

5.8.4.1. Baggage Data

While Spring Sleuth is primarily intended as a distributed tracing tool, its ability to correlate distributed calls can have other practical uses as well. Every created span allows the attachment of arbitrary data, called a baggage item, that will be automatically inserted into the HTTP header and seamlessly carried along with the business request from service to service, for the duration of the span. This application is interested in making the original caller’s IP address available to every microservice. In an OpenShift environment, the calling IP address is stored in the HTTP header under a standard key. To retrieve and set this value on the span:

querySpan.setBaggageItem( "forwarded-for", request.getHeader( "x-forwarded-for" ) );

This value will later be accessible from any service within the same call span under the header key of baggage-forwarded-for. It is used by the Zuul service in a Groovy script to perform dynamic routing.

5.9. Zuul

5.9.1. Overview

This reference architecture uses Zuul as a central proxy for all calls between microservices. By default, the service uses static routing as defined in its application properties:

 zuul:
   routes:
     airports:
       path: /airports/**
       url: http://airports:8080/
     flights:
       path: /flights/**
       url: http://flights:8080/
     sales:
       path: /sales/**
       url: http://sales:8080/

The path provided in the above rules uses the first part of the web address to determine the service to be called, and the rest of the address as the context.

5.9.2. A/B Testing

To implement A/B testing, the Salesv2 service introduces a minor change in the algorithm for calculating fares. Dynamic routing is provided by Zuul through a filter that filters some of the requests.

Calls to other services are not filtered:

if( !RequestContext.currentContext.getRequest().getRequestURI().matches("/sales.*") )
{
    //Won't filter this request URL
    false
}

Only those calls to the Sales service that originate from an IP address ending in an odd digit are filtered:

String caller = new HTTPRequestUtils().getHeaderValue("baggage-forwarded-for");
logger.info("Caller IP address is " + caller)
int lastDigit = caller.reverse().take(1) as Integer
if( lastDigit % 2 == 0 )
{
    //Even IP address won't be filtered
    false
}
else
{
    //Odd IP address will be filtered
    true
}

If the caller has an odd digit at the end of their IP address, the request is rerouted. That means the run method of the filter is executed, which changes the route host:

@Override
Object run() {
    println("Running filter")
    RequestContext.currentContext.routeHost = new URL("http://salesv2:8080")
}

To enable dynamic routing without changing application code, shared storage is made available to the OpenShift nodes and a persistent volume is created and claimed. With the volume set up and the groovy filter in place, the OpenShift deployment config can be adjusted administratively to mount a directory as a volume:

$ oc volume dc/zuul --add --name=groovy --type=persistentVolumeClaim --claim-name=groovy-claim --mount-path=/groovy

This results in all groovy scripts under the groovy directory being found. The zuul application code anticipates the introduction of dynamic routing filters by seeking and applying any groovy script under this path:

for( Resource resource : new PathMatchingResourcePatternResolver().getResources( "file:/groovy/.groovy" ) ) { logger.info( "Found and will load groovy script " + resource.getFilename() ); sources.add( resource ); } if( sources.size() == 1 ) { logger.info( "No groovy script found under /groovy/.groovy" );
}