Red Hat JBoss Data Virtualization for OpenShift

Red Hat JBoss Middleware for OpenShift 3

Using Red Hat JBoss Data Virtualization for OpenShift

Red Hat JBoss Middleware for OpenShift Documentation Team

Abstract

Guide to using Red Hat JBoss Data Virtualization for OpenShift

Chapter 1. Introduction

1.1. What Is Red Hat JBoss Data Virtualization?

Red Hat JBoss Data Virtualization(JDV) is a lean, virtual data integration solution that unlocks trapped data and delivers it as easily consumable, unified, and actionable information. Red Hat JBoss Data Virtualization makes data spread across physically diverse systems — such as multiple databases, XML files, and Hadoop systems — appear as a set of tables in a local database.

Chapter 2. Before You Begin

2.1. Comparison: JDV for OpenShift Image and Red Hat JDV

JDV for OpenShift image is based on Red Hat JBoss Data Virtualization 6.3. There are some differences in functionality between the JDV for OpenShift image and Red Hat JBoss Data Virtualization:

  • Cached results are automatically replicated to all members of the JDV for OpenShift cluster.

In addition, the JDV for OpenShift image is built on the Red Hat JBoss Enterprise Application Platform (EAP) for OpenShift image. As a result, the same differences exist for the JDV for OpenShift image. For more information on the EAP for OpenShift differences, see the Comparison: EAP and EAP for OpenShift Image section in the Red Hat JBoss Enterprise Application Platform for OpenShift Guide.

2.2. Version Compatibility and Support

See the xPaaS part of the OpenShift and Atomic Platform Tested Integrations page for details about OpenShift image version compatibility.

2.3. Initial Setup

The Tutorials in this guide follow on from and assume an OpenShift instance similar to that created in the OpenShift Primer.

Chapter 3. Get Started

3.1. Using the JDV for OpenShift Image Streams and Application Templates

Red Hat JBoss Middleware for OpenShift images are pulled on demand from the Red Hat Registry:
registry.access.redhat.com

3.2. Preparing JDV Project Artifacts

The EAP for OpenShift image, on which the JDV for OpenShift image is built, extends database support in OpenShift using various artifacts. These artifacts are included in the built image through different mechanisms:
- S2I artifacts that are injected into the image during the S2I process, and
- Runtime artifacts from environment files provided through the OpenShift Secret mechanism.

3.2.1. S2I Artifacts

The S2I artifacts include the virtual databases files, modules, drivers, translators, and additional generic deployments that provide the necessary configuration infrastructure required for the JDV for OpenShift deployment. This configuration is built into the image during the S2I process so that only the data sources and associated resource adapters need to be added at runtime.

3.2.1.1. Virtual Databases (VDB)

VDBs contain aggregated data from multiple disparate data sources. This allows applications to access and query the data as though it is in a single database, using a single uniform API.

Deployment

*.vdb and *-vdb.xml files can be included in a Git repository or in the application binary source.

The JDV for OpenShift image uses the file deployment method for deploying virtual databases. Create an empty marker file in the same directory and with the same name as the VDB with the additional extension .dodeploy. For example, if the VDB file is database.vdb, the marker file must be called database.vdb.dodeploy.

For more information on the file deployment method, see the Deploy a VDB via File Deployment section in the Red Hat JBoss Data Virtualization Administration and Configuration Guide.

3.2.1.2. Modules, Drivers, Translators, and Generic Deployments

There are three options for including these S2I artifacts in the JDV for OpenShift image:

  1. Include the artifact in the application source deployment directory. The artifact is downloaded during the build and injected into the image. This is similar to deploying an application on the EAP for OpenShift image.
  2. Include the CUSTOM_INSTALL_DIRECTORIES environment variable, a list of comma-separated list of directories used for installation and configuration of artifacts for the image during the S2I process. There are two methods for including this information in the S2I:
    2a) An install.sh script in the nominated installation directory. The install script will be executed during the S2I process and will operate with impunity.

    Install.sh script example:

    #!/bin/bash
    
    injected_dir=$1
    source /usr/local/s2i/install-common.sh
    install_deployments ${injected_dir}/injected-deployments.war
    install_modules ${injected_dir}/modules
    configure_drivers ${injected_dir}/drivers.env
    source /usr/local/s2i/install-teiid-common.sh
    configure_translators ${injected_dir}/translators.env

    The install.sh script is responsible for customizing the base JDV for OpenShift image using APIs provided by the install-common.sh, which is contained in the underlying base EAP for OpenShift image, and install-teiid-common.sh, which is contained in the base JDV for OpenShift image. It allows for the greatest amount of flexibility for configuring the JDV for OpenShift image. These shell scripts contain functions that are used by the install.sh script to install and configure the modules, drivers, translators, and generic deployments.

    Shell scriptFunctions contained within script

    install-common.sh

    configure_translators

    install-teiid-common.sh

    install_modules
    configure_drivers
    install_deployments

    Modules

    A module is a logical grouping of classes used for class loading and dependency management. Modules are defined in the EAP_HOME/modules/ directory of the application server. Each module exists as a subdirectory, for example EAP_HOME/modules/org/apache/. Each module directory then contains a slot subdirectory, which defaults to main and contains the module.xml configuration file and any required JAR files.

    module.xml example:

    <?xml version="1.0" encoding="UTF-8"?>
    <module xmlns="urn:jboss:module:1.0" name="org.apache.derby">
        <resources>
            <resource-root path="derby-10.12.1.1.jar"/>
            <resource-root path="derbyclient-10.12.1.1.jar"/>
        </resources>
        <dependencies>
            <module name="javax.api"/>
            <module name="javax.transaction.api"/>
        </dependencies>
    </module>

    Drivers and translators are installed as modules. Use the install_modules function in the install.sh to unpack the respective JAR files. These driver and translator modules are then configured in the install.sh by the configure_drivers and configure_translators functions.

    Drivers

    Drivers are installed as modules. Use the install_modules function in the install.sh to unpack the JAR files for the driver. The driver is then configured in the install.sh by the configure_drivers function, the configuration properties for which are in an environment file.

    drivers.env example:

    #DRIVER
    DRIVERS=DERBY
    DERBY_DRIVER_NAME=derby
    DERBY_DRIVER_MODULE=org.apache.derby DERBY_DRIVER_CLASS=org.apache.derby.jdbc.EmbeddedDriver
    DERBY_XA_DATASOURCE_CLASS=org.apache.derby.jdbc.EmbeddedXADataSource

    Translators

    Translators are installed as modules. Use the install_modules function in the install.sh to unpack the JAR files for the translator. The translator is then configured in the install.sh by the configure_translators function, the configuration properties for which are in an environment file.

    translators.env example:

    #TRANSLATOR
    TRANSLATORS=TESTTRANSLATOR
    
    TESTTRANSLATOR_NAME=test
    TESTTRANSLATOR_MODULE=org.jboss.teiid.translator.test

    Generic Deployments

    Deployable archive files, such as JARs, WARs, RARs, or EARs, can be deployed from an injected image using the install_deployments function supplied by the API in the install-common.sh contained in the EAP for OpenShift image.

    2b) If the CUSTOM_INSTALL_DIRECTORIES environment variable has been declared but no install.sh scripts are found in the custom installation directories, the following artifact directories will be copied to their respective destinations in the built image:
    - modules/* copied to $JBOSS_HOME/modules/system/layers/openshift
    - configuration/* copied to $JBOSS_HOME/standalone/configuration
    - deployments/* copied to $JBOSS_HOME/standalone/deployments
    This is a basic configuration approach compared to the install.sh alternative, and requires the artifacts to be structured appropriately.

3.2.2. Runtime Artifacts

3.2.2.1. Data Sources

There are two types of data sources:
- Internal data sources supported by Red Hat JBoss Middleware for OpenShift. These are PostgreSQL, MySQL, and MongoDB. Internal data sources do not require additional environment files to be configured.
- External data sources that do are not supported by default by OpenShift. Configuration of external data sources is provided by environment files added to OpenShift Secrets.

External data source environment file example:

#DATASOURCES
DATASOURCES=ACCOUNTS
ACCOUNTS_DATABASE=accounts
ACCOUNTS_JNDI=java:/accounts-ds
ACCOUNTS_DRIVER=derby
ACCOUNTS_USERNAME=derby
ACCOUNTS_PASSWORD=derby ACCOUNTS_TX_ISOLATION=TRANSACTION_READ_UNCOMMITTED
ACCOUNTS_JTA=true # Connection info for non-xa datasource
ACCOUNTS_URL=jdbc:derby:jar:\(/deployments/accounts-db.jar\)accounts

# Connection info for xa datasource
ACCOUNTS_XA_CONNECTION_PROPERTY_DatabaseName=jar:\(/deployments/accounts-db.jar\)

accounts # _HOST and _PORT are required, but not used
ACCOUNTS_SERVICE_HOST=dummy
ACCOUNTS_SERVICE_PORT=1527

The DATASOURCES property is a comma-separated list of data source property prefixes. These prefixes are then appended to all properties for that data source. Multiple data sources can then be included in a single environment file. Alternatively, each data source can be provided in separate environment files.

Data sources contain two types of properties: connection pool-specific properties and data driver-specific properties. Data-driver specific properties use the generic XA_CONNECTION_PROPERTY, as the driver itself is configured as a driver S2I artifact. The suffix of the driver property is specific to the particular driver for the data source.
In the above example, ACCOUNTS is the data source prefix, XA_CONNECTION_PROPERTY is the generic driver property, and DatabaseName is the property specific to the driver.

The data sources environment files are added to the OpenShift Secret for the project namespace. These environment files are then called within the JDV template using the ENV_FILES environment property, the value of which is a comma-separated list of fully qualified environment files.

For example:

{
    “Name”: “ENV_FILES”,
    “Value”: “/etc/jdv-extensions/datasources1.env,/etc/jdv-extensions/datasources2.env”
}

3.2.2.2. Resource Adapters

Configuration of resource adapters is provided by environment files added to OpenShift Secrets.

Resource adapter environment file example:

#RESOURCE_ADAPTER
RESOURCE_ADAPTERS=QSFILE

QSFILE_ID=fileQS
QSFILE_MODULE_SLOT=main
QSFILE_MODULE_ID=org.jboss.teiid.resource-adapter.file
QSFILE_CONNECTION_CLASS=org.teiid.resource.adapter.file.FileManagedConnectionFactory
QSFILE_CONNECTION_JNDI=java:/marketdata-file
QSFILE_PROPERTY_ParentDirectory=/home/jboss/source/injected/injected-files/data
QSFILE_PROPERTY_AllowParentPaths=true

The RESOURCE_ADAPTERS property is a comma-separated list of resource adapter property prefixes. These prefixes are then appended to all properties for that resource adapter. Multiple resource adapter can then be included in a single environment file. Alternatively, each resource adapter can be provided in separate environment files.

The resource adapter environment files are added to the OpenShift Secret for the project namespace. These environment files are then called within the JDV template using the ENV_FILES environment property, the value of which is a comma-separated list of fully qualified environment files.

For example:

{
    “Name”: “ENV_FILES”,
    “Value”: “/etc/jdv-extensions/resourceadapter1.env,/etc/jdv-extensions/resourceadapter2.env”
}

3.3. Preparing and Deploying the JDV for OpenShift Application Templates

3.3.1. Configuring Keystores

The JDV for OpenShift image requires two keystores:
- An SSL keystore to provide private and public keys for https traffic encryption.
- A JGroups keystore to provide private and public keys for network traffic encryption between nodes in the cluster.

These keystores are expected by the JDV for OpenShift image, even if the application uses only http on a single-node OpenShift instance. Self-signed certificates do not provide secure communication and are intended for internal testing purposes.

Warning

For production environments Red Hat recommends that you use your own SSL certificate purchased from a verified Certificate Authority (CA) for SSL-encrypted connections (HTTPS).

See the JBoss Enterprise Application Platform Security Guide for more information on how to create a keystore with self-signed or purchased SSL certificates.

3.3.2. Generating the Keystore Secret

OpenShift uses objects called Secrets to hold sensitive information, such as passwords or keystores. See the Secrets chapter in the OpenShift documentation for more information.

The JDV for OpenShift image requires a secret that holds the two keystores described earlier. This provides the necessary authorization to applications in the project.

Use the SSL and JGroups keystore files to create a secret for the project:

$ oc secret new <jdv-secret-name> <ssl.jks> <jgroups.jceks>

After the secret has been generated, it can be associated with a service account.

3.3.3. Generating the Artifact Secrets

Files for runtime artifacts are passed to the JDV for OpenShift image using the OpenShift secret mechanism. This includes the environment files for the data sources and resource adapters, as well as any additional data files. These files need to be present locally so as to create secrets for them.

$ oc secret new <jdv-datasource-secret> <datasource.env>
$ oc secret new <jdv-resourceadapter-secret> <resourceadapter.env>
$ oc secret new <jdv-datafiles-secret> <additional/data/files/>

These files can be placed in a single secret, however creating separate secrets simplifies the process of independently updating or replacing one of them without affecting the other.

3.3.4. Creating the Service Account

Service accounts are API objects that exist within each project and allow users to associate certain secrets and roles with applications in a project namespace. This provides the application with the necessary authorization to run with all required privileges.

The service account that you create must be configured with the correct permissions to view pods in Kubernetes. This is required in order for clustering with the JDV for OpenShift image to work. You can view the top of the log files to see whether the correct service account permissions have been configured.

The JDV for OpenShift templates have a default SERVICE_ACCOUNT_NAME variable of datavirt-service-account. If a different service account name is used, this variable must be configured with the appropriate service account name.

  1. Create a service account to be used for the JDV deployment:

    $ oc create serviceaccount <service-account-name>
  2. Add the view role to the service account. This enables the service account to view all the resources in the application namespace in OpenShift, which is necessary for managing the cluster.

    $ oc policy add-role-to-user view system:serviceaccount:<project-name>:<service-account-name> -n <project-name>
  3. Add the secrets created for the project to the service account:

    $ oc secret link <service-account-name> <jdv-secret-name> <jdv-datasource-secret> <jdv-resourceadapter-secret> <jdv-datafiles-secret>

3.3.5. Using the OpenShift Web Console

Log in to the OpenShift web console:

  1. Click Add to project to list all of the default image streams and templates.
  2. Use the Filter by keyword search bar to limit the list to those that match datavirt. You may need to click See all to show the desired application template.
  3. Select an application template and configure the deployment parameters as required.
  4. Click Create to deploy the application template.

Chapter 4. Tutorials

4.1. Example Workflow: Deploying a JDV Project Using the JDV for OpenShift Image

This workflow demonstrates the basic steps for deploying the JDV for OpenShift image. It presumes that the JDV project has been correctly prepared for configuration as per the Preparing JDV Project Artifacts section of the Get Started chapter.

4.1.1. Preparing JDV for OpenShift Deployment

  1. Create a new project:

    $ oc new-project jdv-app-demo
  2. Create a service account to be used for the JDV for OpenShift deployment:

    $ oc create serviceaccount datavirt-service-account
  3. Add the view role to the service account. This enables the service account to view all the resources in the jdv-app-demo namespace, which is necessary for managing the cluster.

    $ oc policy add-role-to-user view system:serviceaccount:jdv-app-demo:datavirt-service-account
  4. The JDV for OpenShift template requires an SSL keystore and a JGroups keystore.
    These keystores are expected even if the application will not use https.
    This example uses ‘keytool’, a package included with the Java Development Kit, to generate self-signed certificates for these keystores. The following commands will prompt for passwords.

    1. Generate a secure key for the SSL keystore:

      $ keytool -genkeypair -alias https -storetype JKS -keystore keystore.jks
    2. Generate a secure key for the JGroups keystore:

      $ keytool -genseckey -alias jgroups -storetype JCEKS -keystore jgroups.jceks
  5. Use the SSL and JGroup keystore files to create the keystore secret for the project:

    $ oc secret new datavirt-app-secret keystore.jks jgroups.jceks
  6. Create separate secrets using the data source and resource adapter environment files. These files may need to be pulled from the Git repository in order to create secrets with them.

    $ oc secret new jdv-datasource-secret datasource1.env
    $ oc secret new jdv-resourceadapter-secret resourceadapter1.env
    Note

    The environment files can be placed in a single secret, however creating separate secrets simplifies the process of independently updating or replacing one of them without affecting the other.

  7. Link the keystore, data source, and resource adapter secrets to the service account created earlier:

    $ oc secret link datavirt-service-account datavirt-app-secret jdv-datasource-secret jdv-resourceadapter-secret

4.1.2. Deployment

  1. Log in to the OpenShift web console and select the jdv-app-demo project space.
  2. Click Add to Project to list all of the default image streams and templates.
  3. Use the Filter by keyword search bar to limit the list to those that match datavirt. You may need to click See all to show the desired application template.
  4. Select and configure the desired template.
  5. Click Deploy.

Chapter 5. Reference

5.1. Glossary

DATA SOURCE
Repository for data. Query languages enable users to retrieve and manipulate data stored in these repositories.
MODULE

A module is a logical grouping of classes used for class loading and dependency management. Modules are defined in the EAP_HOME/modules/ directory of the application server. Each module exists as a subdirectory, for example EAP_HOME/modules/org/apache/. Each module directory then contains a slot subdirectory, which defaults to main and contains the module.xml configuration file and any required JAR files.

See the Red Hat JBoss Enterprise Application Platform Configuration Guide for more information on modules and module structure.

RESOURCE ADAPTERS

Deployable Java EE component that provides communication between a Java EE application and an Enterprise Information System (EIS) using the Java Connector Architecture (JCA) specification. A resource adapter is often provided by EIS vendors to allow easy integration of their products with Java EE applications.

An EIS can be any other software system within an organization. Examples include: Enterprise Resource Planning (ERP) systems, db systems, email servers, and proprietary messaging systems.

A resource adapter is packaged in a RAR (Resource Adapter Archive) file which can be deployed to JBoss EAP. A RAR file may also be included in an EAR (Enterprise Archive) deployment.

TRANSLATOR
Provides an abstraction layer between the query engine and the physical data source. This layer converts query commands into source specific commands and executes them using a resource adapter. The translator also converts the result data that comes from the physical source into the form that the query engine requires
VIRTUAL DATABASE (VDB)

A VDB is a container for components that integrate data from multiple disparate data sources, allowing applications to access and query the data as though it is a single database, using a single uniform API.

A VDB is composed of various data models and configuration information that describes which, and how, data sources are to be integrated. In particular, 'source models' are used to represent the structure and characteristics of the incorporated physical data sources, and 'view models' represent the structure and characteristics of the integrated data that is exposed to applications.

Legal Notice

Copyright © 2017 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.