Chapter 4. Creating a virtual database as a Maven artifact

You can provide a full settings.xml file that specifies a single common Maven repository for the Operator to use in building any virtual database. Use this method if your Maven repository requires authentication, or if you have multiple repositories and you want to use a specific one for all your virtual database builds.

When the Operator builds the image for your virtual database, it checks for a ConfigMap named teiid-maven-settings. If it finds it, it then uses the settings.xml file in the ConfigMap for the build.

In some cases, you might want the Data Virtualization Operator to use a specific Maven repository when it builds a particular virtual database. You can specify the Maven repository to use in a settings.xml file that you add to a ConfigMap or secret that applies only to a virtual database with a specific name.

The configuration that you add for a specific virtual database build takes precedence over any global configuration that you set for the namespace.

The repositories that you add designate for use in building an individual virtual database cannot require authentication.

metadata:
  name: dv-customer-maven-settings

You can specify the private Maven repositories to use in building an individual virtual database in the custom resource for the virtual database. The method of specifying Maven repositories in a custom resource applies only to repositories that do not require authentication.

The repositories that you add to the CR cannot require authentication.

When you run the Data Virtualization Operator to build a virtual database, it uses the settings in the CR. For example,

apiVersion: teiid.io/v1alpha1
kind: VirtualDatabase
metadata:
  name: dv-customer
spec:
  replicas: 1
  build:
    source:
      ddl: |
        CREATE DATABASE customer OPTIONS (ANNOTATION 'Customer VDB');
        USE DATABASE customer;
        ...
      mavenRepositories: 1
        private: https://myprivate.host.com/maven2
        private2: https://myprivate.host2.com/maven2

If you specify Maven repositories in the CR, the Operator uses them in addition to the repositories specified in the settings.xml file.

A virtual database developer can deploy a virtual database to a Maven repository, where it is assigned an identifier, version number, and a defined location. The resulting Maven artifact can then be defined as a dependency to the build process of a second virtual database. When this second virtual database is built, the build reads the contents of the original virtual database from the Maven repository and incorporates it into current new virtual database.

For example, imagine that a Sales team developer creates the virtual databases AccountsDB and SalesDB, and then deploys them to a Maven repository. When the Operations team developer creates a Maven project for a secondary virtual database, OperationsDB, the project defines the AccountsDB and SalesDB databases as dependencies. The new project can then use an IMPORT statement to extract content from the two original virtual databases. After the developer deploys the OperationsDB database, users can connect to it to access data from both of its source databases.

Because the OperationsDB database is also available as a Maven artifact, it can be used as a data source by other virtual databases.

To deploy the OperationsDB virtual database, you must supply the Maven coordinates of the OperationsDB to the Data Virtualization Operator. During the build, the Operator retrieves the contents from the Maven repository, including the dependency information, and deploys the virtual database to OpenShift.

The Maven build process enables build tasks to be automated as part of an automated CI/CD workflow in which you can configure Maven to rebuild virtual databases automatically with no user intervention after changes occur in the sources.

You use a Maven build process to create a virtual database that import from other virtual databases.

The process and the project structure for developing virtual databases that use importing is the same as the process for creating any virtual database as a Maven artifact. You use the standard Maven project structure for your virtual database. Resources in the project must be added to the expected locations in the project structure. If you want to include additional metadata files with the virtual database, add them within this structure.

For more information about the structure for creating virtual databases as Maven artifacts, see Chapter 4, Creating a virtual database as a Maven artifact.

Along with the standard entries, the POM for an importing virtual database must list the following entries:

For example, if you want to import the virtual database accountsdb into the secondary virtual database OperationsDB, the POM file for the OperationsDB database must include a dependency entry for the accountsdb database as in the following example:

<dependencies>
  <dependency>
    <groupId>org.example</groupId>
    <artifactId>accountsdb</artifactId>
    <version>1.0</version>
  </dependency>
</dependencies>

If you want to import multiple virtual databases, the <dependencies> section of the POM must include separate <dependency> entries for each virtual database. The <dependencies> block is contained within the <projects section of the POM.

The DDL for the importing virtual database must contain an IMPORT statement similar to the following statement:

IMPORT DATABASE IMPORTED_VIRTUAL_DATABASE_ VERSION VERSION_NUMBER__;

For example, if you want to import the virtual database accountsdb into a virtual database with the name OperationsDB, you would include the following entry in the DDL for the OperationsDB database:

IMPORT DATABASE AccountsDB VERSION 1.0;

The Maven build reads the contents of the vdb.ddl file for the AccountsDB and merges it into the virtual database that is being built. The result is a virtual database comprised of multiple virtual databases.

After you configure the source files for the Maven build, you can run the build to deploy the Maven artifact to a repository. The process for building the virtual database is the same as with any Maven-based virtual database. For more information, see Section 4.1, “Building a virtual database artifact”

You should avoid the use of some statements in the DDL for a virtual database.

IMPORT FOREIGN SCHEMA public FROM SERVER sampledb INTO accounts;

The IMPORT FOREIGN SCHEMA statement is an expensive operation, that queries the underlying physical data source every time a pod restarts. Introducing this query places a strain on the underlying physical data source, increasing the time that it takes for the pod to start. The problem is magnified if you have multiple pods trying to access the data source at once.

Another problem with importing the foreign schema from another virtual database is that images deployed to OpenShift are assumed to be in an immutable state. That is, no matter how many times the image is stopped and started, their behavior should persist. However, if you define a SQL IMPORT SCHEMA operation in the DDL, the virtual database loads the schema from the source virtual database every time that the image starts up. As a result, the image contents can be modified, which runs counter to the principle of immutability for this architecture.

If you can guarantee that the underlying data source always returns the same metadata, there is no problem. Problems arise if the data source returns different metadata when the image starts.

To ensure that the contents of an image remain stable, it is best to define all metadata explicitly in the virtual database DDL, including tables, procedures, and any functions that data source represents.

ALTER TABLE statements are intended to modify the structure of a table by adding, removing, or modifying columns. However, in the case of imported virtual databases, the actual structure of the imported tables is not available at build time. From the perspective of the virtual database, the runtime metadata does not yet exist to be modified.

When you include an ALTER TABLE statement in the DDL with an IMPORT statement, it attempts to change a table or column that does not exist in the the virtual database metadata. A deployment failure can result, or if the virtual database is deployed, its contents might not include the expected data.