Red Hat Training

A Red Hat training course is available for Red Hat Fuse

Chapter 5. Creating integrations

After some planning and preparation, you are ready to create an integration. In the Fuse Online web interface, when you click Create Integration, Fuse Online guides you through the procedure to create an integration.

Prerequisites

The following topics provide information and instructions for creating an integration:

5.1. Preparation for creating an integration

Preparation for creating an integration starts with answers to the questions listed in Considerations for planning your integrations. After you have a plan for the integration, you need to do the following before you can create the integration:

  1. Determine whether an application that you want to connect to uses the OAuth protocol. For each application that uses OAuth, register Fuse Online as a client that is authorized to access that application. Applications that use the OAuth protocol include:

    • Dropbox
    • Google applications (Gmail, Calendar, Sheets)
    • Salesforce
    • SAP Concur
    • Twitter
  2. Determine whether an application that you want to connect to uses HTTP basic authentication. For each application that does, identify the user name and password for accessing that application. You need to provide this information when you create the connection.
  3. For each application that you want to integrate, create a connection.

5.2. Procedure for creating a simple integration

Fuse Online guides you through the procedure for creating a simple integration. It prompts you to choose the start connection, the finish connection, optional middle connections, and other steps. When your integration is complete, you can publish it so that it is running or you can save it for publication at a later time.

To learn about the procedure for creating an API provider integration, see Section 6.3, “Creating an API provider integration”.

Prerequisites

  • You have a plan for what the steps in the integration will be.
  • You created a connection to each application or service that you want to connect to in this integration.

Procedure

  1. In the left panel in Fuse Online, click Integrations.
  2. In the upper right, click Create Integration.
  3. Choose and configure the start connection:

    1. On the Choose a Start Connection page, click the connection that you want to use to start the integration. When this integration is running, Fuse Online will connect to this application and obtain data that you want the integration to operate on.
    2. On the Choose an Action page, click the action you want this connection to perform. The available actions vary for each connection.
    3. On the page for configuring the action, enter values in the fields.
    4. Optionally, if the connection requires data type specification, Fuse Online prompts you to click Next to specify the input and/or output type of the action.
    5. Click Done to add the start connection.

    As an alternative to connecting to an application, a start connection can be a timer that triggers integration execution at intervals that you specify or it can be a webhook that accepts HTTP requests.

  4. Choose and configure the finish connection:

    1. On the Choose a Finish Connection page, click the connection you want to use to complete the integration. When this integration is running, Fuse Online will connect to this application with the data that the integration has been operating on.
    2. On the Choose an Action page, click the action you want this connection to perform. The available actions vary for each connection.
    3. On the page for configuring the action, enter values in the fields.
    4. Optionally, if the connection requires data type specification, Fuse Online prompts you to click Next to specify the input and/or output type of the action.
    5. Click Next to add the finish connection.

    As an alternative to connecting to an application, a finish connection can send information to the integration’s log about the messages that the integration processed. To do this, select Log when Fuse Online prompts you to choose the finish connection.

  5. Optionally, add one or more connections between the start connection and the finish connection. For each connection, choose its action and enter any required configuration details.
  6. Optionally, add one or more steps that operate on integration data between connections. See About adding steps between connections.
  7. In the flow visualization panel on the left, look for any Warning icons. These warnings indicate that a data mapper step is needed before this connection. Add the required data mapper steps.
  8. When the integration contains all needed steps, click Save or Publish according to whether you want to start running the integration.
  9. In the Integration Name field, enter a name that distinguishes this integration from any other integrations.
  10. In the Description field, enter a description, for example, you can indicate what this integration does.
  11. If you are ready to start running the integration, in the upper right, click Publish. Otherwise, click Save.

Results

In the Fuse Online Integrations page, you can see your new integration in the list of integrations. If you published the integration, then you can see that Fuse Online is in the process of publishing it. It may take a few moments for the status of the integration to become Running. If you saved the integration, then Stopped appears on the integration’s entry.

Click the integration’s entry to see a summary of the integration, including its version history, logs for any executions, and aggregate execution metrics.

5.3. How to process a collection in a flow

Sometimes, a connection returns a collection, which contains multiple values that are all the same type. When a connection returns a collection, the flow can operate on the collection in a number of ways, including:

  • Execute each step once for the collection.
  • Execute each step once for each element in the collection.
  • Execute some steps once for the collection and execute other steps once for each element in the collection.

To decide how to operate on a collection in a flow, you need to know which applications the flow connects to, whether they can handle collections, and what you want the flow to accomplish. You can then use the information in the following topics to add steps to a flow that processes a collection:

5.3.1. About processing collections

The easiest way for a flow to process a collection is to use the data mapper to map fields that are in a source collection to fields that are in a target collection. For many flows, this is all that is required. For example, a flow might obtain a collection of employee records from a database and then insert those records into a spreadsheet. Between the database connection and the Google Sheets connection, a data mapper step maps the database fields to the Google Sheets fields. Since both the source and the target are collections, when Fuse Online executes the flow, it calls the Google Sheets connection once. In that call, Fuse Online iterates through the records and correctly populates the spreadsheet.

In some flows, you might need to split a collection into individual objects. For example, consider a flow that connects to a database and obtains a collection of employees who will lose allotted time off if they do not use it before a certain date. The flow then needs to send an email notification to each of these employees. In this flow, you would add a split step after the database connection. You would then add a data mapper step that maps the source fields for an employee record to target fields in a Gmail connection that sends a message. When Fuse Online executes the flow, it executes the data mapper step and the Gmail connection once for each employee.

Sometimes, after you split a collection in a flow, and after the flow executes some steps once for each element that was in the collection, you want the flow to operate on the collection again. Consider the example in the previous paragraph. Suppose that after a Gmail connection sends a message to each employee, you want to add a list of the employees who were notified to a spreadsheet. In this scenario, after the Gmail connection, add an aggregate step to create a collection of employee names. Then add a data mapper step that maps fields in the source collection to fields in the target Google Sheets connection. When Fuse Online executes the flow, it executes the new data mapper step and the Google Sheets connection once for the collection.

These are the most common scenarios for processing a collection in a flow. However, much more complex processing is also possible. For example, when the elements in a collection are themselves collections, you can nest split and aggregate steps inside other split and aggregate steps.

5.3.2. Using the data mapper to process collections

In a flow, when a step outputs a collection and when a subsequent connection that is in the flow expects a collection as the input, you can use the data mapper to specify how you want the flow to process the collection.

When a step outputs a collection, the flow visualization panel displays Collection in the details about the step. For example:

Data Type: SQL Result (Collection)

Add a data mapper step after the step that provides the collection and before the step that needs the mappings. Exactly where in the flow this data mapper step needs to be depends on the other steps in the flow. The following image shows mappings from source collection fields to target collection fields:

mapping collection

In the source and target panels, the data mapper displays this icon to indicate a collection. When a source collection or a target collection contain only primitive types, the data mapper does not display collection fields because there is no need to. You can map from/to the collection itself.

When a collection contains more than one kind of primitive type or when it contains at least one complex type then the data mapper displays the collection’s child fields. You can map from/to each field.

When Fuse Online executes the flow, it iterates over the source collection elements to populate the target collection elements. If you map one or more source collection fields to a target collection or to target collection fields, the target collection elements contain values for only the mapped fields.

If you map a source collection or a field in a source collection to a target field that is not in a collection, then when Fuse Online executes the flow, it assigns the value from only the last element in the source collection. Any other elements in the collection are ignored in that mapping step. However, any subsequent mapping steps can access all elements in the source collection.

When a connection returns a collection that is defined in a JSON or Java document, the data mapper can usually process the source document as a collection.

5.3.3. Adding a split step

During execution of a flow, when a connection returns a collection of objects, Fuse Online executes subsequent steps once for the collection. If you want to execute subsequent steps once for each object that is in the collection, add a split step. For example, a Google Sheets connection returns a collection of row objects. To execute subsequent steps once for each row, add a split step after the Google Sheets connection.

Ensure that the input to a split step is always a collection. If a split step gets a source document that is not a collection type, the step splits the input at each space. For example, Fuse Online splits “Hello world!” input into two elements: “Hello” and “world!”, and passes those two elements to the next step in the flow. In particular, XML data is not a collection type.

Prerequisites

  • You are creating or editing a flow.
  • The flow already has all the connections that it requires.
  • In the flow visualization, the connection that obtains the source data indicates that the data is a (Collection).

Procedure

  1. In the visualization panel on the left, where you want to add a split step to the flow, click the PlusSignToAddStepOrConnection .
  2. Click Split. This step does not require any configuration.

Additional information

Typically, you want to add any split steps and aggregate steps before you add data mapper steps. This is because whether the data is a collection or individual objects affects the mappings. If you add a data mapper step and then add a split step, you usually need to redo the mappings. Likewise, if you remove a split or aggregate step, then you would need to redo any mappings.

5.3.4. Adding an aggregate step

In a flow, add an aggregate step where you want Fuse Online to create a collection from individual objects. During execution, after an aggregate step, instead of executing subsequent steps once for each object, Fuse Online executes subsequent steps once for the collection.

When deciding whether to add an aggregate step to a flow, consider the connections in the flow. After a split step, for each subsequent connection, Syndesis connects to that application once for each element in the flow’s data. For some connections, it might be preferable to connect once rather than multiple times.

Prerequisites

  • You are creating or editing a flow.
  • The flow already has all the connections that it requires.
  • A previous step split a collection into individual objects.

Procedure

  1. In the flow visualization panel on the left, where you want to add an aggregate step to the flow, click the PlusSignToAddStepOrConnection .
  2. Click Aggregate. This step does not require any configuration.

Additional information

Typically, you want to add any split and aggregate steps before you add data mapper steps. This is because whether the data is a collection or individual objects affects the mappings. If you add a data mapper step and then add an aggregate step, you usually need to redo the mappings. Likewise, if you remove an aggregate step, then you would need to redo any mappings.

5.3.5. Example of processing a collection in a flow

This simple integration obtains a collection of tasks from the sample database provided with Fuse Online. The flow splits the collection into individual task objects and then filters these objects to find the tasks that have been done. The flow then aggregates the completed tasks in a collection, maps the fields in that collection to fields in a spreadsheet, and finishes by adding a list of completed tasks to a spreadsheet.

The procedure below provides instructions for creating this simple integration.

Prerequisites

  • You created a Google Sheets connection.
  • In the account that the Google Sheets connection accesses, there is a spreadsheet for receiving the database records.

Procedure

  1. Click Create Integration.
  2. Add the start connection:

    1. On the Choose a Start Connection page, click PostgresDB.
    2. On the Choose an Action page, click Periodic SQL Invocation.
    3. In the SQL Statement field, enter select * from todo and click Next.

    This connection returns a collection of task objects.

  3. Add the finish connection:

    1. On the Choose a Finish Connection page, click your Google Sheets connection.
    2. On the Choose an Action page, click Append values to a sheet.
    3. In the SpreadsheetId field, enter the ID of the spreadsheet to add the list of tasks to.
    4. In the Range field, enter A:B as the target columns that you want to append values to. The first column, A, is for the task IDs. The second column, B, is for the task names.
    5. Accept the defaults for Major Dimension and for Value Input Option, and click Next.

    The Google Sheets connection finishes the flow by adding each element in a collection to a spreadsheet.

  4. Add a split step to the flow:

    1. In the visualization panel, click the plus sign.
    2. Click Split.

    After the flow executes the split step, the result is a set of individual task objects. Fuse Online executes the subsequent steps in the flow once for each individual task object.

  5. Add a filter step to the flow:

    1. In the visualization panel, after the split step, click the plus sign.
    2. Click Basic Filter and configure the filter as follows:

      1. Click in the first field and select completed, which is the name of the field that contains the data that you want to evaluate.
      2. In the second field, select equals as the condition that the completed field value must satisfy.
      3. In the third field, specify 1 as the value that must be in the completed field. 1 indicates that the task has been completed.
    3. Click Done.

    During execution, the flow executes the filter step once for each task object. The result is a set of individual, completed task objects.

  6. Add an aggregate step to the flow:

    1. In the visualization panel, after the filter step, click the plus sign.
    2. Click Aggregate.

    Now the result set contains one collection, which contains an element for each completed task.

  7. Add a data mapper step to the flow:

    1. In the visualization panel, after the aggregate step, click the plus sign.
    2. Click Data Mapper and map the following fields from the SQL result source collection to the Google Sheets target collection:

      • id to A
      • task to B
    3. Click Done.
  8. Click Publish.

Results

When the integration is running, it obtains tasks from the sample database every minute and then adds the completed tasks to the first sheet in the spreadsheet. The integration maps the task ID to the first column, A, and it maps the task name to the second column, B.

5.4. About adding steps between connections

Although it is not a requirement, the recommendation is to add all needed connections to a flow and then, according to the processing that you want the flow to execute, add additional steps between connections. In a flow, each step operates on data obtained from the previous connection(s) and any previous steps. The resulting data is available to the next step in the flow.

Often, you must map data fields that are received from a connection to data fields that the next connection in the flow can operate on. After you add all connections to a flow, check the flow visualization panel on the left. For each connection that requires data mapping before it can operate on the input data, Fuse Online displays DataTypeMismatchWarning . Click this icon to see Data Type Mismatch: Add a data mapping step before this connection to resolve the difference.

You can click the link in the message to display the Configure Mapper page in which you add and specify a data mapping step. However, the recommendation is to add other needed steps, and then add data mapper steps last.

5.5. Adding a data mapping step

Almost all integrations require data mapping. A data mapper step maps data fields from the previous connection(s) and any other steps to data fields that the next connection in the flow can operate on. For example, suppose the integration data contains a Name field and the next connection in the flow has a CustomerName field. You need to map the source Name field to the target CustomerName field.

Prerequisite

You are creating or editing a flow.

Procedure

  1. In the left panel, where you want to add a data mapper step, click the PlusSignToAddStepOrConnection .
  2. Click Data Mapper to display source and target fields in the data mapper canvas.

5.6. Adding a basic filter step

You can add a step to a flow to filter the data that the flow operates on. In a filter step, Fuse Online inspects the data and continues only if the content meets criteria that you define. For example, in a flow that obtains data from Twitter, you can specify that you want to continue execution by operating only on tweets that contain "Red Hat".

Prerequisites

  • The flow contains all connections that it needs to.
  • You are creating or editing a flow.

Procedure

  1. In the left panel, where you want to add a filter step to the flow, click the PlusSignToAddStepOrConnection .
  2. Click Basic Filter.
  3. On the Configure Basic Filter Step page, in the Continue only if incoming data match field, select one of the following options:

    • Accept the default that all defined rules must be satisfied.
    • Indicate that only one rule must be satisfied by selecting ANY of the following.
  4. Define the filter rule:

    1. The data you want to evaluate: Enter the name of the field that contains the content you want the filter to evaluate. For example, suppose the data coming in to the step consists of tweets that mention your Twitter handle. You want to continue execution only when the tweet contains certain content. The tweet is in a field named text so you enter or select text as the value in the first field.

      You can define the field value in the following ways:

      • Start typing. The data name field has a typeahead feature that provides a list of possible completions for you in a pop-up box. Select the correct one from the box.
      • Click in the field. A dropdown box appears with a list of available fields. Select the field of interest from the list.
    2. Must meet this condition: Select a condition from the dropdown box. The setting defaults to Contains. For execution to continue, the condition that you select in this field must be true for the value that you enter in the third field.
    3. For this value: Enter a value to filter on. For example, if you want to operate on mentions of a certain product from the Twitter feed, you would enter the product name here.
  5. Optionally, click + Add another rule and define another rule.

    You can delete a rule by clicking the trash can icon next to the entry.

  6. When the filter step is complete, click Done to add it to the flow.

Additional resource

If you cannot define the filter you need in a basic filter step, see Adding an advanced filter step.

5.7. Adding an advanced filter step

In a filter step, Fuse Online inspects the data and continues executing the flow only if the content meets criteria that you define. If the basic filter step does not let you define the exact filter that you need, then add an advanced filter step.

Prerequisites

  • The flow contains all connections that it needs to.
  • You are creating or editing a flow.

Procedure

  1. In the left panel, where you want to add an advanced filter step to the flow, click the PlusSignToAddStepOrConnection .
  2. Click Advanced Filter.
  3. In the edit box, use the Camel Simple Expression language to specify a filter expression. For example, the following expression evaluates to true when the message header’s type field is set to widget:

    ${in.header.type} == 'widget'

    In the following example, the expression evaluates to true when the body of the message contains a title field:

    ${in.body.title}
  4. Click Done to add the advanced filter step to the flow.

5.8. Adding a template step

In a flow, a template step takes data from a source and inserts it into the format that is defined in a template that you upload to Fuse Online. The benefit of a template step is that it provides data output in a consistent format that you specify.

In the template, you define placeholders and specify static text. When you create the flow, you add a template step, map source fields to the template placeholders, and then map template content to the next step in the flow. When Fuse Online executes the flow, it inserts the values that are in the mapped source fields into an instance of the template and makes the result available to the next step in the flow.

If a flow includes a template step then it is most likely the only template step in that flow. However, more than one template step in a flow is allowed.

Fuse Online supports the following kinds of templates: Freemarker, Mustache, Velocity.

Prerequisites

  • You must be creating or editing a flow.
  • If you are creating a simple integration then it must already have its start and finish connections.

Procedure

  1. In the visualization panel on the left, click the Plus Sign where you want to add a template step.
  2. Click Template. The Upload Template page opens.
  3. Specify the template type, which is Freemarker, Mustache, or Velocity.
  4. To define the template, do one of the following:

    • Drag and drop a template file or a file that contains text that you want to modify to create a template, into the template editor.
    • Click browse to upload, navigate to a file, and upload it.
    • In the template editor, start typing to define a template.
  5. In the template editor, ensure that the template is valid for use with Fuse Online. Examples of valid templates are after this procedure. Fuse Online displays a red error indicator to the left of a line that contains a syntax error. Hovering over a syntax error indicator displays hints about how to resolve the error.
  6. Click Done to add the template step to the flow.

    If the Done button is not enabled then there is at least one syntax error that you must correct.

    Input to a template step must be in the form of a JSON object. Consequently, you must add a data mapping step before a template step.

  7. To add a data mapper step before the template step:

    1. In the left panel, click the Plus Sign that is immediately before the template step that you just added.
    2. On the Choose a Step page, click Data Mapper.
    3. In the data mapper, map a source field to each template placeholder field.

      For example, using the example templates that are after this procedure, map a source field to each of these template fields:

      • time
      • name
      • text
    4. In the upper right, click Done to add the data mapper step to the flow.

    Output from a template step is always a JSON object. Consequently, you must add a data mapper step after a template step.

  8. To add a data mapper step after the template step:

    1. In the left panel, click the Plus Sign that is immediately after the template step that you just added.
    2. On the Choose a Step page, click Data Mapper.
    3. In the data mapper, map the template’s message field, which always contains the result of inserting source fields into the template, to a target field. For example, suppose that a Gmail connection is next in the flow and you want to send the result of the template step as the content of a Gmail message. To do this, you would map the message source field to the text target field.
    4. In the upper right, click Done.

Examples of templates

Example of a Mustache template:

At {{time}}, {{name}} tweeted:
{{text}}

Freemarker and Velocity support this example template:

At ${time}, ${name} tweeted:
${text}

Velocity also supports syntax without braces, as shown in this example:

At $time, $name tweeted:
$text

A placeholder cannot contain a . (period).

Additional resources

For details about mapping fields, see Mapping integration data to fields for the next connection in the flow.

5.9. Adding a custom step

If Fuse Online does not provide a step that you need in a flow, a developer can define one or more custom steps in an extension. A custom step operates on data between connections in a flow.

You add a custom step to a flow in the same way that you add a built-in step. For a simple integration, choose the start and finish connections, add other connections as needed and then add additional steps. For an API provider integration, select the operation whose flow executes the custom step, add connections as needed to the flow, and then add other steps. When you add a step, Fuse Online operates on the data it receives from the previous step(s) in the flow.

Prerequisites

  • You uploaded the custom step extension to Fuse Online. See Making extensions available.
  • You are creating or editing a flow.
  • The flow already has all the connections that it requires.

Procedure

  1. In the visualization panel on the left, where you want to add a custom step to the flow, click the PlusSignToAddStepOrConnection .
  2. Click the custom step that you want to add.

    The available steps includes any custom steps that are defined in extensions that were uploaded to your Fuse Online environment.

  3. Respond to prompts for any information that is required to perform the step. This information varies for each custom step.