Chapter 2. Connecting to Amazon Web Services

A Fuse Online integration can connect to the following Amazon web services:

  • Simple Notification Service (SNS)
  • Simple Queue Service (SQS)
  • Simple Storage Service (S3)

You must obtain AWS credentials before you can create a connection to an Amazon web service. For details, see:

2.1. Obtaining AWS credentials for creating Fuse Online connections

To create a connection to an Amazon service, you must have an access key that is associated with an AWS account. This is the AWS account that created, or will create, the resource that you want the Fuse Online connection to access. The resource can be an S3 bucket, an SNS topic, or an SQS queue.

An AWS access key has two parts:

  • Amazon access key ID
  • Amazon secret access key

AWS uses access keys to authenticate requests from a Fuse Online connection. There is plenty of AWS documentation about creating and managing access keys. If you already created an AWS Identity and Access Management (IAM) user and captured the access key associated with that user, you can specify those values to create a Fuse Online connection to S3, SNS, or SQS.

If you need to obtain an access key, the procedure below is one way to do it.

Prerequisites

  • Login credentials for the AWS account that created the resource that you want a connection to access.
  • Or, login credentials for the AWS account that you want the connection to use to create a new resource.

Procedure

  1. Go to https://aws.amazon.com/ and sign in to the console.
  2. In the console, in the upper right, click the down arrow next to the user name and click My Security Credentials.
  3. In the popup dialog, click Get Started with IAM Users.

    If a popup with this button does not appear, then on the left, click Users.

  4. Add a user:

    1. Click Add User
    2. Enter a user name and select Programmatic Access.
    3. Click Next: Permissions.
    4. Respond to the prompts for adding the new user to a group.
    5. Click Next: Tags.
    6. Skip adding tags and click Next: Review.
    7. Click Create User.
  5. Click Download .csv to obtain a local copy of the access key. The downloaded file contains the access key ID and the secret access key, which you must specify to create a connection to an Amazon service.
  6. Click Close.

2.2. Connecting to Amazon Simple Notification Service (SNS)

An integration can send a message to an AWS Simple Notification Service topic. To do this, create an AWS SNS connection and then add that connection to an integration flow or as the finish connection for a simple integration. For details, see:

2.2.1. Creating Amazon SNS connections

You must create an Amazon SNS connection before you can add an Amazon SNS connection to an integration.

Prerequisites

  • AWS access key. See Obtaining AWS credentials.
  • You must know the region in which the SNS topic is located. This is the topic that you want the SNS connection that you are creating to send a message to.

Procedure

  1. In Fuse Online, in the left panel, click Connections to display any available connections.
  2. In the upper right, click Create Connection to display Fuse Online connectors.
  3. Click the Amazon SNS connector.
  4. In the Access Key field, enter an Amazon access key ID that is part of a user access key in the AWS account that manages the topic that you want the connection to send a message to.

    If the topic that you want the connection to access does not already exist then when Fuse Online tries to start running the integration, it uses the AWS account associated with this access key to create the topic. However, if the topic already exists in some other AWS account, then the connection cannot create the topic and the integration cannot start.

  5. In the Secret Key field, enter the Amazon secret access key for the Amazon access key ID that you specified.
  6. In the Region field, select the AWS region in which the topic resides. If the connection creates the topic, then it creates it in the selected region.
  7. Click Validate. Fuse Online immediately tries to validate the connection and displays a message that indicates whether or not validation is successful. If validation fails, revise the configuration details as needed and try again.
  8. When validation is successful, click Next.
  9. In the Name field, enter your choice of a name that helps you distinguish this connection from any other connections. For example, enter SNS North.
  10. In the Description field, optionally enter any information that is helpful to know about this connection. For example, enter Sample SNS connection that sends messages to our north region topic.
  11. Click Save to see that the connection you created is now available. If you entered the example name, you would see that SNS North appears as a connection that you can choose to add to an integration.

2.2.2. Sending messages to Amazon SNS

To send a message to an Amazon SNS topic, add an Amazon SNS connection to the middle of a flow or as the finish connection in a simple integration.

Prerequisites

  • You created an Amazon SNS connection.
  • Fuse Online is prompting you to add to the integration, or to add the finish connection to a simple integration.

Procedure

  1. On the Add to Integration page, click the plus sign where you want to add the connection. Skip this step if Fuse Online is prompting you to choose the finish connection.
  2. Click the Amazon SNS connection that you want to use to send a message.
  3. Click the Send Object action.
  4. In the Topic Name or Amazon Resource Name field, enter the name of the SNS topic to send a message to or enter the topics’s Amazon Resource Name (ARN).

    If the SNS topic does not already exist in either the AWS account that the connection is authorized to access, or in any other AWS account, the connection creates the topic in the AWS account that the connection is configured to access.

  5. Click Next to add the connection to the integration.

Result

The connection appears in the integration flow where you added it.

2.3. Connecting to Amazon Simple Queue Service (SQS)

An integration can retrieve messages from an Amazon SQS queue or send messages to an Amazon SQS queue. To do this, create an Amazon SQS connection and then add that connection to an integration flow. For details, see:

2.3.1. Creating Amazon SQS connections

You must create an Amazon SQS connection before you can add an Amazon SQS connection to an integration.

Prerequisites

  • AWS access key. See Obtaining AWS credentials.
  • You must know the region in which the SQS queue is located. This is the queue that you want the SQS connection that you are creating to send messages to or retrieve messages from.

Procedure

  1. In Fuse Online, in the left panel, click Connections to display any available connections.
  2. In the upper right, click Create Connection to display Fuse Online connectors.
  3. Click the Amazon SQS connector.
  4. In the Access Key field, enter an Amazon access key ID that is part of a user access key in the AWS account that manages the queue that you want the connection to access.

    If the queue that you want the connection to access does not already exist then when Fuse Online tries to start running the integration, it uses the AWS account associated with this access key to create the queue. However, if the queue already exists in some other AWS account, then the connection cannot create the queue and the integration cannot start.

  5. In the Secret Key field, enter the Amazon secret access key for the Amazon access key ID that you specified.
  6. In the Region field, select the AWS region in which the queue resides. If the connection creates the queue, then it creates it in the selected region.
  7. Click Validate. Fuse Online immediately tries to validate the connection and displays a message that indicates whether or not validation is successful. If validation fails, revise the configuration details as needed and try again.
  8. When validation is successful, click Next.
  9. In the Name field, enter your choice of a name that helps you distinguish this connection from any other connections. For example, enter SQS South.
  10. In the Description field, optionally enter any information that is helpful to know about this connection. For example, enter Sample SQS connection that sends messages to our southern region queue.
  11. Click Save to see that the connection you created is now available. If you entered the example name, you would see that SQS South appears as a connection that you can choose to add to an integration.

2.3.2. Obtaining messages from Amazon SQS to trigger integration execution

To start an integration by obtaining messages from an Amazon SQS queue, add an Amazon SQS connection as a simple integration’s start connection.

Prerequisites

  • You created an Amazon SQS connection that is configured to access the queue that you want to obtain messages from.
  • You know the name of the SQS queue that you want the connection to obtain messages from.

Procedure

  1. In the Fuse Online panel on the left, click Integrations.
  2. Click Create Integration.
  3. On the Choose a connection page, click the Amazon SQS connection that you want to use to start the integration.
  4. On the Choose an action page, select Poll an Amazon SQS Queue to periodically retrieve messages from an SQS queue.
  5. Configure this action:

    1. In the Delay field, accept the default of 500 milliseconds as the time that elapses between polls. Or, to specify a different polling interval, enter a number and select its time unit.
    2. In the Maximum Objects to Retrieve field, enter the largest number of messages that one poll operation can obtain. The default is 10.

      To have no limit on the number of messages that can be obtained, specify 0 or a negative integer. When Maximum Objects to Retrieve is unlimited, the poll action obtains all messages in the SQS queue.

    3. In the Queue Name or Amazon Resource Name field, specify the name of the SQS queue to retrieve messages from, or the queue’s ARN. If a queue with this name does not exist, the connection creates it. This is the only required field.
    4. Indicate whether you want to Obtain messages and then delete them from the queue. Obtaining messages and then deleting them from the queue is the default behavior. Unselect this option only if you want the connection to retrieve messages and also leave them on the queue, which means that the messages will be retrieved again.
    5. Indicate whether you want to Obtain messages and delete the message if it does make it through a Camel filter. This option is selected by default. Leave it selected when you want to pass a retrieved message through a filter to determine whether to delete it from the queue. You do not need to unselect Obtain messages and then delete from the queue. Fuse Online ensures the correct behavior when you select Obtain messages and delete the message if it does make it through a Camel filter.
  6. Click Next.

Result

The connection appears at the beginning of the integration flow.

Next steps

If you selected Obtain messages and delete the message if it does make it through a Camel filter then you need to add a filter step after this SQS connection. The filter step does not need to be immediately after the SQS connection. Add all connections to the integration, and then add the filter step. During execution, if there are messages that pass the filter, Fuse Online uses the same connection, the SQS start connection, to delete those messages from the queue.

2.3.3. Sending messages to an Amazon SQS queue

In the middle of a flow, or to finish a simple integration, you can send messages to an Amazon SQS queue.

Prerequisites

  • You created an Amazon SQS connection that is configured to access the queue that you want to send messages to.
  • You know the name of the SQS queue that you want the connection to send messages to.
  • Fuse Online is prompting you to add to the integration, or to choose the finish connection for a simple integration.

Procedure

  1. On the Add to Integration page, click the plus sign where you want to add the connection. Skip this step if Fuse Online is prompting you to choose the finish connection.
  2. Click the Amazon SQS connection that you want to use to send messages.
  3. Select the action that you want the connection to perform:

    • Send Object sends one message to the queue.
    • Send Batch Object sends a batch of messages to the queue.
  4. In the Message group ID strategy field, for a first-in, first-out (FIFO) queue, accept ConstantMessageGroupIdStrategy or click in the field and select a different strategy.

    The setting of this parameter has no effect on simple queues. The message group ID identifies the group that a message belongs to. Messages that belong to the same message group are always processed one by one, in a strict order relative to the message group. The message group ID strategy determines how the connection assigns a group ID to a message. The options are:

    • ConstantMessageGroupIdStrategy — The connection uses a constant to group messages.
    • ExchangeIdMessageGroupIdStrategy — The connection uses the exchange ID, which is in each message, to group messages.
    • PropertyValueMessageGroupIdStrategy — The connection uses the value of an internal property to group messages.

    FIFO queues are designed to ensure that the order in which messages are sent and received is strictly preserved and that each message is processed exactly once. A FIFO queue name has .fifo as a suffix.

  5. In the Message deduplication ID strategy field, for a FIFO queue, accept NullMessageDeduplicationIdStrategy or click in the field to select ExchangeIdMessageDeduplicationIdStrategy.

    The setting of this parameter has no effect on simple queues. For a FIFO queue, the connection uses the selected strategy to ensure that it does not send duplicate messages to the queue.

    • NullMessageDeduplicationIdStrategy uses the message body as the deduplication element. In other words, the connection compares message bodies to identify duplicate messages.
    • ExchangeIdMessageDeduplicationIdStrategy uses the message’s exchange ID as the deduplication element. In other words, the connection compares exchange IDs to identify duplicate messages.
  6. In the Queue Name or Amazon Resource Name field, enter the name of the queue to send messages to. If the queue does not exist, the connection creates it.
  7. In the Delay field, accept the default, which is 0 for no delay. Or, to add a delay, specify a number. The integration waits this number of seconds before sending messages. This is useful when the message consumers might not be immediately ready for the messages that the connection is sending.
  8. Click Next.

Result

The connection appears in the integration flow where you added it.

2.4. Connecting to Amazon Simple Storage Service (S3)

An integration can retrieve data from an Amazon S3 bucket or copy data into an Amazon S3 bucket. To do this, create an Amazon S3 connection and then add that connection to an integration flow. For details, see:

2.4.1. Creating Amazon S3 connections

You must create an Amazon S3 connection before you can add an Amazon S3 connection to an integration.

Prerequisites

  • AWS access key. See Obtaining AWS credentials.
  • If the bucket that you want the connection to access already exists, you must know:

    • The region that the bucket is in.
    • The bucket’s name or Amazon Resource Name (ARN).

Procedure

  1. In Fuse Online, in the left panel, click Connections to display any available connections.
  2. In the upper right, click Create Connection to display Fuse Online connectors.
  3. Click the Amazon S3 connector.
  4. In the Access Key field, enter an Amazon access key ID that is part of a user access key in the AWS account that manages the bucket that you want the connection to access.

    If the bucket that you want the connection to access does not already exist then when Fuse Online tries to start running the integration, it uses the AWS account associated with this access key to create the bucket. However, if the bucket already exists in some other AWS account, then the connection cannot create the bucket and the integration cannot start.

  5. In the Secret Key field, enter the Amazon secret access key for the Amazon access key ID that you specified.
  6. In the Region field, select the AWS region in which the bucket resides. If the connection creates the bucket, then it creates it in the selected region.
  7. In the Bucket Name or Amazon Resource Name field, enter the name of the bucket that you want this connection to access or enter the bucket’s ARN.

    If the bucket you specify does not yet exist then the connection tries to create a bucket with the name that you specify. Because S3 allows a bucket to be used as a URL that can be accessed publicly, the bucket name that you specify must be globally unique. Also, it must meet S3 bucket naming requirements.

  8. Click Validate. Fuse Online immediately tries to validate the connection and displays a message that indicates whether or not validation is successful. If validation fails, revise the configuration details as needed and try again.
  9. When validation is successful, click Next.
  10. In the Name field, enter your choice of a name that helps you distinguish this connection from any other connections. For example, enter Obtain S3 Data.
  11. In the Description field, optionally enter any information that is helpful to know about this connection. For example, enter Sample S3 connection that obtains data from a northeast bucket.
  12. Click Save to see that the connection you created is now available. If you entered the example name, you would see that Obtain S3 Data appears as a connection that you can choose to add to an integration.

2.4.2. Obtaining data from Amazon S3 to trigger integration execution

To start an integration by obtaining data from an Amazon S3 bucket, add an Amazon S3 connection as a simple integration’s start connection.

Prerequisite

You created an Amazon S3 connection.

Procedure

  1. In the Fuse Online panel on the left, click Integrations.
  2. Click Create Integration.
  3. On the Choose a connection page, click the Amazon S3 connection that you want to use to start the integration.
  4. On the Choose an action page, select the action that you want the connection to perform:

    • Get Object obtains a file from the bucket that the connection accesses. In the File Name field, enter the name of the file that you want to obtain. If the specified file is not in the bucket, it is a runtime error.
    • Poll an Amazon S3 Bucket periodically obtains files from the bucket that the connection accesses. To configure this action:

      1. In the Delay field, accept the default of 500 milliseconds as the time that elapses between polls. Or, to specify a different polling interval, enter a number and select its time unit.
      2. In the Maximum Objects to Retrieve field, enter the largest number of files that one poll operation can obtain. The default is 10.

        To have no limit on the number of files that can be obtained, specify 0 or a negative integer. When Maximum Objects to Retrieve is unlimited, the poll action obtains all files in the bucket.

        If the bucket contains more than the specified maximum number of files then the action obtains the files that were most recently modified or created.

      3. In the Prefix field, optionally specify a regular expression that evaluates to a string. If you specify a prefix then this action retrieves a file only when its name starts with that string.
      4. Indicate whether you want to Obtain files and then delete them from the bucket.
  5. After you configure the action, click Next to specify the action’s output type.
  6. In the Select Type field, if the data type does not need to be known, accept Type specification not required and click Next. You do not need to follow the rest of these instructions.

    Otherwise, click in the Select Type field and select one of the following as the schema type:

    • JSON schema is a document that describes the structure of JSON data. The document’s media type is application/schema+json.
    • JSON instance is a document that contains JSON data. The document’s media type is application/json.
    • XML schema is a document that describes the structure of XML data. The document’s file extension is .xsd.
    • XML instance is a document that contains XML data. The document’s file extension is .xml.
  7. In the Definition input box, paste a definition that conforms to the schema type you selected. For example, if you select JSON schema then you would paste the content of a JSON schema file, which has a media type of application/schema+json.
  8. In the Data Type Name field, enter a name that you choose for the data type. For example, if you are specifying a JSON schema for vendors then you might specify Vendor as the data type name.

    You will see this data type name when you are creating or editing an integration that uses the connection for which you are specifying this type. Fuse Online displays the type name in the integration visualization and in the data mapper.

  9. In the Data Type Description field, provide information that helps you distinguish this type. This description appears in the data mapper when you hover over the step that processes this type.
  10. Click Next.

Result

The integration now has a start connection and Fuse Online is prompting you to choose the finish connection.

2.4.3. Adding data to Amazon S3 to finish an integration

To finish an integration by copying data to Amazon S3, add an Amazon S3 connection as a simple integration’s finish connection.

Prerequisites

  • You created an Amazon S3 connection.
  • You are creating or editing a simple integration and Fuse Online is prompting you to choose a finish connection.

Procedure

  1. On the Choose a connection page, click the Amazon S3 connection that you want to use to finish the integration.
  2. Select the action that you want the connection to perform:

    • Copy Object adds one or more objects to the bucket.

      To add one file to the bucket, you can enter its name in the File Name field.

      To add multiple files to the bucket, do not specify a file name. In this case, the action adds all objects that it obtains from the previous integration step(s).

      If you used the poll action to obtain multiple files and you specify a file name then the Copy Object action adds only the last file that was received from the poll action.

    • Delete Object deletes an object from the bucket. In the File Name field, specify the name of the object that you want to delete. If the specified file is not in the bucket, the integration continues with no error.
  3. After you configure the chosen action, click Next to specify the action’s input type.
  4. In the Select Type field, if the data type does not need to be known, accept Type specification not required and click Next. You do not need to follow the rest of these instructions.

    Otherwise, click in the Select Type field and select one of the following as the schema type:

    • JSON schema is a document that describes the structure of JSON data. The document’s media type is application/schema+json.
    • JSON instance is a document that contains JSON data. The document’s media type is application/json.
    • XML schema is a document that describes the structure of XML data. The document’s file extension is .xsd.
    • XML instance is a document that contains XML data. The document’s file extension is .xml.
  5. In the Definition input box, paste a definition that conforms to the schema type you selected. For example, if you select JSON schema then you would paste the content of a JSON schema file, which has a media type of application/schema+json.
  6. In the Data Type Name field, enter a name that you choose for the data type. For example, if you are specifying a JSON schema for vendors then you might specify Vendor as the data type name.

    You will see this data type name when you are creating or editing an integration that uses the connection for which you are specifying this type. Fuse Online displays the type name in the integration visualization and in the data mapper.

  7. In the Data Type Description field, provide information that helps you distinguish this type. This description appears in the data mapper when you hover over the step that processes this type.
  8. Click Next.

Result

The connection appears at the end of the integration visualization.

2.4.4. Adding data to Amazon S3 in the middle of an integration

In the middle of an integration, to add data to Amazon S3, add an Amazon S3 connection to the middle of a flow.

Prerequisite

  • You created an Amazon S3 connection.
  • You are creating or editing a flow and Fuse Online is prompting you to add to the integration.

Procedure

  1. In the flow visualization, click the plus sign where you want to add an Amazon S3 connection.
  2. Click the Amazon S3 connection that you want to use in the middle of a flow.
  3. Select the action that you want the connection to perform:

    • Copy Object adds one or more objects to the bucket.

      To add one file to the bucket, you can enter its name in the File Name field.

      To add multiple files to the bucket, do not specify a file name. In this case, the action adds all objects that it obtains from the previous integration step(s).

      If you used the poll action to obtain multiple files and you specify a file name then the Copy Object action adds only the last file that was received from the poll action.

    • Delete Object deletes an object from the bucket. In the File Name field, specify the name of the object that you want to delete. If the specified file is not in the bucket, the integration continues with no error.
  4. After you configure the chosen action, click Next to specify the action’s input type.
  5. In the Select Type field, if the data type does not need to be known, accept Type specification not required and click Next. You do not need to follow the rest of these instructions.

    Otherwise, click in the Select Type field and select one of the following as the schema type:

    • JSON schema is a document that describes the structure of JSON data. The document’s media type is application/schema+json.
    • JSON instance is a document that contains JSON data. The document’s media type is application/json.
    • XML schema is a document that describes the structure of XML data. The document’s file extension is .xsd.
    • XML instance is a document that contains XML data. The document’s file extension is .xml.
  6. In the Definition input box, paste a definition that conforms to the schema type you selected. For example, if you select JSON schema then you would paste the content of a JSON schema file, which has a media type of application/schema+json.
  7. In the Data Type Name field, enter a name that you choose for the data type. For example, if you are specifying a JSON schema for vendors then you might specify Vendor as the data type name.

    You will see this data type name when you are creating or editing an integration that uses the connection for which you are specifying this type. Fuse Online displays the type name in the integration visualization and in the data mapper.

  8. In the Data Type Description field, provide information that helps you distinguish this type. This description appears in the data mapper when you hover over the step that processes this type.
  9. Click Next.

Result

The connection appears in the flow visualization where you added it.