Chapter 17. Connecting to MongoDB

MongoDB is a distributed NoSQL database. A Fuse Online integration can obtain content from a MongoDB database or update content in a MongoDB database. To do this, create a MongoDB connection, and add it to an integration.

Details for connecting to a MongoDB database are in the following topics:

17.1. Creating a connection to a MongoDB database

Create a MongoDB connection so that you can connect to a MongoDB database in an integration.

Prerequisites

  • You must know the host name for the MongoDB database that you want to connect to.
  • You must have a user name and password that authorizes access to the MongoDB database.

Procedure

  1. In Fuse Online, in the left panel, click Connections to display any available connections.
  2. Click Create Connection to display connectors.
  3. Click the MongoDB connector.
  4. Configure the connection:

    1. In the Database host(s) field, specify one or more MongoDB database hosts that this connection can access. Separate multiple specifications of host:port with commas. Replace host with a host name, a virtual host name, or an IP address. For example: mongodb0.example.com:1234,mongodb1.example.com:1234.
    2. In the Username field, enter a user name that is authorized to access the database.
    3. In the Password field, enter the password for the user name that you specified.
    4. Optional. In the Administration database field, if the database has an administration database, enter the name of the MongoDB administration database that is used to grant access permission. A MongoDB administrator should let you know whether the database that you want to access has an administration database.
    5. Optional. In the Replica set name field, specify the name of the MongoDB replica set (cluster) that the connection can access.
    6. In the Database field, enter the name of the MongoDB database that contains the collection(s) that you want the connection to read.
  5. Click Validate. Fuse Online immediately tries to validate the connection and displays a message that indicates whether validation is successful. If validation fails, revise the input parameter(s) and try again.
  6. When validation is successful, click Next.
  7. In the Name field, enter your choice of a name that helps you distinguish this connection from any other connections. For example, you might enter MongoDB North.
  8. In the Description field, optionally enter any information that is helpful to know about this connection.
  9. Click Save to see that the connection you created is now available. If you entered the example name, you would see that MongoDB North appears as a connection that you can choose to add to an integration.

Next step

Add your MongoDB connection to an integration.

17.2. Triggering an integration when a MongoDB collection is updated

To trigger execution of an integration when a document is added to a MongoDB collection, add a MongoDB connection to a simple integration as its start connection. When the integration is running, the MongoDB connection monitors the database collection that you specified when you created the integration. Insertion of a new document into the collection that the connection is configured to monitor triggers execution of the integration. The MongoDB connection returns the new document to the integration and this data is available to the next step in the integration.

Prerequisites

  • You created a MongoDB connection.
  • You know the name of the MongoDB collection that you want the connection to read.
  • If the collection that you want the connection to monitor is a capped collection, then you must also know:

    • The name of the MongoDB collection field that tracks incoming documents.
    • Whether persistent tracking is enabled for the database that contains the collection.
    • How to specify tail tracking information related to the data that you want the connection to read. MongoDB connections can use tail tracking information to resume an action after an unexpected connection termination. Tail tracking ensures that when the connection resumes its activity, it does not return duplicate data to the integration nor does it drop any data that needs to be returned to the integration.

Procedure

  1. In the Fuse Online panel on the left, click Integrations.
  2. Click Create Integration.
  3. On the Choose a connection page, click the MongoDB connection that is configured to access the database that you want the integration to obtain updates from.
  4. On the Choose an action page, select one of these actions:

    • Retrieve documents (stream) is typically the action that you want to select. It retrieves data from non-capped collections.
    • Retrieve documents (tail) is the action for retrieving data from capped collections.
  5. Configure the action:

    1. In the Collection name field, enter the name of the MongoDB collection that you want the connection to read.

      If you are configuring the Retrieve documents (stream) action, this is the only required parameter. Go to step 6. If you are configuring the Retrieve documents (tail) action, follow the remaining substeps.

    2. In the Collection field used to track incoming documents field, enter the name of the relevant MongoDB collection field. This field contains data that can be sorted, such as a date or an ID.
    3. Select the Persistent tracking enabled checkbox if persistent tracking is enabled for the database that you want the connection to read.
    4. Optional. In the Unique id to identify this tracking process field, specify a unique ID if there are multiple consumers that monitor the same collection. For more information, see the Camel documentation for Tailable Cursor Consumer.
    5. Optional. In the DB used to store tail tracking field, enter the name of the MongoDB database that contains tail tracking information.
    6. Optional. In the Collection used to store tail tracking field, enter the name of the collection that contains tail tracking information.
    7. Optional. In the Field used to store tail tracking field, enter the name of the field that stores tail tracking information for the collection that this connection is reading.

    To identify the tail tracking field, you can specify the tail tracking database or tail tracking collection. With one of these pieces of information, the connection can find the tail tracking field.

  6. Click Next to specify the action’s output type.
  7. In the Select Type field, if the data type does not need to be known, accept Type specification not required and click Next. You do not need to follow the rest of these instructions.

    However, a structured data type is recommended. For example, if you want to map the connection output in a data mapper step then you must specify the data type. The data mapper cannot display fields for unstructured data.

    To specify the data type, click in the Select Type field and select one of the following as the schema type:

    • JSON schema is a document that describes the structure of JSON data. The document’s media type is application/schema+json.
    • JSON instance is a document that contains JSON data. The document’s media type is application/json.
    • XML schema is a document that describes the structure of XML data. The document’s file extension is .xsd.
    • XML instance is a document that contains XML data. The document’s file extension is .xml.
    • CSV instance is a document that contains comma-separated value (CSV) data. The document’s file extension is .csv.

      Note Data Mapper support for CSV data is a Technology Preview feature only.

  8. In the Definition input box, paste a definition that conforms to the schema type you selected. For example, if you select JSON schema then you would paste the content of a JSON schema file, which has a media type of application/schema+json.
  9. In the Data Type Name field, enter a name that you choose for the data type. For example, if you are specifying a JSON schema for vendors then you might specify Vendor as the data type name.

    You will see this data type name when you are creating or editing an integration that uses the connection for which you are specifying this type. Fuse Online displays the type name in the integration visualization and in the data mapper.

  10. In the Data Type Description field, provide information that helps you distinguish this type. This description appears in the data mapper when you hover over the step that processes this type.
  11. Click Next.

Result

The integration now has a start connection and Fuse Online is prompting you to choose the finish connection.

During execution, the connection returns the inserted document in JSON format.

17.3. Operating on documents in a MongoDB collection

In an integration, you can search for, update, delete, count, or add a document to a MongoDB collection. To do this, add a MongoDB connection to the middle of a flow or as a simple integration’s finish connection.

Prerequisites

  • You created a MongoDB connection.
  • You know the name of the MongoDB collection that you want the connection to operate on.
  • You are creating or editing an integration and Fuse Online is prompting you to add to the integration. Or, Fuse Online is prompting you to choose a finish connection.

Procedure

  1. On the Add to Integration page, click the plus sign where you want to add a MongoDB connection. Skip this step if you are adding a simple integration’s finish connection.
  2. Click the MongoDB connection that you want to use.
  3. On the Choose an action page, select the action that you want the connection to perform.
  4. To configure the action, in the Collection name field, enter the name of the collection that you want the connection to operate on.

    If you selected the Insert action:

    • You are done configuring the connection.
    • Ensure that the data that the integration passes to this MongoDB connection contains JSON notation that specifies the document to add to the collection.
    • Click Next to add this connection to the integration.
  5. For all other actions, the following table provides information about what to specify in the second field for configuring the action.

    The second field is a JSON notation expression that identifies the document(s) to operate on and possibly, details for the operation. If you plan to map an incoming source field to an identifier in the expression, specify :#variable-name in the expression.

    For completeness, the Insert action is described in this table as well.

    ActionWhat it doesParameter fieldReturns

    Count

    Counts the number of documents that satisfy the count expression that you specify.

    In the Count expression field, specify a JSON filter expression in the form of {"test":"unit"}. The action counts the number of documents that have a test attribute value of unit.

    An instance of java.lang.Long that specifies the number of documents that were found.

    Delete

    Removes one or more documents from the collection.

    In the Delete expression field, specify a JSON expression in the form of {"test":"unit"}. The operation removes each document whose test attribute has a value of unit.

    An instance of java.lang.Long that specifies the number of documents that were deleted.

    Find

    Looks for all documents that match the filter expression that you specify.

    In the Filter expression field, specify a JSON notation string that identifies one or more documents.

    JSON notation that specifies a list (an array). Each list element identifies a document that matches the filter expression and provides the document content. If no documents are found, the action returns an empty list.

    You must be familiar with the data model of the collection. If the collection adheres to a JSON schema, you must be familiar with this schema because the returned list elements adhere to this schema. For additional information, see MongoDB Data Models.

    To use output data from the Find action in a subsequent data mapper step, the collection must adhere to a JSON schema. This enables the data mapper to display data fields. For more information, see MongoDB documentation for JSON schema validation.

    Insert

    Adds one or more documents to the collection.

    There is no action configuration parameter for inserting a document. The data coming in to the MongoDB connection must contain JSON notation that specifies one ore more documents to add to the collection.

    JSON notation that specifies a list (an array). Each list element is a document that this action inserted.

    Update

    Updates one or more documents in the collection.

    In the Filter criteria field, specify an array of 2 arguments. The first argument is a filter expression that identifies the documents to update. The second argument is a MongoDB update statement that specifies how to update the documents. Update statement format is described in https://docs.mongodb.com/manual/reference/method/db.collection.update/. For example: [{"_id":11},{$set: {"test":"updated!"}}].

    An instance of java.lang.Long that specifies the number of documents that were updated.

    Upsert

    Updates one or more documents in the collection or creates a document if no documents match the Upsert expression.

    In the Upsert expression field, specify JSON notation that identifies one or more documents. The action uses this expression to detect any matching documents. If the action finds a matching document(s) in the collection, it updates it. If the action does not find a matching document, the action creates a new document. The data coming into the MongoDB connection must contain JSON notation that specifies:

    * An _id field to identify whether it is an insert or an update operation.

    * The content to add or update.

    An instance of com.mongodb.client.result.UpdateResult that specifies either the number of updated documents or the ID of the new, inserted document.

    Each action returns a response. If this is a finish connection, then the response is lost, but you can add a Log step to capture the response.

  6. Click Next.
  7. Optional. If this is a finish connection, add a Log step to finish the integration instead. Configure the Log step to capture the response.

Result

The connection appears in the integration visualization where you added it.

Next steps

Add any other needed connections to the flow. Add any other steps that process data. Complete the integration by adding any needed data mapping steps. When you configured the MongoDB connection action, if you specified any :#variable-name identifiers in an action expression, be sure to map a source field to the MongoDB connection variable field.