Chapter 10. Connecting to an FTP or SFTP server

In an integration, you can connect to an FTP or SFTP server to download or upload files. To do this, create an FTP or SFTP connection and then add it to an integration flow.

For the first step in an integration, you can download files that you define with an Apache Camel File Language expression.

For the middle or last step in an integration, you can upload files that you define with an Apache Camel File Language expression. For an FTP server only, you can also download or upload a named file (where the previous step in the integration provides the name of the file).

The following topics provide details:

10.1. Creating an FTP or SFTP connection

In an integration, to download or upload files from/to an FTP or SFTP server, create a connection to that FTP or SFTP server. You can add the same connection to any number of integrations.

Prerequisite

You must know the host name of the server you want to connect to.

Procedure

  1. In Fuse Online, in the left panel, click Connections to display any available connections.
  2. Click Create Connection to display Fuse Online connectors.
  3. To create a connection that uses File Transfer Protocol, click the FTP connector.

    Or, to create a connection that uses Secure File Transfer Protocol, click the SFTP connector.

  4. Configure the connection.

    • For an FTP connection:

      • Host is the only parameter that you must specify. Enter the host name of the server that you want to connect to. For example, if the name of your FTP host is FTP.WEST, then you would enter exactly that, FTP.WEST. Do not specify the protocol, for example, you should not specify something like this: ftp://FTP.WEST.
      • Port is required and has a default value of 21. This is the port that the FTP server is listening on.
      • All other parameters are either not required or have default values. The defaults are suitable for most integrations. Descriptions of these parameters are after this procedure.
    • For an SFTP connection, there must be values for these parameters:

      • Host is the host name of the SFTP server that you want to connect to. For example, if the name of your SFTP host is SFTP.EAST, then you would enter exactly that, SFTP.EAST. Do not specify the protocol, for example, you should not specify something like this: sftp://SFTP.EAST.
      • Port has a default of 22. This is the port that the SFTP server is listening on.
      • User name of the account that you want to use to access the SFTP server.
      • Password that is associated with that user name.
      • All other parameters have default values. The defaults are suitable for most integrations. Descriptions of these parameters are after this procedure.
  5. Click Validate. Fuse Online immediately tries to validate the connection and displays a message that indicates whether or not validation is successful. If validation fails, revise the configuration details as needed and try again.
  6. When validation is successful, click Next.
  7. In the Name field, enter your choice of a name that helps you distinguish this connection from any other connections. For example, enter XLight FTP Server.
  8. In the Description field, optionally enter any information that is helpful to know about this connection.
  9. Click Save to see that the connection you created is now available. If you entered the example name, you would see that XLight FTP Server appears as a connection that you can choose to add to an integration.

Descriptions of other parameters

  • Connect timeout defaults to 10000 milliseconds and indicates a maximum wait of 10 seconds to establish the connection. If 10 seconds elapse without a connection then Fuse Online waits for the number of milliseconds defined by Reconnect delay and then tries to reconnect.
  • Reconnect delay defaults to 1000 milliseconds and indicates the wait time before trying to reconnect again.
  • Maximum reconnect attempts defaults to 3. Fuse Online tries as many as 3 times to establish a connection.
  • Binary file transfer mode is used by default. Select No for ASCII transfer mode.
  • Passive connection mode defaults to Yes, which is usually the preferred mode. In passive mode, the client opens communication channels with the server as a way to avoid firewall issues. If you select No then active mode is used.
  • Disconnect from the server after use defaults to No. The connection remains established after it performs the action. Select Yes if you want to disconnect from the server after the connection performs the upload or download.
  • Data timeout defaults to 30000 milliseconds and indicates the maximum length of time that Fuse Online waits for a reply.

10.2. Obtaining files from an FTP or SFTP server

To trigger integration execution when an FTP or SFTP connection finds the file(s) you are interested in, add an FTP or SFTP connection as an integration’s start connection.

Prerequisite

You created an FTP or SFTP connection.

Procedure

  1. In the Fuse Online panel on the left, click Integrations.
  2. Click Create Integration.
  3. On the Choose a connection page, click the FTP or SFTP connection that you want to use to poll an FTP or SFTP server.
  4. On the Choose an action page, select Download.
  5. In the File name expression field, type an Apache Camel File language expression that specifies the file or files to download. (Note that you cannot specify a regular expression.)

    For example, you can filter for all files that have a specific extension from the FTP server. The following example specifies all files with a .doc extension:

    '${file:onlyname.noext}.doc'

    Leave the File name expression field blank if you want to download all files in the FTP directory.

    At runtime, the connection polls (periodically checks) the server for the file or files and downloads them when found.

  6. In the FTP directory field, enter the absolute or relative path of the server directory to poll. The connection watches this directory for any content and downloads all files when it finds any content.
  7. In the Milliseconds before polling starts field, accept the default of 1000 milliseconds or change the number of milliseconds.
  8. In the Milliseconds before the next poll field, accept the default of 500 milliseconds or change the number of milliseconds. This is the interval between polls.
  9. In the Delete after download field, accept the default of No or select Yes to download the file(s) and then delete it(them) from the server.
  10. Click Next to specify the action’s output type.
  11. In the Select Type field, if the data type does not need to be known, accept Type specification not required and click Next. You do not need to follow the rest of these instructions.

    However, a structured data type is recommended. For example, if you want to map the connection output in a data mapper step then you must specify the data type. The data mapper cannot display fields for unstructured data.

    To specify the data type, click in the Select Type field and select one of the following as the schema type:

    • JSON schema is a document that describes the structure of JSON data. The document’s media type is application/schema+json.
    • JSON instance is a document that contains JSON data. The document’s media type is application/json.
    • XML schema is a document that describes the structure of XML data. The document’s file extension is .xsd.
    • XML instance is a document that contains XML data. The document’s file extension is .xml.
    • CSV instance is a document that contains comma-separated value (CSV) data. The document’s file extension is .csv.
  12. In the Definition input box, paste a definition that conforms to the schema type you selected. For example, if you select JSON schema then you would paste the content of a JSON schema file, which has a media type of application/schema+json.
  13. In the Data Type Name field, enter a name that you choose for the data type. For example, if you are specifying a JSON schema for vendors then you might specify Vendor as the data type name.

    You will see this data type name when you are creating or editing an integration that uses the connection for which you are specifying this type. Fuse Online displays the type name in the integration visualization and in the data mapper.

  14. In the Data Type Description field, provide information that helps you distinguish this type. This description appears in the data mapper when you hover over the step that processes this type.
  15. Click Next.

Result

The connection appears at the beginning of the integration visualization.

10.3. Obtaining a named file from an FTP server

In an integration, you can download a specific file from an FTP server in the middle of a flow or to finish a simple integration. To do this, add an FTP connection to the middle of a flow or as a simple integration’s finish connection.

Prerequisites

  • You created an FTP connection.
  • You are creating or editing a flow and Fuse Online is prompting you to add to the integration. Or, Fuse Online is prompting you to choose a finish connection.

Procedure

  1. On the Add to Integration page, click the plus sign where you want to add the connection. Skip this step if Fuse Online is prompting you to choose the finish connection.
  2. Click the FTP connection that you want to use to download files.
  3. On the Choose an action page, select DownloadNamedFile.
  4. In the Delete after download field, accept the default of No or select Yes to download the file and then delete it from the server.
  5. In the FTP directory field, enter the absolute or relative name of a server directory. At runtime, the connection watches this directory for the named file and downloads it when it finds it.
  6. Click Next.

Result

The connection appears in the integration visualization where you added it.

10.4. Uploading files to an FTP or SFTP server

In an integration, you can upload files to an FTP or SFTP server in the middle of a flow or to finish a simple integration. To do this, add an FTP or SFTP connection to the middle of a flow or as a simple integration’s finish connection.

Prerequisites

  • You created an FTP or SFTP connection.
  • You are creating or editing a flow and Fuse Online is prompting you to add to the integration. Or, Fuse Online is prompting you to choose a finish connection.

Procedure

  1. On the Add to Integration page, click the plus sign where you want to add the connection. Skip this step if Fuse Online is prompting you to choose the finish connection.
  2. Click the FTP or SFTP connection that you want to use to upload files.
  3. On the Choose an action page, select Upload.
  4. In the File name expression field, type an Apache Camel File language expression that specifies the file or files to upload. (Note that you cannot specify a regular expression.)

    For example, you can filter for all files that have a specific extension from the FTP server. The following example specifies all files with a .doc extension:

    '${file:onlyname.noext}.doc'

    Leave the File name expression field blank if you want to upload all files in the FTP directory.

  5. In the FTP directory field, enter the absolute or relative name of a server directory. If the File name expression field contains an expression, then the connection stores the specified file in this directory. If the File name expression field is blank, then the connection uploads to this directory all files that were received from the previous step.
  6. In the If file exists field, indicate the behavior when you are uploading a file that has the same path and name as a file that is on the server. Accept the default, Override, to overwrite the file that is on the server with the file that you are uploading. Or, select one of the following:

    • Append adds the content in the file being uploaded to the file that is on the server.
    • Fail throws GenericFileOperationException. The integration does not enter an error state.
    • Ignore does not upload the file. The integration continues running under the assumption that everything is okay.
    • Move renames one of the files.
    • TryRename uploads the file with a temporary name and renames the file to the desired name. This operation does not check for the existence of a file with the desired name, which makes the operation faster on most servers than when existence checks are done.
  7. In the Temporary file prefix while copying field, specify a string. The connection prepends this string to the name of a file while it is being uploaded. This enables the connection to write to a temporary file on the server and then rename that temporary file to have the correct name. This is useful for reducing locks when uploading very large files.
  8. In the Temporary file name while copying field, specify a string. The connection renames a file being uploaded to have this name while it is being uploaded. This enables the connection to write to a temporary file on the server and then rename that temporary file to have the correct name. This is useful for reducing locks when uploading very large files.
  9. Click Next to specify the action’s input type.
  10. In the Select Type field, if the data type does not need to be known, accept Type specification not required and click Next. You do not need to follow the rest of these instructions.

    However, a structured data type is recommended. For example, if you want to map the connection input in a data mapper step then you must specify the data type. The data mapper cannot display fields for unstructured data.

    To specify the data type, click in the Select Type field and select one of the following as the schema type:

    • JSON schema is a document that describes the structure of JSON data. The document’s media type is application/schema+json.
    • JSON instance is a document that contains JSON data. The document’s media type is application/json.
    • XML schema is a document that describes the structure of XML data. The document’s file extension is .xsd.
    • XML instance is a document that contains XML data. The document’s file extension is .xml.
    • CSV instance is a document that contains comma-separated value (CSV) data. The document’s file extension is .csv.
  11. In the Definition input box, paste a definition that conforms to the schema type you selected. For example, if you select JSON schema then you would paste the content of a JSON schema file, which has a media type of application/schema+json.
  12. In the Data Type Name field, enter a name that you choose for the data type. For example, if you are specifying a JSON schema for vendors then you might specify Vendor as the data type name.

    You will see this data type name when you are creating or editing an integration that uses the connection for which you are specifying this type. Fuse Online displays the type name in the integration visualization and in the data mapper.

  13. In the Data Type Description field, provide information that helps you distinguish this type. This description appears in the data mapper when you hover over the step that processes this type.
  14. Click Next.

Result

The connection appears in the integration visualization where you added it.

10.5. Uploading a named file to an FTP server

In an integration, you can upload a specific file to an FTP server in the middle of a flow or to finish a simple integration. To do this, add an FTP connection to the middle of a flow or as a simple integration’s finish connection.

Prerequisites

  • You created an FTP connection.
  • You are creating or editing a flow and Fuse Online is prompting you to add to the integration. Or, Fuse Online is prompting you to choose a finish connection.

Procedure

  1. On the Add to Integration page, click the plus sign where you want to add the connection. Skip this step if Fuse Online is prompting you to choose the finish connection.
  2. Click the FTP connection that you want to use to upload files.
  3. On the Choose an action page, select UploadNamedFile.
  4. In the FTP directory field, enter the absolute or relative name of a server directory. At runtime, the connection uploads the named file (that was received from the previous step) to this directory.
  5. In the If file exists field, indicate the behavior when you are uploading a file that has the same path and name as a file that is on the server. Accept the default, Override, to overwrite the file that is on the server with the file that you are uploading. Or, select one of the following:

    • Append adds the content in the file being uploaded to the file that is on the server.
    • Fail throws GenericFileOperationException. The integration does not enter an error state.
    • Ignore does not upload the file. The integration continues running under the assumption that everything is okay.
    • Move renames one of the files.
    • TryRename uploads the file with a temporary name and renames the file to the desired name. This operation does not check for the existence of a file with the desired name, which makes the operation faster on most servers than when existence checks are done.
  6. In the Temporary file prefix while copying field, specify a string. The connection prepends this string to the name of a file while it is being uploaded. This enables the connection to write to a temporary file on the server and then rename that temporary file to have the correct name. This is useful for reducing locks when uploading very large files.
  7. In the Temporary file name while copying field, specify a string. The connection renames a file being uploaded to have this name while it is being uploaded. This enables the connection to write to a temporary file on the server and then rename that temporary file to have the correct name. This is useful for reducing locks when uploading very large files.
  8. Click Next.

Result

The connection appears in the integration visualization where you added it.