FTP consumer thread safety - what is an 'endpoint'
Environment
- Fuse ESB
- 7.1
Issue
- The Camel documentation says "The FTP consumer (with the same endpoint) does not support concurrency (the backing FTP client is not thread safe). You can use multiple FTP consumers to poll from different endpoints. It is only a single endpoint that does not support concurrent consumers." http://camel.apache.org/ftp2.html.
Issues is with a route configuration that has multiple consumer endpoints on the same ftp server, with the same accounts, but different path:
Pseudo code:
Endpoint input01 = getContext().getEndpoint("ftp:localhost/input01?username=test&password=test");
Endpoint input02 = getContext().getEndpoint("ftp:localhost/input02?username=test&password=test");
In view of above mentioned limitation - does this count as two endpoints or as one ? In other words, can we expect the two consumers to be thread-safe ?
Resolution
- We have already known that camel resolves endpoint based on it's uri. The uri string also includes options. Different option will result a new endpoint to be created. However, the order (or position) of the option in your uri string will not count. So following two uri strings will represent the same FTP endpoint since they have the same set of options although ordering of the options were different:
String uri1="ftp:localhost/ftpsample/input01?username=joe&password=test&readLock=changed&delay=5000&binary=true&passiveMode=true";
String uri2="ftp:localhost/ftpsample/input01?username=joe&password=test&delay=5000&binary=true&passiveMode=true&readLock=changed";
And if you are trying to use the above two uri strings in two separated camel routes under the same camel context as FTP consumers, you will get following error message:
10:05:24,653 | WARN | NAPSHOT-thread-1 | BlueprintCamelContext | 150 - org.apache.camel.camel-blueprint - 2.10.0.fuse-71-047 | Error occurred during starting CamelContext(blueprintContext)
org.apache.camel.FailedToStartRouteException: Failed to start route route01audit because of Multiple consumers for the same endpoint is not allowed: Endpoint[ftp://localhost/ftpsample/input01?binary=true&delay=5000&ftpClient=%23forloopFtpClient&passiveMode=true&password=******&readLock=changed&username=joe]
Camel will create an endpoint first and an endpoint is an object with all necessary configuration attached to it. Then endpoint will create consumers and producers and these objects will be concrete objects to function. For the most of components, one endpoint can create multiple consumers and producers. And you might also configure concurrent consumers. For instance, you can configure "concurrentConsumers" option for camel-jms component like:
from("activemq:testQueue?concurrentConsumers=10")...
However, for camel-ftp (or camel-file) component, it is not possible to use "concurrentConsumers" option since it does not support multiple consumers per camel-ftp endpoint and in fact this option does not exist for camel-ftp and camel-file endpoint either.
So for camel-ftp component, it is one to one relationship between FTP endpoint and FTP consumer. And it is also true that one FTP endpoint will have only one FTPClient instance. Therefore, a FTP consumer will start polling file periodically from it's own thread and use the FTPClient instance to send ftp commands over to FTP server such as change directory or get file and so on.
It should be no problem if two FTP consumers (obviously from two different FTP endpoints) are trying to poll files from the same directory as by default, each FTP consumers will have it's own FTPClient instances. The only downside will be that the two FTP consumer (or we should say two FTPClient) will compete for resources/files from the FTP server on the same directory though.
Let us see what is going to happen if you are trying to force two FTP consumers to share the same FTPClient instance. Say one consumer is about to change to directory "input01/" from current "ftpsample/" directory while the other consumer is about to change to directory "input02/" from the same location "ftpsample/". In the first thread, the FTPClient instance sent a command "CWD input01" to FTP server and it was successful so the ftpclient was already in the directory "ftpsample/input01/" and about to start scanning list of files and polling them. Then at the same time, the same FTPClient instance sent another command "CWD input02" from a different thread to the FTP server and asked to change directory again. Obviously the command "CWD input02" would fail because the FTPClient was already in "ftpsample/input01" folder and it won't be able find "ftpsample/input01/input02" directory to change. The second thread will receive a failure 550 as a reply.
The endpoint "input01" and "input01_processed" are two different endpoints but they share the same FTPClient instance because you configured "CUSTOM_CLIENT" value to "true so it appended one more option "&ftpClient=#forloopFtpClient" to the endpoint uri.
The bean reference "#forloopFtpClient" was configured in OSGI-INF/blueprint/blueprint.xml as:
<bean id="forloopFtpClient" class="com.mycompany.ftpsample.FtpClient">
</bean>
by default it has a singleton scope which means that there will be only one bean instance of "com.mycompany.ftpsample.FtpClient" type to be created and all FTP consumers will have to share this FTPClient instance. That's why you were seeing 550 error.
- To fix this problem, you might want to
- change "CUSTOM_CLIENT" value to false so every FTP endpoint will create it's own FTPClient instance for associated FTP consumer;
- or you can change the blueprint bean "forloopFtpCient" scope to "prototype":
<bean id="forloopFtpClient" class="com.mycompany.ftpsample.FtpClient" scope="prototype">
</bean>
so the blueprint bean will create one instance of the bean per requester, here per FTP endpoint. Therefore, they will not share the same FTPClient instance.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
