Red Hat Training

A Red Hat training course is available for Red Hat Fuse

Chapter 136. Splunk

Splunk Component

Available as of Camel 2.13
The Splunk component provides access to Splunk using the Splunk provided client api, and it enables you to publish and search for events in Splunk.
Maven users will need to add the following dependency to their pom.xml for this component: 
	<dependency>
    	<groupId>org.apache.camel</groupId>
    	<artifactId>camel-splunk</artifactId>
    	<version>${camel-version}</version>
	</dependency>

URI format

  splunk://[endpoint]?[options]

Producer Endpoints:

Endpoint
Description
stream
Streams data to a named index or the default if not specified. When using stream mode be aware of that Splunk has some internal buffer (about 1MB or so) before events gets to the index. If you need realtime, better use submit or tcp mode.
submit
submit mode. Uses Splunk rest api to publish events to a named index or the default if not specified.
tcp
tcp mode. Streams data to a tcp port, and requires a open receiver port in Splunk.
When publishing events the message body should contain a SplunkEvent.
Example 
      from("direct:start").convertBodyTo(SplunkEvent.class)
          .to("splunk://submit?username=user&password=123&index=myindex&sourceType=someSourceType&source=mySource")...
In this example a converter is required to convert to a SplunkEvent class.

Consumer Endpoints:

Endpoint
Description
normal
Performs normal search and requires a search query in the search option.
savedsearch
Performs search based on a search query saved in splunk and requires the name of the query in the savedSearch option.
Example 
      from("splunk://normal?delay=5s&username=user&password=123&initEarliestTime=-10s&search=search index=myindex sourcetype=someSourcetype")
          .to("direct:search-result");
camel-splunk creates a route exchange per search result with a SplunkEvent in the body.

URI Options

Name
Default Value
Context
Description
host
localhost
Both
Splunk host.
port
8089
Both
Splunk port
username
null
Both
Username for Splunk
password
null
Both
Password for Splunk
connectionTimeout
5000
Both
Timeout in MS when connecting to Splunk server
useSunHttpsHandler
false
Both
Use sun.net.www.protocol.https.Handler Https hanlder to establish the Splunk Connection. Can be useful when running in application servers to avoid app. server https handling.
index
null
Producer
Splunk index to write to
sourceType
null
Producer
Splunk sourcetype arguement
source
null
Producer
Splunk source arguement
tcpReceiverPort
0
Producer
Splunk tcp receiver port when using tcp producer endpoint.
initEarliestTime
null
Consumer
Initial start offset of the first search. Required
earliestTime
null
Consumer
Earliest time of the search time window.
latestTime
null
Consumer
Latest time of the search time window.
count
0
Consumer
A number that indicates the maximum number of entities to return. Note this is not the same as maxMessagesPerPoll which currently is unsupported
search
null
Consumer
The Splunk query to run
savedSearch
null
Consumer
The name of the query saved in Splunk to run
streaming
false
Consumer
Camel 2.14.0 : Stream exchanges as they are received from Splunk, rather than returning all of them in one batch. This has the benefit of receiving results faster, as well as requiring less memory as exchanges aren't buffered in the component.

Message body

Splunk operates on data in key/value pairs. The SplunkEvent class is a placeholder for such data, and should be in the message body for the producer. Likewise it will be returned in the body per search result for the consumer.

Use Cases

Search Twitter for tweets with music and publish events to Splunk 
      from("twitter://search?type=polling&keywords=music&delay=10&consumerKey=abc&consumerSecret=def&accessToken=hij&accessTokenSecret=xxx")
          .convertBodyTo(SplunkEvent.class)
          .to("splunk://submit?username=foo&password=bar&index=camel-tweets&sourceType=twitter&source=music-tweets");
To convert a Tweet to a SplunkEvent you could use a converter like 
@Converter
public class Tweet2SplunkEvent {
    @Converter
    public static SplunkEvent convertTweet(Status status) {
        SplunkEvent data = new SplunkEvent("twitter-message", null);
        //data.addPair("source", status.getSource());
        data.addPair("from_user", status.getUser().getScreenName());
        data.addPair("in_reply_to", status.getInReplyToScreenName());
        data.addPair(SplunkEvent.COMMON_START_TIME, status.getCreatedAt());
        data.addPair(SplunkEvent.COMMON_EVENT_ID, status.getId());
        data.addPair("text", status.getText());
        data.addPair("retweet_count", status.getRetweetCount());
        if (status.getPlace() != null) {
            data.addPair("place_country", status.getPlace().getCountry());
            data.addPair("place_name", status.getPlace().getName());
            data.addPair("place_street", status.getPlace().getStreetAddress());
        }
        if (status.getGeoLocation() != null) {
            data.addPair("geo_latitude", status.getGeoLocation().getLatitude());
            data.addPair("geo_longitude", status.getGeoLocation().getLongitude());
        }
        return data;
    }
}
Search Splunk for tweets 
      from("splunk://normal?username=foo&password=bar&initEarliestTime=-2m&search=search index=camel-tweets sourcetype=twitter")
          .log("${body}");

Other comments

Splunk comes with a variety of options for leveraging machine generated data with prebuilt apps for analyzing and displaying this. For example the jmx app. could be used to publish jmx attributes, eg. route and jvm metrics to Splunk, and displaying this on a dashboard.

See Also