Hi folks,

Try the following to send rsyslog output as JSON:

/etc/rsyslog.d/logstash.conf

template(name="ls_json"
         type="list"
         option.json="on") {
           constant(value="{")
             constant(value="\"@timestamp\":\"")         property(name="timegenerated" dateFormat="rfc3339")
             constant(value="\",\"@version\":\"1")
             constant(value="\",\"message\":\"")         property(name="msg")
             constant(value="\",\"host\":\"")            property(name="hostname")
             constant(value="\",\"logsource\":\"")       property(name="fromhost")
             constant(value="\",\"severity_label\":\"")  property(name="syslogseverity-text")
             constant(value="\",\"severity\":\"")        property(name="syslogseverity")
             constant(value="\",\"facility_label\":\"")  property(name="syslogfacility-text")
             constant(value="\",\"facility\":\"")        property(name="syslogfacility")
             constant(value="\",\"program\":\"")         property(name="programname")
             constant(value="\",\"pid\":\"")             property(name="procid")
             constant(value="\",\"priority\":\"")        property(name="pri")
             constant(value="\",\"rawmsg\":\"")          property(name="rawmsg")
             constant(value="\",\"syslogtag\":\"")       property(name="syslogtag")
             constant(value="\",\"appname\":\"")         property(name="app-name")
           constant(value="\"}\n")
         }

*.* @logserver.example.com:5500;ls_json

On the Logstash side (I'm using the latest logstash-1.3.3-flatjar too for now):
/opt/logstash/syslog.conf

input {
  syslog {
    type => syslog
    port => 5544
  }
  udp {
    type => syslogjson
    port => 5500
    codec => "json"
  }
}

output {
  elasticsearch {
    embedded => true
  }
}

I've been using port 5544 to take raw rsyslog data from machines, but was getting frustrated that some messages would produce __grokparsefailure (run-parts most often, but also sssd sometimes). Raw files also parse the "program" field for things like postfix into "postfix/local", "postfix/pickup", "postfix/qmgr", "postfix/cleanup", etc.

Getting rsyslog to chuck out native JSON directly to port 5500 leaves me with no __grokparsefailures, but am now wondering what I can do about the "program" fields. Haven't used the rsyslog property replacer before, so am trying to figure out how to create a new "subprogram" field. My current setup puts (using postfix as an example again) "postfix" in the program field, but the remainder of the program field - the "local", "pickup", "qmgr", "cleanup" don't go anywhere. I'm sending through the "rawmsg" at the moment while I try to figure it out.

Would be nice to send JSON to logstash that requires no parsing and populates extra fields with some intelligence. Also, Kibana has habit of splitting the programs up a bit too much when looking at a program pie chart. 47 "run-parts" log entries will show up as 47 "run" entries and 47 "parts" entries. Not the worst example, but splitting puppet-agent into "puppet" and "agent", postfix into "postfix", "qmgr", cleanup", "pickup", "local" etc starts to blur the numbers a bit too much.

Cheers

Duncan

There's also an RPM out there, but I've not got round to trying it out yet.

D

That's my next step as it happens. I'm planning to roll out a design with failover Redis servers at each site, failover Logstash servers at each site, and an ElasticSearch cluster at one site.

I've heard that ES could get confused if the inter-site link went down, but both sides were still receiving data from their agents.

I've looked into the Paramedic plugin for ES clusters. Can't give any feedback other than that it looks good on the demo site.