Multi-line Java stack traces out of order in Logstash and Splunk

Do you have customers frustrated by getting multi-line Java stack traces out of order? We're working on a clean solution in our enhanced metrics work, but here is a workaround courtesy @DavidLaing .

With the Java Logback library you can do this by adding 

"%replace(%xException){'\n','\u2028'}%nopex" 

to your logging config [1] , and then use the following logstash conf.[2]

Replace the unicode newline character \u2028 with \n, which Kibana will display as a new line.

mutate {
  gsub => [ "[@message]", '\u2028', "

"]

^^^ Seems that passing a string with an actual newline in it is the only way to make gsub work

}

[1] https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12

[2] https://github.com/cloudfoundry-community/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64

In Splunk you will need to configure the the event stream to recombine multine exceptions into one event. See http://docs.splunk.com/Documentation/Splunk/5.0/Data/Indexmulti-lineevents