Grok Patterns
Although this article is a part of the Self-managed ELK Stack articles, it can be read as an independent module as well due to the fact that Grok patterns are a common standard and not ELK specidic.
Grok Patterns
Parsing timestamps
%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}
Ideal method
filter {
if [type] == "artim-learning" {
grok {
match => {
"message" => [
"%{TIMESTAMP_ISO8601:logdate} ....other fields..."
}
}
}
date {
match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS" ]
}
}
}
Add field to grok pattern
filter {
grok {
match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP}%{SPACE}%{BASE16NUM:docker_id}%{SYSLOG5424SD}%{GREEDYDATA:python_log_message}" ]
add_field => { "container_id" => "%{docker_id}" }
}
}
Example Log
# 1,30-12-2020 15:56:59,Thor,Asgard,Mjolnir and Stormbreaker,27-Jan-2021,100,97.29
1,%{DATESTAMP:log_ts},%{WORD:hero},%{WORD:origin},%{DATA:weapons},%{DATA:curr_date},%{INT:dps},%{NUMBER:hp}
NOTE 1 : WORD will parse only a single word whereas DATA uses a greedy approach to parse multiple words
NOTE 2 : Specifying {INT:dps} doesn’t parse the value to an integer but a string only. To convert the value to an integer use mutate
filter {
mutate {
convert => { "dps" => "integer" }
}
}
Custom formats of timestamps
date {
match => ["log_ts", "YYYY-MM-dd HH:mm:ss.SSS"]
target => "@timestamp"
}