Introduction

ELK is a combination of Elasticsearch, Logstash and Kibana. Together with Beats the combination is a self-sufficient pipeline to collect logs, parse and extract data points, store them in a non relational database that provides querying and indexing data followed by creating powerful and stunning visualizations in real time, in one swift motion. This post would describe the setup of a self-managed ELK stack to ingest custom logs using Filebeat.

Note : Given that ELK setup consitutes of multiple moving parts along with a huge number of add-ons, custom configurations as well as additional concepts like Index templates and Lifecycle policies, I would be adding multiple links to documentations and this article would not be an exhaustive guide to understand and setup the complete stack. This article offers a basic understanding of what ELK is, what are the important documentations you should go through, what are the latest configurations you would need, along with solutions to multiple errors or known bugs that consume appreciable time and effort to discover and resolve.

The entire ELK stack documentation is ginormous and I mention only the documentations you would require for the initial setup and depending on your use-case you might need to read up additional documentations.

Components

Beats : Data collection
- Filebeat - collect and ship log files, most commonly used beat
- Auditbeat - audit user and process activity on Linux servers
- Metricbeat - monitor various PC and OS stats like system-level CPU usage, memory, file system, disk IO, and network IO statistics
Logstash : Data aggregation and processing
ElasticSearch : Indexing and storage
Kibana : Analysis and visualization

Policies and Index Patterns

Before starting this section I would ask the reader to understand what is an Elasticsearch Index, Lifecycle policy and Index lifecycle management. The complete documentation can be found here.

To create a fresh data stream, create a custom lifecycle policy.

Create a new index pattern to select data belonging to a particular index and define their properties. Once additional data fields are added to an index, refresh the field list for the changes to take place.Once the fields are declared with a specific data type, they cannot be changed and a new index pattern has to be created.

Filebeat

Logstash

Kibana

Logging and Troubleshooting

Error : ```config error => “mapper_parsing_exception” reason => “failed to parse field [host] of type [text] in documents with id ‘blah blah’. Preview of field’s value: ‘{name=my.host.name}’” caused_by => “illegal state exception” reason => “Can’t get text on START_OBJECT at 1:974”

	[Solution 1](https://discuss.elastic.co/t/failed-to-parse-field-host-of-type-text-cant-get-text-on-a-start-object-at-1-974/235221/2) :

 Solution 2 : I had to add index in the output section of logstash config
 ```yaml
 output {
   elasticsearch {
     index => "hft-filebeat"
     hosts => ["localhost:9200"]
   }
 }

Filebeat logs

 journalctl -f -u filebeat.service
 cat /var/log/filebeat/filebeat.log

Logstash logs

 sudo journalctl -f -u logstash.service
 cat /var/log/logstash/logstash-plain.log

FAQ

Deleting a range of index logs

Open dev console of Kibana at http://<ip_address>:5601/app/kibana#/dev_tools/console
Make a delete query

POST hft-filebeat/_delete_by_query
{
 "query": {
  "range" : {
   "@timestamp" : {
    "gte" : "12/05/2020",
    "lte" : "12/09/2020",
    "format": "MM/dd/yyyy||yyyy"
     }
    }
  }
} 

Sources

Docs
Forum

Making GET and PUT requests

To make GET and PUT requests, open dev console of Kibana(http://<ip_address>:5601/app/kibana#/dev_tools/console)

Setting up ELK Stack