This article is the fourth module of the Self-managed ELK Stack article behind Introduction, Filebeat and Logstash. Hence, it is recommended to go through the aforementioned articles before proceeding with Kibana.

Creating visualizations from filtered data

  1. Open Kibana -> Discover
  2. Select appropriate index (sample_index*) and necessary columns
  3. Click on Save to save your discover search so you can use it in visualizations and dashboards
  4. If this is your first time using Kibana visualizations, you must reload your field list before proceeding. To reload, Stack Management -> Index Patterns -> hft_file* -> Refresh file list icon (on top right)
  5. Click on Visualize and create a new visualization
  6. Click on Dashboard and create a new dashboard and add the visualization

Aggregation query

POST /sample_index/_search
{
  "aggs": 
  {
    "eq_entries": 
    {
      "aggs": {
        "min_ts": { "min": { "field": "@timestamp" } }
      },
      "terms": {
        "field": "agent.keyword",
        "size": 10
      }
    }
  }
}

Visualization tools

Most of visualization tools like bar and line charts are pretty straightforward and intuitive to work with. I have mentioned a couple of more advanced tools that are not completely GUI based.

Timelion

Can be used to plot time series sequential data and allows mathematical operations. For instance, it can be used to perform point wise subtraction of two timestamp sequences to obtain latency.

Sample syntax

  1. Offset a sequence of data by 1 hour and plot the count of logs with label last_hour.
  .es(offset=-1h,index=sample-filebeat,timefield='@timestamp',metric=count).label('last_hour')
  1. Plot a bar graph of the frequency of logs containing the field condition_type such that condition_type == exit 
  .es(index=sample-filebeat,timefield='@timestamp', metric=count,q='_exists_ : condition_type AND condition_type : exit').label('exit').bars(width=10,stack=yes),
  .es(index=sample-filebeat,timefield='@timestamp', metric=count,q='_exists_ : condition_type AND condition_type : entry').label('entry').bars(width=10,stack=yes)
  1. Plot the sum of field quantity of logs containing the field condition_type such that condition_type == entry and assign the label entry_sum 
  .es(index=sample-filebeat,timefield='@timestamp', metric='sum:quantity.value',q='_exists_ : condition_type AND condition_type : entry').sum().label('entry_sum')
  1. Visualization
  2. Conditional Logic
  3. Mathematical funtions
  4. Aggregations(Highly useful)
  5. Sparse time series

Vega

Similar to Timelion but provides support for more complex queries such as groupby aggregations.

  1. Visualization
  2. Aggregation