Do you know: how to get the size of your document (log message) as a field value in Kibana?

Each week, a new “Do You Know” will be posted on our Elastic Technical Knowledge Hub to share useful knowledge to improve the observability using Elasticsearch. These topics originate from day-to-day challenges we solved for our clients. A stepwise description helps you to successfully implement solutions on improving the performance of your deployment and get the best monitoring of your applications using dashboards and alerting.

This week I will discuss: how to get the size of your document (log message) as a field value in Kibana?

Background

It might be wise to know the size in bytes of your document (log message) in Kibana. Especially in the situation that images or PDF documents may occur in your document. Having large size documents in your index may cause issues on the performance of Kibana (long loading time or even time outs due to the maximum querying size that has been reached). It is explained here how to add the field “_size” in your log document so you can easily detect large sized documents to ultimately remove unnecessary fields from that document.

Solution

Step 1

Install the mapper-size plugin.

Go to your deployment → choose Edit in the left side menu → at the Elasticsearch section click on: Manage user settings and extensions → go to the top right tab: Extensions and check the mapper-size box.

Now go back to the Edit page and press Save at the bottom of the page and confirm the configuration changes. Your deployment will be updated with the mapper-size plugin installed.

Step 2

Add the field “_size” in the mapping of your index.

In Kibana go to: Stack Management → Index Management → Templates. Edit your existing template and go to the tab: Mappings.

Choose the tab: Advanced Options and activate the index the _source field size in bytes at the bottom of the page.

If you do not have a template yet just put the following command in the Dev Tool page in Kibana:

PUT <index_name>
{
  "mappings": {
    "_size": {
      "enabled": true
    }
  }
}

Step 3

Update the metaFields setting and add “_size” to the list of metaFields. The metaFields can be configured in Kibana from the Advanced Settings page in Management.

Step 4

Perform a rollover on your alias name or index name to activate the change.

Post <alias name>/_rollover

From now on you will see the field “_size” in your logging. It can be used in your filter settings as well!