Do you know: how to list multiple results from one aggregation in the output of an advanced watcher?

Each week, a new “Do You Know” will be posted on our Elastic Technical Knowledge Hub to share useful knowledge to improve the observability using Elasticsearch. These topics originate from day-to-day challenges we solved for our clients. A stepwise description helps you to successfully implement solutions on improving the performance of your deployment and get the best monitoring of your applications using dashboards and alerting.

This week I will discuss: how to list multiple results from one aggregation in the output of an advanced watcher in Elasticsearch.

Background

Alerting is one of the most important tools in monitoring. After setting up your alerting in the Advanced Watcher, you want the users to get the complete list of results in the message they receive. So in case you alert on the errors, you want to get the details on all the errors occurred during the time window. 

Solution

Step 1

Set up your Advanced Watcher. Go to the Kibana menu: Stack Management → Watcher → Create → Create advanced watch. For this example, I made an aggregation on ErrorMessage.keyword.

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [],
              "filter": [
                {
                  "match_phrase": {
                    "Severity.keyword": "ERROR"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "format": "strict_date_optional_time",
                      "gte": "now-5m"
                    }
                  }
                }
              ],
              "should": [],
              "must_not": []
            }
          },
          "aggs": {
            "ErrorMessage": {
              "terms": {
                "field": "ErrorMessage.keyword",
                "order": {
                  "_count": "desc"
                },
                "size": 500,
                "shard_size": 25
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 1
      }
    }
  },
   "actions": {
    "my-logging-action": {
      "logging": {
        "text": "The following error occured {{ctx.payload.aggregations.ErrorMessage.buckets.0.key}}"
      }
    }
  }
}

In this example, I have pulled the ErrorMessage that occurred within the time window using the request: {{ctx.payload.aggregations.ErrorMessage.buckets.0.key}}. However, multiple different errors may have occurred and only the first hit in the ErrorMessage bucket is shown now.

So, let’s assume two different error messages occurred. You can list them using: {{ctx.payload.aggregations.ErrorMessage.buckets.0.key}}, {{ctx.payload.aggregations.ErrorMessage.buckets.1.key}}.

Since we do not know how many Errors may occur, we must write the request differently in order to get the complete list of Error Messages.

Step 2

To get all the Error Messages present in the bucket you should write: 

{{#ctx.payload.aggregations.ErrorMessage.buckets}}{{key}} {{/ctx.payload.aggregations.ErrorMessage.buckets}}

By using this way of writing you automatically get all the error messages in the bucket:

{{ctx.payload.aggregations.ErrorMessage.buckets.0.key}} 

{{ctx.payload.aggregations.ErrorMessage.buckets.1.key}}

{{ctx.payload.aggregations.ErrorMessage.buckets.2.key}}

……………………………………………………………………………………..

{{ctx.payload.aggregations.ErrorMessage.buckets.n.key}}