Do you know: how to use multiple sources as input for the Advanced Watcher?

Each week, a new “Do You Know” will be posted on our Elastic Technical Knowledge Hub to share useful knowledge to improve the observability using Elasticsearch. These topics originate from day-to-day challenges we solved for our clients. A stepwise description helps you to successfully implement solutions on improving the performance of your deployment and get the best monitoring of your applications using dashboards and alerting.

This week I will discuss: how to use multiple sources as input for the Advanced Watcher.

Background

Alerting is one of the most important tools in monitoring. It may occur that you want to query multiple data sources to perform different queries per data source. Elastic offers the possibility to do that with the use of a chain input.

Solution

Step 1

In Kibana go to the menu on the left side and choose Stack Management → Watcher → Create Advanced Watch. By default you get this json:

{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "input": {
    "search": {
      "request": {
        "body": {
          "size": 0,
          "query": {
            "match_all": {}
          }
        },
        "indices": [
          "*"
        ]
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 10
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 10."
      }
    }
  }
}

Step 2

In order to query multiple data sources in a single alert, you must use the chain input function.

"input" : {
  "chain" : {
    "inputs" : [ 
      {
        "first" : {
         "search": {
          "request": {
            "body": {
             "size": 0,
               "query": {
                 "match_all": {}
          }
        },
        "indices": [
          "*"
        ]
      }
    }
        }
      },
      {
        "second" : {
          "search": {
            "request": {
              "body": {
                "size": 0,
                 "query": {
                  "match_all": {}
          }
        },
        "indices": [
          "*"
        ]
      }
    }
        }
      }
    ]
  }
}

Step 3

The following syntax must be used for the outcome of the input per individual data source:

ctx.payload.first.aggregations… for the first data source,

ctx.payload.second. aggregations… for the second data source.

This reference data loaded by a particular input can subsequently be used in the condition or the output of the watcher.