Do you know: how to enrich your document with data from another document in Kibana?

Each week, a new “Do You Know” will be posted on our Elastic Technical Knowledge Hub to share useful knowledge to improve the observability using Elasticsearch. These topics originate from day-to-day challenges we solved for our clients. A stepwise description helps you to successfully implement solutions on improving the performance of your deployment and get the best monitoring of your applications using dashboards and alerting.

This week I will discuss: how to enrich your document with data from another document in Kibana.

Background

It is possible to add information from one document into another document from the same index or another index. This is done using the enrich processor. See the example used here:

We monitor the frontend and backend calls generated by clients who are using a web portal. Client information is generated in the request-in calls. However, when an error occurs, the client information is not included in the error log. To overcome this, we use the enrichment processor to add the client’s account number to the error log. Also, an enrich policy must be created that includes the indices, a match_field, and the enrich_fields. In our example, the index name is webportal calls, match_field is the businessTransactionId, and the enrich field is the account number. The match field is used to map the account number from the request-in into the error log.

Solution

Step 1

Put in the enrich policy in which you identify the matched index, matched field, and the enrich field. In Kibana, go to the menu and choose the Dev Tools. Put in the following PUT function:

PUT /_enrich/policy/webportalcalls-policy (given policy name)
{
  "match": {
    "indices": "webportalcalls",
    "match_field": "businessTransactionId",
    "enrich_fields": ["accountnumber"]
  }
}

Step 2

Execute the policy. In Kibana, go to the menu and choose the Dev Tools. Put in the following POST function.

POST /_enrich/policy/webportalcalls-policy/_execute

Step 3

Add the enrich processor in the pipeline that includes the condition for activation. In Kibana, go to the menu and choose the Dev Tools. Put in the following PUT function:

PUT /_ingest/pipeline/webportalcallspipeline (given pipeline name: webportalcallspipeline)
{
  "processors" : [
    {
      "enrich" : {
        "if": "ctx.Severity == 'ERROR'",
        "description": "Add 'accountnumber' data based on 'businessTransactionId' and Serverity: ERROR",
        "policy_name": "webportalcalls-policy",
        "field" : "businessTransactionId",
        "target_field": "enriched",
        "max_matches": "1"      }
    }
  ]
}

After setting this up, every time an error occurs, the account number based on matching businessTransactionId will be added to the document!