What if I told you that Elastic offers robust support for stored search features? With this capability, you can execute queries with a wide range of parameters, much like creating a parametric function in MySQL.
A Search Template is essentially a stored search that you can execute with varying input values. When Elasticsearch serves as your search backend, it enables you to seamlessly pass user inputs from a search bar as parameters for a Search Template. This empowers you to perform searches without exposing the intricate query syntax of Elasticsearch to your end users.
If your application relies on Elasticsearch as a key component, utilizing Search Templates offers you the flexibility to modify your searches without the need to alter your application’s codebase. It’s truly a remarkable feature, isn’t it? So, let’s embark on a deeper exploration of this incredible capability.
Prerequisites
Before writing a Search Template, you need to understand how to write basic DSL Queries. In my previous blog I have already explained this in very simple steps.
In case you are new to Elasticsearch, you can simply go through the blog on mastering DSL essential queries first. Only then you can understand the story of the Search Template.
Let’s begin with the Search Templates
Elasticsearch Search Templates (formerly known as “scripted templates”) allow you to predefine complex queries with placeholders for values that can be filled in at runtime. This can be useful for creating reusable and parameterized queries. I will try to explain the use of Elasticsearch Search Templates through some real world examples.
Suppose you have a requirement from a client in which you have to fetch all the news articles published between 2021-01-01 and 2021-01-08 on index name news_articles.
GET news_articles/_search
{
“query”: {
“range”: {
“publish_date”: {
“gte”: “2021-01-01”,
“lt”: “2021-01-08”
}
}
}
}
If you go through the basic DSL queries, it won’t take much time to write queries for such requirements.
Now, let’s break down this query step by step:
- GET news_articles/_search: This part of the query specifies the HTTP method (GET) and the index (news_articles) on which you want to perform the search. In Elasticsearch, data is organized into indices, and in this case, you are searching within an index named “news_articles.”
- The query body is enclosed within curly braces {} and is defined using JSON format.
- “query”: This is the root element of the query DSL and indicates that you are defining a query for Elasticsearch.
- “range”: Within the query, you are using a “range” query type. A range query allows you to search for documents where a specified field falls within a specified range of values.
- “publish_date”: This specifies the field that you want to apply the range query to. In this case, you are looking at the “publish_date” field of the documents in the “news_articles” index.
- “gte” and “lt”: These are parameters within the range query.
- “gte” stands for “greater than or equal to,” and you have set it to “2021-01-01”, which means you want to find documents where the “publish_date” is greater than or equal to January 1, 2021.
- “lt” stands for “less than,” and you have set it to “2021-01-08”, which means you want to find documents where the “publish_date” is less than January 8, 2021.
- In summary, this Elasticsearch query is searching for documents in the “news_articles” index where the “publish_date” falls within the date range of January 1, 2021, (inclusive) to January 8, 2021 (exclusive). It will return all matching documents that satisfy this condition.
Time to replace above query with a Search Template
Subsequently, another requirement emerged: you needed to retrieve additional weekly articles, spanning various date ranges such as 2021-01-08 to 2021-01-15, 2021-01-15 to 2021-01-22, and more, from the ‘news_article’ index.
As you found yourself repeatedly rewriting the same query with different date ranges, you began to find it tedious. That’s when you stumbled upon an elegant feature called a Search Template. With this feature, you could define a Search Template with customizable parameters, akin to creating a parametric function that takes inputs, like x and y, performs operations on them, and returns a result. In this case, you could achieve the same level of flexibility and efficiency with your Elasticsearch queries.
You can define a Search Template using the Elasticsearch PUT or POST API. You typically store templates in the .scripts index, but you can specify a custom index as well. In our scenario, we will just use the above query and put parameters as mentioned below:
PUT _scripts/weekly_articles
{
“script”: {
“lang”: “mustache”,
“source”: {
“query”: {
“range”: {
“publish_date”: {
“gte”: “{{start_date}}”,
“lt”: “{{start_date}}||+1w”
}
}
}
}
}
}
Note: In this example, {{start_date}} is a placeholder that can be replaced with an actual value at runtime.
Let’s break down this query step by step:
- PUT _scripts/weekly_articles: This part of the query indicates that you are sending an HTTP PUT request to Elasticsearch to create or update a stored script named “weekly_articles.” In Elasticsearch, stored scripts are reusable pieces of code that can be used in various search queries and aggregations.
- The script definition is enclosed within curly braces {} and is defined using JSON format.
- “script”: This is the root element of the script definition and specifies that you are defining a script.
- “lang”: “mustache”: This line indicates the scripting language used for the script. In this case, it’s set to “mustache.” Elasticsearch supports different scripting languages, and “mustache” is one of them.
- “source”: This is where you define the actual script logic. Inside “source”, you have another JSON object.
- “query”: This is the key where you start defining the query logic that the script will use.
- “range”: Within the query, you are using a “range” query type, similar to the previous example. It allows you to search for documents where a specified field falls within a specified range of values.
- “publish_date”: This specifies the field that you want to apply the range query to. In this case, it’s the “publish_date” field.
- “gte” and “lt”: These are parameters within the range query.
- “gte” stands for “greater than or equal to.” It uses the value {{start_date}}, which is a placeholder for a date. The actual date value for “start_date” will be provided when you use this script.
- “lt” stands for “less than.” It uses {{start_date}}||+1w as the value. {{start_date}} here represents the same placeholder for a date, and ||+1w is a Mustache template filter that adds one week to the “start_date” value. This means you want to find documents where the “publish_date” is less than one week after the provided “start_date”.
In summary, this script definition creates a stored script named “weekly_articles” that you can later use in Elasticsearch queries. The script’s purpose is to find documents where the “publish_date” falls within a date range starting from the provided “start_date” and ending one week later. The actual value for “start_date” will be supplied when you use this script in a query.
It’s time to test this
To test a template with different params, use the render Search Template API.
POST _render/template
{
“id”: “weekly_articles”, # name of your Search Template as we created weekly_articles.
“params”: {
“start_date”: “2021-01-01″
}
}
When rendered, the template outputs a search request body. In this request, “id” specifies the name of the Search Template, and “params” provide the values that should replace the placeholders defined in the template.
Execution
GET your_index_name/_search/template
{
“id”: “weekly_articles”,
“params”: {
“start_date”: “2021-04-01”
}
}
In this request, “id” specifies the name of the Search Template, and “params” provide the values that should replace the placeholders defined in the template.
As you are doing so well, so you have got another requirement (requirement 3)
This time the date range is flexible using start_date and end_date parameters.
As you already know about Search Templates and you are following the examples in this blog. So, It is not a tough requirement for you.
Let’s write the Search Template.
We will copy the above template as it is
PUT _scripts/weekly_articles
{
“script”: {
“lang”: “mustache”,
“source”: {
“query”: {
“range”: {
“publish_date”: {
“gte”: “{{start_date}}”,
“lt”: “{{start_date}}||+1w”
}
}
}
}
}
}
In it we will add another parameter {{end_date}} and the job is done. This is how your final query will look like.
PUT _scripts/weekly_articles
{
“script”: {
“lang”: “mustache”,
“source”: {
“query”: {
“range”: {
“publish_date”: {
“gte”: “{{start_date}}”,
“lt”: “{{end_date}}”
}
}
}
}
}
}
It’s testing time
POST _render/template
{
“id”: “weekly_articles”, # name of your Search Template as we created weekly_articles.
“params”: {
“start_date”: “2021-01-01″
“end_date” : “2021-01-08”
}
}
You will notice that the template outputs a search request body with success. That means it is working.
It is time to optimize it
To streamline the management of Search Templates, the client has opted for a more efficient approach: instead of creating separate Search Templates for requirements 2 and 3, they propose developing a single template that can cater to both.
For instance, if the ‘end_date’ parameter is either absent or empty, the template should automatically return weekly articles by simply extending the ‘start_date’ value by one week. On the other hand, if both ‘start_date’ and ‘end_date’ are provided as parameters, the template should retrieve articles within the specified date range.
This template-driven search mechanism employs a default value strategy. To assign a default value to a variable, use the following syntax:
{{my-var}}{{^my-var}}default value{{/my-var}}
In our scenario, the default value is set to {{start_date}}+1w.
Now our query like this:
PUT _scripts/top_blogs
{
“script”: {
“lang”: “mustache”,
“source”: {
“query”: {
“range”: {
“publish_date”: {
“gte”: “{{start_date}}”,
“lt”: “{{end_date}}{{^end_date}}{{start_date}}||+1w{{/end_date}}”
}
}
}
}
}
}
Cleanup process
You can delete a Search Template when it’s no longer needed:
DELETE /_scripts/my_search_template
This will remove the template from Elasticsearch.
Here are some important points to keep in mind
- Elasticsearch supports Mustache templates by default. You can use other template languages by installing plugins.
- Search Templates can make your queries more maintainable and reusable, especially when you have complex queries that need to be reused with slight variations.
- Templates can help prevent SQL injection-like attacks by parameterizing your queries.
- Be cautious when allowing user inputs to directly influence the template parameters, as it can still pose security risks if not properly sanitized.
- Ensure that you have the appropriate permissions to create, read, and delete scripts/templates in your Elasticsearch cluster.
- Elasticsearch Search Templates can be a powerful tool for managing and reusing complex queries, but they should be used with care to avoid security risks and maintain query performance.
I want to extend my heartfelt gratitude to you for dedicating your valuable time to explore this blog. I sincerely hope you found it insightful and gained new knowledge about Elasticsearch Search Templates. If you have any queries or seek further exploration of this topic, please don’t hesitate to reach out.
For those eager to delve deeper into this subject, I encourage you to explore the official Elastic documentation, which provides an extensive resource on Elasticsearch Search Templates.