Do you know: how to use Elastic API Keys in Logstash code?

Each week, a new ‘Do You Know’ post will be added to our Elastic Technical Knowledge Hub to share valuable insights for improving observability using Elasticsearch. These topics stem from day-to-day challenges we have solved for our clients. Stepwise descriptions help you successfully implement solutions to enhance the performance of your deployment and achieve optimal monitoring of your applications using dashboards and alerting.

This week, I will discuss: how to use Elastic API Key in Logstash code.

Background

Integrating Elastic API keys into Logstash enhances the security and efficiency of data pipelines. This process involves configuring Logstash to authenticate with Elasticsearch using API keys instead of traditional credentials. By enabling API authentication and creating API keys through Kibana, users can securely manage access and streamline their data workflows. This article provides a guide on setting up and implementing Elastic API keys within Logstash for a robust and secure integration.

What are the security best practices that can be used instead of exposing Elastic credentials in Logstash pipeline?

Solution

Step 1 – API authentication

First, enable it by using the parameter below in the elasticsearch.yml file:

xpack.security.authc.api_key.enabled: true

According to Elastic documentation, it is enabled by default, but it is best to verify this and set it to true if not done already.

Step 2 – Creating API key

Go to Kibana → Stack Management → API Keys → Create API Key.

1. Name the API key
2. Update the values if required
3. Create it
4. Change the key format to JSON and copy the entire JSON file
5. Sample shown below:

{
  "id":"Nr0UroABgN_gBm6G",
  "name":"logstash_key",
  "api_key":"1XgAtLFuSk2lNb7rXHg",
  "encoded":"TnIwVXJvQUJnTlVWX2dCbUhtNkc6QXRMRnVTazJtRnpsTmI3clhIZw="
}

Created API Key

This can also be copied directly in Logstash format like “Nr0UroABgV_gBmHm6G:1XgAtLFuSkzlNb7rXHg”.

Logstash Format

Step 3 – Logstash implementation

Finally, paste this into the pipeline output with other parameters:

output {
  elasticsearch {
    apikey => "Nr0UroABgN_gBm6G:1XgAtLFuSk2lNb7rXHg"
  }
}

Elastic output

Using a plain text username and password in the configuration file to connect to Elasticsearch is not the recommended way for a secure Logstash output connection to Elasticsearch. Using an API key is the preferred method.