The challenge
The government body in question receives a large amount of information from hundreds of different sources. They wanted to store this information in one central system. To establish a central logging platform, where infrastructure logging, application logging, and authentication logging could be stored centrally, they decided to start with Elastic in 2018.
The solutions
Clearly defining the requirements
To ensure that the large volume of incoming log sources is processed in a structured and consistent manner, it was important to clearly define the requirements regarding the type of logs and align them with the Elastic Common Schema (ECS). The ECS is a shared data model designed to capture log data from various sources in a standardized way. This allows for more efficient log analysis and quicker interpretation and application of insights derived from the logs. As a result, the processing of logs is streamlined, enabling faster and more accurate response to any issues or trends observed in the logs.
Control of the environment
As the platform grows, the required resources must be considered. The government body has everything self-managed in different data centers. Elasticsearch employs the concept of data tiers: hot-warm-cold. This means that different resource criteria apply to each tier.
For the hot tier, the most powerful resources are used with less storage space, while less powerful (less expensive) resources are used for the warm tier, but with more storage capacity. This pattern continues for the cold and frozen tiers.
With such a scale of the hardware platform, it is important to maintain control over costs. Additionally, there is a desire to have control over licensing costs, which depend on the chosen subscription.
Automation
Large-scale environments, such as the one in this government body, are impossible to maintain manually. Hence, the need arose for a solution to automate this process. Through a structured and standardized approach, all resources and variables can be planned, enabling their consistent and reusable usage.
Since the organization lacked the necessary expertise in Elastic, experts from Devoteam were engaged to assist. Our experts automated and simplified the installation and configuration of Elastic using Ansible and Terraform.
Knowledge and expertise
In addition to Elastic as the core technology, we provide expertise in other technologies such as Kafka, Docker, and Logstash. This was necessary to bring the logging from the source systems to the Elastic platform.
The benefits
By leveraging Devoteam’s expertise in Elastic, there are several benefits for the government body, including:
- All logs are now stored in one central location.
- The environment that Devoteam helped build has become manageable. Despite its size making it complex, it is still manageable.
- The government body can now search through all stored data in one central place.
- In addition to searching logs, the government body can develop dashboards and visualize the data.
- The DevOps teams are also able to monitor application logs and technical logs.