What about Sustainable Data and DevOps?

The IT industry generates vast amounts of data, and the amount of data is growing exponentially. This exponential growth in data is mirrored by similar growth in data centers and carbon emissions.

Data centers need electricity, and this growing demand for electricity is straining our transmission grid. London and Amsterdam’s transmission grids are already at capacity and cannot currently support more data centers ^[1]. Microsoft is discussing powering data centers with mini nuclear power stations due to the lack of available power^[2].

We have fast-growing data storage requirements, driving the need for more electricity and emitting more carbon. The carbon emissions from data storage are becoming a significant percentage of global emissions.

Furthermore, much of the data we store is of little value. Various estimates suggest that between 60-90% of the data we store is never used. We call this unused data “Dark Data”.

A recent report states, “66% of organizations believe that navigating the complexities of their data landscape is now among their top sustainability challenges”.^[3]

Managing the amount of data we create (for example, by eliminating Dark Data) is a simple way to reign in uncontrolled growth in data storage and data centers, reducing emissions and making our industry more sustainable. It also saves money and increases IT agility!

It’s frequently said that “data is the new gold.” While this influx of data can be monetized, it resembles mining real gold because it generates far more waste than valuable nuggets. Just as we manage valuable resources, we must also manage our data by preserving the valuable components and discarding the waste.

DevOps Data Anti-Patterns

In the DevOps world, several anti-patterns contribute to our ballooning data requirements. For example:

1. Data explosion through automation
2. The Bad Data Virus
3. Delete Nothing, Ever

Follow the links above for more information on these anti-patterns and how we can rectify these anti-patterns to reduce data storage requirements and carbon emissions.

Data Management

To address this challenge, we must adopt a systematic approach to data management, focusing on data of value while limiting and eliminating data of no value. If estimates about “Dark Data” are accurate and much of our stored data serves no purpose, then 

removing such data is a pivotal step toward lowering our carbon emissions and reducing the need for more data centers and electricity.

Follow these steps to begin the journey of Data Management:

1. Conduct a data audit. This will help you to identify all the data that your organisation is storing and to assess how much of it is actually used.
2. Implement a data classification scheme. This will help you to identify the different types of data that your organization stores, and to assign different retention periods to each type of data.
3. Develop a data retention/deletion policy. This policy should specify the criteria that you will use to determine when data should be deleted.
4. Automate your data management processes. This will help you to ensure that your data is managed in a consistent and efficient manner.