New Data Platforms: The End of ETLs?

This topic could be quite provocative for many experienced Business Intelligence specialists, but let’s raise the debate. Are cloud data platforms like Snowflake and BigQuery the end of ETLs? Explore their impact on data integration.

A Paradigm Shift in the Data World

One might say that new cloud data platforms like Snowflake, BigQuery, and Databricks are redefining how we work with data. These giants raise a crucial question: are we witnessing the end of traditional Extract, Transform and Load (ETL) processes? Let’s dive into this provocative reflection.

The Impact of Limitless Processing and Low-Cost Storage on ETL

Cloud platforms have changed the game. With breathtaking processing capabilities and competitive data storage costs, one wonders if the old ETL model still has its place. Platforms like Snowflake and BigQuery handle data on a scale and speed unimaginable just a few years ago, all while keeping costs under control.

So, is this the end of ETLs? Let’s investigate.

Limitations of Traditional ETLs

ETLs are like old habits that are hard to break. They require specific skills to create and maintain, leading to technical debt that becomes challenging to manage. Data engineers are also shifting their skills towards more modern solutions. These processes can seem heavy and rigid compared to the agile solutions offered by the cloud. Moreover, they originated in a world where storage and processing were expensive. Back then, data engineers preferred to process data before ingesting it.

Data Cloud Platforms and Automated Ingestion

New data platforms offer a range of automated ingestion methods. They provide a seductive alternative, simplifying data integration and making traditional ETL processes almost obsolete. These solutions are more flexible and in tune with the current needs of companies, including streaming and real-time data processing.

Evolution of Talent and Cloud-Native Approaches

The job market in the data field is evolving. Today’s talents are increasingly turning away from traditional solutions. They are embracing cloud-native approaches and CI/CD practices, not to mention the fast-growing dbt community. This trend marks a significant turning point, reducing the appeal of classic ETL processes.

Synchronisation and Data Replication

Ultimately, we all want a faithful and up-to-date copy of operational data on our data platforms. Cloud-oriented solutions seem much more suitable for data synchronisation and replication. ETLs, with their heavier and less flexible approach, are losing ground.

Conclusion: The Future of ETL in the Age of Data Platforms

Are ETLs a relic of a bygone era? Perhaps not entirely. They are often connected to source systems, which can be practical for feeding a data platform. However, modern data platforms are redefining the rules of the game. The transition to more suitable, agile, and cloud-aligned solutions is underway.

What do you think? Are we witnessing the end of ETLs, or is it just another evolution in the world of data?