The snapshot
1
Agora S.A., had an on-premises Hadoop-based analytics platform with growing data and infrastructure challenges
2
The company chose Google Cloud to improve performance, scalability, and cost-efficiency
3
Devoteam implemented a solution using BigQuery, Cloud Storage, Cloud Composer, and other Google Cloud-managed services
About Agora
Agora is the 4th largest media group in Poland. Its offerings include the Helios SA cinema chain, the Next Film film production and distribution company, 9 radio stations operating under the Eurozet Group banner, outdoor advertising (AMS SA), respected news media – ‘Gazeta Wyborcza’ and Wyborcza.pl, the Gazeta.pl portal and its services (including Sport.pl, Moto.pl and Plotek.pl), the supra-regional Radio TOK FM, book, music and film publishing, as well as catering operations.
The Challenge
In 2014, Agora began collecting data on service user activity using its own technology.
A Hadoop cluster, data model and data collection mechanisms were developed for data storage and processing.
Hive was chosen for analytical purposes. The business requirement was constant access to up-to-date, aggregated data. Data management processes were built and a number of ETL processes based on it. In addition to traffic data, Hive also stored content metadata from monitored websites and other data from various systems.
The ever-increasing volume of data, the growing reporting needs on the Business side, combining different data sets, maintaining the physical infrastructure and software, addressing errors and failures – these were the main challenges of the on-prem setup.
Concluding, the challenge featured:
- 70 TB of data to be migrated from Hive to BigQuery
- Integration with Kafka and other solutions providing real-time data
- Optimisation of service utilisation
- Variable load on the computing unit
Devoteam G Cloud was a reliable partner in areas such as training, knowledge transfer and DevOps for our team responsible for migrating a large-scale data warehouse from an on-prem solution to the cloud. It was a pleasure to work with Devoteam engineers.
Andrzej Purchla
Director of the Analytics Platform and Identity Area
The Goal
Taking into account the risks and limitations of the current analytical platform, Agora’s Management Board decided to migrate its analytical processes to the Google Cloud.
Due to limited experience in the cloud area, Agora was looking for a partner to support employees in the migration process at the stage of modelling the target data structure and creating the architecture of the cloud provisioning system – in order to optimise the above elements of the solution in terms of costs.
The Solution
The Devoteam team prepared a concept using Managed and Serverless solutions that would guarantee that Agora S.A.’s analytics teams would retain their existing style of working.
Following client approval, a project was implemented to migrate the analytics platform to the Google Cloud, using components such as BigQuery, Google Cloud Storage, Cloud Composer (Airflow), Cloud Functions or Dataflow.
The cooperation between Devoteam and the Agora S.A. team in the project has allowed for a smooth transfer of knowledge, enabling Agora S.A. employees to successfully carry out further development work, as well as to maintain the Big Data environment created together with us. The partnership approach and very good preparation of the Agora S.A. team allowed all the necessary work to be carried out as planned.
Emil Dąbrowski
Senior Cloud Architect
The Results
The results we achieved:
- All historical traffic data was transferred to the cloud
- A framework/system of procedures related to data management and access to data was created
- Unified feed processes for the Datalake and Data Warehouse were introduced
- Data from several key sources is now available in one place
- The working environment for analysts has been unified
Key benefits for the organisation:
- No infrastructure-related maintenance work
- Higher computing performance / faster operation
- Greater stability
- On-demand scalability
- Democratisation of data access
- Cost optimisation
Currently, the Agora S.A. team is supported by Devoteam in supporting the continuity of Google Cloud services.
The Better Change
Migrated 70 TB of historical traffic data to BigQuery.
Unified data lake and warehouse feeds, improving data access.
Eliminated infrastructure maintenance, boosting performance and cutting costs.
Your success starts here
What’s your Google Cloud challenge?