Case Study: AWS Aurora to GCP BigQuery Integration

By intelia | @intelia | January 31, 2023

The intelia team worked with a large global wagering provider to build a robust, parametrised & easily repeatable solution that can be replicated to any client database/table combination in future to ingest data from AWS to GCP in automated fashion.

The challenge

The customer’s current reporting platform which was built on AWS Aurora Serverless V1 takes around 24 hours to complete the end-to-end process of making data available for internal and external reporting needs. The customer’s data team is looking to move to a modern data platform that allows them to ingest, store and transform data from the sources with lower latency and in turn improve their reporting.

The solution

The solution involved data pipelines sourcing data from AWS, landing into GCS and then moving through BigQuery in a number of raw, transformed and consumption layers. Cloud Composer was used as an end-to-end pipeline orchestration tool. Ingesting the data using Cloud Functions, loading data from GCS to BigQuery, running BigQuery Merge queries and finally archiving the GCS data.

The results

Delivered a robust and easily repeatable solution to the large global wagering providers team which they can easily utilise for all of their future client reporting databases allowing for scalability and performance.