Data Engineering Pipeline
The Challenge — Our client required a large external data source with over 20 million records to be polled every month for refreshing the data and insourcing the updated data after normalisation. The Solution — Our team built a message queue system with the pub/sub architecture. We used Kafka to…