Handling e-commerce data in an automated way
The client, merXu, is a leading online trading platform for B2B companies throughout the European Union, providing a secure and trustworthy environment that connects verified buyers and sellers. This platform serves businesses of all sizes and ensures a smooth trading experience through its user-friendly interface and various features. In the role of a partner in the project, Datumo's experts developed generic data processing pipelines using appropriate GCP services. Our team implemented Apache Airflow on GKE with several DAG templates, while also sharing cloud and data engineering best practices with the client's developers. The challenge faced was that rapid growth brought impressive business results but also led to an expanding volume of data. This growth resulted in frequent modifications to merXu's platform, the creation of numerous ad-hoc solutions, and a deficiency in automation, standardization, and monitoring. Therefore, Datumo needed to stabilize operations and replace certain tools with suitable GCP services, provide best practice guidance, and create solutions to effectively manage future requirements.
The solution centered around Apache Airflow, deployed on GKE as per client needs. Datumo experts leveraged Airflow, BigQuery, GCS, and other GCP services to build flexible solutions that could easily accommodate the addition of new data pipelines to meet complex business demands. Data backups and monitoring alerts were implemented to ensure the platform could handle challenging scenarios effectively.
As a result of Datumo’s enhancements, managing key data pipelines became both reliable and straightforward. For the most frequent tasks, providing just a few configuration values was enough to establish a robust, multi-service solution. With monitored orchestration, all processes were executed correctly, and in the event of an issue, the support team received clear and transparent information to swiftly identify the cause.