Design and migration of data warehouses at MTS
Data migration may be necessary when updating software, switching to a new platform, combining databases, or transitioning to the cloud. In this context, the ITFB Group outlines how to design data warehouse architectures, migrate from non-target systems (such as SAS and Teradata) to target systems (like Greenplum and Hadoop), and create ETL processes. Objective MTS, one of the top three telecommunications companies in Russia, encountered the challenge of migrating data from SAS and Teradata to Greenplum and Hadoop. To successfully carry out this project, they required experts capable of performing the data migration while ensuring the security and integrity of the system remained intact.
One major challenge was migrating ETL processes from the SAS and Teradata data warehouses to the new Hadoop and Greenplum environments. The team needed to fully adapt and rewrite these processes in a different programming language and develop mechanisms for data loading from all sources. Another challenge was the performance of the ETL processes. The team optimized modifications to the data model and table structure while also revising their data loading methodology. This optimization involved reevaluating the index structure, breaking large tables into smaller ones, enhancing the query structure, and implementing various strategies to boost ETL performance.
As a result, the team successfully integrated the data sources and improved the data model, table structure, data loading methods, and queries. The time required for various processes—such as data retrieval, processing, and loading from numerous sources, along with data analysis and reporting—was cut by 50%. The system is now capable of handling vast volumes of data, including 60,000 terabytes of storage, 14 million events per second, and 100 terabytes of data daily (equating to 700 terabytes weekly and 33,600 terabytes annually).