Rebuilding the Big Data processing system
Collaborating with a company that aggregated company-related data in the DACH market—including information on companies, shareholders, annual reports, financial reports, and more—helped reveal insights that weren't immediately obvious. As the company expanded, the volume of data surged due to an increase in data sources and entry into new markets, necessitating a system redesign to facilitate further growth. Challenges included:
- Completely overhauling a continuously operating system without causing regressions
- The complexity of algorithms used for data aggregation
- A variety of external sources with disparate data structures and quality
- A significant rise in processing power requirements with each new market entry
- The need to manage incoming data that may arrive out of order
Key actions taken involved:
- Developing a new system in parallel with the existing one, followed by agile releases of individual modified processes
- Collaborating with data scientists to identify optimal data processing algorithms
- Enhancing database utilization
- Transforming the system into smaller, standalone microservices
- Ensuring scalability through cloud migration
Outcome: The incremental approach allowed for a seamless system transition for clients. The agile methodology enabled the team to implement changes progressively, providing incremental value through short development cycles. The revamped architecture and migration to the cloud significantly enhanced performance and scalability, positioning the company to expand its business and enter new markets. Working with data scientists not only improved algorithm selection but also led to the introduction of a new data interface, facilitating the integration of additional data sources and creating new business opportunities.