Improved Data Processing Time and Quality
betPawa, a sports betting firm, was managing substantial OLTP data through a data warehouse (DWH) system that operated with continuous ETL processes. However, they encountered several problems, including slow support, absent OLTP data in their reports, and inflexibility regarding new business procedures. Sometimes, the ETL processes would run overnight but still produce untimely results, and reports were based on outdated DWH data, limiting thorough analysis. To tackle these issues, betPawa sought the expertise of Valiotti Analytics in business intelligence (BI) reporting, DWH, and ETL design.
**Results: Flexible & Scalable DWH System**
After evaluating betPawa's DWH system, data structure, and workflows, we revamped the setup to include:
- **Extended DWH**: Added new fact tables for various data sources.
- **Data Flow**: Transitioned to Kafka for real-time updates and utilized MaxWell’s Daemon for integrating MySQL.
- **Data Quality**: Validated ETL records and monitored inconsistencies through ETL logs in Redash.
**Data Transfer Methods**
We assessed four methods for transferring OLTP data:
1. **Materialized MySQL**: Replicates MySQL tables into Clickhouse, but it's not compatible with the Percona Update tool.
2. **Full and Incremental Load**: Accesses MySQL data directly via Clickhouse SQL, suitable for small tables or indexed incremental loads.
3. **Kafka Topics**: Transfers corporate data to Kafka in Google’s Protocol Buffers format for Clickhouse’s Kafka Engine to extract.
4. **MySQL Transactions to Kafka Events**: Captures data from MySQL transaction logs (binlog), sending events to Kafka.
**Improved Data Accuracy and Real-Time Insights**
By employing Kafka and MaxWell’s Daemon tool, we established standardized processing and real-time updates. This improvement in managing fact and dimensional tables helped eliminate data duplication and enhance accuracy.
**A New BI Tool – Redash**
We evaluated, trained, and implemented Redash into production. Reports that were originally created in Tableau were successfully replicated in Redash. After thorough testing and debugging, the client was able to access enhanced data visualization and reporting features.