Big Data
Project A, a leading Communications Services Provider (CSP), required a near real-time, big data solution for processing and analyzing vast quantities of data, handling 30 million files monthly and around 3TB of data daily.
**Challenge:** The goal was to process and analyze significant volumes of data in near real-time, which arrived in various formats (structured and semi-structured) from diverse sources, including files, CDRs, XML, and CSV.
**Solution:** IBM Streams gathers data from multiple sources, such as files (XML, CSV, ASN.1), databases, sockets, and REST Web Services. Once collected, the data is filtered, processed, enriched, and aggregated using calculations (sum, average, max, min, etc.) through workflows coded in Streams Processing Language (SPL). The processed data is then directed to various destinations, including databases, HDFS, and filesystem storage. Meanwhile, Apache Flink aggregates data from other sources (Kafka, files, and REST Web Services) and applies similar filtering, processing, enrichment, and aggregation techniques using Java. The final output is sent to various sinks, including SQL and NoSQL databases and the filesystem. Both solutions operate in a clustered environment.
**Result:** This solution effectively lowered costs and significantly accelerated the client's decision-making process, reducing the time frame from 48-72 hours to just 2-3 minutes. Additionally, it greatly enhanced the client's capacity to handle large volumes of data efficiently.