Redshift Migration
We revamped Hürriyet's on-premises architecture using various AWS technologies, including AWS Elastic Beanstalk, Kinesis, Firehose, Lambda, S3, DynamoDB, Redshift, EMR (Elastic Map Reduce), SNS (Simple Notification Service), and SQS (Simple Queue Service). The data collection pipeline initiates with HTTP requests to a REST API hosted on AWS Elastic Beanstalk. This API transmits clickstream data to Kinesis, where real-time filtering occurs through AWS Lambda Functions. Additionally, Lambda Functions increment the video view counter in DynamoDB based on the video view events. Filtered data is then stored in the S3 clickstream bucket via Firehose, adhering to a specified time threshold. To generate reports promptly, we developed a Lambda function that updates Redshift when new files arrive in S3. For segmentation tasks, another Lambda function is triggered to start the segmentation job, which reads data from the S3 clickstream bucket and outputs results to Redshift on EMR according to a set schedule. Given the importance of tracking whether each step on EMR is successful, we established a notification system using SNS to alert us via email about the outcomes of EMR jobs. To send segmented user data to DFP (DoubleClick For Publishers), we implemented an SQS pipeline that retrieves segmentation results and writes them to the DFP Segment Database. This architecture offers several advantages: it is scalable, easy to manage from data collection to big data analytics, and facilitates the generation of reports related to clickstream data. Since the data collection API is deployed on AWS Elastic Beanstalk, it can easily scale with varying loads. The auto-scaling capability in Beanstalk, based on defined load metrics, is also cost-effective, ensuring you only pay for the resources you actually use.