Instagram Data Analysis Tool
**Tech Stack:** Web Scraping, Python, Instagram, AWS, S3 bucket, PostgreSQL, Lambda, Instagrapi, Pydantic, SQLAlchemy, requests, HikerAPI, Web Scraping Software, Python Script, Data Scraping, Web Crawler, Contact Information Data.
**Task:** Created a sophisticated data extraction tool for Instagram that collects extensive data such as posts, followers, likes, comments, profile details, avatars, stories, and interaction profiles. This data is then cleaned, organized, deduplicated, transformed, and stored in a PostgreSQL database hosted on AWS. Additionally, images or videos from posts must be stored in an S3 bucket. The tool conducts daily scans to identify any changes in profiles and posts, effectively handling large volumes of data.
**Challenge:** The main challenge involved extracting followers from profiles with millions of followers. Manually scrolling through followers only reveals about 50 at a time on Instagram. Detecting changes in data was also quite complex. Moreover, the database structure needed to be designed to ensure clarity for non-specialists. Another requirement was to achieve rapid data extraction, with the target of processing around 2 million profiles in 2-3 days.
**Solution:** To address these challenges with a cost-effective and fast approach, we devised a strategy that captured all necessary data. We configured a resource-efficient server and database and executed over 40 functions concurrently. We also utilized HikerAPI to obtain followers and those who liked or commented on posts.
**Result:** Successfully implemented a custom solution that provided the client with a well-organized database of over 10 million records and approximately 2 million unique Instagram accounts within PostgreSQL. This system assists the client in data analysis, forecasting human behavior, and creating tables and charts.