LLM Platform Engineering & Cost Optimization
Redbird (redbird.io) is an AI-driven data analytics platform that caters to enterprise clients such as Disney, Mars, and Warner Bros. As they expanded their analytics capabilities powered by large language models (LLMs), they required a platform engineer to design a production-worthy AI infrastructure on AWS without exceeding their compute budget. I joined Redbird as a Senior DevOps Engineer/Tech Lead, responsible for the comprehensive deployment of LLMs on AWS EKS. My role encompassed model serving, containerization, Helm chart creation, and coordinating production deployments across development, staging, and production environments. I carefully configured GPU-enabled node groups to align with the inference workload demands while ensuring adherence to strict latency service level agreements (SLAs). A significant improvement came from rethinking cluster scaling. I introduced KEDA to facilitate pod-level autoscaling based on real-time queue depth and request throughput, and then implemented Karpenter to enhance dynamic EC2 provisioning—utilizing Spot and Graviton instances—over static managed node groups. The combined use of KEDA and Karpenter achieved effective scale-to-zero during off-peak periods, resulting in a 15% reduction in monthly AWS compute costs while maintaining P99 latency targets during peak inference workloads. To ensure the platform's reliability and traceability, I managed all infrastructure as code using Terraform across all environments, and established CI/CD pipelines with GitHub Actions and ArgoCD for progressive deployments and quick rollbacks. Observability was enhanced using Prometheus and Grafana, with dashboards monitoring inference latency, GPU utilization, autoscaling activities, and node provisioning times—providing the engineering team with complete insight into performance and cost factors. The outcome was a production-ready LLM platform that scales efficiently with demand, operates at a significantly lower cost, and provides Redbird with a robust foundation to meet the needs of its enterprise customers confidently.