https://varmazyar.de/tags/apache-airflow/ Recent content in Apache Airflow on Cloud Engineering Chronicles with Mohammad Hugo en Tue, 27 Aug 2024 11:46:20 +0200 https://varmazyar.de/posts/optimizing-airflow-scheduler-gke/ Tue, 27 Aug 2024 11:46:20 +0200 https://varmazyar.de/posts/optimizing-airflow-scheduler-gke/ Introduction Link to heading Running hundreds of daily tasks on Apache Airflow is a routine for data-driven companies like StoreMaven. We use Google’s managed Composer environment to streamline our data pipelines. However, as the number of tasks grows, the default Composer setup on Google Kubernetes Engine (GKE) reveals its limitations, particularly when it comes to scaling efficiently and cost-effectively. In this post, I’ll share our journey of optimizing the Airflow Scheduler on GKE, which led to significant cost savings while maintaining high performance.