https://varmazyar.de/categories/cloud-engineering/ Recent content in Cloud Engineering on Cloud Engineering Chronicles with Mohammad Hugo en Wed, 28 Aug 2024 11:46:20 +0200 https://varmazyar.de/posts/genai-impact-cloud-devops-practices/ Wed, 28 Aug 2024 11:46:20 +0200 https://varmazyar.de/posts/genai-impact-cloud-devops-practices/ Introduction Link to heading In today’s rapidly evolving technological landscape, software development and operations have become more complex and interconnected than ever. The rise of cloud computing, microservices, and continuous delivery has transformed how we build and deploy applications. However, with these advancements comes increased complexity, necessitating the convergence of various disciplines such as DevSecOps, Site Reliability Engineering (SRE), Platform Engineering, and Cloud Engineering. Each of these disciplines addresses unique challenges but also shares common goals—improving efficiency, scalability, security, and collaboration across teams. https://varmazyar.de/posts/optimizing-airflow-scheduler-gke/ Tue, 27 Aug 2024 11:46:20 +0200 https://varmazyar.de/posts/optimizing-airflow-scheduler-gke/ Introduction Link to heading Running hundreds of daily tasks on Apache Airflow is a routine for data-driven companies like StoreMaven. We use Google’s managed Composer environment to streamline our data pipelines. However, as the number of tasks grows, the default Composer setup on Google Kubernetes Engine (GKE) reveals its limitations, particularly when it comes to scaling efficiently and cost-effectively. In this post, I’ll share our journey of optimizing the Airflow Scheduler on GKE, which led to significant cost savings while maintaining high performance.