The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
The Data Flowcast is a podcast dedicated to Apache Airflow, a workflow management system for data engineering and AI. Each week, the show explores the current state, future, and potential of Airflow with leading thinkers in the community. It provides insights on how to leverage Airflow to meet the evolving needs of data engineering and AI ecosystems. The podcast is produced by Astronomer, a company specializing in Airflow solutions.
Episodes
Scaling to 1,000+ Airflow Dags at Descartes / Idelic
Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at Idelic (now part of Descartes), about how his team standardized DAG creation and logging, built a Jenkins-driven testing workflow with per-PR Airflow environments, and is opening up DAG configuration and triggering to their own customers thr
Orchestrating local LLMs for sensitive data with Airflow
Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems can depend on their output. In this episode, Kenten is joined by Chhayank Jain, Senior ML Engineer at Visa, to talk through how Airflow orchestrates local LLM pipelines end to end: nightly inference over millions of recor
Improving security in Airflow 3
Security is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production. In this episode, Kenten Danas talks with Anish Giri, Senior Software Engineer and active Apache Airflow contributor focused on security, about three recent contributions that touch nearly every Airflow 3 deployment: hardening connection testing, adding JWT revocation,
Orchestrating data across 30 companies at itti
Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano, Data Engineer at itti, the technology company within Grupo Vázquez in Paraguay. Lucas walks through the custom YAML framework his Center of Excellence built on top of Airflow, how they baked data quality and custom operators
Building self-healing Airflow pipelines at ATC Drivetrain
Manufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can cost real money. In this episode, Kenten Danas talks with Kumuda Sreenivasa, Founder of Receitly and Senior Data Architect at ATC Drivetrain, about how her team built self-healing Airflow pipelines, where AI fits into the recovery loop, an
Orchestrating strictly sequential ETL pipelines at Synechron
Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic, Senior Big Data Engineer at Synechron, joins Marc Lamberti to walk through the three-layer DAG architecture her team built to solve exactly that, plus how they generate 200-task DAGs and how they rebuilt subdag-style group ret
Managing financial datasets with Airflow at Wise
Financial data pipelines have to be right the first time. On this episode, Kenten sits down with [Antonello Benedetto](linkedin.com/in/anbento4), Staff Data Engineer at Wise, to talk about how the central data and analytics engineering team runs Airflow for critical financial datasets, tiers pipelines by reliability, and orchestrates LLM-enabled workflows with validation layers and agent-checking-
Managing travel platform data with Airflow at Headout
Running a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping data trustworthy for a data-driven org. In this episode, Mrinalini Singh, Data Platform Engineer at [Headout](headout.com), walks through how her team uses Airflow as the nervous system of their stack: orchestrating dbt with a write-audit-pub
How Airflow orchestration decisions impact Spark performance
Airflow and Spark are one of the most common combinations in modern data platforms, but the orchestration decisions made on the Airflow side often determine whether Spark jobs run fast and cheap or slow and expensive. In this episode, Kenten is joined by [Meni Shmueli](linkedin.com/in/meni-shmueli-dataflint), Co-Founder and CEO at [DataFlint](dataflint.io), to dig into how Airflow and Spark fit to
Using Airflow for diverse client projects at Accion Labs
When a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, [Chandan Gowda](linkedin.com/in/chandan-gowda-a-h-744908195), Data Engineer at [Accion Labs](accionlabs.com), joins Kenten to discuss how his team uses Airflow across client projects, including a financial services scoring use case and a POC applying production-grade orchestrat
Consolidating legacy job schedulers with Airflow at Trading Technologies
Migrating two decades of legacy job orchestration is the kind of project most teams quietly avoid. In this episode, Marc Lamberti is joined by [Sanket Patel](linkedin.com/in/sanket-patel-a2a57b3a), Director of Engineering at Trading Technologies, to talk through how his team is consolidating 20 years of C#/.NET-based scheduling and a recently acquired company's stack onto Airflow, Astronomer,
Orchestrating AI video intelligence and evaluation pipelines at Firework
How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at [Firework](firework.com), joins the show to walk through how his team uses Airflow to coordinate ingestion, embeddings, and evaluation pipelines for AI features in a video commerce platform. The conversation covers the technical stack, the ch
Orchestrating Retail Data Pipelines at Saks Global
Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems into Snowflake. In this episode, [Shailesh Kadam](linkedin.com), Architect at [Saks Global](saks.com), joins Kenten to walk through how Airflow acts as the nervous system tying it all together, why they moved from self-manag
What's New in Apache Airflow® 3.3
Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer](astronomer.io), joins Kenten Danas to walk through what's new in the release and where eac
Running Airflow 3 in a regulated environment at OTPP
Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan](otpp.com), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execut
Managing a Customer Analytics Platform with Airflow at Skimlinks
Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode, Julian Larralde, Director of Data Engineering at Skimlinks, walks through the stack, the migration from external task sensors to event-driven Assets, and a YAML-based DAG factory the team built to onboard new publishers without rewriting Pytho
Building a custom Tableau provider for Airflow at JLR
JLR is the UK's largest automotive manufacturer, behind brands like Range Rover, Jaguar, Defender, and Discovery. In this episode, Najeeb Sulaiman, Senior Data Engineer at JLR, walks through how Airflow orchestrates data across manufacturing, supply chain, and finance — including a custom Tableau provider his team built (after the community version dropped PAT authentication) and a CI/CD pipe
Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira
Running Airflow at the scale of a national retailer means more than just scheduling. It means giving non-engineers a path to ship DAGs, and classifying thousands of runs to know which ones need attention. In this episode, Mateus Ferreira, Senior Data Engineer at Luiza Labs (the technology arm of Magazine Luiza, one of Brazil's largest retailers), joins Marc to talk about the patterns his team
Enhancing DAGs for Data Processing with William Orgertrice III at Cargill
In the data engineering world, the difference between a pipeline that works and one that's truly production-ready often comes down to a handful of deliberate decisions. William Orgertrice III, Data Engineer at Cargill, joins us to share the DAG design and monitoring practices he presented at Airflow Summit 2025 and how his team is rolling out Airflow across 60+ internal teams as part of Cargi
Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer
In this episode, we take a step back from implementation-specific topics to explore what it actually takes to build a career in data engineering — and how AI is reshaping that path.Shrividya Hegde, a data and AI engineer and an Airflow champion in Astronomer’s Champions program, joins us to discuss getting into data engineering, contributing to open source and why good data engineering should mak
Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development
We explore how a third-party logistics platform built its entire data orchestration layer on Airflow, and what that makes possible for developer teams and merchant-facing products alike.Filip Kunčar, Platform Director at ShipMonk Product Development, discusses migrating from a closed source tool to Airflow, orchestrating dbt with both Cosmos and the BashOperator and using Airflow to power customer
Building Airflow CTL with Buğra Öztürk at Mollie
Buğra Öztürk, Senior Data Engineer at Mollie and Committer and PMC member on the Apache Airflow project, joins us to walk through Airflow CTL — what it is, how it differs from the existing Airflow CLI and where it is headed under AIP-94.Key Takeaways:00:00 Introduction.03:10 Buğra has contributed to Airflow since 2022, from docs changes up to Committer and PMC member — a path he hopes inspires oth
Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik
In this episode, we explore the newly released Apache Airflow common AI provider — what problem it solves, how it was built and what's coming next.Kaxil Naik, Senior Director of Engineering at Astronomer and Apache Airflow PMC member, and Pavan Kumar Gopidesu, Lead Data Engineer at Experian and Apache Airflow PMC member, join us to walk through the provider's first release and the techni
Building AI Debugging Agents Into Airflow DAGs at Jeppesen ForeFlight with Samantha Blaney Cuevas
Aviation data pipelines run on strict 28-day publication cycles, and the margin for error is zero. In this episode, we're joined by Samantha Blaney Cuevas, Software Engineer at Jeppesen ForeFlight, to explore how her team orchestrates a complex, time-sensitive data pipeline with Airflow and where AI is starting to fit into that picture.Key Takeaways:00:00 Introduction.04:05 Airflow orchestrat
Introducing Airflow 3.2
We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines.Astronomer’s Head of Customer Education, Marc Lamberti, and Senior Manager of Developer Relations, Kenten Danas, break down what’s new, from asset partitioning to Async Python tasks and DAG versioning. They explore how these updates improve scheduling, performance and observability in production workflows.Key
Reflections on a Decade of Data Engineering at Seattle Data Guy
Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent.In this episode, Benjamin Rogojan, Owner and Data Consultant at Seattle Data Guy, joins us to reflect on how the data engineering landscape has evolved alongside Apache Airflow. We explore when Airflow makes sense as an orchestrator, why batch processing is still d
Managing Data Quality and Governance With Airflow at Credit Karma with Ashir Alam
Data quality is not optional when you manage credit data at scale.In this episode, Ashir Alam, Senior Data Engineer at Credit Karma, joins us to share how his team acts as the gatekeeper for credit data ingestion, how they standardize data quality with Airflow and DAG Factory and how they scale safely across thousands of DAGs. We explore how governance, PII protection and orchestration come togeth
Open Source Airflow Contributions and Performance Improvements at G-Research with Christos Bisias
Modern Airflow isn’t just orchestration. It's a contribution.In this episode, we explore how open source investment drives real performance gains and deeper observability.We’re joined by Christos Bisias, Open Source Software Engineer, Apache Airflow at G-Research, to discuss how his team uses Airflow for large-scale data transformations, contributes upstream and improves scheduler throughpu
Automating Threat Intelligence Using Airflow with Karan Alang
In this episode, Karan Alang, Principal Software Engineer at Versa Networks, joins the conversation to discuss how Airflow can be used to automate threat intelligence in modern cybersecurity environments. He explains the growing scale of cloud computing, the profitability of hacking and the shortage of SOC analysts. Karan also outlines a novel architecture that combines Airflow, XDR, graph databas
Using Plugins To Customize Airflow at Ponder Labs with Egor Tarasenko
In this episode, we explore how teams scale Apache Airflow in complex environments and what it takes to make orchestration work across many stakeholders. We look at real-world challenges around visibility, ownership and predictability as data platforms grow.Egor Tarasenko, Data and AI Engineer at Ponder Labs, joins us to share how Ponder Labs customizes Airflow for education organizations using pl
Scaling Airflow at Wix for Analytics and AI with Ethan Shalev
Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge.In this episode, we’re joined by Ethan Shalev, Data Engineer at Wix, to discuss how Wix operates Airflow at massive scale, migrates to Airflow 3 and uses AI to accelerate development.Key Tak
Using Airflow To Orchestrate Billions of Events at Addi with Carlos Daniel Puerto Niño
Strong data orchestration is as much about culture and visibility as it is about technology. As data platforms scale, teams need systems that reduce cognitive load while increasing reliability and observability.In this episode, Carlos Daniel Puerto Niño, Senior Analytics Engineer and Data Analyst at Addi, joins us to share how Addi uses Airflow to support batch orchestration, manage organizational
Building Event-Driven Data Pipelines With Airflow 3 at Astrafy with Andrea Bombino
Real-time data expectations are reshaping how modern data teams think about orchestration and dependencies. As event-driven architectures become more common, teams need to rethink how pipelines react to data changes, rather than schedules.In this episode, Andrea Bombino, Co-Founder and Head of Analytics Engineering at Astrafy, joins us to discuss how event-driven scheduling in Airflow is evolving
Uphold’s Approach to Orchestrating Modern Data Workflows with Jaime Oliveira
A strong data-driven mindset underpins how fintech teams scale analytics, infrastructure and decision-making across the business.In this episode, Jaime Oliveira, Lead Data Engineer at Uphold, joins us to discuss how Uphold structures its data organization and orchestration strategy. Jaime shares how the team uses Airflow and dbt to support analytics, reporting and data activation while evolving th
Modern Airflow Best Practices for Scalable Data Pipelines with Bhavani Ravi
Building reliable data pipelines at scale requires more than writing code. It depends on thoughtful design, infrastructure trade-offs and an understanding of how orchestration platforms evolve over time.In this episode, Airflow best practices shaped by real-world implementation are examined. Bhavani Ravi, Independent Software Consultant and Apache Airflow Champion, shares lessons on pipeline desig
Inside Conviva’s Decision To Power Its Data Platform With Airflow with Han Zhang
Conviva operates at a massive scale, delivering outcome-based intelligence for digital businesses through real-time and batch data processing. As new use cases emerged, the team needed a way to extend a streaming-first architecture without rebuilding core systems.In this episode, Han Zhang joins us to explain how Conviva uses Apache Airflow as the orchestration backbone for its batch workloads, ho
Why Airflow Became the Scheduling Backbone at Condé Nast Technology Lab with Arun Karthik
Data platforms are moving from batch-first pipelines to near real-time systems where orchestration, observability, scalability and governance all have to work together.In this episode, Arun Karthik, Director, Data Solutions Engineering at Condé Nast Technology Lab, joins us to share how data engineering evolves from relational databases and ETL into distributed processing, modern orchestration wit
The Role of Airflow in Building Smarter ML Pipelines at Vivian Health with Max Calehuff
The integration of data orchestration and machine learning is critical to operational efficiency in healthcare tech. Vivian Health leverages Airflow to power both its ETL pipelines and ML workflows while maintaining strict compliance standards.Max Calehuff, Lead Data Engineer at Vivian Health, joins us to discuss how his team uses Airflow for ML ops, regulatory compliance and large-scale data orch
Scaling Airflow to 11,000 DAGs Across Three Regions at Intercom with András Gombosi and Paul Vickers
The evolution of Intercom’s data infrastructure reveals how a well-built orchestration system can scale to serve global needs. With thousands of DAGs powering analytics, AI and customer operations, the team’s approach combines technical depth with organizational insight.In this episode, András Gombosi, Senior Engineering Manager of Data Infra and Analytics Engineering, and Paul Vickers, Principal
How Covestro Turns Airflow Into a Simulation Toolbox with Anja Mackenzie
Building scalable, reproducible workflows for scientific computing often requires bridging the gap between research flexibility and enterprise reliability.In this episode, Anja MacKenzie, Expert for Cheminformatics at Covestro, explains how her team uses Airflow and Kubernetes to create a shared, self-service platform for computational chemistry.Key Takeaways:00:00 Introduction.06:19 Custom script
Building Secure Financial Data Platforms at AgileEngine with Valentyn Druzhynin
The use of Apache Airflow in financial services demands a balance between innovation and compliance. Agile Engine’s approach to orchestration showcases how secure, auditable workflows can scale even within the constraints of regulatory environments.In this episode, Valentyn Druzhynin, Senior Data Engineer at AgileEngine, discusses how his team leverages Airflow for ETF calculations, data validatio
How Redica Transformed Their Data With Airflow and Snowflake with Shankar Mahindar
The life sciences industry relies on data accuracy, regulatory insight and quality intelligence. Building a unified system that keeps these elements aligned is no small feat.In this episode, we welcome Shankar Mahindar, Senior Data Engineer II at Redica Systems. We discuss how the team restructures its data platform with Airflow to strengthen governance, reduce compliance risk and improve customer
How Airflow and AI Power Investigative Journalism at the Financial Times with Zdravko Hvarlingov
The Financial Times leverages Airflow and AI to uncover powerful stories hidden within vast, unstructured data.In this episode, Zdravko Hvarlingov, Senior Software Engineer at the Financial Times, discusses building multi-tenant Airflow systems and AI-driven pipelines that surface stories that might otherwise be missed. Zdravko walks through entity extraction and fuzzy matching, linking the UK Reg
Inside Vinted’s Code-Generated Airflow Pipelines with Oscar Ligthart and Rodrigo Loredo
The shift from monolithic to decentralized data workflows changes how teams build, connect and scale pipelines.In this episode, we feature Oscar Ligthart, Lead Data Engineer, and Rodrigo Loredo, Lead Analytics Engineer, both at Vinted, as we unpack their YAML-driven abstraction that generates Airflow DAGs and standardizes cross-team orchestration.Key Takeaways:00:00 Introduction.05:28 Challenges o
Transforming Data Pipelines at XENA Intelligence with Naseem Shah
The shift from simple cron jobs to orchestrated AI-powered workflows is reshaping how startups scale. For a small team, these transitions come with unique challenges and big opportunities.In this episode, Naseem Shah, Head of Engineering at Xena Intelligence, shares how he built data pipelines from scratch, adopted Apache Airflow and transformed Amazon review analysis with LLMs.Key Takeaways:00:00
Scaling Geospatial Workflows With Airflow at Overture Maps Foundation and Wherobots with Alex Iannicelli and Daniel Smith
Using Airflow to orchestrate geospatial data pipelines unlocks powerful efficiencies for data teams. The combination of scalable processing and visual observability streamlines workflows, reduces costs and improves iteration speed.In this episode, Alex Iannicelli, Staff Software Engineer at Overture Maps Foundation, and Daniel Smith, Senior Solutions Architect at Wherobots, join us to discuss leve
Scaling Airflow for Enterprise Data Platforms at PepsiCo with Kunal Bhattacharya
PepsiCo’s data platform drives insights across finance, marketing and data science. Delivering stability, scalability and developer delight is central to its success, and engineering leadership plays a key role in making this possible.In this episode, Kunal Bhattacharya, Senior Manager of Data Platform Engineering at PepsiCo, shares how his team manages Airflow at scale while ensuring security, pe
Building a Unified Data Platform at Pattern with William Graham
The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.In this episode, we are joined by William Graham, Senior Data Engineer at Pattern, who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orc
How Astronomer Turns Proactive Monitoring Into Customer Success with Collin McNulty
The evolution of Airflow continues to shape data orchestration and monitoring strategies. Leveraging it beyond traditional ETL use cases opens powerful new possibilities for proactive support and internal operations.In this episode, we are joined by Collin McNulty, Sr. Director of Global Support at Astronomer, who shares insights from his journey into data engineering and the lessons learned from
Overcoming Data Engineering Challenges at Daiichi Sankyo Europe GmbH with Evgenii Prusov
The shift to a unified data platform is reshaping how pharmaceutical companies manage and orchestrate data. Establishing standards across regions and teams ensures scalability and efficiency in handling large-scale analytics.In this episode, Evgenii Prusov, Senior Data Platform Engineer of Daiichi Sankyo Europe GmbH, joins us to discuss building and scaling a centralized data platform with Airflow
Building a Data-Driven Beauty and Wellness Marketplace at StyleSeat with Paschal Onuorah
StyleSeat is revolutionizing how beauty and wellness professionals grow their businesses through data-driven tools. From streamlining scheduling to optimizing marketing, their platform empowers professionals to focus on their craft while expanding their client base.In this episode, Paschal Onuorah, Senior Data Engineer at StyleSeat, shares how the company leverages Airflow, dbt, and Cosmos to driv
Building the Future of Airflow Execution at Astronomer with Ian Buss and Piotr Chomiak
The evolution of orchestration in Airflow continues with innovations that address both scalability and security. From improving executor reliability to enabling remote execution, these advancements reshape how organizations manage data pipelines.In this episode, we’re joined by Ian Buss, Principal Software Engineer at Astronomer, and Piotr Chomiak, Principal Product Manager at Astronomer, who shar
Scaling On-Prem Airflow With 2,000 DAGs at Numberly with Sébastien Crocquevieille
Scaling 2,000+ data pipelines isn’t easy. But with the right tools and a self-hosted mindset, it becomes achievable.In this episode, Sébastien Crocquevieille, Data Engineer at Numberly, unpacks how the team scaled their on-prem Airflow setup using open-source tooling and Kubernetes. We explore orchestration strategies, UI-driven stakeholder access and Airflow’s evolving features.Key Takeaways:00:0
How Moniepoint Group Uses Airflow for Exposure Monitoring with Adeolu Adegboye
Managing financial data at scale requires precise orchestration and proactive monitoring to maintain operational efficiency.In this episode, we are joined by Adeolu Adegboye, Data Engineer at Moniepoint Group, who shares how his team uses data pipelines and workflow automation to manage high volumes of transactions, ensure timely alerts and support diverse stakeholders across the business.Key Take
Inside Bosch’s Airflow 3 Revolution: Remote Execution with Jens Scheffler
The evolution of Airflow has reached a milestone with the introduction of remote execution in Airflow 3, enabling flexible orchestration across distributed environments.In this episode, Jens Scheffler, Test Execution Cluster Technical Architect at Bosch, shares insights on how his team’s need for large-scale, cross-environment testing influenced the development of the Edge Executor and shaped this
Inside Modern Data Infrastructure at Massdriver with Cory O’Daniel and Jake Ferriero
Managing modern data platforms means navigating a web of complex infrastructure, competing team needs and evolving security standards. For data teams to truly thrive, infrastructure must become both accessible and compliant without sacrificing velocity or reliability.In this episode, we’re joined by Cory O’Daniel, CEO and Co-Founder at Massdriver, and Jacob Ferriero, Senior Software Engineer at As
The Future of Airflow Telemetry with Bolke de Bruin
Telemetry has the potential to guide the future of Airflow, but only if it’s implemented transparently and with community trust. In this episode, we’re joined by Bolke de Bruin, Director at Metyis and a long-time Airflow PMC member. Bolke discusses how telemetry has been handled in the past, why it matters now and what it will take to get it right.Key Takeaways:(03:20) The role of foundations in e
Transforming the Airflow UI for Cloudera’s Users with Shubham Raj
Contributing to open-source projects can be daunting, but it can also unlock unexpected innovation. This episode showcases how one engineer’s journey with Apache Airflow led to impactful UI enhancements and infrastructure solutions at scale. Shubham Raj, Software Engineer II at Cloudera, shares how his team built a drag-and-drop DAG editor for non-coders, contributions which helped shape the Airfl
Streamlining Thousands of Data Pipelines at Lyft with Yunhao Qing
Managing data pipelines at scale is not just a technical challenge. It is also an organizational one. At Lyft, success means empowering dozens of teams to build with autonomy while enforcing governance and best practices across thousands of workflows.In this episode, we speak with Yunhao Qing, Software Engineer at Lyft, about building a governed data-engineering platform powered by Airflow that ba
Transforming Customer Education in Data Engineering at Astronomer with Marc Lamberti
Understanding the complexities of Apache Airflow can be daunting for newcomers and seasoned data engineers. But with the right guidance, mastering the tool becomes an achievable milestone.In this episode, Marc Lamberti, Head of Customer Education at Astronomer, joins us to share his journey from Udemy instructor to driving education at Astronomer, and how he's helping over 100,000 learners de
Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi
The flexibility of Airflow plays a pivotal role in enabling decentralized data architectures and empowering cross-functional teams.In this episode, we speak with Alberto Crespi, Data Architect at lastminute.com, who shares how his team scales Airflow across 12 teams while supporting both vertical and horizontal structures under a data mesh approach.Key Takeaways:(02:17) Defining responsibilities w
The AI-Ready Pipeline: Reimagining Airflow at Veyer® Logistics with Anu Pabla
Innovation in orchestration is redefining how engineers approach both traditional ETL pipelines and emerging AI workloads. Understanding how to harness Airflow’s flexibility and observability is essential for teams navigating today’s evolving data landscape.In this episode, Anu Pabla, Principal Engineer at The ODP Corporation, joins us to discuss her journey from legacy orchestration patterns to A
Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel
The orchestration layer is foundational to building robust AI- and ML-powered data pipelines, especially in complex hybrid enterprise environments. IBM’s partnership with Astronomer reflects a strategic alignment to simplify and scale Airflow-based workflows across industries.In this episode, we’re joined by IBM’s Senior Product Manager, BJ Adesoji, and GTM PM and Growth Leader, Ryan Yackel. We di
Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich
Efficient orchestration and maintainability are crucial for data engineering at scale. Gil Reich, Data Developer for Data Science at Wix, shares how his team reduced code duplication, standardized pipelines, and improved Airflow task orchestration using a Python-based framework built within the data science team.In this episode, Gil explains how this internal framework simplifies DAG creation, imp
Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero
Legacy architecture and AI workloads pose unique challenges at scale, especially in a global enterprise with complex data systems. In this episode, we explore strategies to proactively monitor and optimize pipelines while minimizing downstream failures.Adonis Castillo Cordero, Senior Automation Manager at Procter & Gamble, joins us to share actionable best practices for dependency mapping, ano
Building an End-to-End Data Observability System at Netflix with Joseph Machado
Building reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy.Joseph Machado, Senior Data Engineer at Netflix, joins us to share practical insights gleaned from sup
Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli
Creating consistency across data pipelines is critical for scaling engineering teams and ensuring long-term maintainability.In this episode, Snir Israeli, Senior Data Engineer at Next Insurance, shares how enforcing coding standards and investing in developer experience transformed their approach to data engineering. He explains how implementing automated code checks, clear documentation practices
Data Quality and Observability at Tekmetric with Ipsa Trivedi
Airflow’s adaptability is driving Tekmetric’s ability to unify complex data workflows, deliver accurate insights and support both internal operations and customer-facing services — all within a rapidly growing startup environment.In this episode, Ipsa Trivedi, Lead Data Engineer at Tekmetric, shares how her team is standardizing pipelines while supporting unique customer needs. She explains how Ai
Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham
The Airflow 3.0 release marks a significant leap forward in modern data orchestration, introducing architectural upgrades that improve scalability, flexibility and long-term maintainability.In this episode, we welcome Vikram Koka, Chief Strategy Officer at Astronomer, and Jed Cunningham, Principal Software Engineer at Astronomer, to discuss the architectural foundations, new features and future im
Airflow in Action: Powering Instacart's Complex Ecosystem
The evolution of data orchestration at Instacart highlights the journey from fragmented systems to robust, standardized infrastructure. This transformation has enabled scalability, reliability and democratization of tools for diverse user personas.In this episode, we’re joined by Anant Agarwal, Software Engineer at Instacart, who shares insights into Instacart's Airflow journey, from its earl
From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori
Data orchestration at scale presents unique challenges, especially when aiming for flexibility and efficiency across cloud environments. Choosing the right tools and frameworks can make all the difference. In this episode, Raviteja Tholupunoori, Senior Engineer at Deloitte Digital, joins us to explore how Airflow enhances orchestration, scalability and cost efficiency in enterprise data workflows.
A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer
The 2025 State of Airflow report sheds light on how global users are adopting, evolving and innovating with Apache Airflow. With over 5,000 responses from 116 countries, the survey reveals critical insights into Airflows’ role in business operations, new use cases and what’s ahead for the community.In this episode, Tamara Fingerlin, Developer Advocate at Astronomer, walks us through her process of
The Software Risk That Affects Everyone and How To Address It with Michael Winser and Jarek Potiuk
The security of open-source software is a growing concern, especially as dependencies and regulations become more complex, making it essential to understand how to manage software supply chains effectively. In this episode, we sit down with Michael Winser, Co-Founder at Alpha-Omega and Security Strategy Ambassador at Eclipse Foundation, and Jarek Potiuk, Member of the Security Committee at the Apa
Building Scalable ML Infrastructure at Outerbounds with Savin Goyal
Machine learning is changing fast, and companies need better tools to handle AI workloads. The right infrastructure helps data scientists focus on solving problems instead of managing complex systems. In this episode, we talk with Savin Goyal, Co-Founder and CTO at Outerbounds, about building ML infrastructure, how orchestration makes workflows easier and how Metaflow and Airflow work together to
Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy
Keeping data pipelines reliable at scale requires more than just the right tools — it demands constant innovation. In this episode, Nick Bilozerov, Senior Data Engineer at Stripe, and Sharadh Krishnamurthy, Engineering Manager at Stripe, discuss how Stripe customizes Airflow for its needs, the evolution of its data orchestration framework and the transition to Airflow 2. They also share insights o
Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot
Turning complex datasets into meaningful analysis requires robust data infrastructure and seamless orchestration. In this episode, we’re joined by Jennifer Melot, Technical Lead at the Center for Security and Emerging Technology (CSET) at Georgetown University, to explore how Airflow powers data-driven insights in technology policy research. Jennifer shares how her team automates workflows to supp
Hybrid Testing Solutions for Autonomous Driving at Bosch with Jens Scheffler and Christian Schilling
Testing autonomous vehicles demands precision, scalability and powerful orchestration tools — enter Apache Airflow, a key component of Bosch’s cutting-edge testing framework. In this episode, we sit down with Jens Scheffler, Test Execution Cluster Technical Architect, and Christian Schilling, Product Owner Open Loop Testing Automated Driving, both at Bosch, to explore how Bosch harnesses Airflow t
Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer
Scaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer, Former Platform Engineer at Monzo Bank, joins us to share his journey optimizing data pipelines, over
Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar
The future of data engineering lies in seamless orchestration and automation. In this episode, Arjun Anandkumar, Data Engineer at Telia, shares how his team uses Airflow to drive analytics and AI workflows. He highlights the challenges of scaling data platforms and how adopting best practices can simplify complex processes for teams across the organization. Arjun also discusses the transformative
The Role of Airflow in Finance Transformation at Etraveli Group with Mihir Samant
Transforming bottlenecked finance processes into streamlined, automated systems requires the right tools and a forward-thinking approach. In this episode, Mihir Samant, Senior Data Analyst at Etraveli Group, joins us to share how his team leverages Airflow to revolutionize finance automation. With extensive experience in data workflows and a passion for open-source tools, Mihir provides valuable i
Recommended
紐約鳥｜New York Aperture
The Swerve Podcast: Obscure Topics | Conspiracy Theories
The Bread and Banter Podcast
The Church of What's Happening Now: The New Testament
Busted Open
Deadline: White House
Crickets Corner
History of the Earthquake and Fire in San Francisco
Fox news
K-Drama Obsessed Mamas | Beyond “Just One More Episode!”
True Crime Classics
The Healer Revolution
