Data Engineering Weekly
Data Engineering Weekly is a podcast that covers topics related to data engineering, including news, tools, and best practices. Hosted by Ananth Packkildurai, it aims to keep data professionals informed about the latest developments in the field. The podcast is based on the popular Substack newsletter of the same name.
Episodes
Knowledge, Metrics, and AI: Rethinking the Semantic Layer with David Jayatillake
Semantic layers have been with us for decades—sometimes buried inside BI tools, living in analysts’ heads. But as data complexity grows and AI pushes its way into the stack, the conversation is shifting. In a recent conversation with David Jayatillake, a long-time data leader with experience at Cube, Delphi Labs, and multiple startups, we explored how semantic layers move from BI lock-in to invisi
Insights from Jacopo Tagliabue, CTO of Bauplan: Revolutionizing Data Pipelines with Functional Data Engineering
Data Engineering Weekly recently hosted Jacopo Tagliabue, CTO of Bauplan, for an insightful podcast exploring innovative solutions in data engineering. Jacopo shared valuable perspectives drawn from his entrepreneurial journey, his experience building multiple companies, and his deep understanding of data engineering challenges. This extensive conversation spanned the complexities of data engineer
AI and Data in Production: Insights from Avinash Narasimha [AI Solutions Leader at Koch Industries]
In our latest episode of Data Engineering Weekly, co-hosted by Aswin, we explored the practical realities of AI deployment and data readiness with our distinguished guest, Avinash Narasimha, AI Solutions Leader at Koch Industries. This discussion shed significant light on the maturity, challenges, and potential that generative AI and data preparedness present in contemporary enterprises.Introducin
Is Apache Iceberg the New Hadoop? Navigating the Complexities of Modern Data Lakehouses
The modern data stack constantly evolves, with new technologies promising to solve age-old problems like scalability, cost, and data silos. Apache Iceberg, an open table format, has recently generated significant buzz. But is it truly revolutionary, or is it destined to repeat the pitfalls of past solutions like Hadoop?In a recent episode of the Data Engineering Weekly podcast, we delved into this
The State of Lakehouse Architecture: A Conversation with Roy Hassan on Maturity, Challenges, and Future Trends
Lakehouse architecture represents a major evolution in data engineering. It combines data lakes' flexibility with data warehouses' structured reliability, providing a unified platform for diverse data workloads ranging from traditional business intelligence to advanced analytics and machine learning. Roy Hassan, a product leader at Upsolver, now Qlik, offers a comprehensive reality check on Lakeho
Beyond Kafka: Conversation with Jark Wu on Fluss - Streaming Storage for Real-Time Analytics
Fluss is a compelling new project in the realm of real-time data processing. I spoke with Jark Wu, who leads the Fluss and Flink SQL team at Alibaba Cloud, to understand its origins and potential. Jark is a key figure in the Apache Flink community, known for his work in building Flink SQL from the ground up and creating Flink CDC and Fluss.You can read the Q&A version of the conversation here, an
The Future of Data Lakehouses: A Fireside Chat with Vinoth Chandar - Founder CEO Onehouse & PMC Chair of Apache Hudi
Exploring the Evolution of Lakehouse Technology: A Conversation with Vinoth Chandar and Onehouse CEOIn this episode, Ananth, author of Data Engineering Weekly and CEO of Onehouse, discusses the latest developments in the Lakehouse technology space, particularly focusing on Apache Hudi, Iceberg, and Delta Lake. They discuss the intricacies of building high-scale data ecosystems, the impact of table
Agents of Change: Navigating 2025 with AI and Data Innovation
Agents of Change: Navigating 2025 with AI and Data InnovationIn this episode of Dew, the hosts and guests discuss their predictions for 2025, focusing on the rise and impact of agentic AI. The conversation covers three main categories:1. The role of agent AI2. The future workforce dynamic involving human and AI agent3. Innovations in data platforms heading into 2025.Highlights include insights fro
Data Engineering Trends With Aswin & Ananth
Welcome to another insightful edition of Data Engineering Weekly. As we approach the end of 2023, it's an opportune time to reflect on the key trends and developments that have shaped the field of data engineering this year. In this article, we'll summarize the crucial points from a recent podcast featuring Ananth and Ashwin, two prominent voices in the data engineering community.Understanding the
DEW #133: How to Implement Write-Audit-Publish (WAP), Vector Database - Concepts and examples & Data Warehouse Testing Strategies for Better Data Quality
Welcome to another episode of Data Engineering Weekly. Aswin and I select 3 to 4 articles from each edition of Data Engineering Weekly and discuss them from the author’s and our perspectives.On DEW #133, we selected the following articleLakeFs: How to Implement Write-Audit-Publish (WAP)I wrote extensively about the WAP pattern in my latest article, An Engineering Guide to Data Quality - A Data Con
DEW #132: The New Generative AI Infra Stack, Databricks cost management at Coinbase, Exploring an Entity Resolution Framework Across Various Use Cases & What's the hype behind DuckDB?
Welcome to another episode of Data Engineering Weekly. Aswin and I select 3 to 4 articles from each edition of Data Engineering Weekly and discuss them from the author’s and our perspectives.On DEW #132, we selected the following articleCowboy Ventures: The New Generative AI Infra StackGenerative AI has taken the tech industry by storm. In Q1 2023, a whopping $1.7B was invested into gen AI startup
DEW #131: dbt model contract, Instacart ads modularization in LakeHouse Architecture, Jira to automate Glue tables, Server-Side Tracking
Welcome to another episode of Data Engineering Weekly. Aswin and I select 3 to 4 articles from each edition of Data Engineering Weekly and discuss them from the author’s and our perspectives.On DEW #131, we selected the following articleRamon Marrero: DBT Model Contracts - Importance and Pitfallsdbt introduces model contract with 1.5 release. There were a few critics of the dbt model implementatio
DEW #129: DoorDash's Generative AI, Europe data salary, Data Validation with Great Expectations, Expedia's Event Sourcing
Welcome to another episode of Data Engineering Weekly. Aswin and I select 3 to 4 articles from each edition of Data Engineering Weekly and discuss them from the author’s and our perspectives.On DEW #129, we selected the following articleDoorDash identifies Five big areas for using Generative AI.Generative AI took the industry by storm, and every company is trying to figure out what it means to the
DEW #124: State of Analytics Engineering, ChatGPT, LLM & the Future of Data Consulting, Unified Streaming & Batch Pipeline, and Kafka Schema Management
Welcome to another episode of Data Engineering Weekly. Aswin and I select 3 to 4 articles from each edition of Data Engineering Weekly and discuss them from the author’s and our perspectives. On DEW #124, we selected the following articledbt: State of Analytics Engineeringdbt publishes the state of analytical [data???🤔] engineering. If you follow Data Engineering Weekly, We actively talk about dat
DEW 123: Generative AI at BuzzFeed, Building OnCall Culture & Dimensional Modeling at WhatNot
Welcome to another episode of Data Engineering Weekly Radio. Ananth and Aswin discussed a blog from BuzzFeed that shares lessons learned from building products powered by generative AI. The blog highlights how generative AI can be integrated into a company's work culture and workflow to enhance creativity rather than replace jobs. BuzzFeed provided their employees with intuitive access to APIs and
Podcast: dbt Reimagined, Change Data Capture @ Brex, on Data Products and how to describe them
DBT Reimagined by Pedram NavidThe challenge with this, having the Jinja templating, I found out two things. One is like; it is on runtime. So you have to build it and then run some simulations to understand whether you did it correctly or not.Jinja Templates also add cognitive load. The developers have to know how the Jinja template will work; how SQL will work, and it becomes a bit difficult to r
What Happened at Data Council 2023?
Hey folks, have you heard about the Data Council conference in Austin? The three-day event was jam-packed with exciting discussions and innovative ideas on data engineering and infrastructure, data science and algorithms, MLOps, generative AI, streaming infrastructure, analytics, and data culture and community. "People are so nice in the data community. Meeting them and brainstorming with many ide
Podcast: Analysis on MAD [Machine Learning, Artificial Intelligence & Data] Landscape
In this episode of Data Engineering Weekly Radio, we delve into modern data stacks under pressure and the potential consolidation of the data industry. We refer to a four-part article series that explores the data infrastructure landscape and the Software as a Service (SaaS) products available in data engineering, machine learning, and artificial intelligence.We discussed that the siloed nature of
Podcast: Data Product @ Oda, Reflection Talking with Data Leaders & Great Migration To Snowflake
We are back in our Data Engineering Weekly Radio for edition #121. We will take 2 or 3 articles from each week's Data Engineering Weekly edition and go through an in-depth analysis. Please subscribe to our Podcast on your favorite apps.From editor #121, we took the following articlesOda: Data as a product at OdaOda writes an exciting blog about “Data as a Product,” describing why we must treat dat
Data Engineering Weekly Radio #120
We are back in our Data Engineering Weekly Radio for edition #120. We will take 2 or 3 articles from each week's Data Engineering Weekly edition and go through an in-depth analysis. From editor #120, we took the following articlesTopic 1: Colin Campbell: The Case for Data Contracts - Preventative data quality rather than reactive data qualityIn this episode, we focus on the importance of data cont
Podcast: Data Engineering Weekly #119
We are super excited to be back to discussing Data Engineering Weekly Newsletter articles every week. We will take 2 or 3 articles from each week's Data Engineering Weekly edition and go through an in-depth analysis. On Data Engineering Weekly edition #119, We are taking three articles.#1 Netflix's article about Scaling Media Machine Learning at Netflixhttps://netflixtechblog.com/scaling-media-mac
Recommended
Speak And Shine English
Speak Local - English Listening and Speaking
Legal Off the Leash
Beyond the Syllabus: Pedagogy and Purpose
Mid-Age Tech
Hoops Collectors: Basketball & Sports Cards Podcast
LETRAS NÓRDICAS
Ås biblioteks podcast
Flora Funga Podcast
Vox Polony
Loving BDSM
The Sex Addiction Podcast For High-achievers and Entrepreneurs
