Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
This weekly show from Confluent, hosted by Tim Berglund, Adi Polak, and Viktor Gamov, features in-depth interviews with software developers about the real-world challenges they've faced in their careers. The conversations explore how each person's experiences shaped their understanding of building systems, particularly around open-source data streaming technologies like Apache Kafka and Apache Flink. Whether you're an experienced streaming engineer or new to real-time data, the podcast aims to offer engaging stories and high-quality technical discussion.
Episodes
155 Contributors Later: Where Kafka Is Headed ft. Andrew Schofield | Ep. 34
Tim Berglund talks to Andrew Schofield (Confluent) about his career in Apache Kafka. Andrew’s first job: working on queuing systems in 1991. His challenge: working at Confluent and in the Kafka community to bring queue semantics into Kafka while also helping shape major efforts like diskless Kafka, disaster recovery, and the project’s broader evolution.► Queues for Kafka Explained (KIP-932): https
Agent Mesh: A Microservice With a Brain ft John Miller & Eric Broda | Ep. 33
Tim Berglund talks to John Miller (Enid Technologies) and Eric Broda (Agentic Mesh Company) about their work on agent mesh and enterprise agents. Their challenge: rethinking agents as enterprise-grade participants in business processes.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by
10 Years of Kafka Streams with Matthias J. Sax | Ep. 32
Tim Berglund talks to Matthias J. Sax (Confluent) about 10 whole years of Kafka Streams! Matthias’ first job: electrician-in-training on BMW’s assembly lines. His challenge: reflecting on 10 years of Kafka Streams growth, major milestones, and what comes next.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by
Gunnar Morling Built a New Parquet Engine with AI | Ep. 31
Tim Berglund talks to Gunnar Morling (Confluent) about his career in open source Java and data infrastructure. Gunnar’s first job: a student PHP developer in AMD’s e-learning group. His challenge: building Hardwood, a fast, multi-threaded Parquet engine for Java with minimal dependencies.► The One Billion Row Challenge blog post: https://www.morling.dev/blog/one-billion-row-challenge/SEASON 2 Host
How AI Is Changing Apache Iceberg with Russell Spitzer | Ep. 30
Adi Polak talks to Russell Spitzer (Snowflake) about his career in open source data infrastructure. Russell’s first job: software engineer in test at DataStax. His challenge: making Apache Iceberg ready for AI and streaming.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧
Being Wrong in the Right Direction with Caleb Grillo | Ep. 29
Tim Berglund talks to Caleb Grillo (Confluent / WarpStream) about his career in data streaming product management. Caleb’s first job: washing windows. Their challenge: reshaping Confluent Cloud’s billing and pioneering diskless Kafka to trade latency for huge cost savings.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Moha
Building Banking Systems with Kafka Streams with Mateo Rojas | Ep. 28
Adi Polak talks to Mateo Rojas (LittleHorse) about his career working with Kafka Streams. Mateo’s first job: building a real-money policy management platform on early Kafka Streams. His challenge: working at LittleHorse with Kafka as a workflow engine and deciding whether it should be the source of truth.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gall
The AI Impact on the Developer Future with Joseph Morais | Ep. 27
Tim Berglund talks to Joseph Marais (Confluent) about his career in data streaming. Joseph’s first job: SAN administrator. His challenge: AI radically changing how developers build software.The Pragmatic Engineer episode ft. Grady Booch: https://newsletter.pragmaticengineer.com/p/software-architecture-with-grady-boochSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited b
Tech vs. People: The Hardest Problem with Will LaForest | Ep. 26
Tim Berglund talks to Will LaForest (Confluent) about his career in software and data streaming. Will’s first job: a high school internship at DARPA. His challenge: turning advanced technology into something people actually care about.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by
How Maven Changed Java Forever with Baruch Sadogursky | Ep. 25
Viktor Gamov talks to Baruch Sudakurski (TuxCare) about his career in developer advocacy. Baruch's first job: fixing electric kettles. His challenge: figuring out how to map a non-relational database (MongoDB) into Spring Data’s SQL-oriented model.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal
Inside OpenAI’s Streaming Backbone with Aravind Suresh | Ep. 24
Adi Polak talks to Aravind Suresh (OpenAI) about his career in distributed systems and real-time streaming. Arvind’s first job: coding at school. His challenge: turning OpenAI’s fragile Kafka setup into a reliable, multi-region streaming backbone.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites
The 1 Billion Row Challenge with Gunnar Morling | Ep. 23
Tim Berglund talks to Gunnar Morling (Confluent) about his career in open source Java and data streaming. Gunnar’s first job: a student PHP developer in AMD’s e-learning group. His challenge: working at Decodable on the 1 Billion Row Challenge.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Ar
From Git Blame to Principal Engineer with Sage Pierce | Ep. 22
Adi Polak talks to Sage Pierce (Indeed) about his career in software engineering and event-driven architectures. Sage’s first job: Java Swing development at a Department of Defense–affiliated research lab. His challenge: working at Indeed on event-driven views and IMI to join data across domains in a polyglot microservices world.Sage's Atleon project: https://github.com/atleon SEASON 2 Hosted
From Coding Machines to Leading Humans ft. Leonid Igolnik | Ep. 21
Viktor Gamov talks to Leonid Igolnik (Former CTO at Clari) about his career in B2B SaaS engineering leadership. Leonid’s first job: teaching kids Pascal. His challenge: changing buyer behavior and scale complex systems.Books mentioned:► Influence without Authority: https://www.amazon.com/Influence-Without-Authority-Allan-Cohen/dp/0471463302► Drive: https://www.danpink.com/books/drive/► Blink: The
Killing Clusters & Orchestrating Chaos with Colt McNealy | Ep. 20
Tim Berglund talks to Colt McNealy (LittleHorse Enterprises) about his career in distributed systems. Colt’s first job: software engineer at a real estate company. His challenge: working in a complex microservices environment and turning that pain into Little Horse.Colt's Current 2024 talk: https://current.confluent.io/2024-sessions/kafka-streams-as-a-data-store-for-a-workflow-engineGunnar Mo
Deleting Architecture for Better Systems ft. Daniel Doubrovkine | Ep. 19
Adi Polak talks to Daniel Doubrovkine (Shopify) about his career building data‑intensive systems. Daniel’s first job: delivering pharmacy medications by bike. His challenge: building Artsy’s Art Genome and auctions as simple as possible.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork b
Fail Fast & Ship It with Jeremy Custenborder | Ep. 18
Viktor Gamov talks to Jeremy Custenborder (Confluent) about his career in large-scale systems. Jeremy’s first job: paper boy. His challenge: keeping MySpace running at a massive pre-cloud scale while building the tools that didn’t exist yet and learning to fail fast.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Mu
From “This May Never Work” to WarpStream with Richie Artoul | Ep. 17
Tim Berglund talks to Richie Artoul (WarpStream/Confluent) about his career in data infrastructure. Richie’s first job: working at Howie’s Game Shack, a walk‑in LAN gaming cafe. His challenge: working at Datadog on a new log storage system.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwor
Inside $3M GPU Racks: Powering Modern AI with Bryan Oliver | Ep. 16
Adi Polak talks to Bryan Oliver (Thoughtworks) about his career in platform engineering and large-scale AI infrastructure. Bryan’s first job: building pools and teaching swimming lessons. His challenge: running large-scale GPU data centers while keeping AI workloads predictable and reliable.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter F
Hacking Kafka Streams with Sophie Blee‑Goldman | Ep. 15
Tim Berglund talks to Sophie Blee-Goldman (Responsive) about her career in container orchestration and Kafka Streams. Sophie’s first job: interning at Google. Her challenge: helping a hyper-growth customer whose Kafka Streams app was about to hit partition-based scalability limits.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and N
Turning Chaos into Push-Button Provisioning with Dhiraj Suri| Ep. 14
Viktor Gamov talks to Dhiraj Suri (Confluent) about his career in systems engineering and stream governance. Dhiraj’s first job: software developer at NetApp. His challenge: working at Splunk to stitch together disparate systems into an event-driven provisioning platform.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Moham
The Late Night Hack That Changed Daniel Hinojosa's Career | Ep. 13
Tim Berglund talks to Daniel Hinojosa (an independent consultant) about his career in software development, data engineering, and event-driven architecture. Daniel’s first job: Sears credit card telemarketing. His challenge: working at a company with internal bad blood and being called at 11 p.m. to pull off a late night “security research” hack on Windows and Lotus Notes systems.SEASON 2 Hosted b
From Early Startups to Product Leadership with Mike Agnich | Ep. 12
Tim Berglund talks to Mike Agnich (Confluent) about his career in product leadership and startups. Mike’s first job: refereeing youth basketball. His challenge: leading product across connectors, governance, stream processing, and partnerships at Confluent.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coa
Adventures in Data Infrastructure with Gwen Shapira | Ep. 11
Adi Polak talks to Gwen Shapira (Nile) about her career in databases and data infrastructure. Gwen’s first job: a side hustle fixing computers. Her challenge: figuring out why a production report at HP slowed down dramatically after daylight saving time.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coasta
Decreasing Java Build Times with Pratik Patel | Ep. 10
Tim Berglund talks to Pratik Patel (Azul Systems) about his career in developer relations and Java. Pratik’s first job: computer lab assistant at UNC Chapel Hill. His challenge: working at a large enterprise with manual, slow build processes and transforming them through automation.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and
Reimagining Stream Processing with Matthias J. Sax | Ep. 9
Viktor Gamov talks to Matthias J. Sax (Confluent) about his career in stream processing and, specifically, Kafka Streams. Matthias’ first job: an electrician-in-training on BMW’s assembly lines. His challenge: building Kafka Streams at Confluent with a focus on API design, backward compatibility, and a library-first approach that also fits microservices.SEASON 2 Hosted by Tim Berglund, Adi Polak a
How Time Kills All Deals in Pre-Sales with Rachel Pedreschi | Ep. 8
Listen: https://confluent.buzzsprout.com | In this episode, Tim Berglund talks to his guest, Rachel Pedreschi (DeltaStream), about her career in pre-sales engineering. Her first job: rectory office assistant at her local parish. Her challenge/theme: working at early-stage startups to bridge sales, marketing, and engineering to reach product-market fit.Check out Tim and Rachel's previous podca
Scaling AI in Engineering with Peter Bell | Ep. 7
Listen: https://confluent.buzzsprout.com | Today, Adi Polak talks to her guest, Peter Bell (gather.dev), about his career in software engineering leadership, CTO community building, and AI-driven development. Peter’s first job: electronics lab technician at their school (alongside shifts at Tesco). His challenge/theme: working at scale with AI adoption and change management.Check out gather.dev: h
How Kafka Expert Robin Moffat Tackles Open Source Problems | Ep. 6
Today, Viktor Gamov talks to his colleague Robin Moffat (Confluent) about his career in data engineering. His first job: paperboy. His challenge: working at a retailer with Oracle materialized views as well as teaching others how to productively approach Kafka’s internal systems.Blog posts mentioned in the podcast:► Oracle Materialized Views troubleshooting: https://rnm1978.wordpress.com/2011/01/0
Building Parquet into Apache Pinot ft. Neha Pawar | Ep. 5
Today, Tim Berglund talks to Neha Pawar (StarTree) about her career in real-time analytics and open source database engineering. Her first job: a year-long internship at NVIDIA. Her challenge: leading the technical effort to add native Parquet support into Apache Pinot.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
The Fix That Secured 1000s of Credit Cards ft. Brian Sletten | Ep. 4
In this episode, Tim talks to Brian Sletten (Bosatsu Consulting) about his career in software development. His first job: working at a small communications company that built network matrix switch interfaces. His challenge/theme: overhauling credit card storage and security at a major hospitality company.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gall
How Viktor Gamov Stays Curious as Tech Rapidly Evolves | Ep. 3
Adi Polak interviews her co-host, Viktor Gamov, about his career’s evolution from distributed systems to streaming technology. Viktor’s first job: apple picking. His challenge/theme: staying curious and non-judgmental in the ever-changing landscape of tech.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coa
How Tim Berglund Found His Calling | Ep. 2
Viktor Gamov interviews his co-host, Tim Berglund, about his career in the world of streaming data. Tim’s first job: Burger King broiler steamer. His challenge/theme: pivoting from working in hardware and firmware to finding his calling in enterprise software and developer relations.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and
Building Real-time Systems for Apple, Nike & more ft. Adi Polak | Ep. 1
The Confluent Developer Podcast is here! For this first episode, Tim Berglund talks to his co-host, Adi Polak (Confluent), about her career in distributed data systems. Her first job: neighborhood dogwalker. Her challenge/theme: early Hadoop, working at Akamai on data optimization and real-time threat detection for huge global customers like Apple, Nike, Facebook and others, and the power of colla
We're back! Welcome to the Confluent Developer Podcast.
Weekly episodes launching Sept. 22! | Hi, I'm Tim Berglund. It's been about four years since I've been podcasting at Confluent, and "Streaming Audio" has been on hiatus for a little more than two, but I've got great news: we are back! We're back with a new name, a new format, and new hosts. Welcome to the Confluent Developer Podcast, where we talk to software dev
Apache Kafka 3.5 - Kafka Core, Connect, Streams, & Client Updates
Apache Kafka® 3.5 is here with the capability of previewing migrations between ZooKeeper clusters to KRaft mode. Follow along as Danica Fine highlights key release updates.Kafka Core:KIP-833 provides an updated timeline for KRaft.KIP-866 now is preview and allows migration from an existing ZooKeeper cluster to KRaft mode.KIP-900 introduces a way to bootstrap the KRaft controllers with SCRAM creden
A Special Announcement from Streaming Audio
After recording 64 episodes and featuring 58 amazing guests, the Streaming Audio podcast series has amassed over 130,000 plays on YouTube in the last year. We're extremely proud of these achievements and feel that it's time to take a well-deserved break. Streaming Audio will be taking a vacation! We want to express our gratitude to you, our valued listeners, for spending 10,000 hours wit
How to use Data Contracts for Long-Term Schema Management
Have you ever struggled with managing data long term, especially as the schema changes over time? In order to manage and leverage data across an organization, it’s essential to have well-defined guidelines and standards in place around data quality, enforcement, and data transfer. To get started, Abraham Leal (Customer Success Technical Architect, Confluent) suggests that organizations associate t
How to use Python with Apache Kafka
Can you use Apache Kafka® and Python together? What’s the current state of Python support? And what are the best options to get started? In this episode, Dave Klein joins Kris to talk about all things Kafka and Python: the libraries, the tools, and the pros & cons. He also talks about the new course he just launched to support Python programmers entering the event-streaming world.Dave has been
Next-Gen Data Modeling, Integrity, and Governance with YODA
In this episode, Kris interviews Doron Porat, Director of Infrastructure at Yotpo, and Liran Yogev, Director of Engineering at ZipRecruiter (formerly at Yotpo), about their experiences and strategies in dealing with data modeling at scale.Yotpo has a vast and active data lake, comprising thousands of datasets that are processed by different engines, primarily Apache Spark™. They wanted to provide
Migrate Your Kafka Cluster with Minimal Downtime
Migrating Apache Kafka® clusters can be challenging, especially when moving large amounts of data while minimizing downtime. Michael Dunn (Solutions Architect, Confluent) has worked in the data space for many years, designing and managing systems to support high-volume applications. He has helped many organizations strategize, design, and implement successful Kafka cluster migrations between diffe
Real-Time Data Transformation and Analytics with dbt Labs
dbt is known as being part of the Modern Data Stack for ELT processes. Being in the MDS, dbt Labs believes in having the best of breed for every part of the stack. Oftentimes folks are using an EL tool like Fivetran to pull data from the database into the warehouse, then using dbt to manage the transformations in the warehouse. Analysts can then build dashboards on top of that data, or execute tes
What is the Future of Streaming Data?
What’s the next big thing in the future of streaming data? In this episode, Greg DeMichillie (VP of Product and Solutions Marketing, Confluent) talks to Kris about the future of stream processing in environments where the value of data lies in their ability to intercept and interpret data.Greg explains that organizations typically focus on the infrastructure containers themselves, and not on the t
What can Apache Kafka Developers learn from Online Gaming?
What can online gaming teach us about making large-scale event management more collaborative in real-time? Ben Gamble (Developer Relations Manager, Aiven) has come to the world of real-time event streaming from an usual source: the video games industry. And if you stop to think about it, modern online games are complex, distributed real-time data systems with decades of innovative techniques to t
Apache Kafka 3.4 - New Features & Improvements
Apache Kafka® 3.4 is released! In this special episode, Danica Fine (Senior Developer Advocate, Confluent), shares highlights of the Apache Kafka 3.4 release. This release introduces new KIPs in Kafka Core, Kafka Streams, and Kafka Connect.In Kafka Core:KIP-792 expands the metadata each group member passes to the group leader in its JoinGroup subscription to include the highest stable generation t
How to use OpenTelemetry to Trace and Monitor Apache Kafka Systems
How can you use OpenTelemetry to gain insight into your Apache Kafka® event systems? Roman Kolesnev, Staff Customer Innovation Engineer at Confluent, is a member of the Customer Solutions & Innovation Division Labs team working to build business-critical OpenTelemetry applications so companies can see what’s happening inside their data pipelines. In this episode, Roman joins Kris to discuss tr
What is Data Democratization and Why is it Important?
Data democratization allows everyone in an organization to have access to the data they need, and the necessary tools needed to use this data effectively. In short, data democratization enables better business decisions. In this episode, Rama Ryali, a Senior IT and Data Executive, chats with Kris Jenkins about the importance of data democratization in modern systems.Rama explains that tech has unp
Git for Data: Managing Data like Code with lakeFS
Is it possible to manage and test data like code? lakeFS is an open-source data version control tool that transforms object storage into Git-like repositories, offering teams a way to use the same workflows for code and data. In this episode, Kris sits down with guest Adi Polak, VP of DevX at Treeverse, to discuss how lakeFS can be used to facilitate better management and testing of data.At its co
Using Kafka-Leader-Election to Improve Scalability and Performance
How does leader election work in Apache Kafka®? For the past 2 ½ years, Adithya Chandra, Staff Software Engineer at Confluent, has been working on Kafka scalability and performance, specifically partition leader election. In this episode, he gives Kris Jenkins a deep dive into the power of leader election in Kafka replication, why we need it, how it works, what can go wrong, and how it's bein
Real-Time Machine Learning and Smarter AI with Data Streaming
Are bad customer experiences really just data integration problems? Can real-time data streaming and machine learning be democratized in order to deliver a better customer experience? Airy, an open-source data-streaming platform, uses Apache Kafka® to help business teams deliver better results to their customers. In this episode, Airy CEO and co-founder Steffen Hoellinger explains how his company
The Present and Future of Stream Processing
The past year saw new trends emerge in the world of data streaming technologies, as well as some unexpected and novel use cases for Apache Kafka®. New reflections on the future of stream processing and when companies should adopt microservice architecture inspired several talks at this year’s industry conferences. In this episode, Kris is joined by his colleagues Danica Fine, Senior Developer Advo
Top 6 Worst Apache Kafka JIRA Bugs
Entomophiliac, Anna McDonald (Principal Customer Success Technical Architect, Confluent) has seen her fair share of Apache Kafka® bugs. For her annual holiday roundup of the most noteworthy Kafka bugs, Anna tells Kris Jenkins about some of the scariest, most surprising, and most enlightening corner cases that make you ask, “Ah, so that’s how it really works?”She shares a lot of interesting details
Learn How Stream-Processing Works The Simplest Way Possible
Could you explain Apache Kafka® in ways that a small child could understand? When Mitch Seymour, author of Mastering Kafka Streams and ksqlDB, wanted a way to communicate the basics of Kafka and event-based stream processing, he decided to author a children’s book on the subject, but it turned into something with a far broader appeal.Mitch conceived the idea while writing a traditional manuscript
Building and Designing Events and Event Streams with Apache Kafka
What are the key factors to consider when developing event-driven architecture? When properly designed, events can connect existing systems with a common language and allow data exchange in near real time. They also help reduce complexity by providing a single source of truth that eliminates the need to synchronize data between different services or applications. They enable dynamic behavior, allo
Rethinking Apache Kafka Security and Account Management
Is there a better way to manage access to resources without compromising security? New employees need access to a variety of resources within a company's tech stack. But manually granting access can be error-prone. And when employees leave, their access must be revoked, thus potentially introducing security risks if an admin misses one. In this podcast, Kris Jenkins talks to Anuj Sawani (Secu
Real-time Threat Detection Using Machine Learning and Apache Kafka
Can we use machine learning to detect security threats in real-time? As organizations increasingly rely on distributed systems, it is becoming more important to analyze the traffic that passes through those systems quickly. Confluent Hackathon ’22 finalist, Géraud Dugé de Bernonville (Data Consultant, Zenika Bordeaux), shares how his team used TensorFlow (machine learning) and Neo4j (graph databas
Improving Apache Kafka Scalability and Elasticity with Tiered Storage
What happens when you need to store more than a few petabytes of data? Rittika Adhikari (Software Engineer, Confluent) discusses how her team implemented tiered storage, a method for improving the scalability and elasticity of data storage in Apache Kafka®. She also explores the motivating factors for building it in the first place: cost, performance, and manageability. Before Tiered Storage, ther
Decoupling with Event-Driven Architecture
In principle, data mesh architecture should liberate teams to build their systems and gather data in a distributed way, without having to explicitly coordinate. Data is the thing that can and should decouple teams, but proper implementation has its challenges.In this episode, Kris talks to Florian Albrecht (Solution Architect, Hermes Germany) about Galapagos, an open-source DevOps software tool fo
If Streaming Is the Answer, Why Are We Still Doing Batch?
Is real-time data streaming the future, or will batch processing always be with us? Interest in streaming data architecture is booming, but just as many teams are still happily batching away. Batch processing is still simpler to implement than stream processing, and successfully moving from batch to streaming requires a significant change to a team’s habits and processes, as well as a meaningful u
Security for Real-Time Data Stream Processing with Confluent Cloud
Streaming real-time data at scale and processing it efficiently is critical to cybersecurity organizations like SecurityScorecard. Jared Smith, Senior Director of Threat Intelligence, and Brandon Brown, Senior Staff Software Engineer, Data Platform at SecurityScorecard, discuss their journey from using RabbitMQ to open-source Apache Kafka® for stream processing. As well as why turning to fully-man
Running Apache Kafka in Production
What are some recommendations to consider when running Apache Kafka® in production? Jun Rao, one of the original Kafka creators, as well as an ongoing committer and PMC member, shares the essential wisdom he's gained from developing Kafka and dealing with a large number of Kafka use cases.Here are 6 recommendations for maximizing Kafka in production:1. Nail Down the Operational PartWhen setti
Build a Real Time AI Data Platform with Apache Kafka
Is it possible to build a real-time data platform without using stateful stream processing? Forecasty.ai is an artificial intelligence platform for forecasting commodity prices, imparting insights into the future valuations of raw materials for users. Nearly all AI models are batch-trained once, but precious commodities are linked to ever-fluctuating global financial markets, which require real-ti
Optimizing Apache JVMs for Apache Kafka
Java Virtual Machines (JVMs) impact Apache Kafka® performance in production. How can you optimize your event-streaming architectures so they process more Kafka messages using the same number of JVMs? Gil Tene (CTO and Co-Founder, Azul) delves into JVM internals and how developers and architects can use Java and optimized JVMs to make real-time data pipelines more performant and more cost effective
Apache Kafka 3.3 - KRaft, Kafka Core, Streams, & Connect Updates
Apache Kafka® 3.3 is released! With over two years of development, KIP-833 marks KRaft as production ready for new AK 3.3 clusters only. On behalf of the Kafka community, Danica Fine (Senior Developer Advocate, Confluent) shares highlights of this release, with KIPs from Kafka Core, Kafka Streams, and Kafka Connect. To reduce request overhead and simplify client-side code, KIP-709 extends the Offs
Application Data Streaming with Apache Kafka and Swim
How do you set data applications in motion by running stateful business logic on streaming data? Capturing key stream processing events and cumulative statistics that necessitate real-time data assessment, migration, and visualization remains as a gap—for event-driven systems and stream processing frameworks according to Fred Patton (Developer Evangelist, Swim Inc.) In this episode, Fred explains
International Podcast Day - Apache Kafka Edition | Streaming Audio Special
What’s your favorite podcast? Would you like to find some new ones? In celebration of International Podcast Day, Kris Jenkins invites 12 experts from the Apache Kafka® community to talk about their favorite podcasts. Unlike other episodes where guests educate developers and tell stories about Kafka, its surrounding technological ecosystem, or the Cloud, this special episode provides a glimpse into
How to Build a Reactive Event Streaming App - Coding in Motion
How do you build an event-driven application that can react to real-time data streams as they happen? Kris Jenkins (Senior Developer Advocate, Confluent) will be hosting another fun, hands-on programming workshop—Coding in Motion: Watching the River Flow, to demonstrate how you can build a reactive event streaming application with Apache Kafka®, ksqlDB using Python.As a developer advocate, Kris of
Real-Time Stream Processing, Monitoring, and Analytics With Apache Kafka
Processing real-time event streams enables countless use cases big and small. With a day job designing and building highly available distributed data systems, Simon Aubury (Principal Data Engineer, Thoughtworks) believes stream-processing thinking can be applied to any stream of events. In this episode, Simon shares his Confluent Hackathon ’22 winning project—a wildlife monitoring system to observ
Reddit Sentiment Analysis with Apache Kafka-Based Microservices
How do you analyze Reddit sentiment with Apache Kafka® and microservices? Bringing the fresh perspective of someone who is both new to Kafka and the industry, Shufan Liu, nascent Developer Advocate at Confluent, discusses projects he has worked on during his summer internship—a Cluster Linking extension to a conceptual data pipeline project, and a microservice-based Reddit sentiment-analysis proje
Capacity Planning Your Apache Kafka Cluster
How do you plan Apache Kafka® capacity and Kafka Streams sizing for optimal performance? When Jason Bell (Principal Engineer, Dataworks and founder of Synthetica Data), begins to plan a Kafka cluster, he starts with a deep inspection of the customer's data itself—determining its volume as well as its contents: Is it JSON, straight pieces of text, or images? He then determines if Kafka is a go
Streaming Real-Time Sporting Analytics for World Table Tennis
Reimagining a data architecture to provide real-time data flow for sporting events can be complicated, especially for organizations with as much data as World Table Tennis (WTT). Vatsan Rama (Director of IT, ITTF Group) shares why real-time data is essential in the sporting world and how his team reengineered their data system in 18 months, moving from a solely on-premises infrastructure to a clou
Real-Time Event Distribution with Data Mesh
Inheriting software in the banking sector can be challenging. Perhaps the only thing harder is inheriting software built by a committee of banks. How do you keep it running, while improving it, refactoring it, and planning a bigger future for it? In this episode, Jean-Francois Garet (Technical Architect, Symphony) shares his experience at Symphony as he helps it evolve from an inherited, monolithi
Apache Kafka Security Best Practices
Security is a primary consideration for any system design, and Apache Kafka® is no exception. Out of the box, Kafka has relatively little security enabled. Rajini Sivaram (Principal Engineer, Confluent, and co-author of “Kafka: The Definitive Guide” ) discusses how Kafka has gone from a system that included no security to providing an extensible and flexible platform for any business to build a se
What Could Go Wrong with a Kafka JDBC Connector?
Java Database Connectivity (JDBC) is the Java API used to connect to a database. As one of the most popular Kafka connectors, it's important to prevent issues with your integrations. In this episode, we'll cover how a JDBC connection works, and common issues with your database connection. Why the Kafka JDBC Connector? When it comes to streaming database events into Apache Kafka®, the JDB
Apache Kafka Networking with Confluent Cloud
Setting up a reliable cloud networking for your Apache Kafka® infrastructure can be complex. There are many factors to consider—cost, security, scalability, and availability. With immense experience building cloud-native Kafka solutions on Confluent Cloud, Justin Lee (Principal Solutions Engineer, Enterprise Solutions Engineering, Confluent) and Dennis Wittekind (Customer Success Technical Archite
Event-Driven Systems and Agile Operations
How do the principles of chaotic, agile operations in the military apply to software development and event-driven systems? As a former Royal Marine, Ben Ford (Founder and CEO, Commando Development) is also a software developer, with many years of experience building event streaming architectures across financial services and startups. He shares principles that the military employs in chaotic condi
Streaming Analytics and Real-Time Signal Processing with Apache Kafka
Imagine you can process and analyze real-time event streams for intelligence to mitigate cyber threats or keep soldiers constantly alerted to risks and precautions they should take based on events. In this episode, Jeffrey Needham (Senior Solutions Engineer, Advanced Technology Group, Confluent) shares use cases on how Apache Kafka® can be used for real-time signal processing to mitigate risk befo
Blockchain Data Integration with Apache Kafka
How is Apache Kafka® relevant to blockchain technology and cryptocurrency? Fotios Filacouris (Staff Solutions Engineer, Confluent) has been working with Kafka for close to five years, primarily designing architectural solutions for financial services, he also has expertise in the blockchain. In this episode, he joins Kris to discuss how blockchain and Kafka are complementary, and he also highlight
Automating Multi-Cloud Apache Kafka Cluster Rollouts
To ensure safe and efficient deployment of Apache Kafka® clusters across multiple cloud providers, Confluent rolled out a large scale cluster management solution.Rashmi Prabhu (Staff Software Engineer & Eng Manager, Fleet Management Platform, Confluent) and her team have been building the Fleet Management Platform for Confluent Cloud. In this episode, she delves into what Fleet Management is,
Common Apache Kafka Mistakes to Avoid
What are some of the common mistakes that you have seen with Apache Kafka® record production and consumption? Nikoleta Verbeck (Principal Solutions Architect at Professional Services, Confluent) has a role that specifically tasks her with performance tuning as well as troubleshooting Kafka installations of all kinds. Based on her field experience, she put together a comprehensive list of common is
Recommended
The Joe Rogan Experience
World News Tonight with David Muir
Talk About Talk - Executive & Leadership Communication Skills
This Past Weekend w/ Theo Von
Stand In The Circle
Conspiracy Files with Paige Carter
Learn English B1 with Daily News | English Listening Practice
Bad Friends
The Swerve Podcast: Obscure Topics | Conspiracy Theories
The Bread and Banter Podcast
The Church of What's Happening Now: The New Testament
Deadline: White House
