The Databricks Data Engineer
Helping 18k+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.
Episodes
Your Databricks director: what they fear, why they fund the duller proposal, what gets yours through
Two proposals sit on a head of data's desk, both about the nightly pipeline that feeds the finance dashboard. One swaps the full reload for incremental loads with Auto Loader: it finishes in a fraction of the time and it's cheaper every month after the first. The other keeps the full reload, puts it on a bigger job cluster and starts it an hour earlier. It costs more. Forever. She asks one
Data skew is a checkout line - how to unjam Databricks jobs without bigger clusters
Your nightly Databricks job ran long, so you open the Spark UI. Two hundred tasks. One hundred ninety-nine finished in about a minute. One is still running, and it's been running for forty. You double the cluster and rerun it. Same forty minutes. Nearly twice the bill.The cluster was never the problem. The rule that keeps your joins and group-bys correct is the same rule that lets one task car
Can You Reach Senior Databricks Engineer Without Ever Leading an Architecture Review?
Sam is four years into a Databricks role, up for senior this cycle. Merged pull requests on the ingestion layer. The nightly job he rescued at three in the morning when the orders table stopped landing. Then his manager asks for one call where the answer wasn't obvious, he framed it, and he lived with the result. Sam starts three different sentences. None of them finish.The packet is full of w
Why Databricks serverless exists, and which of your jobs should stay on classic clusters
Open the workspace. Start a cluster. Get coffee. Come back, and sometimes it's ready. If you've been doing this a few years you can feel that wait in your shoulders, and it's the least interesting part of the story.Before any of it, you had to answer a question you could not possibly answer: how big should this be? Start small and you pay in latency. Start big and you pay for machines
How Databricks engineers lose the final round (and the 4 places it happens)
You cleared the recruiter screen. You cleared the Databricks technical round, and cleared it well. Then the final round, then the email that says strong candidate, tough decision. You go back looking for the mistake and there isn't one. You were good. Somebody was just easier to picture.That sentence never makes it into the feedback. And it isn't a bad panel or a bias against you. It's
The real reason your Databricks week never got simpler - it isn't the lakehouse
Your org finished the migration. Two systems became one. The nightly copy nobody wanted to own is gone, and so is the quarterly meeting where finance and the data science team each showed up with a real revenue number from tables in different buildings. That pain was real, and it is genuinely dead. So why does your week feel heavier than the deck promised?Because most of what got sold as simplific
The Databricks incident where the most senior engineer never touched the keyboard
Thursday, just after two in the afternoon. The first message doesn't come from an alert, it comes from a person in revenue operations: the biggest region jumped overnight and nobody sold anything new. Nobody answers.The engineer who owns the nightly Lakeflow job finds it in the run history. An overnight failure, a restart before anyone was awake, a window of orders written a second time under
9 Databricks decisions that reveal your real seniority (and one that only looks senior)
Three in the morning. The nightly job that loads the orders table died halfway. Half the tasks are green, and the rerun button is right there. What you do in the next ten seconds says more about your level than anything on your resume.Nobody places you by your title or your years. They place you by a pattern of routine calls made fast, in the dark, and everyone around you can read it except you.In
The Databricks job market in 2026: there are two of them, and only one answers
One engineer tells you hiring is back and recruiters are warm again. The next one is sixty applications deep and hearing nothing. Same month, same platform, sometimes the same city.That sounds like noise. It isn't. It's a split, and what sorts you onto one side of it has almost nothing to do with how good you are.In this episode:- Why forty applications in a week returns two dead screens,
Databricks broadcast joins: when the memo beats the meeting (and when it kills a task)
Two engineers on the same team join the same big orders table to the same small lookup table. Same cluster, same data, one line of code different. Sarah's finishes in the time it takes to get a coffee. Mike's dies, and the error isn't about the data at all. A task ran out of memory building a hash map.Broadcast joins get passed around as a tip instead of a model: small table equals fas
Build the platform or use the platform: the two Databricks data engineer tracks nobody names
Two Databricks engineers sit at adjacent desks. Same title, same pay, same words in the ladder document. Watch a year go by, and they are not doing the same job.Both get rated strong. Both get told they're ready for more scope. Nobody says the useful thing: the evidence those two are stacking isn't interchangeable, and one of those years makes no sense on the other one's promotion pack
Why Unity Catalog exists: the Databricks governance chaos it was built to end
You inherited Unity Catalog already switched on. You learned the catalogs, the schemas, the grant statements, and never once saw the problem all of it was built to solve. Then someone in a review asks why the company suffered through that migration, and the best you've got is one word: governance.Here's what that word hides. Before Unity Catalog, one company ran five separate Databricks wo
How to get credit for Databricks platform work without becoming the on-call martyr
Your pipeline hasn't failed in months, and nobody noticed. Then it breaks at 2am, you fix it in twenty minutes, and Slack fills with fire emojis and thank-yous. You're the backbone of the team. You've also been at the same level for three years, and nobody can quite explain why.It's not bad luck, and it's not a skill gap. The exact thing everyone praises you for, being the one
Why Databricks Built Delta Lake When Parquet Was Already Good Enough
A nightly job is halfway through writing a batch when the cluster dies. Nobody runs a rollback, because there's nothing to roll back to. The next morning a dashboard is quietly serving half-written garbage, and no one can tell the good files from the wreckage.That's not a bug in Parquet. Every one of those files is valid, beautifully compressed, doing its job perfectly. The problem lives o
The Databricks pipeline that ran green for a year and was wrong the whole time
A revenue-attribution pipeline runs overnight, bronze to silver to gold, and lands a clean set of numbers the whole company uses to decide where the money goes. Every morning: green. Every morning, everyone moves on with their day. For a year, nobody looked any closer.The job succeeded every single night, so everyone trusted the data. But "finished" and "correct" are two differ
How Photon actually makes your Databricks queries faster (and when it silently doesn't)
Two engineers run the same SQL on the same Delta table. Same data, same cluster size, copy-pasted code. Alex goes to make a coffee and comes back to a query still running. Sam's is done before they finish reading the first Slack message. The only difference is one checkbox on the cluster called Photon.Most Databricks data engineers have that box ticked, pay a premium for it on every DBU, and c
The Databricks interview round nobody studies for (and almost everybody fails)
Picture the debrief room after a Databricks loop. Two candidates went through that day. On paper, a coin flip: SQL tied, Spark internals solid, system design clean for both. Score only the rounds with a rubric and you cannot separate them. And yet the room isn't split. One gets the offer, and the thing that decided it wasn't any of the rounds they studied for.It was the conversation everyo
The Spark Shuffle is baggage claim: why your job waits instead of computes (and more workers won't fix it)
Your Spark job has been running for forty minutes. The dashboard shows your cluster isn't even busy. So you do the obvious thing: add more workers. And it changes nothing.Here's why. During a shuffle, Spark is barely computing at all. It's tagging every row by destination, piling rows together, spilling the overflow to disk, and hauling data across the network between executors. It'
Your Databricks data quality framework is a Yeti: everyone talks about it, nobody has seen it work
An architecture review. A platform team is presenting their data quality setup, and honestly, it's impressive. Expectations on every ingestion table. Drift metrics on the dashboard. A dedicated alerts channel. Then a finance engineer asks the only question that counts: when did this last catch something before one of us did? Silence.That silence is the whole problem. The decks, the suites, the
Why senior Databricks engineers write less code than mid-level ones
Two engineers, same team, both five years in. Last quarter Mark shipped forty-seven pull requests across three pipelines. Sam shipped nine. On any dashboard, Mark wins by a mile. Sam got the staff offer. Mark got a kind note about continuing to demonstrate impact.This isn't politics, and it isn't luck. It's a pattern that specifically catches the engineers who are best at shipping, bec
4 habits that quietly turn your Databricks Delta Lake into a swamp
You built the table right. Well-partitioned, documented, fast enough that the row count came back before you finished reading your own Slack. Six months later it takes four minutes to return that same count, and nobody on your team ever decided to make it that way. There was no meeting, no design doc, no ticket titled "let's make this unqueryable by Q3."A swamp is not a decision. It&
Liquid Clustering vs Z-Ordering: 4 questions that decide
You open your Databricks workspace. Two Delta tables. Same size, same downstream BI workload. Table A was partitioned and z-ordered in 2023, runs fine. Table B is greenfield this quarter, liquid clustering by default. Your tech lead asks how aggressive you want to be with migration tickets. Whatever you type back is probably wrong.This is not a feature swap. It's a paradigm shift, and the migratio
The compounding curve: why some Databricks engineers' salaries grow 5x faster than others
Year one. Two new juniors join the same Databricks platform org. Same starting salary, same skills, same desk. Year three, five thousand bucks apart. Year eight, household-car-and-a-half apart. Every year. Forever.Both worked hard. Both stayed technical. Both got positive reviews. Neither did anything wrong. So what happened? Salary in this field isn't one curve. It's two that look identic
The 90/9/1 rule of Databricks performance work - how to triage Spark optimization in 60 seconds
Your team is three weeks into a Databricks performance push. Broadcast hints in PRs. AQE flags toggled like christmas lights. Partition counts re-tuned for the third time. The manager is asking, gently, when the gains are showing up in the bill.The staff DE on the next team finished theirs in two afternoons. Same workloads, bigger drop. They were running a triage you have never been taught.In this
The Databricks data engineer in 2026 - the four shifts that just changed your job
You scroll past the cancelled junior req, the "serverless first" line on your director's planning slide, and the third Lakebase mention from your Databricks rep this quarter. Each one looks like a news item. None of them feel like they're about you.They are. Four structural shifts have already happened in the field, and the words "Databricks data engineer" don't mea
9 Behaviors Quietly Killing Your Promotion To Senior Databricks Data Engineer
Mid-level is a down escalator. It looks like flat ground. You feel productive, your tickets close on Friday, your burndown chart is healthy, and your review says "reliable executor of well-defined work" for the third cycle in a row.That sentence is the official label for "not getting promoted this year" - and most Databricks data engineers never decode it. It isn't a skill
The Dashboard Theater: What Databricks Engineers Build That Nobody Opens
You check the usage logs on a dashboard you spent two weeks building. Zero views. Not low views. Zero. The stakeholder who requested it hasn't logged in once. Three months later they ask the exact question the dashboard answers, in a meeting, out loud, as if the dashboard doesn't exist. Because for them, it doesn't.In this episode:- Why the most technically impressive Databricks dashbo
The Preparation Gap: What Interviewers Actually Evaluate in Databricks Data Engineers
She could explain shuffle hash join versus sort merge join. She knew when Adaptive Query Execution kicks in. She had six weeks of notes on Delta Lake, Spark memory, and cluster configs. She walked into the Databricks senior interview feeling genuinely confident. Then the interviewer asked her to walk through a diagnosis, not recite a definition, and everything she studied was aimed at the wrong ta
The Invisible Engineer: Why Your Best Work Gets the Least Recognition
She kept a terabyte-scale pipeline running for six months without a single incident. Not one page, not one late dashboard. Then review season came and the engineer who spent two weekends fixing an outage he partly caused got the promotion instead. Her name wasn't in a single incident report, because when you prevent problems, there's no report to put your name on.In this episode:- Why the
Recommended
زن بودن
Medicine with Meaning
Yours Truly Johnny Dollar Collection
Take Back Your Health®
CBS Radio Mystery Theater | Old Time Radio
Solved Murders - True Crime Stories
紐約鳥｜New York Aperture
The Swerve Podcast: Obscure Topics | Conspiracy Theories
The Bread and Banter Podcast
The Church of What's Happening Now: The New Testament
Busted Open
Deadline: White House
