The Databricks Data Engineer
Helping 18k+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.
Episodes
9 Databricks decisions that reveal your real seniority (and one that only looks senior)
Three in the morning. The nightly job that loads the orders table died halfway. Half the tasks are green, and the rerun button is right there. What you do in the next ten seconds says more about your level than anything on your resume.Nobody places you by your title or your years. They place you by a pattern of routine calls made fast, in the dark, and everyone around you can read it except you.In
The Databricks job market in 2026: there are two of them, and only one answers
One engineer tells you hiring is back and recruiters are warm again. The next one is sixty applications deep and hearing nothing. Same month, same platform, sometimes the same city.That sounds like noise. It isn't. It's a split, and what sorts you onto one side of it has almost nothing to do with how good you are.In this episode:- Why forty applications in a week returns two dead screens,
Databricks broadcast joins: when the memo beats the meeting (and when it kills a task)
Two engineers on the same team join the same big orders table to the same small lookup table. Same cluster, same data, one line of code different. Sarah's finishes in the time it takes to get a coffee. Mike's dies, and the error isn't about the data at all. A task ran out of memory building a hash map.Broadcast joins get passed around as a tip instead of a model: small table equals fas
Build the platform or use the platform: the two Databricks data engineer tracks nobody names
Two Databricks engineers sit at adjacent desks. Same title, same pay, same words in the ladder document. Watch a year go by, and they are not doing the same job.Both get rated strong. Both get told they're ready for more scope. Nobody says the useful thing: the evidence those two are stacking isn't interchangeable, and one of those years makes no sense on the other one's promotion pack
Why Unity Catalog exists: the Databricks governance chaos it was built to end
You inherited Unity Catalog already switched on. You learned the catalogs, the schemas, the grant statements, and never once saw the problem all of it was built to solve. Then someone in a review asks why the company suffered through that migration, and the best you've got is one word: governance.Here's what that word hides. Before Unity Catalog, one company ran five separate Databricks wo
How to get credit for Databricks platform work without becoming the on-call martyr
Your pipeline hasn't failed in months, and nobody noticed. Then it breaks at 2am, you fix it in twenty minutes, and Slack fills with fire emojis and thank-yous. You're the backbone of the team. You've also been at the same level for three years, and nobody can quite explain why.It's not bad luck, and it's not a skill gap. The exact thing everyone praises you for, being the one
Why Databricks Built Delta Lake When Parquet Was Already Good Enough
A nightly job is halfway through writing a batch when the cluster dies. Nobody runs a rollback, because there's nothing to roll back to. The next morning a dashboard is quietly serving half-written garbage, and no one can tell the good files from the wreckage.That's not a bug in Parquet. Every one of those files is valid, beautifully compressed, doing its job perfectly. The problem lives o
The Databricks pipeline that ran green for a year and was wrong the whole time
A revenue-attribution pipeline runs overnight, bronze to silver to gold, and lands a clean set of numbers the whole company uses to decide where the money goes. Every morning: green. Every morning, everyone moves on with their day. For a year, nobody looked any closer.The job succeeded every single night, so everyone trusted the data. But "finished" and "correct" are two differ
How Photon actually makes your Databricks queries faster (and when it silently doesn't)
Two engineers run the same SQL on the same Delta table. Same data, same cluster size, copy-pasted code. Alex goes to make a coffee and comes back to a query still running. Sam's is done before they finish reading the first Slack message. The only difference is one checkbox on the cluster called Photon.Most Databricks data engineers have that box ticked, pay a premium for it on every DBU, and c
The Databricks interview round nobody studies for (and almost everybody fails)
Picture the debrief room after a Databricks loop. Two candidates went through that day. On paper, a coin flip: SQL tied, Spark internals solid, system design clean for both. Score only the rounds with a rubric and you cannot separate them. And yet the room isn't split. One gets the offer, and the thing that decided it wasn't any of the rounds they studied for.It was the conversation everyo
The Spark Shuffle is baggage claim: why your job waits instead of computes (and more workers won't fix it)
Your Spark job has been running for forty minutes. The dashboard shows your cluster isn't even busy. So you do the obvious thing: add more workers. And it changes nothing.Here's why. During a shuffle, Spark is barely computing at all. It's tagging every row by destination, piling rows together, spilling the overflow to disk, and hauling data across the network between executors. It'
Your Databricks data quality framework is a Yeti: everyone talks about it, nobody has seen it work
An architecture review. A platform team is presenting their data quality setup, and honestly, it's impressive. Expectations on every ingestion table. Drift metrics on the dashboard. A dedicated alerts channel. Then a finance engineer asks the only question that counts: when did this last catch something before one of us did? Silence.That silence is the whole problem. The decks, the suites, the
Why senior Databricks engineers write less code than mid-level ones
Two engineers, same team, both five years in. Last quarter Mark shipped forty-seven pull requests across three pipelines. Sam shipped nine. On any dashboard, Mark wins by a mile. Sam got the staff offer. Mark got a kind note about continuing to demonstrate impact.This isn't politics, and it isn't luck. It's a pattern that specifically catches the engineers who are best at shipping, bec
4 habits that quietly turn your Databricks Delta Lake into a swamp
You built the table right. Well-partitioned, documented, fast enough that the row count came back before you finished reading your own Slack. Six months later it takes four minutes to return that same count, and nobody on your team ever decided to make it that way. There was no meeting, no design doc, no ticket titled "let's make this unqueryable by Q3."A swamp is not a decision. It&
Liquid Clustering vs Z-Ordering: 4 questions that decide
You open your Databricks workspace. Two Delta tables. Same size, same downstream BI workload. Table A was partitioned and z-ordered in 2023, runs fine. Table B is greenfield this quarter, liquid clustering by default. Your tech lead asks how aggressive you want to be with migration tickets. Whatever you type back is probably wrong.This is not a feature swap. It's a paradigm shift, and the migratio
The compounding curve: why some Databricks engineers' salaries grow 5x faster than others
Year one. Two new juniors join the same Databricks platform org. Same starting salary, same skills, same desk. Year three, five thousand bucks apart. Year eight, household-car-and-a-half apart. Every year. Forever.Both worked hard. Both stayed technical. Both got positive reviews. Neither did anything wrong. So what happened? Salary in this field isn't one curve. It's two that look identic
The 90/9/1 rule of Databricks performance work - how to triage Spark optimization in 60 seconds
Your team is three weeks into a Databricks performance push. Broadcast hints in PRs. AQE flags toggled like christmas lights. Partition counts re-tuned for the third time. The manager is asking, gently, when the gains are showing up in the bill.The staff DE on the next team finished theirs in two afternoons. Same workloads, bigger drop. They were running a triage you have never been taught.In this
The Databricks data engineer in 2026 - the four shifts that just changed your job
You scroll past the cancelled junior req, the "serverless first" line on your director's planning slide, and the third Lakebase mention from your Databricks rep this quarter. Each one looks like a news item. None of them feel like they're about you.They are. Four structural shifts have already happened in the field, and the words "Databricks data engineer" don't mea
9 Behaviors Quietly Killing Your Promotion To Senior Databricks Data Engineer
Mid-level is a down escalator. It looks like flat ground. You feel productive, your tickets close on Friday, your burndown chart is healthy, and your review says "reliable executor of well-defined work" for the third cycle in a row.That sentence is the official label for "not getting promoted this year" - and most Databricks data engineers never decode it. It isn't a skill
The Dashboard Theater: What Databricks Engineers Build That Nobody Opens
You check the usage logs on a dashboard you spent two weeks building. Zero views. Not low views. Zero. The stakeholder who requested it hasn't logged in once. Three months later they ask the exact question the dashboard answers, in a meeting, out loud, as if the dashboard doesn't exist. Because for them, it doesn't.In this episode:- Why the most technically impressive Databricks dashbo
The Preparation Gap: What Interviewers Actually Evaluate in Databricks Data Engineers
She could explain shuffle hash join versus sort merge join. She knew when Adaptive Query Execution kicks in. She had six weeks of notes on Delta Lake, Spark memory, and cluster configs. She walked into the Databricks senior interview feeling genuinely confident. Then the interviewer asked her to walk through a diagnosis, not recite a definition, and everything she studied was aimed at the wrong ta
The Invisible Engineer: Why Your Best Work Gets the Least Recognition
She kept a terabyte-scale pipeline running for six months without a single incident. Not one page, not one late dashboard. Then review season came and the engineer who spent two weekends fixing an outage he partly caused got the promotion instead. Her name wasn't in a single incident report, because when you prevent problems, there's no report to put your name on.In this episode:- Why the
Recommended
Bible Tea
Pod Save America
Dateline NBC
صداستان: ساعتی با موسیقی
Becoming: HER with Nikki Spoelstra
Exposing Workplace Bullying
Everyday AI Made Simple - AI For Everyday Tasks
FemTech Focus
Not Too Sensitive - Empowering Highly Sensitive People (HSPs) To Own Their Sensitivity
Mother Daughter Relationship Show
Skin Deep MDs with Dr. Mamina Turegano, Dr. Lindsey Zubritsky and Dr. Jenny Liu
girl talk 🎧🫧
