The 5 Mental Models Every Backend Engineer Needs
That nobody teaches you in school
It’s 11:47 AM on a Tuesday. Your e-commerce checkout service is humming along beautifully. Orders are flowing. Your team is in a standup congratulating themselves on the clean deploy from last night.
Then someone opens Slack.
“Hey, we just refunded $40,000 in duplicate orders. Customers clicked ‘Buy’ once but got charged twice.”
The room goes quiet. Someone pulls up the logs.
Two pods — both perfectly healthy, both doing exactly what they were coded to do — each saw the same order at the same millisecond, each decided “this hasn’t been processed yet,” and each completed the purchase. Simultaneously. Against the same user account.
No errors. No exceptions. No alerts. Just two pods politely agreeing to disagree about reality.
Welcome to distributed systems. Population: everyone who thought this couldn’t happen to them.
The Mindset Shift: From “Code That Works” to “Code That’s Correct”
Most engineers are taught to think about code correctness in isolation — does this function return the right value for this input? That’s necessary but nowhere near sufficient when your code runs on multiple machines simultaneously, sharing state, competing for resources, and failing in ways that only manifest under load at 2 AM.
The mindset shift you need is this:
Every piece of shared mutable state is a potential crime scene. Your job is to either eliminate the shared state entirely, or arm it with the right concurrency primitive before someone gets hurt.
This isn’t pessimism. It’s engineering.
Pattern 1: Optimistic Concurrency — The Polite Bouncer
The classic solution to the duplicate-order problem is a database lock. But distributed locks are expensive, require a lock service, and if the pod holding the lock crashes, the lock stays held and your entire checkout queue freezes.
There’s a better way: Compare-And-Swap (CAS), also called optimistic concurrency control.
The idea:
When you read a document, you get a token (a version number, an ETag, a timestamp).
When you write it back, you say “update this, but only if it still matches this token.”
If another pod modified the document in between, the database rejects your write and you retry with a fresh read.
No lock service. No deadlocks. No frozen queues. Just two pods politely disagreeing, with the database as referee.
This is the same mechanism behind Git merge conflicts, Google Docs collaborative editing, and the CMPXCHG instruction in your CPU. It scales beautifully because contention is the exception, not the rule — and you only pay for coordination when you actually need it.
Pattern 2: Stateless Pods — The Restaurant That Never Remembers You
Imagine a restaurant where your regular order is memorized by one specific waiter. Works great until that waiter is sick. Now you’re explaining your entire order from scratch to someone who’s never seen you before — and the kitchen has no record either.
That’s what happens when your pods store session state in memory.
The fix is embarrassingly simple and deeply counterintuitive to engineers coming from monolith backgrounds:
Pods should remember nothing. All state — session data, job progress, queue items — lives in the database. Any pod can handle any request. Scale out by adding pods. Scale down by removing them. No data loss, no sticky sessions, no “but what if Pod 3 dies” panic.
This is Factor VI of the Twelve-Factor App methodology — written in 2011, still violated daily by engineers who “just needed a quick cache.”
The quick cache becomes a load-bearing wall. The load-bearing wall becomes the reason you can’t scale past one pod without data corruption.
