A single unhandled race condition let customers order stock that no longer existed. Here's how we diagnosed it and rebuilt the inventory logic to be atomic.
Every system looks correct until it meets real concurrency. This is the story of a bug that only showed up under load — and the architectural fix that made our ordering system trustworthy.
TheProblem
The ordering flow looked simple on paper: a customer places an order, the system checks stock, and if there's enough, it deducts the quantity and confirms the order.
Under normal, sequential conditions, this works perfectly. The failure only appeared when two orders arrived within milliseconds of each other — a common scenario during peak hours in any food-service platform.
Here's what was actually happening:
This is a textbook race condition: two operations reading shared state before either has finished writing to it.
WhyThisMattersBeyondaSingleBug
Overselling isn't just an inconvenience — it erodes trust at the exact moment a business is trying to build it. A customer who orders something that turns out to be unavailable after payment or commitment is a churn risk, and for a food-service platform specifically, it creates operational chaos in the kitchen.
The deeper lesson: any system tracking finite, shared resources under concurrent access needs to be designed for that concurrency from the start, not patched afterward.
TheFix:MakingStockChecksAtomic
The core fix was combining the "check" and the "deduct" into a single atomic operation, rather than two separate steps that could be interleaved by competing requests.
At the database level, this meant moving from a read-then-write pattern to a conditional write — an operation that only succeeds if the stock is still sufficient at the exact moment of the write, using row-level locking to serialize concurrent attempts on the same inventory record.
Conceptually:
This guarantees that when two orders compete for the same unit, one succeeds and the other is correctly rejected or queued — instead of both silently succeeding.
WhatI'dTellAnyEngineerBuildingSimilarSystems
Race conditions rarely show up in development. They show up in production, under load, at the worst possible time. If your system has any shared, finite resource — inventory, seats, slots, credits — ask early: what happens if two requests hit this at the exact same millisecond? If the answer isn't obvious, that's the system telling you it needs a stronger consistency guarantee before it needs another feature.
Building for scale isn't just about handling more traffic. It's about making sure correctness holds even when that traffic collides.