Two safe pieces of code can combine into an unsafe one
Combining two perfectly correct concurrent modules can still produce a deadlock.
This is the idea from the Cantrill and Bonwick paper that took me the longest to actually believe. It sounds wrong at first. If module A is correct and thread-safe, and module B is correct and thread-safe, surely using them together is also safe? It is not, and understanding why changed how I think about building bigger systems on top of my server.
Setting up the problem
Say I have two completely separate, well-tested pieces of my key-value store. One handles the main db map and its lock. The other, hypothetically, handles a separate expiry map that tracks when keys should auto-delete, with its own lock.
Each one in isolation is fine. SET locks the db, writes, unlocks. SetExpiry locks the expiry map, writes, unlocks. Tested independently, no race conditions, no deadlocks. Both modules pass every test you throw at them.
Where it breaks
Now I want a single atomic operation: SETEX key value 60, which sets a value and an expiry time together, and I want it to be atomic, meaning no other goroutine can see the key set without its expiry, or the expiry set without the key.
To do this safely, I need to lock the db map and the expiry map together, as one unit, for the duration of the operation. That means my SETEX function has to reach inside both modules and grab both of their internal locks directly.
This is the moment the abstraction breaks. The whole point of wrapping the db map in its own struct with its own lock was so the rest of my code never had to think about locking details, it could just call Set and trust it was safe. The moment I need to combine two modules atomically, I have to reach past that wall and manage both locks myself, in the right order, every time.
Why this causes deadlocks
Now imagine a second function, DeleteWithExpiry, that also touches both maps, but happens to lock the expiry map first and the db map second, the opposite order from SETEX.
If SETEX runs on one goroutine and DeleteWithExpiry runs on another at the same time, you get the same deadlock pattern as splitting a single lock badly: SETEX holds the db lock and waits for the expiry lock, while DeleteWithExpiry holds the expiry lock and waits for the db lock. Both wait forever.
Neither module was wrong on its own. The deadlock only exists because combining them required breaking their encapsulation, and the two combining functions happened to acquire locks in different orders.
The deeper point
This is what the paper means by locks not being composable. With most programming concepts, if two pieces work correctly on their own, you can usually trust them together. Concurrency breaks this. The moment you need an atomic operation spanning two lock-protected modules, you are forced to expose their internals, and now every combining function in your codebase has to agree on lock ordering, forever, or risk a deadlock that only shows up under specific timing.
