Wake up one thread, not the whole room

The last idea from the Cantrill and Bonwick paper covers two practical choices you make when writing concurrent code, plus a debugging technique that only makes sense once you understand why concurrency bugs are so hard to catch.

Mutex over semaphore

A mutex has one job: a thread either holds it or it doesn’t. That’s the entire state. When something goes wrong and you are staring at your program afterward trying to figure out what happened, a mutex tells you a clear story. Thread X had the lock. Everyone else was waiting.

A semaphore is more flexible. Instead of a simple held or not-held state, it holds a count, letting some number of threads through at once. That flexibility is also what makes it harder to debug. If a semaphore’s counter is at 3, you cannot easily tell who incremented it, who decremented it, or whether a bug somewhere let one thread through twice. There is no single owner to point to. Just a number with a history you cannot reconstruct after the fact.

Signal versus broadcast

This one applies to condition variables, which let a goroutine sleep until some condition becomes true, then wake back up. Go’s sync.Cond has two ways to wake sleeping goroutines: Signal, which wakes exactly one, and Broadcast, which wakes all of them.

Picture a scenario in a future version of my server: multiple client goroutines are waiting for a specific key to be set, say they all issued a BLPOP-style blocking read on the same key. When that key finally gets set, only one of those waiting goroutines can actually consume the value. Using Broadcast here would wake every single one of them up at once. They all rush to check the condition, only one succeeds, and the rest go straight back to sleep, having burned CPU cycles for nothing. With enough waiters, this pile of wasted wakeups is called the thundering herd problem.

Signal avoids this. It wakes exactly one waiting goroutine, the one that can actually proceed, and leaves everyone else asleep until the next event. The rule of thumb is: use Signal when only one waiter can make progress, and save Broadcast for situations where the condition genuinely affects everyone, like a shutdown signal that every goroutine needs to notice.

Why these bugs are so hard to catch with print statements

Concurrency bugs depend on the exact order in which threads happen to interleave. A race condition might only appear once every ten thousand runs, when two goroutines happen to hit the same line of code in the same microsecond. Adding a fmt.Println to debug it often changes the timing enough that the bug disappears, only to reappear the moment you remove the print statement. This is maddening, and it is why traditional debugging often fails here.

Postmortem debugging

Cantrill’s answer is to stop trying to catch the bug while it happens and instead capture the entire state of the program at the exact moment it fails, then analyze that frozen snapshot afterward. This is what a core dump is: a full picture of every goroutine’s stack, every lock’s state, every variable’s value, captured at the instant of the crash.

Instead of guessing what interleaving caused the deadlock, you can look directly at the frozen state and see it: goroutine 12 is stuck waiting for a lock that goroutine 47 is holding, and goroutine 47 is stuck waiting for a lock that goroutine 12 holds. The deadlock is sitting right there in the data, no reproduction needed.

Tying it together

Together they explain why concurrent programming has a reputation as something close to a black art. It is not that the rules are unknowable. It is that the cost of breaking any one of them often will not show up until your code is under real load, with real timing, on real hardware.