The reader-writer lock I trusted has a hidden cost

I have been using sync.RWMutex in my server for two weeks now and treating it like an obviously correct choice. Multiple readers can read at the same time, only one writer gets exclusive access, what is there not to like? Reading Cantrill and Bonwick’s paper made me realize I had only seen half the picture.

What RWMutex promises

The pitch sounds great. If 16 goroutines all want to read the map at the same time, they all get to proceed in parallel. No one waits on anyone else, as long as nobody is writing. Only when a writer shows up does everyone else have to pause.

This matches my benchmark results from last week: 48 nanoseconds per read with 16 goroutines hammering it concurrently. Looks like a clear win.

The hidden cost: cache-line bouncing

Here is the part that is easy to miss. Even though readers do not block each other from reading the actual data, the lock still has to track how many readers currently hold it. That reader count lives somewhere in memory, and every single reader has to atomically increment it when they acquire the lock and decrement it when they release it.

On a machine with many CPU cores, each core has its own cache. When core 1 increments that shared counter, it has to tell every other core “the value you have cached is now stale.” Core 2, which was about to read or update the same counter, has to fetch the fresh value from core 1’s cache instead of its own. This back-and-forth is called cache-line bouncing, and it happens on every single lock acquisition, even though the actual data being read never changes.

With enough cores all hammering the same RWMutex, this bouncing can get expensive enough that a plain Mutex, the simple kind with no separate reader path, actually performs better. The “smarter” lock loses to the “dumber” one because the smart one has more internal bookkeeping to keep coherent across cores.

The other trap: upgrade deadlocks

There is a second danger that has nothing to do with performance. Imagine a goroutine holds a read lock and then decides it needs to write, so it tries to upgrade to a write lock without releasing the read lock first.

Now imagine a second goroutine does the exact same thing at the same time. Goroutine A holds a read lock and waits for the write lock. Goroutine B also holds a read lock and waits for the write lock. Neither can get the write lock because the other one’s read lock is still active. They wait on each other forever. Permanent deadlock.

Go’s standard library does not even give you a direct upgrade function, which mostly protects you from this by design. But the same shape of bug can show up if you are not careful about releasing read locks before requesting a write lock anywhere in your code.

What this means for my server

My server is small enough that 16 cores hammering the read lock has not caused a measurable problem, based on the benchmark. But the lesson generalizes. A RWMutex is not automatically the right tool just because reads outnumber writes. It is the right tool when reads vastly outnumber writes and the system has few enough cores that the reader-count bookkeeping does not become its own bottleneck.

If I were scaling this server to something with dozens of cores and millions of reads per second, the next experiment would be benchmarking a plain Mutex against the RWMutex directly, instead of assuming the reader-writer version wins by default.