A while back, I wrote this post A curious case of cookie threading issue – Quan Mai’s blog (vimvq1987.com) as an example of how to use WinDbg to diagnose a race condition (with results in an infinite loop). Naturally that issue needs to be fixed. A starting point to fix a race condition is to put locks whenever you change the data. But it’s not that simple. Too much locking would definitely hurt your application performance, and increase the chance of you running into another nasty problem (namely, deadlock). So the art of locking is “lagom” – Swedish word for “just right”, not too much, not too little.
ReaderWriteLockSlim comes to rescue. You only need to lock when writing, not locking. So this for reading
readerWriterLockSlim.EnterReadLock();
try
{
return httpRequest.Cookies[CookieKey];
}
finally
{
readerWriterLockSlim.ExitReadLock();
}And this for writing
readerWriterLockSlim.EnterWriteLock();
try
{
httpRequest.Cookies.Remove(CookieKey);
}
finally
{
readerWriterLockSlim.ExitWriteLock();
}Would be enough, right?
No. The issue runs deeper than that.
If we go back to the stacktrace of the issue
0:039> !clrstack
OS Thread Id: 0x1e34 (39)
Child SP IP Call Site
000000740273cff0 00007ff9297d9f59 System.Collections.Generic.HashSet`1[[System.__Canon, mscorlib]].Remove(System.__Canon)
000000740273d080 00007ff92a8eb4e3 System.Web.HttpCookieCollection.Get(System.String)
000000740273d0c0 00007ff9314de98d and look closer. Something is not right. If you do not spot the issue – it’s fine, I missed it too. The problem is that we have a HashSet.Remove inside a HttpCookieCollection.Get. Uh oh.
If we look at source code of HttpCookieCollection, inside Get, it calls to EnsureKeyValidated
if (cookie != null) {
EnsureKeyValidated(name, cookie.Value);
}referencesource/HttpCookieCollection.cs at master · microsoft/referencesource · GitHub
which itself calls to
_keysAwaitingValidation.Remove(key);referencesource/HttpCookieCollection.cs at master · microsoft/referencesource · GitHub
_keysAwaitingValidation is a HashSet<string> . That explains the Remove we saw. And that explains why ReaderWriterLockSlim is not enough – changes are made within the supposedly read only action as well.
The only valid solution here is to lock both read and write actions on HttpCookieCollection. However, as HttpCookieCollection is per request, so our lock object should only be per request as well (we certainly do not want every thread to be locked when we get cookie on a request).
Moral of the story:
- Look closer. There might always be something underneath. Close to the truth is still not the truth.
- Never assume that a
Getmethod is thread-safe. The implementation can do plenty of unexpected things under the hood. - You might ask why this happen, as
HttpCookieCollectionis per request. Turned out that there is a code that useThreadPool.QueueUserWorkItemto queue tasks which share the sameHttpContextobject. As we learned, that’s the recipe for disaster. Think twice (or thrice) before sharing an object that is not thread-safe between threads.