Exploring Large Object Heap with WinDBG

This is the second part of http://vimvq1987.com/2016/11/debug-net-memory-dump-windbg-crash-course-part-1/, – which is far from complete. In this post, we will explore the Large Object Heap (LOH) of a .NET application with WinDBG

Why LOH? It’s a special heap contains the memory objects which are more than 85000 bytes in size – which, previously, never compacted (that was changed with .NET 4.5 when you have an option to compact LOH, but beware of the consequences).

If you know about the generation garbage collection in CLR, you already know that when an object is no longer used, its memory will be claimed back for later use. GC do more than that, by trying to compact the “free” memory – so it’ll move (via copy and delete) the survived objects next to each other, therefore you’ll be a big, continuous free memory. This helps improving performance in upcoming allocations, but only until a point. If the objects are too big, then the costs of moving the objects around might outweigh the benefits. Microsoft did a bunch of performance test and they decided that 85000 bytes is the break point, where the costs are bigger than the performance gain. Then we have LOH, where the memory is almost never compacted.

Because LOH is expensive, both in creating and reclaiming – it should be used with some cautions. LOH’s objects should be long lived, preferably the entire application lifetime.  So, if you have LOH objects constantly filled up/freed, that would be a bad sign for performance.

The first steps – as usual, is to load the memory dump, and the load the sos and sosex modules and we can start working. To retrieve information about the LOH, sos extension provide an export !eeheap -gc

0:000> !eeheap -gc
Number of GC Heaps: 2
------------------------------
Heap 0 (000000df21fdc4e0)
generation 0 starts at 0x000000dfc6ebad68
generation 1 starts at 0x000000dfc6e87cf8
generation 2 starts at 0x000000df224f1000
ephemeral segment allocation context: none
         segment             begin         allocated              size
000000df224f0000  000000df224f1000  000000dfc890ab80  0xa6419b80(2789317504)
Large object heap starts at 0x000000e1224f1000
         segment             begin         allocated              size
000000e1224f0000  000000e1224f1000  000000e12b54ee30  0x905de30(151379504)
Heap Size:               Size: 0xaf4779b0 (2940697008) bytes.
------------------------------
Heap 1 (000000df22058050)
generation 0 starts at 0x000000e0c3be9e90
generation 1 starts at 0x000000e0c3673a08
generation 2 starts at 0x000000e0224f1000
ephemeral segment allocation context: none
         segment             begin         allocated              size
000000e0224f0000  000000e0224f1000  000000e0c4866848  0xa2375848(2721536072)
Large object heap starts at 0x000000e1324f1000
         segment             begin         allocated              size
000000e1324f0000  000000e1324f1000  000000e13c1d39c0  0x9ce29c0(164506048)
000000e1a9f10000  000000e1a9f11000  000000e1b6541450  0xc630450(207815760)
Heap Size:               Size: 0xb8688658 (3093857880) bytes.
------------------------------
GC Heap Size:            Size: 0x167b00008 (6034554888) bytes.

So we can see this application has two heaps. Each heap has a LOH segment.  We can explore one by one.

To investigate the first one, we can simply use !dumpheap <Starting address of LOH>

0:000> !dumpheap 0x000000e1224f1000
         Address               MT     Size
000000e1224f1000 000000df2192fc50       24 Free
000000e1224f1018 000000df2192fc50       30 Free
000000e1224f1038 00007fff3cbce100     9416     
000000e1224f3500 000000df2192fc50       30 Free
000000e1224f3520 00007fff3cbce100     1048     
000000e1224f3938 000000df2192fc50       30 Free
000000e1224f3958 00007fff3cbce100     8184     
000000e1224f5950 000000df2192fc50       30 Free
000000e1224f5970 00007fff3cbce100    16344     
000000e1224f9948 000000df2192fc50       30 Free
000000e1224f9968 00007fff3cbce100    32664     
000000e122501900 000000df2192fc50       30 Free
000000e122501920 00007fff3cbce100     9416     
000000e122503de8 000000df2192fc50       30 Free
000000e122503e08 00007fff3cbce100    16344     
000000e122507de0 000000df2192fc50       30 Free
000000e122507e00 00007fff3cbce100    32664     
000000e12250fd98 000000df2192fc50       30 Free
000000e12250fdb8 00007fff3cbce100    16344     
000000e122513d90 000000df2192fc50       30 Free
000000e122513db0 00007fff3cbce100    65304     
000000e122523cc8 000000df2192fc50       30 Free
000000e122523ce8 00007fff3cbce100   130584     
000000e122543b00 000000df2192fc50       30 Free
000000e122543b20 00007fff3cbce100     2072     
000000e122544338 000000df2192fc50       30 Free
000000e122544358 00007fff3cbce100     4120     
000000e122545370 000000df2192fc50       30 Free
000000e122545390 00007ffedf296f98    97200     
000000e12255cf40 000000df2192fc50       30 Free
000000e12255cf60 00007fff3cbce100     8216     
000000e12255ef78 000000df2192fc50       30 Free
000000e12255ef98 00007fff3cbce100    32664     
000000e122566f30 000000df2192fc50       30 Free
000000e122566f50 00007fff3cbce100    16408     
000000e12256af68 000000df2192fc50    97262 Free
000000e122582b58 00007fff29edac28   202080     
000000e1225b40b8 000000df2192fc50       30 Free
000000e1225b40d8 00007fff3cbd27a8    87345     
000000e1225c9610 000000df2192fc50     9878 Free
000000e1225cbca8 00007fff29edac28   202080     
000000e1225fd208 000000df2192fc50       30 Free
000000e1225fd228 00007fff29edac28    97200     
000000e122614dd8 000000df2192fc50       30 Free
000000e122614df8 00007fff29edac28    97200     
000000e12262c9a8 000000df2192fc50       30 Free
000000e12262c9c8 00007fff3cbce100   131064     
000000e12264c9c0 000000df2192fc50   203422 Free
000000e12267e460 00007fff3cbce100    32792     
000000e122686478 000000df2192fc50       30 Free
000000e122686498 00007fff3cbce100    65304     
000000e1226963b0 000000df2192fc50       30 Free
000000e1226963d0 00007fff3cbce100   130584     
000000e1226b61e8 000000df2192fc50       30 Free
000000e1226b6208 00007fff3cbce100   131064     
000000e1226d6200 000000df2192fc50       30 Free
000000e1226d6220 00007fff3cbce100   131064     
000000e1226f6218 000000df2192fc50   341590 Free
000000e122749870 00007fff3cbce100    65560     
000000e122759888 000000df2192fc50     2598 Free
000000e12275a2b0 00007fff3cbce100   131064     
000000e12277a2a8 000000df2192fc50  2740110 Free
000000e122a17238 00007fff3cbce100   131064     
000000e122a37230 000000df2192fc50    80294 Free
000000e122a4abd8 00007fff3cbcda88    98830     
000000e122a62de8 000000df2192fc50  5359262 Free
000000e122f7f488 00007fff3cbcc448   202080     
000000e122fb09e8 000000df2192fc50  2554686 Free
000000e123220528 00007fff3cbd27a8   168118     
000000e1232495e0 000000df2192fc50  3812534 Free
000000e1235ec298 00007fff3cbcda88   325884     
000000e12363bb98 000000df2192fc50  1722558 Free
000000e1237e0458 00007fff3cbcda88   287938     
000000e123826920 000000df2192fc50   564910 Free
000000e1238b07d0 00007fff3cbcda88  1133348     
000000e1239c52f8 000000df2192fc50   904798 Free
000000e123aa2158 00007fff3cbd0368   751028     
000000e123b59710 000000df2192fc50       30 Free
000000e123b59730 00007fff3cbcc448 16147872     
000000e124abfcd0 000000df2192fc50       30 Free
000000e124abfcf0 00007fff3cbcc448 16147872     
000000e125a26290 000000df2192fc50 31726614 Free
000000e127867ea8 00007fff3cbcc448 33486336     
000000e1298574a8 000000df2192fc50 22587478 Free
000000e12ade1d00 00007fff3cbcc448  7786800     
000000e1324f1000 000000df2192fc50       24 Free
000000e1324f1018 000000df2192fc50  3039334 Free
000000e1327d7080 00007fff3cbcc448    97200     
000000e1327eec30 000000df2192fc50  1292294 Free
000000e13292a438 00007fff3cbcc448    97200     
000000e132941fe8 000000df2192fc50   387614 Free
000000e1329a0a08 00007fff3cbcda88   341524     
000000e1329f4020 000000df2192fc50  2347590 Free
000000e132c31268 00007fff29edac28    97200     
000000e132c48e18 000000df2192fc50       30 Free
000000e132c48e38 00007fff29edac28    97200     
000000e132c609e8 000000df2192fc50 44391870 Free
000000e1356b67a8 00007fff3cbd0368   751028     
000000e13576dd60 000000df2192fc50       30 Free
000000e13576dd80 00007ffedf487d78  6008056     
000000e135d28a78 000000df2192fc50       30 Free
000000e135d28a98 00007fff3cbd0368   751028     
000000e135de0050 000000df2192fc50       30 Free
000000e135de0070 00007ffedf488730  4506048     
000000e13622c230 000000df2192fc50       30 Free
000000e13622c250 00007ffedf488110  4506048     
000000e136678410 000000df2192fc50  2020446 Free
000000e136865870 00007ffee13063e8   202080     
000000e136896dd0 000000df2192fc50 26599374 Free
000000e1381f4da0 00007fff3cbcc448 33486336     
000000e13a1e43a0 000000df2192fc50       30 Free
000000e13a1e43c0 00007fff3cbcc448 33486336     
000000e1a9f11000 000000df2192fc50 207613678 Free
000000e1b650fef0 00007fff3cbcc448   202080     

Statistics:
              MT    Count    TotalSize Class Name
00007ffedf296f98        1        97200 System.Collections.Generic.Dictionary`2+Entry[[System.Type, mscorlib],[StructureMap.Graph.PluginFamily, StructureMap]][]
00007ffee13063e8        1       202080 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[DynamicFieldInfo, Models]][]
00007fff3cbd27a8        2       255463 System.Byte[]
00007fff29edac28        6       792960 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.IO.Compression.ZipArchiveEntry, System.IO.Compression]][]
00007fff3cbce100       25      1351352 System.Object[]
00007fff3cbcda88        5      2187524 System.String
00007fff3cbd0368        3      2253084 System.Int32[]
00007ffedf488730        1      4506048 System.Collections.Generic.Dictionary`2+Entry[[System.Uri, System],[EPiServer.Web.PermanentLinkMap, EPiServer]][]
00007ffedf488110        1      4506048 System.Collections.Generic.Dictionary`2+Entry[[EPiServer.Core.ContentReference, EPiServer],[EPiServer.Web.PermanentContentLinkMap, EPiServer]][]
00007ffedf487d78        1      6008056 System.Collections.Generic.Dictionary`2+Entry[[System.Guid, mscorlib],[EPiServer.Web.PermanentLinkMap, EPiServer]][]
00007fff3cbcc448       10    141140112 System.Collections.Hashtable+bucket[]
000000df2192fc50       58    360401232      Free
Total 114 objects

So what do we expect to find here? The number of large objects, and the status of them.

You can of course use the -stat argument to have a summarized result:

0:000> !dumpheap -stat 0x000000e1224f1000
Statistics:
              MT    Count    TotalSize Class Name
00007ffedf296f98        1        97200 System.Collections.Generic.Dictionary`2+Entry[[System.Type, mscorlib],[StructureMap.Graph.PluginFamily, StructureMap]][]
00007ffee13063e8        1       202080 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[DynamicFieldInfo, Web.Models]][]
00007fff3cbd27a8        2       255463 System.Byte[]
00007fff29edac28        6       792960 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.IO.Compression.ZipArchiveEntry, System.IO.Compression]][]
00007fff3cbce100       25      1351352 System.Object[]
00007fff3cbcda88        5      2187524 System.String
00007fff3cbd0368        3      2253084 System.Int32[]
00007ffedf488730        1      4506048 System.Collections.Generic.Dictionary`2+Entry[[System.Uri, System],[EPiServer.Web.PermanentLinkMap, EPiServer]][]
00007ffedf488110        1      4506048 System.Collections.Generic.Dictionary`2+Entry[[EPiServer.Core.ContentReference, EPiServer],[EPiServer.Web.PermanentContentLinkMap, EPiServer]][]
00007ffedf487d78        1      6008056 System.Collections.Generic.Dictionary`2+Entry[[System.Guid, mscorlib],[EPiServer.Web.PermanentLinkMap, EPiServer]][]
00007fff3cbcc448       10    141140112 System.Collections.Hashtable+bucket[]
000000df2192fc50       58    360401232      Free
Total 114 objects

So here we have 58 of “freed” objects, taking about 350MB of memory. This information is not exactly helpful on it own. However, if you continue to run the application and take another memory dump, and it has even more “freed” objects – then it might be a problem: Your site might be using LOH more than it should.

Leave a Reply

Your email address will not be published. Required fields are marked *