This is the second part of http://vimvq1987.com/debug-net-memory-dump-windbg-crash-course-part-1/, – which is far from complete. In this post, we will explore the Large Object Heap (LOH) of a .NET application with WinDBG
Why LOH? It’s a special heap contains the memory objects which are more than 85000 bytes in size – which, previously, never compacted (that was changed with .NET 4.5 when you have an option to compact LOH, but beware of the consequences).
If you know about the generation garbage collection in CLR, you already know that when an object is no longer used, its memory will be claimed back for later use. GC do more than that, by trying to compact the “free” memory – so it’ll move (via copy and delete) the survived objects next to each other, therefore you’ll be a big, continuous free memory. This helps improving performance in upcoming allocations, but only until a point. If the objects are too big, then the costs of moving the objects around might outweigh the benefits. Microsoft did a bunch of performance test and they decided that 85000 bytes is the break point, where the costs are bigger than the performance gain. Then we have LOH, where the memory is almost never compacted.
Because LOH is expensive, both in creating and reclaiming – it should be used with some cautions. LOH’s objects should be long lived, preferably the entire application lifetime. So, if you have LOH objects constantly filled up/freed, that would be a bad sign for performance.
The first steps – as usual, is to load the memory dump, and the load the sos and sosex modules and we can start working. To retrieve information about the LOH, sos extension provide an export !eeheap -gc
0:000> !eeheap -gc
Number of GC Heaps: 2
------------------------------
Heap 0 (000000df21fdc4e0)
generation 0 starts at 0x000000dfc6ebad68
generation 1 starts at 0x000000dfc6e87cf8
generation 2 starts at 0x000000df224f1000
ephemeral segment allocation context: none
segment begin allocated size
000000df224f0000 000000df224f1000 000000dfc890ab80 0xa6419b80(2789317504)
Large object heap starts at 0x000000e1224f1000
segment begin allocated size
000000e1224f0000 000000e1224f1000 000000e12b54ee30 0x905de30(151379504)
Heap Size: Size: 0xaf4779b0 (2940697008) bytes.
------------------------------
Heap 1 (000000df22058050)
generation 0 starts at 0x000000e0c3be9e90
generation 1 starts at 0x000000e0c3673a08
generation 2 starts at 0x000000e0224f1000
ephemeral segment allocation context: none
segment begin allocated size
000000e0224f0000 000000e0224f1000 000000e0c4866848 0xa2375848(2721536072)
Large object heap starts at 0x000000e1324f1000
segment begin allocated size
000000e1324f0000 000000e1324f1000 000000e13c1d39c0 0x9ce29c0(164506048)
000000e1a9f10000 000000e1a9f11000 000000e1b6541450 0xc630450(207815760)
Heap Size: Size: 0xb8688658 (3093857880) bytes.
------------------------------
GC Heap Size: Size: 0x167b00008 (6034554888) bytes.
So we can see this application has two heaps. Each heap has a LOH segment. We can explore one by one.
To investigate the first one, we can simply use !dumpheap <Starting address of LOH>
0:000> !dumpheap 0x000000e1224f1000
Address MT Size
000000e1224f1000 000000df2192fc50 24 Free
000000e1224f1018 000000df2192fc50 30 Free
000000e1224f1038 00007fff3cbce100 9416
000000e1224f3500 000000df2192fc50 30 Free
000000e1224f3520 00007fff3cbce100 1048
000000e1224f3938 000000df2192fc50 30 Free
000000e1224f3958 00007fff3cbce100 8184
000000e1224f5950 000000df2192fc50 30 Free
000000e1224f5970 00007fff3cbce100 16344
000000e1224f9948 000000df2192fc50 30 Free
000000e1224f9968 00007fff3cbce100 32664
000000e122501900 000000df2192fc50 30 Free
000000e122501920 00007fff3cbce100 9416
000000e122503de8 000000df2192fc50 30 Free
000000e122503e08 00007fff3cbce100 16344
000000e122507de0 000000df2192fc50 30 Free
000000e122507e00 00007fff3cbce100 32664
000000e12250fd98 000000df2192fc50 30 Free
000000e12250fdb8 00007fff3cbce100 16344
000000e122513d90 000000df2192fc50 30 Free
000000e122513db0 00007fff3cbce100 65304
000000e122523cc8 000000df2192fc50 30 Free
000000e122523ce8 00007fff3cbce100 130584
000000e122543b00 000000df2192fc50 30 Free
000000e122543b20 00007fff3cbce100 2072
000000e122544338 000000df2192fc50 30 Free
000000e122544358 00007fff3cbce100 4120
000000e122545370 000000df2192fc50 30 Free
000000e122545390 00007ffedf296f98 97200
000000e12255cf40 000000df2192fc50 30 Free
000000e12255cf60 00007fff3cbce100 8216
000000e12255ef78 000000df2192fc50 30 Free
000000e12255ef98 00007fff3cbce100 32664
000000e122566f30 000000df2192fc50 30 Free
000000e122566f50 00007fff3cbce100 16408
000000e12256af68 000000df2192fc50 97262 Free
000000e122582b58 00007fff29edac28 202080
000000e1225b40b8 000000df2192fc50 30 Free
000000e1225b40d8 00007fff3cbd27a8 87345
000000e1225c9610 000000df2192fc50 9878 Free
000000e1225cbca8 00007fff29edac28 202080
000000e1225fd208 000000df2192fc50 30 Free
000000e1225fd228 00007fff29edac28 97200
000000e122614dd8 000000df2192fc50 30 Free
000000e122614df8 00007fff29edac28 97200
000000e12262c9a8 000000df2192fc50 30 Free
000000e12262c9c8 00007fff3cbce100 131064
000000e12264c9c0 000000df2192fc50 203422 Free
000000e12267e460 00007fff3cbce100 32792
000000e122686478 000000df2192fc50 30 Free
000000e122686498 00007fff3cbce100 65304
000000e1226963b0 000000df2192fc50 30 Free
000000e1226963d0 00007fff3cbce100 130584
000000e1226b61e8 000000df2192fc50 30 Free
000000e1226b6208 00007fff3cbce100 131064
000000e1226d6200 000000df2192fc50 30 Free
000000e1226d6220 00007fff3cbce100 131064
000000e1226f6218 000000df2192fc50 341590 Free
000000e122749870 00007fff3cbce100 65560
000000e122759888 000000df2192fc50 2598 Free
000000e12275a2b0 00007fff3cbce100 131064
000000e12277a2a8 000000df2192fc50 2740110 Free
000000e122a17238 00007fff3cbce100 131064
000000e122a37230 000000df2192fc50 80294 Free
000000e122a4abd8 00007fff3cbcda88 98830
000000e122a62de8 000000df2192fc50 5359262 Free
000000e122f7f488 00007fff3cbcc448 202080
000000e122fb09e8 000000df2192fc50 2554686 Free
000000e123220528 00007fff3cbd27a8 168118
000000e1232495e0 000000df2192fc50 3812534 Free
000000e1235ec298 00007fff3cbcda88 325884
000000e12363bb98 000000df2192fc50 1722558 Free
000000e1237e0458 00007fff3cbcda88 287938
000000e123826920 000000df2192fc50 564910 Free
000000e1238b07d0 00007fff3cbcda88 1133348
000000e1239c52f8 000000df2192fc50 904798 Free
000000e123aa2158 00007fff3cbd0368 751028
000000e123b59710 000000df2192fc50 30 Free
000000e123b59730 00007fff3cbcc448 16147872
000000e124abfcd0 000000df2192fc50 30 Free
000000e124abfcf0 00007fff3cbcc448 16147872
000000e125a26290 000000df2192fc50 31726614 Free
000000e127867ea8 00007fff3cbcc448 33486336
000000e1298574a8 000000df2192fc50 22587478 Free
000000e12ade1d00 00007fff3cbcc448 7786800
000000e1324f1000 000000df2192fc50 24 Free
000000e1324f1018 000000df2192fc50 3039334 Free
000000e1327d7080 00007fff3cbcc448 97200
000000e1327eec30 000000df2192fc50 1292294 Free
000000e13292a438 00007fff3cbcc448 97200
000000e132941fe8 000000df2192fc50 387614 Free
000000e1329a0a08 00007fff3cbcda88 341524
000000e1329f4020 000000df2192fc50 2347590 Free
000000e132c31268 00007fff29edac28 97200
000000e132c48e18 000000df2192fc50 30 Free
000000e132c48e38 00007fff29edac28 97200
000000e132c609e8 000000df2192fc50 44391870 Free
000000e1356b67a8 00007fff3cbd0368 751028
000000e13576dd60 000000df2192fc50 30 Free
000000e13576dd80 00007ffedf487d78 6008056
000000e135d28a78 000000df2192fc50 30 Free
000000e135d28a98 00007fff3cbd0368 751028
000000e135de0050 000000df2192fc50 30 Free
000000e135de0070 00007ffedf488730 4506048
000000e13622c230 000000df2192fc50 30 Free
000000e13622c250 00007ffedf488110 4506048
000000e136678410 000000df2192fc50 2020446 Free
000000e136865870 00007ffee13063e8 202080
000000e136896dd0 000000df2192fc50 26599374 Free
000000e1381f4da0 00007fff3cbcc448 33486336
000000e13a1e43a0 000000df2192fc50 30 Free
000000e13a1e43c0 00007fff3cbcc448 33486336
000000e1a9f11000 000000df2192fc50 207613678 Free
000000e1b650fef0 00007fff3cbcc448 202080
Statistics:
MT Count TotalSize Class Name
00007ffedf296f98 1 97200 System.Collections.Generic.Dictionary`2+Entry[[System.Type, mscorlib],[StructureMap.Graph.PluginFamily, StructureMap]][]
00007ffee13063e8 1 202080 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[DynamicFieldInfo, Models]][]
00007fff3cbd27a8 2 255463 System.Byte[]
00007fff29edac28 6 792960 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.IO.Compression.ZipArchiveEntry, System.IO.Compression]][]
00007fff3cbce100 25 1351352 System.Object[]
00007fff3cbcda88 5 2187524 System.String
00007fff3cbd0368 3 2253084 System.Int32[]
00007ffedf488730 1 4506048 System.Collections.Generic.Dictionary`2+Entry[[System.Uri, System],[EPiServer.Web.PermanentLinkMap, EPiServer]][]
00007ffedf488110 1 4506048 System.Collections.Generic.Dictionary`2+Entry[[EPiServer.Core.ContentReference, EPiServer],[EPiServer.Web.PermanentContentLinkMap, EPiServer]][]
00007ffedf487d78 1 6008056 System.Collections.Generic.Dictionary`2+Entry[[System.Guid, mscorlib],[EPiServer.Web.PermanentLinkMap, EPiServer]][]
00007fff3cbcc448 10 141140112 System.Collections.Hashtable+bucket[]
000000df2192fc50 58 360401232 Free
Total 114 objects
So what do we expect to find here? The number of large objects, and the status of them.
You can of course use the -stat argument to have a summarized result:
0:000> !dumpheap -stat 0x000000e1224f1000
Statistics:
MT Count TotalSize Class Name
00007ffedf296f98 1 97200 System.Collections.Generic.Dictionary`2+Entry[[System.Type, mscorlib],[StructureMap.Graph.PluginFamily, StructureMap]][]
00007ffee13063e8 1 202080 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[DynamicFieldInfo, Web.Models]][]
00007fff3cbd27a8 2 255463 System.Byte[]
00007fff29edac28 6 792960 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.IO.Compression.ZipArchiveEntry, System.IO.Compression]][]
00007fff3cbce100 25 1351352 System.Object[]
00007fff3cbcda88 5 2187524 System.String
00007fff3cbd0368 3 2253084 System.Int32[]
00007ffedf488730 1 4506048 System.Collections.Generic.Dictionary`2+Entry[[System.Uri, System],[EPiServer.Web.PermanentLinkMap, EPiServer]][]
00007ffedf488110 1 4506048 System.Collections.Generic.Dictionary`2+Entry[[EPiServer.Core.ContentReference, EPiServer],[EPiServer.Web.PermanentContentLinkMap, EPiServer]][]
00007ffedf487d78 1 6008056 System.Collections.Generic.Dictionary`2+Entry[[System.Guid, mscorlib],[EPiServer.Web.PermanentLinkMap, EPiServer]][]
00007fff3cbcc448 10 141140112 System.Collections.Hashtable+bucket[]
000000df2192fc50 58 360401232 Free
Total 114 objects
So here we have 58 of “freed” objects, taking about 350MB of memory. This information is not exactly helpful on it own. However, if you continue to run the application and take another memory dump, and it has even more “freed” objects – then it might be a problem: Your site might be using LOH more than it should.
Thanks