<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maoni0 on Medium]]></title>
        <description><![CDATA[Stories by Maoni0 on Medium]]></description>
        <link>https://medium.com/@maoni0?source=rss-df86aab12c0b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*S3J1iFi4ZKNeoynv8cwwVw.jpeg</url>
            <title>Stories by Maoni0 on Medium</title>
            <link>https://medium.com/@maoni0?source=rss-df86aab12c0b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:22:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@maoni0/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Preparing for the .NET 10 GC]]></title>
            <link>https://maoni0.medium.com/preparing-for-the-net-10-gc-88718b261ef2?source=rss-df86aab12c0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/88718b261ef2</guid>
            <dc:creator><![CDATA[Maoni0]]></dc:creator>
            <pubDate>Wed, 24 Sep 2025 06:25:35 GMT</pubDate>
            <atom:updated>2025-09-24T06:25:35.232Z</atom:updated>
            <content:encoded><![CDATA[<h3>Preparing for the .NET 10 GC</h3><p>In .NET 9 we enabled DATAS by default. But .NET 9 is not an LTS release so for many people they will be getting DATAS for the first time when they upgrade to .NET 10. This was a tough decision because GC features are usually the kind that don’t require user intervention — but DATAS is a bit different. That’s why this post is titled “preparing for” instead of just “what’s new” 😊.</p><p>If you’re using Server GC, you might notice a performance profile that’s more noticeably different than what you saw in previous runtime upgrades. Memory usage may look drastically different (very likely smaller) — and that may or may not be desirable. It all depends on whether the tradeoff is noticeable, and if it is, whether it aligns with your optimization goals. I’d recommend taking at least a quick look at your application performance metrics to see if you are happy with the results of this change. Many people will absolutely welcome it — but if you are not one of them, don’t panic. I encourage you to read on to see whether it makes sense to simply turn DATAS off or if a bit of tuning could make it work in your favor.</p><p>I’ll talk about how we generally decide which performance features to add, why DATAS is so different from typical GC features, and the tuning changes introduced since my last DATAS blog post. I’ll also share two examples of how I tuned DATAS in first-party scenarios.</p><p>If you’re mainly here to see which scenarios DATAS isn’t designed for — to help decide whether to turn it off — feel free to skip ahead to <a href="https://maoni0.medium.com/88718b261ef2#cc2c">this section</a>.</p><h3><strong>General policies of adding GC performance features</strong></h3><p>Most GC performance features — whether it’s a new GC flavor, a new mechanism that enables the GC to do something it couldn’t before, or optimizations that improve an existing mechanism — are typically lit up automatically when you upgrade to a new runtime version. We don’t require users to take action because these features are designed to improve a wide range of scenarios. In fact, that’s often why we choose to implement them: we analyze many scenarios to understand the most common problems, figure out what it would take to solve them, and then prioritize which ones to design and implement.</p><p>Of course, with any performance changes, there’s always the risk of regressions — and for a framework used by millions, you’re guaranteed to regress someone. These regressions can be especially visible in microbenchmarks, where the behavior is so extreme that even small changes can cause wild swings in results.</p><p>A recent example being the change we made in how we handle the free regions for UOH (ie, LOH + POH) generations. We changed from the budget based trimming policy to an age based because it’s more robust in general (so we don’t either quickly decommit memory and have to recommit again, or keep extra free regions around even after a long time because we continue not consuming nearly all of the UOH budgets). But this can totally change a microbenchmark that used to observe the primary memory go down to a very low value after one <em>GC.Collect()</em> now requires 3 <em>GC.Collect()</em> calls (because we have to wait for the UOH free regions to age out in 2 gen2 GCs and the 3rd one will put it on the decommit list).</p><p>But for DATAS, we knew it was by definition not necessarily for a wide range of scenarios. As I mentioned in my last blog post, there were 2 specific kinds of scenarios that DATAS targeted. I’ll reiterate them here –</p><p>1. Bursty workloads running in memory constraint environments. DATAS aims to retract the heap size back when the application doesn’t require as much memory and grow it when the app requires more. This is especially important for apps running in containers with memory limits.</p><p>2. Small workloads using Server GC — for example, if someone wants to try out a small asp.net core app to see what the experience is like in .NET, DATAS aims provide a heap size much more inline with what the small app actually needs.</p><p>I should give more explanation about 1). It’s not uncommon to see bursty workloads at all. If you have an app that handles requests, which is completely common, naturally you could have many more users during a specific time of the day than the rest of the day. However, the key here is the action that follows it — if you have memory freed up during the non peak hours, what would you do with this memory? It turns out that sometimes folks don’t actually know — they want to see the memory go down when the workload lightens, but they have no plans to do anything with this memory. And for some teams, they don’t need to the memory usage to go down because they already budgeted all that memory to their apps. I was just talking to a customer recently and when I asked them “if DATAS frees up memory for you, what would you use for it?”. The answer was “that’s a good question, we never thought about it”.</p><p>For folks who do want to make use of the freed up memory, a common way is to use an orchestrated environment . DATAS makes this scenario more robust as heap sizes will be much more predictable, as I’ll explain below, which helps with setting sensible memory limits. For example, in k8s, you can determine appropriate request and limit values for both non-peak and peak workloads to better leverage HPA. I have also seen teams that schedule tasks to run when the machines/VMs have free memory — this is more involved (and these teams usually are equipped with a team of dedicated perf engineers) but gives them more control.</p><p>Then there are plenty of teams that have dedicated fleets of machines and want to maximize their throughput during peak hours as much as possible. They do not want to tolerate any type of slow down. They are definitely not the target of DATAS which will almost always regress their throughput — when it comes to perf it’s rarely an all or none situation and I will discuss below how to make a decision if you should turn DATAS off.</p><p>All these made it difficult to make DATAS the default because we know there are a lot of teams that don’t want to sacrifice throughput at all or don’t make use of freed up memory.</p><p>I will discuss in detail below if you do want to look at the perf differences and make a decision if DATAS is for you or not (maybe when you see the memory reduction you will have ideas of using the freed up memory).</p><h3><strong>Performance differences between DATAS and the traditional Server GC</strong></h3><p>DATAS is a GC feature that I spent more time explaining to my coworkers than any other — being such a user visible feature, it naturally attracted more questions than pretty much any other GC features I added. And there were <em>lots</em> of misconceptions. Some thought that DATAS only affected startup; some assumed it would just “reduce memory by x% and throughput by y%”; some expected it to “magically reducing memory without any other perf differences” (okay, I added the “magically” part 😆); and etc.</p><p>To understand the differences properly, we need to understand the difference in policies. First and foremost, Server GC does not adapt to the application size — it was never a goal. Server GC looks mostly at the survival rate of each generation and does GCs based on that (there are a number of other factors that affect when GCs are triggered but survival rate is one of the most significant). In the last DATAS post I talked about the number of heaps which can affect the heap size significantly, especially in workloads that allocate a lot of temporary data. Since Server GC creates the same number of heaps as the number of cores the process is allowed to use, it means you <em>can</em> see very different heap sizes when running the same app with a different number of cores (by running it on a machine with a different number of cores or let your process use different number of cores on the same machine).</p><p>DATAS, on the other hand, aims to adapt to the application size which means you should see similar heap sizes even when the number of cores varies a lot. So there’s no “DATAS will reduce memory by X%” compared to Server GC.</p><p>If we look at the “Max heap size” metric for asp.net benchmarks, it’s obvious that Server GC behaves very differently when running on a 28-core machine (28c) vs a 12-core machine (12c) –</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1GdAW1fUigGsSZSg17YYjA.jpeg" /></figure><p>Careful readers will notice that the order of which color is on top is not consistent. For example, for MultipleQueriesPlatform, the max heap size is actually much larger for 12c than 28c. Looking at the data in more detail reveals that the max heap size happens at the very beginning of the test for the 12c case –</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJi6aC8z-zU6-d9MdlXfBA.jpeg" /></figure><p>(Heap size (before) is right before a GC before that GC could possibly shrink the heap size. So “Max Heap Size” would be the max of this metric)</p><p>This is because at the beginning, there were a lot more allocations happened before the first GC happened on 28c with 28 heaps. So after that GC, a smaller survival rate was observed which caused the gen0 budget to be much smaller than on 12c. 12c quickly dropped to the steady state which has a much lower heap size than 28c. For steady state, these benchmarks always exhibit a much higher heap size on 28c. This illustrates 2 points –if you just measure “max heap size”, it can easily be affected by the non-steady state behavior; secondly, the heap size can vary a lot due to the machine the test runs on. Note that these effects can be magnified because we are looking at small benchmarks, but the reasoning applies to real-world apps.</p><p>With DATAS we see this picture -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNHA6YwZLgDXyZJMJ4Q0UA.jpeg" /></figure><p>The max heap sizes are very similar on 28c and 12c which is exactly what DATAS is for — it adapts to the application size.</p><p><strong>Do I need to care if I’m using Workstation GC?</strong></p><p>The answer depends on why you are using Workstation GC. If you are using Workstation GC because your workload simply does not call for using Server GC at all, then there’s no need to change. This could be due to your app being single threaded or the allocation is simply not stressful and you are totally fine with having one thread doing the collection work, in which case Workstation GC not only suffices but is exactly the correct choice to make.</p><p>But if you are using it because Server GC’s memory usage was too large and you are just using Workstation to limit the memory usage, you could find DATAS very attractive because it can both limit the memory usage and make the GC pauses lower with more GC threads doing the collection work.</p><h3><strong>How DATAS does its job</strong></h3><p>If you understood how DATAS does it job, it would be natural to arrive at the recommendations below for deciding if DATAS is for you. You could also skip this section, but I always like to understand how something works if I care about it, so I can come to my own conclusions instead of just memorizing some rules. In the last blog post I mentioned some details of DATAS at the time (.NET 8), noting that it would likely change dramatically — and it did, both in design and implementation. The implementation we had in .NET 8 was mostly for functional — we spent very little time in tuning. The majority of the tuning work happened after .NET 8.</p><p>The goal of DATAS is to adapt to the application size, or the LDS (Live Data Size). So there needs to be some way to adapt to it. Because the .NET GC is generational, it means we don’t collect the whole heap often. And since most full GCs we do are background GCs which don’t compact, it’s reasonable to approximate the LDS with the space objects take up in the old generations, i.e., (total size — fragmentation). Another convenient number to use when you do your perf investigations is to look at the promoted size when a full GC is done.</p><p>In the last blog post I mentioned the <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#conserve-memory">conserve memory </a>config is part of the DATAS implementation — that part did not change. But <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#conserve-memory">conserve memory</a> only affects when full GCs are triggered. For apps that allocate very frequently, unless these are temporary UOH objects, most of the GCs are ephemeral GCs. And ephemeral generation sizes can be a significant portion of the whole heap especially for small heaps.</p><p>After experimenting with various approaches, I settled on the approach of “adapting to the app size while maintaining reasonable performance” which consisted of 2 key components -</p><p>1) introduced a concept of “Budget Computed via DATAS (BCD)” which is calculated based on the application size and gives us an upper bound of the gen0 budget for that size, which can approximate the generation size for gen0 (since there’s pinning it may not be exactly the generation size for gen0).</p><p>2) within this upper bound, we can further reduce memory if we can still maintain reasonable performance. And we define this “reasonable performance” with a target Throughput Cost Percentage (TCP). This takes into consideration both GC pauses and how much allocating threads have to wait. But you can approximate TCP with % pause time in GC in steady state. The idea is to keep TCP around this target if we can, which means if the workload gets lighter, we’d be adjusting the gen0 budget smaller. And that in turn means gen0 will be smaller before the next GC, which translates to smaller heap size. The default target TCP is 2%. This can be changed via the <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#target-tcp">GCDTargetTCP</a> config.</p><p>Let’s look at 2 example scenarios to see how this manifests. For simplicity, I’m ignoring background GCs, and I’ll use % pause time in GC to approximate TCP.</p><p><strong>Scenario A</strong> — I have an e-commerce app which stores the whole catalog in memory, and this remains the same during the process lifetime. This is our LDS. Now the process starts to process requests and for each request there’s memory allocated and only used for the duration of that request.</p><p>During peak hours, it processes many concurrent requests. We hit our max budget which is our BCD. Let’s say this is 1gb, it means we are doing a GC each time 1GB is allocated. If we use the % pause time in GC to approximate TCP, let’s say during each second it allocates 1GB and observes one GC that has a 20ms pause. So the % time in GC is 2%. And that’s the same as our target TCP.</p><p>When it’s outside the peak hours and handling way fewer concurrent requests, let’s say we allocate ~200MB per second. If we keep our 1GB budget, it means we are doing a GC every 5s. And our % time in GC would be (20ms / 5s = 0.4%), much lower than 2%. So to reach the target TCP we’d want to reduce the budget and trigger a GC much sooner. If we reduce the budget to 200MB, and we’ll still use 20ms as our GC pause just to make it simple (it’ll likely be shorter as it’s roughly proportional to the survival and there’s likely less survival out of 200MB vs 1GB), now we are achieving 2% TCP again.</p><p>So for this scenario, the heap size is reduced by ~800MB when it’s outside peak hours. Depending on your total heap size, this can be a very significant reduction.</p><p><strong>Scenario B </strong>is built on top of A but we’ll throw in a cache that’s part of the LDS but gets smaller during lighter workload as we don’t need to cache as much. Because the LDS is smaller it means your BCD will be smaller as it’s a function of LDS. So during the lighter workload, the gen0 budget will be further reduced which again reflects the adapting to size nature. The <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#conserve-memory">conserve memory</a> mechanism is still in effect too and would adjust the old generation budget and size accordingly.</p><p>Notice that so far I have not talked about the number of heaps at all! This is completely taken care of by DATAS itself so you don’t need to worry about it. Previously, some of our customers were using the <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#heap-count"><em>GCHeapCount</em></a> config to specify the number of heaps for Server GC. But DATAS makes it more robust as it can take advantage of more heaps if needed (which usually means shorter individual pause times) and reduces the heap size when the LDS goes down, without your having to specify a heap count yourself.</p><p>DATAS has <a href="https://maoni0.medium.com/88718b261ef2#e19d">specific events</a> that indicate the actual TCP and LDS but that requires you to programmatically get them via the <a href="https://github.com/microsoft/perfview/blob/main/documentation/TraceEvent/TraceEventLibrary.md">TraceEvent</a> library. The approximations I mentioned above are sufficient for almost all perf investigations.</p><h3><strong>When DATAS might not be applicable to your scenario</strong></h3><p>If you read the previous sections, what’s listed below hopefully makes sense.</p><p><em>1) If you have no use for free memory, you don’t need DATAS</em></p><p>This one should be obvious — why change it at all if you don’t have any use for the memory that gets freed up by DATAS anyway? You can turn DATAS off by the <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#enabling-or-disabling-datas">GCDynamicAdaptationMode</a> config.</p><p>I’ve come across a few first party teams who simply didn’t need DATAS — they have dedicated machines to run their processes and have no use for free memory as they don’t plan to run anything else on the machine. So they have no use for DATAS. One team did say “now we probably want to think about taking advantage of free memory” (they were not thinking about it because Server GC isn’t aggressive at reducing memory usage). So for them, they will disable DATAS for now but will enable it when they can take advantage of memory during non peak hours.</p><p><em>2) If startup perf is critical, DATAS is not for you</em></p><p>DATAS always starts with 1 heap. We cannot predict how stressful your workload will be and since we are optimizing for size here, it starts with the smallest heap count which is 1. So if your startup perf is critical, you will see a regression because it takes time to go from 1 heap to multiple.</p><p><em>3) If you do not tolerate any throughput regression, DATAS may not be for you</em></p><p>If this includes the startup throughput, as 2) also states, DATAS is not for you. However, some scenarios aren’t concerned with startup perf so DATAS may or may not be desirable. Let’s say your % pause time in GC is 1% with Server GC, you can just set the <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#target-tcp">GCDTargetTCP</a> config to 1. If you were restricting the heap count you could very possibly see a perf improvement because the pause time can be shorter with DATAS. If the adaptation to the size aspect is beneficial to you, using DATAS can be a much better choice. But as stated in 1) if you don’t have any use for the freed up memory anyway, it wouldn’t justify spending time on using DATAS.</p><p>4) If you are doing mostly gen2 GCs, DATAS <em>may</em> not be for you</p><p>One case I haven’t spent much time tuning is when your scenario mostly does gen2 GCs (this is almost always due to excessive allocation of temporary large objects). If this is the case for you, and if you’ve tried DATAS and weren’t happy with the results, I would suggest to disable DATAS. You could investigate to see if you can make it work by following the tuning section if it’s justified to spend the time.</p><h3><strong>Tuning DATAS if necessary</strong></h3><p>I’ve tried DATAS on some first party workloads and in general it worked out great. I’ll show a couple of examples where the default parameters of DATAS weren’t great but tuning one or 2 configs made it work.</p><p><strong>Customer case 1</strong></p><p>This is a server app running on dedicated machines. But they are in the process of containerizing it so there’s definitely merit to use DATAS. With DATAS they observed a 6.8% regression in throughput with a 10% reduction in working set. For now they’ve disabled DATAS — I will explain how I debugged it and determined what DATAS config to use to make it work if/when they want to enable DATAS.</p><p>Because DATAS limits the largest gen0 budget based on the LDS, we want to see if we are hitting that limit. It’d be easiest if you <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#How-to-collect-top-level-GC-metrics">captured a GC trace</a> with DATAS and one without DATAS. If you are seeing more GCs triggered, that means most likely you are hitting that limit.</p><p>You can approximate the TCP with what’s shown in the “% Pause Time” column, and gen0 budget with the “Gen0 Alloc MB” column. And you’d want to find the phase when you have the highest % pause time and see if you are triggering more GCs.</p><p>So for this particular customer, here are some excerpts of the GC (I’ve trimmed down the columns of the <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#displaying-top-level-gc-metrics">GCStats </a>view) –</p><p><em>Without DATAS</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44a909c0c13de217b7ae9df4c1e23fb2/href">https://medium.com/media/44a909c0c13de217b7ae9df4c1e23fb2/href</a></iframe><p>With DATAS</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff0973ac680e0e33da3c58c2dea1f372/href">https://medium.com/media/ff0973ac680e0e33da3c58c2dea1f372/href</a></iframe><p>Comparing their gen0 budget and % pause time in GC -</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/519a30cdda4e55ed14c3bc53128ca784/href">https://medium.com/media/519a30cdda4e55ed14c3bc53128ca784/href</a></iframe><p>So gen0 budget without DATAS is 2.6x with DATAS. Another useful thing we notice is the % Pause Time is basically exactly the target TCP — 2%. That tells us that this is working exactly as by design from DATAS’s POV. But without DATAS we got 2.6x budget so naturally we triggered GC less frequently and % pause time is 1.2 instead of 2.1.</p><p>But if we want to enable DATAS and not regress throughput for this phase, we’d like to have DATAS use a larger gen0 budget. To do that we should understand how DATAS determines the BCD. Since we are adapting to the size, we want to multiply the size with something. But this should not be a constant value because when the size is very small, this multiplier should be quite large — if the LDS is only 2MB (which is totally possible for a tiny app), we wouldn’t want to trigger a GC for every 0.2MB of allocation — the overhead would be too high. Let’s say we want to allow 20MB of allocation before triggering a GC, that makes the multiplier 10. But if the LDS is 20GB, we wouldn’t want to allocate 200GB before doing a GC, which means we want a much smaller multiplier. This means a power function but we also want to clamp it between a min and max value -</p><pre>m = constant / sqrt (LDS);<br>// default for max_m is 10<br>m = min (max_m, m);<br>// default for min_m is 0.1<br>m = max (min_m, m);</pre><p>The actual formula for the power function is</p><pre>m = (20 - conserve_memory) / sqrt (LDS / 1000 / 1000);</pre><p>which can be simplified to</p><pre>m = (20 - conserve_memory) * 1000 / sqrt (LDS);<br>m = (20 - 5) * 1000 / sqrt (LDS);<br>m = 15000 / sqrt (LDS);</pre><p>So the constant is 15000, or we could just say it’s 15 if we use MB for size. here’re some example with different LDS -</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ef6457c8d9fa8e4c650526b130a1396/href">https://medium.com/media/1ef6457c8d9fa8e4c650526b130a1396/href</a></iframe><p>This constant, max_m and min_m can all be adjusted by configs. Please see <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#gen0-max-budget-settings">the config page</a> for detailed explanation.</p><p>Now it’s quite obvious why DATAS came up with the gen0 budget and how we can adjust it. If we want to bring this up to the same budget without DATAS, we’d want to use the <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#gen0-max-budget-settings">GCDGen0GrowthPercent</a> config to increase the constant to 2.6x, and increase min_m with the <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#gen0-max-budget-settings">GCDGen0GrowthMinFactor</a> config so it’s not clamped to 0.1 — you don’t need to be very accurate since you just need to make it not be the limiting factor. So in this case if we use 15GB to approximate the LDS (the “Promoted (mb)” column for both gen2 GCs says ~15GB), and without DATAS the gen0 budget is 4.22GB. So min_m should be around (4.22/15 = 0.28). We can just set min_m to 300 which translates to 0.3 of LDS.</p><p><strong>Customer case 2</strong></p><p>This is an asp.net app on a staging server from the customer that represents one of their key scenarios. I used a load test tool to generate variable workloads.</p><p>The team was already using some GC configs -</p><p>· <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#heap-count">GCHeapCount</a> is set to 2 to use 2 heaps</p><p>· Affinity is turned off with the <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#affinitize">GCNoAffinitize</a> config.</p><p>If the GCHeapCount config is specified, DATAS would be disabled because it’s telling the GC to not change the heap count. And since changing the heap count is one of the key mechanisms to adjust perf for DATAS, it’s an indication to disable DATAS.</p><p>Because this is a process that co-exist with many others on the same machine, before DATAS was available they chose to give it 2 heaps to limit the memory usage while still getting reasonable throughput. But this is not flexible — when the load becomes higher the throughput can suffer with 2 heaps and also the GC pauses can be noticeably higher since there’s only 2 GC threads collecting. They can adjust the number of GC heaps but that means more work and since Server GC isn’t very aggressive at reducing memory usage they can end up with a much bigger heap than desired when the load is lighter.</p><p>I’ll demonstrate how using DATAS makes this robust. When I made the load pretty high I could see that % pause time in GC is quite high — not surprising with just 2 heaps. So I enabled DATAS by simply getting rid of the GCHeapCount config (I kept the GCNoAffinitize config as I still wanted the GC threads to not be affinitized). I could see the % pause time in GC was also high because even with BCD we still ended up triggering GCs quite often. So I decided to make BCD 2x the default value with the <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#gen0-max-budget-settings">GCDGen0GrowthPercent</a> config (I didn’t need to use the <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#gen0-max-budget-settings">GCDGen0GrowthMinFactor</a> config since 2x is still well within our max_m/min_m clamping values). And now the process behaves in a much more desirable way with the following characteristics –</p><ul><li>the % pause time is dramatically lower. With the default DATAS the % pause time is basically comparable and the heap size is noticeably lower. Depending on your optimization goal this could be exactly what you want. DATAS is able to achieve this with smaller budgets and more GC threads doing the collection work. But I know for this customer, they don’t want % pause time in GC to be this high as it affects their throughput. I could also make DATAS <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#target-tcp">use a smaller target TCP </a>but in this case the default TCP seems quite sufficient.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h48FM2kBsWB3XV0htWgRBQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tsCil0FQnJNukUvTeVOLpw.png" /></figure><ul><li>individual GC pauses are a lot lower since we have a lot more GC threads collecting.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XoACTDEUJjS4TiTN5GQ-Cg.png" /></figure><ul><li>when the load becomes lighter (# of concurrent client threads went from 200 to 100), the heap also becomes smaller. And we are still maintaining a much lower % pause time in GC and individual GC pauses.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U1smce_mcFtZJB7FBH4pnw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rnFDI3bAOQpd0XS3lEhoDA.png" /></figure><p>I hope this helps with your DATAS tuning, if you need to do any.</p><h3><strong>DATAS Events</strong></h3><p>I expect most users never needing to look at these events, so I’ll keep it brief. The approximations that I mentioned above should suffice. For the small number of folks who want to do a detailed analysis for whatever reason, DATAS fires an event that accurately represents the metrics we discussed. Note that we only use these events programmatically, so they are not surfaced in PerfView’s Events view (all you’ll see is the GC/DynamicTraceEvent which shows you the name but not individual fields of that event). See <a href="https://devblogs.microsoft.com/dotnet/556-2/">this blog article</a> for an example how to programmatically retrieve GC info as a list of <a href="https://github.com/microsoft/perfview/blob/0691921c6cc22eacaff2b41efa5b4d38ef1ffd20/src/TraceEvent/Computers/TraceManagedProcess.cs#L1856"><em>TraceGC</em></a> objects from a trace.</p><p>LDS and TCP are indicated in the SizeAdaptationTuning event, assuming you have a <em>gc</em> object of the type <a href="https://github.com/microsoft/perfview/blob/0691921c6cc22eacaff2b41efa5b4d38ef1ffd20/src/TraceEvent/Computers/TraceManagedProcess.cs#L1856"><em>TraceGC</em></a><em> —</em></p><pre>// LDS<br>gc.DynamicEvents().SizeAdaptationTuning?.TotalSOHStableSize<br>// TCP<br>gc.DynamicEvents().SizeAdaptationTuning?.TcpToConsider</pre><p>This event is not fired every GC since we only check to see if we need to change the tuning for DATAS every few GCs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88718b261ef2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamically Adapting To Application Sizes]]></title>
            <link>https://maoni0.medium.com/dynamically-adapting-to-application-sizes-2d72fcb6f1ea?source=rss-df86aab12c0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d72fcb6f1ea</guid>
            <dc:creator><![CDATA[Maoni0]]></dc:creator>
            <pubDate>Thu, 24 Aug 2023 18:17:00 GMT</pubDate>
            <atom:updated>2023-08-25T00:26:32.177Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog post I’m going to talk about our .NET 8.0 GC feature called DATAS (<strong>D</strong>ynamic <strong>A</strong>daptation <strong>T</strong>o <strong>A</strong>pplication Sizes). To talk about this work I think it’s necessary to first give some background on what kind of scenarios this feature is aiming to make better. This section is quite long because I will discuss some existing solutions for this problem space and why they are not sufficient for the scenarios DATAS is targeting. Feel free to <a href="https://medium.com/@maoni0/dynamically-adapting-to-application-sizes-2d72fcb6f1ea#2871">skip it</a> if you want to dive into the feature right away.</p><p>II’m adopting the foot note style to explain things because I find it helpful for providing more info while keeping the main portion of the post more focused on the topic at hand.</p><h3><strong>Background and problem space</strong></h3><p>In the past couple of .NET releases, we really focused on making the GC work more efficient and the GC pauses shorter<a href="https://medium.com/@maoni0/dynamically-adapting-to-application-sizes-2d72fcb6f1ea#3143">¹</a>. Most of those optimizations we did were targeting larger workloads<a href="https://medium.com/@maoni0/dynamically-adapting-to-application-sizes-2d72fcb6f1ea#1fc2">²</a>. These workloads usually use Server GC. Server GC was designed with maximizing throughput in mind and has a reputation that it uses a lot of memory. In fact I’ve heard some folks say “Server GC tries to use as much memory as possible” which is well, not the case😊. Server GC doesn’t <em>try</em> to use a lot of memory — it just by default has the potential to use more memory when <em>there is a lot of memory available</em>. Let me explain why.</p><p>One of the most important factors that contribute to more memory usage is the number of heaps. Having more heaps doesn’t automatically mean you’ll use more memory — you could just have less memory per heap. And this is the case for long lived data — there’s only so much long-lived data so if you have more heaps each heap will have less long live data. But if you look at it from the user allocation point of view, it means you now have multiple heaps to allocate on instead of one with Workstation GC. Or at least that portion could be N times more if you have N heaps<a href="https://medium.com/@maoni0/dynamically-adapting-to-application-sizes-2d72fcb6f1ea#877b">³</a>. And by default, Server GC will have the same number of heaps as the number of logical processors on the machine<a href="https://medium.com/@maoni0/dynamically-adapting-to-application-sizes-2d72fcb6f1ea#d1b8">⁴</a>. This means the problem is not just that your heap may grow to a bigger size. It’s that it’s unpredictable — if you run your workload on a machine with more processors you might see a bigger size.</p><p>Another important but more subtle factor is <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#Concurrent-GCBackground-GC">Background GC</a>. In general, for gen2 GCs we choose to do Background GCs instead of compacting (which incur long GC pauses if the heap is large). What this means is during peak hours, your GC heap could grow to a large size simply because of the demand. But after that if we are not under memory pressure, we will most likely not do a full compacting GC so the heap size will stay that large. So next time you hit peak hours the heap will be fully utilized again. Note that this means if you happen to look at the heap size not during peak hours you will see a lot of <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#Fragmentation-free-objects-is-part-of-the-heap-size">fragmentation</a> on the heap (it’s BGC’s job to build up fragmentation so it can use that space to allocate objects into).</p><p>This situation is fine if you aren’t using the machine to do other things during non-peak hours. But if you are running in a memory constraint environment like a container with a memory limit specified, often folks’ intention is to pack as many containers as possible on a VM so they don’t want to leave memory sitting around unused. In other words, when the application itself isn’t needing so much memory, it’s desirable to shrink the GC heap size accordingly.</p><p><strong>Ways to control the GC heap size before .NET 8</strong></p><ol><li><em>We could change the </em><a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#the-allocation-budget"><em>allocation budget</em></a><em>.</em> For example, for the generations user code actually allocates in which means gen0, LOH and <a href="https://github.com/dotnet/runtime/blob/main/docs/design/features/PinnedHeap.md">POH</a>, if we reduce their budgets, the GC heap isn’t allowed to grow as big before the next GC is triggered, since the user code will not be able to allocate as much in-between GCs.</li><li><em>We could decide to do a full compacting GC instead of a BGC.</em> It’s very common to do a compacting GC for ephemeral collections. But in large workloads, most of the memory does not live in ephemeral generations. In order to reduce the gen2 size, we could choose to do a BGC sooner or, if gen2 has already grown very large with a lot of fragmentation (ie, the peak vs non peak situation I described above) we could do a full compacting GC. Note that when the memory load gets high, ie, there’s not a lot of available memory, a full compacting GC will <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#gc-is-per-process-but-is-aware-of-physical-memory-load-on-the-machine">automatically happen</a>.</li><li><em>A config called </em><a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#heap-count"><em>GCHeapCount</em></a><em> is provided to change the number of heaps</em>. This is a technique that many folks have used. However, it’s not very flexible for the following reasons –</li></ol><ul><li>The heap count cannot change after the runtime is initialized, which means you may need to specify a different value for this config if you are running in a different environment. A different environment could mean you need the process to handle more concurrent requests because it’s during peak hours, or on a different machine with more memory so you can afford to give this process more memory.</li><li>Instead of changing the heap count, we could distribute the total allocation budget over the number of heaps. But this doesn’t scale. We wouldn’t want a heap count that’s very disproportional⁵. In other words, having a reasonable heap count is important to keep things running smoothly for the GC.</li></ul><p><strong>Surprising results for small apps with Server GC</strong></p><p>It’s usually the right choice to turn Server GC on for a real web application instead of a simple test app. But it does not help if folks who are just “trying it out” don’t get a good experience because they see a big heap when the app doesn’t do much. We really want to improve this experience.</p><h3><strong>DATAS</strong></h3><p>The .NET 8 GC introduces an experimental feature called DATAS which stands for <strong>D</strong>ynamic <strong>A</strong>daptation <strong>T</strong>o <strong>A</strong>pplication <strong>S</strong>izes. What does “application size” mean exactly? This is the LDS (Live Data Size) from GC’s point of view, meaning that if we did the most aggressive GC possible, this is how much memory your application uses. Another way to look at it is this is your long lived data + whatever inflight data you have when a GC occurs.</p><p>The goal for DATAS is that you no longer need to do various configurations to try to achieve a heap size proportional to your application usage. The 2 main cases we target with DATAS are -</p><p>1) Bursty workloads running in memory constraint environments. DATAS aims to retract the heap size back when the application doesn’t require as much memory and grow it when the app requires more. This is especially important for apps running in containers with memory limits.</p><p>2) Small workloads using Server GC — for example, if someone wants to try out a small asp.net core app to see what the experience is like in .NET, DATAS aims provide a heap size much more inline with what the small app actually needs.</p><p>When we started this feature, within teams in the .NET org some folks were referring to it as “the Dynamic GC”. I did have to clarify there that GC has been dynamic in many ways over the history of .NET. What’s key in DATAS is the “adapting to application size” part. Of course GC has always been dynamic in some ways — it’s unrealistic to have a GC that serves the amount of workloads running .NET without it being dynamically tuned<a href="https://medium.com/@maoni0/dynamically-adapting-to-application-sizes-2d72fcb6f1ea#caf6">⁶</a>.</p><p><strong>How does the GC achieve that?</strong></p><p>At a high level, the current DATAS design does the following to adapt -</p><ol><li>adjust the number of heaps appropriately to allow things to run fairly smoothly and</li><li>adjust allocation budgets and policy for doing full compacting GCs to adapt to application sizes</li></ol><p>The kinds of tuning we can do are limited by the mechanisms available to us. The mechanism in 1) which is being able to adjust the number of heaps was developed as part of our 8.0 work (it’s still being worked on — I will talk about this in a future post). As part of the GC development work, we are always adding new mechanisms so in the future we’ll have even more flexibility to adjust things in the GC.</p><p>Let’s take a look at the policies we use to do our adjustments as they are the major factor that determines the outcome. In the next couple of blog posts I will provide details on some of the mechanisms themselves as they are quite intricate and I know many of my readers really enjoy reading about internals of the GC.</p><p>Because this is currently heavily under development, some of what I describe will very likely change. I was thinking to wait till this is more baked but lots of people have expressed interest in this work and I also thought there’s merit to talk about things in early stages so we can see how they evolve.</p><p><strong>Policies to adjust the number of heaps</strong></p><p>When DATAS is on, we don’t want users to have to specify any configs to try to influence the heap size — it should happen automatically. Since we cannot predict how much memory a workload will need, DATAS will always start with one heap and adjust up or down as needed. “As needed” is determined by the combination of a <em>throughput cost</em> percentage and a <em>space cost</em> percentage that we’ll discuss in detail below.</p><p>We continuously measure these cost percentages and once every 3 GCs we pick a median value and use that to do our calculations.</p><p><strong><em>Throughput cost</em></strong></p><p>From the throughput perspective, the cost is how long you have to wait to do an allocation. For SOH allocations, they are usually very fast because most of the time they don’t even need to come to the GC (each time GC gives out memory, it’s enough for many small allocations). For UOH (meaning LOH and POH) allocations they will need to come to the GC for every allocation. When an allocating thread does need to come to the GC, it needs to take a per heap SOH or UOH lock. If this lock is contended, it means some allocating threads will need to wait. So we record the amount of time threads have to wait when the lock is contended. When we have enough heaps, this lock is contended very infrequently which means the wait time from all allocating threads should be insignificant.</p><p>When a GC is triggered, we record the duration of the GC.</p><p>The throughput cost percentage is then</p><pre>throughput_cost = (total_SOH_wait time + total_UOH_wait time) / num_of_heaps + <br>                   GC duration<br>throughput_cost_percentage = throughput_cost / elapsed_time_since_the_last_GC</pre><p>For ephemeral GCs this formula is quite reasonable — they are usually short and happen frequently. And during ephemeral GCs managed threads are paused. For full GCs this is very questionable — if we do a full blocking GC, it’s very likely the cost percentage will be very high which means it’ll never get picked because it’ll never be the median value. And of course for BGCs the GC duration is very different from pause duration. I have some changes that treat full GCs separately from ephemeral GCs. The idea is still the same but we take the elapsed time between full GCs.</p><p>The throughput cost percentage is a good indication to adjust the heap count up. Since we start from 1 heap, if the app starts to spin up lots of threads allocating, it’s definitely good to have more GC heaps so the contention goes down. On the other hand, if the app doesn’t do much, let’s say it’s a single threaded app, which means there will be no contention at all on the per heap lock. Unless the GC cost is high it will just stay at 1 heap.</p><p><strong><em>Space cost</em></strong></p><p>The space cost percentage is to determine if we are making the right tradeoff between how much GC runs and how much memory it uses. In general, the more memory GC can use, the less frequent it has to run. We calculate the per heap space cost percentage as</p><pre>size_of_an_extra_heap / total_heap_size</pre><p>How to define “size of an extra heap” requires some thoughts. An extra heap could be “what the user code allocates on a heap” (we will move some of the already survived memory from other heaps to this extra heap so that part already exists). Currently this is temporarily defined as the minimum gen0 budget. For one thing, if the heap size gets big enough, this will always be very tiny so it’s not useful. So this will certainly change.</p><p><strong><em>Combining them</em></strong></p><p>The way we look at the tradeoffs is –</p><p>· Would it be worth it to adjust up? Ie, would we save more in throughput cost than the extra space cost incurred by a significant amount? If so we adjust up.</p><p>· Otherwise, would it be worth to adjust down? Ie, would we save more in space cost than the extra throughput cost by a significant amount? If so we adjust down.</p><p>“A significant amount” is currently set as 1%. I also expect this to change to take the total heap size and other factors into consideration.</p><p><strong><em>How much to adjust?</em></strong></p><p>The current policy for adjustment is the following -</p><p>· If the throughput cost is really high (&gt; 10%), we adjust up aggressively — our goal is to see if we can bring it down to 5%.</p><p>· If the throughput cost is fairly high (between 5% and 10%), we adjust by steps. So far we’ve used a simple policy for calculating the step for adjusting up and down based on the current number of heaps.</p><p>· Otherwise we look at our tradeoff to see if we should adjust up or down.</p><p>We apply exponential smoothing to make it more stable so we do fewer heap count changes. However, there are quite a few problems remaining to be solved.</p><p>· We can still be in a situation that one run’s max heap count is noticeably higher than another run’s because currently we just use the current number of heaps to calculate how much to adjust up/down. We’d definitely want to take the amount of work GC has to do (ie, the survived bytes) into consideration — if we have a small survivorship, that should contribute to calculating how much we should adjust.</p><p>· Right now we are not adjusting down aggressively enough. It’s true that we don’t want to change the heap count too often because it obviously takes work (I’ll talk about this in more detail in a future post). When the heap size is small, the amount of this work is small as well so we should be more free to change the heap count.</p><p>I also expect to make this more configurable. In general 5% is fine but some folks may want that number to be different. Since this is something easily measurable by users, we can take this as an input for our adjustment.</p><p><strong>Policies to adapt to application sizes</strong></p><p>Most of the policies are around adjusting the allocation budgets. In .NET 6 (and .NET Framework 4.8) we introduced a <a href="https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#conserve-memory">ConserveMemory config</a> that I know many teams have made use of. Since I haven’t talked about it in my blog I’ll take the opportunity to describe it as we are definitely taking advantage of it for DATAS.</p><p>The idea behind this config is exactly about how much we allow the heap size to grow to relative to the LDS (at the time we implemented this config, it was only for gen2 as for a sizeable heap that’s where most of the data is). When we do a full GC, we know how much is survived in gen2 which is close to the LDS assuming most of your long term data is in gen2 (gen2 here refers to the logical generation 2 which includes UOH). The calculation below equates LDS to the gen2 survived bytes. Let’s say the gen2 allocation budget was X. If the gen2 budget is X, it means when a gen2 GC is triggered, the gen2 size is (X + LDS). The config value is interpreted as “how conservative do you want the GC to be at keeping the heap size down” so the larger it is, the smaller the heap is. Let’s say you specify 7, which is interpreted as 70% and it means we allow only 30% on the heap that’s not live data. So</p><pre>(10 — C) / 10 = (10–7) / 10 = 30%</pre><p>And this 30% is (budget / (budget + LDS)). So</p><pre>(10 — C) / 10 = (budget / (budget + LDS))<br>=&gt;<br>budget = (10 / C - 1) * LDS</pre><p>Then we take half of that just to be conservative (otherwise we might very well grow gen2 size which we try to avoid) and use that to set the gen2 budget.</p><p>Because BGC does not compact, if we are observing the transition from peak hours to non peak hours, our policy is to do a full compacting GC to reduce the heap size if we detect we are over the desired heap size.</p><p>DATAS effectively sets this config to 5 unless it’s set by the user. In DATAS we also modify the gen0 budget because for smaller workloads that can be a significant percentage of the total heap. But we do this differently because gen0 mostly holds temporary data so we calculate the budget the same way as we used to which is based on the gen0 survival rate. However, for DATAS we want the LDS to influence the gen0 budget when it’s small so gen0 doesn’t make the heap grow too large. As always we also want to make sure it doesn’t become too small so we set 2.5MB as the minimum gen0 budget.</p><p><strong>Example Results</strong></p><p>One of the test suites we’ve been using to do our validation for the feature is benchmarks<a href="https://medium.com/@maoni0/dynamically-adapting-to-application-sizes-2d72fcb6f1ea#6381">⁷</a> from the asp.net team. The server is running on <a href="https://github.com/aspnet/Benchmarks/tree/main/scenarios#profiles">“aspnet-citrine-win” </a>which is an Intel machine with 28 logical cores on 1 socket and 32GB DRAM.</p><p>On average, max working set has been reduced by 73% and throughput (RPS) has been regressed by 2.3%.</p><p>Individual benchmark results are in the charts below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_mR40JM7sHel0VJ1v8EfHg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PFsccx0MZE18WqkD7ioqkw.jpeg" /></figure><p>Note that some of these tests are very noisy and the 2 that regressed &gt; 10% in throughput, JsonPlatformInline and UpdatesPlatform, are among the noisy ones. Going forward, we’ll report a volatility score for each benchmark. We have yet to do in-depth investigation on these benchmarks. Based on a cursory look, the % time in GC went from 0.0%/0.4% with Server GC to 0.2%/2.4% with DATAS for these 2 benchmarks respectively. So it doesn’t seem like that’s the cause but we will obviously need to investigate to understand why. In general, we certainly do not expect to regress the throughput this much.</p><p>Since the current way of running these tests do not have a variable load, I’m including results from running the JsonMVC benchmark with a variable load to show how the heap size adapts when we change from a heavier load to a lighter load.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vKWoPBty0IDGrW6qZdbMdQ.png" /></figure><p>If we look at the heap count, we can see how it’s adjusted. The max it got adjusted to is 17.</p><p>(note this chart’s x-axis is different from the last one — it’s the GC indices instead of time)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dzr5UUECj5edBqW1pga1Lg.png" /></figure><p><strong>How to use the feature</strong></p><p>If you’d like to try it out, DATAS can be turned on via one of the following ways in .NET 8 (note that you always need to turn on Server GC). Note that if you set it to use Server GC but also config the heap count yourself, it will simply use Server GC with that many heaps and ignore that the adaptation mode is set.</p><p>1) via runtimeconfig.json –</p><pre>“configProperties”: {<br>  &quot;System.GC.Server&quot;: true,<br>  &quot;System.GC.DynamicAdaptationMode&quot;: 1<br>}</pre><p>2) via an msbuild property –</p><pre>&lt;PropertyGroup&gt;<br>  &lt;ServerGarbageCollection&gt;true&lt;/ServerGarbageCollection&gt;<br>  &lt;GarbageCollectionAdaptationMode&gt;1&lt;/GarbageCollectionAdaptationMode&gt;<br>&lt;/PropertyGroup&gt;</pre><p>3) via an env var –</p><pre>set DOTNET_gcServer=1<br>set DOTNET_GCDynamicAdaptationMode=1</pre><p>In future releases, we plan to turn DATAS on by default so you’ll no longer need to do any configuration if you want to have the heap size adapt to what your application requires.</p><h3>Footnotes</h3><p>1. For example, <a href="https://github.com/dotnet/runtime/pull/73375">use prefetch during marking</a>. And by moving the GC heap from segments to regions (turned on by default in .NET 7) it enabled us to do optimizations that were much harder before such as being able to <a href="https://github.com/dotnet/runtime/pull/64206">balance survivors</a> easily for the generations we are collecting by moving the regions around between heaps.</p><p>2. By that I mean these workloads can afford to have a fairly large memory footprint but they have high demands on latency (so reducing the GC pauses can be critical for them). And they are usually expected to be active very often. Or when they are not active, they don’t need to share memory on the machine with other applications.</p><p>3. With Workstation GC you have one heap and let’s say the gen0 <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#looking-at-the-gc-heap-size-wrt-when-gcs-happen">budget</a> (ie, the amount of allocations that when exceeded will trigger a GC) is 10mb. If you have 8 heaps with Server GC it will be able to allocate 80mb before the next GC is triggered. Now you might’ve just added 70mb to your process’s memory usage than with Workstation GC. I say “might’ve” because this is a very simplistic way of looking at this problem as there are multiple factors that would affect the computation of the budget. For example, if the memory load is high, GC will reduce the budget accordingly so the total gen0 budget could be less than 8x that of Workstation GC’s. It could also be more than 8x because Server GC might allow a larger budget per heap.</p><p>4. Strictly speaking, this is not the # of processors on the machine — it’s the # that your process is allowed to use. So if your process is only affinitized to 4 processors out of 32 that the machine has, then you will get 4 heaps by default.</p><p>5. It’s very unlikely you’d want each heap to have a budget say is only 32k because that would mean you are triggering GCs too often. You also wouldn’t want each heap to get too much budget because that would mean allocating threads might have to wait for too long to get to allocate on a heap (when an allocating thread comes to the GC to get memory, it needs to take a lock for the heap it’s assigned to allocate on).</p><p>6. I think of the GC dynamic tuning as a threefold problem –</p><p>· How many conditions it adapts to and how well it adapts to them. GC already dynamically adapts to a wide range of things such as survival rates and physical memory load.</p><p>· How the dynamic tuning work relates to application metrics. Survival rate is not an app metric while memory usage is. We should be able to make statements like “my application uses ~Xmb memory so my GC heap memory usage should be ~Ymb where Y is proportional to X”. We’ve done tuning specifically in this space (eg, the ConserveMemory config) but DATAS takes it to a new level. In the future we also plan to adapt to other application metrics like a CPU percentage GC is allowed to take specified by the user.</p><p>· The tuning is never “let’s just optimize for this one metric and not care about anything else”. It really means <em>having things run fairly smoothly in general while optimizing for a metric/set of metrics</em>. There can be a point when the performance is no longer good because there’s simply too much stress on the GC. Obviously, we continue to make improvements in the GC which make GC be able to handle more stress.</p><p>7. I’ve excluded all AoT tests — since we do our perf testing with clrgc.dll and AoT currently does not supporting loading the standalone GC dll, we don’t have results for them. We are currently adding this support to AoT.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d72fcb6f1ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How the heap verification mode helps with debugging heap corruption]]></title>
            <link>https://maoni0.medium.com/how-the-heap-verification-mode-helps-with-debugging-heap-corruption-58b3d08f5e8c?source=rss-df86aab12c0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/58b3d08f5e8c</guid>
            <category><![CDATA[debugging]]></category>
            <dc:creator><![CDATA[Maoni0]]></dc:creator>
            <pubDate>Thu, 22 Dec 2022 21:33:23 GMT</pubDate>
            <atom:updated>2022-12-23T06:28:36.248Z</atom:updated>
            <content:encoded><![CDATA[<p>There’s a very handy tool called the heap verification mode that’s used to help figure out heap corruption issues, ie, those gnarly AVs (access violations) when accessing an object that’s no longer valid. It’s by default off and to turn it on, set this env var –</p><pre>set DOTNET_HeapVerify=1</pre><p>When it’s on, it verifies that the GC heap looks sane on entry and on exit of each GC. And when it doesn’t look sane, heap verification will report an error. This is an example of such an error –</p><pre>void gc_heap::verify_heap (BOOL begin_gc_p)<br>{<br>    // omitted<br>    prev_object = curr_object;<br>    // omitted<br>    curr_object = curr_object + Align(s, align_const);<br>    if (curr_object &lt; prev_object)<br>    {<br>        dprintf (1, (&quot;overflow because of a bad object size: %p size %zx&quot;, prev_object, s));<br>        FATAL_GC_ERROR();<br>    }<br>    // omitted<br>}</pre><p>I’ve omitted all the other lines in the <em>verify_heap</em> method. This particular code snippet says that if we calculate the size of <em>curr_object</em> to obtain the address of the next object and that address is smaller than the <em>current_object</em> start, something is wrong so we call <em>FATAL_GC_ERROR() </em>which does the following –</p><pre>inline void FATAL_GC_ERROR()<br>{<br>  GCToOSInterface::DebugBreak();<br>  _ASSERTE(!&quot;Fatal Error in GC.&quot;);<br>  GCToEEInterface::HandleFatalError((unsigned int)COR_E_EXECUTIONENGINE);<br>}</pre><p>When folks observe a heap corruption, which often shows up somewhere in the GC code like <em>mark_object_simple1</em> (unsurprisingly, since GC looks at the heap the most extensively, it very likely gets to observe the corruption before something else looks the corrupted object and observes a failure there), I would ask them to turn this mode on and see if it breaks in <em>FATAL_GC_ERROR</em> as the first step of figuring out if the corruption is likely caused by the GC or something else.</p><p>If it does break in <em>FATAL_GC_ERROR</em>, what does that mean? First of all, it’s useful to know how to distinguish “on entry” and “on exit” –</p><p>· <em>On entry of a GC</em> means <em>verify_heap</em> is called directly by <em>garbage_collect</em></p><p>· <em>On exit of a GC</em> means it’s called directly by <em>gc1</em>.</p><p>Let’s say the heap looked sane on entry of a GC but not on exit of that GC. It means the heap looked fine when this GC was triggered, so GC was handed a heap that heap verification did not detect any problems with. But after the GC was done, the heap no linger looked fine. So something went wrong during the GC. Note that I’m very careful here not to say that “something went wrong with the GC” because it can still mean something else went wrong, for example, if something reported roots incorrectly to the GC. But we can treat it as “it’s very likely something went wrong with the GC”. And vice versa, if the heap looks sane on exit on a GC but not on entry of the next GC, it’s very likely that something went wrong with something other than the GC. <strong>If the corruption repros with the heap verification mode on, the policy we generally use to decide if a GC team member should debug it is if it breaks on exit of a GC.</strong> And this should make sense now because that tells us “it’s very likely something went wrong with the GC”.</p><p>Of course having a heap verification mode provides more value than just deciding who should look at the heap corruption issue first 😊. It’s extremely useful to look at the corrupted object(s) as early as possible after it occurred. As I mentioned above, the reason you see callstacks that indicate heap corruption often in GC code is because GC looks at objects very extensively. Think about how objects are normally used — you allocate an object, or get a hold of an object from something you call, then you access some fields. There could be a long time between when it’s corrupted till it’s accessed in a way that causes an AV. <strong>To understand why/what corrupted it, it’s better to get as close to when it happened as possible.</strong> Even doing GCs normally may not cause an AV to show up when an object is corrupted because of reasons like GC doesn’t look at all objects when doing ephemeral GCs, or that GC doesn’t need to look at an object again after it corrupts it therefore no AV would show up during that GC. <strong>But when heap verification is on, we verify the whole heap, not just the generations that get collected.</strong> So if say something went wrong during a GC, it’s much better to be able to look at it at the end of that GC instead of an undeterministic number of GCs later.</p><p>There is a ton of stuff heap verification checks which gives us clues what to look at next. Below is the list of things that get verified during the <em>verify_heap</em> call (I’ll mention regions but segments works very similarly) -</p><p>· The whole heap is walkable, meaning we can advance from the beginning of a region all the way to the last object on that region.</p><p>· For each object we walk that’s not a free object, we verify various aspects of it, eg, if we find it in SOH its size should be less than the threshold for a large object. We verify its immediate children when we can (during some phases we may not be able to do this).</p><p>· Region lists themselves look fine. For example, we can go from the start region of a generation to the tail region, a region’s generation number is set correctly, and etc.</p><p>· The GC bookkeeping data structures look fine, ie, cards, bricks, mark bits and mark array bits look fine, eg, when a card should be set it is indeed set, when mark array bits should be cleared they are indeed cleared.</p><p>· Free lists look fine.</p><p>· It calls the verification code in components that GC interacts with, such as the handle table and finalize queue.</p><p>Each of these will give clues to what we want to look at next. For example, if it breaks because it hits <em>FATAL_GC_ERROR</em> when a card that should be set is not, we look at places where GC is supposed to set cards and go from there.</p><p>Note that <strong><em>verify_heap</em> is always called during STW</strong> (Stop The World), <strong>which means for a Background GC, we will do an additional STW phase</strong>. Normally the end of a BGC runs concurrently but when heap verification mode is on, we call <em>SuspendEE</em> to suspend managed thread before calling <em>verify_heap</em> and <em>RestartEE</em> afterwards to resume them.</p><p>You may have noticed I said “<em>if</em> the corruption repros with the heap verification mode on” instead of “when”. Because <em>verify_heap</em> does so much, it will make your application run a lot slower (unless you barely do GCs of course) so it could disturb the timing enough that it makes the bug no longer repro. I’ve had folks who worked on GC or GC related issues tell me that heap verification wasn’t useful to them when they couldn’t repro the bug with it on. I suggested to them that <strong>they could heavily cut down a lot of what the verification verifies by default</strong>. For example, you could start with just walking the heap, or just walking the heap and verifying cards, and only in the generations that you suspect might be of interest. That can reduce the work by a lot and often make the bug repro.</p><p>In GC development, we use the heap verification mode extensively and it’s normal to make change to the verification code itself if we want to cut down the cost there. When customers report a heap corruption issue, I start with having them turn the mode on and see if they can repro it, and if they can, see if it reports an error on entry or exit of a GC and go from there. <strong>If the error is on exit of a GC, a GC team member would take a look. If it’s on entry of a GC, then it’s likely caused by something other than the GC.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58b3d08f5e8c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is the regression in GC or something else?]]></title>
            <link>https://maoni0.medium.com/is-the-regression-in-gc-or-something-else-38f10018dd21?source=rss-df86aab12c0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/38f10018dd21</guid>
            <dc:creator><![CDATA[Maoni0]]></dc:creator>
            <pubDate>Wed, 07 Dec 2022 01:30:39 GMT</pubDate>
            <atom:updated>2022-12-07T01:30:39.793Z</atom:updated>
            <content:encoded><![CDATA[<p>When you change your product to run with one version of .NET to another, sometimes there’s a regression in memory. This could show up as heap size increases, GC pause increases (either individual GC pause increase and/or total GC pause increase), or GC CPU time increases.</p><p>It’s very often that I’ve observed folks immediately think this is a GC problem, after all, GC is in charge of managing memory for you, right? GC is indeed in charge of managing memory for you but if your workload changes of course the GC behavior will change accordingly.</p><p>I’ve had many people say to me, “NO! My code did not change! It’s exactly the same code. I just upgraded from .NET version X to .NET version Y!”.</p><p>Your code does not run just on the GC (it’d be hard to do anything interesting if there was only the GC :-)). It relies on the GC, the rest of the .NET runtime (which includes JIT, type system, interop, and etc), BCL (Base Class Library — this is where the implementation for classes like string/StringBuilder/Stream and many, many others live) and the higher level libraries (asp.net and etc). And with each new version of .NET there are not just changes in the GC but in all those other components. Each new release contains a huge amount of changes.</p><p>Obviously, changes in the GC can change memory behavior. But changes in the library code you are calling can contribute to the changes in memory behavior just as easily.</p><p>With this many changes, debugging why you see a memory regression can range from trivial to very difficult. Trivial cases are like “this particular class we use a lot started to allocate a lot more objects/pinned handles which causes a lot more GCs/fragmentation on the heap”. But with a ton of changes, many of which can affect memory behavior, it could be a lot of work to figure out what contributes to regressions.</p><p>As someone who often gets asked to do this kind of memory regression analysis, I thought I’d make the initial step of identifying “is the regression in GC or something else?” much quicker. This mechanism is shipping in .NET 7. Details are explained below.</p><p><strong>Using clrgc.dll to determine if the regression is caused by changes in GC</strong></p><p>Starting .NET 7.0 we’ll be shipping a standalone clrgc.dll that includes the GC implementation from .NET 6.0. This will be in the same directory as coreclr.dll. When you see a memory regression, please run your scenario again with clrgc.dll by setting this runtime config -</p><pre>{<br>   &quot;runtimeOptions&quot;: {<br>      &quot;configProperties&quot;: {<br>         &quot;System.GC.Name&quot;: &quot;clrgc.dll&quot;<br>      }<br>   }<br>}</pre><p>or this environment variable -</p><pre>set DOTNET_GCName=clrgc.dll</pre><p>There are 3 possible outcomes -</p><p>1. if the regression is gone, it’s clearly in GC so please file an issue on our <a href="https://github.com/dotnet/runtime/issues">GH repo</a> with the following data when running on .NET 7, with and without clrgc.dll -</p><p><a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#How-to-collect-top-level-GC-metrics">Top level GC traces</a></p><p><a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#perf-data">CPU samples traces</a></p><p>and the GC team will look at it promptly! We’ll likely ask you for more data and while we are doing our investigation and producing a fix, please feel free to keep using clrgc.dll.</p><p>2. if the regression is the same, it’s clearly not in GC. The best starting point is also by capturing <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#How-to-collect-top-level-GC-metrics">traces with top level GC metrics</a> for running on .NET 7 and the version you upgraded from. You can <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#displaying-top-level-gc-metrics">display the trace in PerfView</a> and compare the “Total Allocs” in the GCStats summary for your process. If the total allocation is simply more on 7.0, you can follow the <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#Measure-allocations">measuring allocations</a> section to diagnose what’s making the allocations. Otherwise, you can follow the section for diagnosing <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#High--Pause-time-in-GC">high % pause time in GC</a> and/or <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#Large-GC-heap-size">large GC heap size</a>.</p><p>3. The regression is only partially gone. You can start with the same step described in 1).</p><p><strong>History of clrgc.dll</strong></p><p>In .NET 2.0 we started working on the LocalGC project which was to allow you to load an alternate GC implementation, ie, a “local implementation”. By default, the GC for .NET Core is part of coreclr.dll. LocalGC enabled the runtime to load a GC implementation from a separate DLL. This separate DLL needed to communicate with coreclr.dll via well-defined interfaces, the 2 main ones are -</p><p>• <em>IGCHeap</em> which contains methods on the VM side (in this context, VM side really just means the code that lives in the coreclr\vm directory) uses to call into the GC. This is defined in coreclr\gc\gcinterface.h. Many methods here are actually to support calls indirectly made by the BCL side such as <em>GetMemoryInfo </em>which is used to implement the <em>GC.GetGCMemoryInfo</em> method. There are also methods called by the allocation helpers to actually call into the GC to get memory.</p><p>• <em>IGCToCLR</em> which contains methods that the GC side uses to call into the VM, such as <em>SuspendEE</em>/<em>RestartEE </em>for suspension and <em>GcScanRoots</em> to scan stack roots. This is defined in coreclr\gc\gcinterface.ee.h.</p><p>If you want to plug in your own GC implementation, you will need to implement what is on the <em>IGCHeap</em> interface. If you do this just for research purposes or for fun, you don’t need to implement every single method <em>IGCHeap </em>define, just enough to get the runtime functional. Since many methods are to support APIs defined on the <em>GC </em>class, they are not required to make the runtime functional.</p><p>We actually ported all this back to .NET Framework 4.8 because aside from using the standalone GC in our own day-to-day development work, it has been really handy to use it to verify fixes or provide instrumentation in customers’ production environment. Especially for .NET Framework it’s a really big deal to change clr.dll because you’d be changing it for everything running on that machine, but with standalone GC you can just have the process of interest to load clrgc.dll and all other processes will remain using clr.dll. As part of the early testing for regions, I gave our partner teams two clrgc.dll’s — one with the regions implementation and the other with segments. This way they didn’t need to upgrade their .NET version at all — they just needed to load a clrgc.dll with their product running on .NET 6.0.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38f10018dd21" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Write barrier optimizations in regions]]></title>
            <link>https://maoni0.medium.com/write-barrier-optimizations-in-regions-984bde6c0ffc?source=rss-df86aab12c0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/984bde6c0ffc</guid>
            <dc:creator><![CDATA[Maoni0]]></dc:creator>
            <pubDate>Sun, 02 Oct 2022 01:16:45 GMT</pubDate>
            <atom:updated>2022-10-02T01:16:45.811Z</atom:updated>
            <content:encoded><![CDATA[<p>This is another blog entry about regions. As I mentioned in my last blog entry, regions on their own will naturally give us some perf benefits already but they are really meant to be a foundation for us to unlock optimizations that are difficult to do with segments. Given the time we had in .NET 7.0, we unlocked some of these optimizations and obviously will continue to unlock more in the upcoming releases.</p><p>One of the optimizations we did was to have a more precise write barrier. To talk about that, let&#39;s talk about what purposes the write barrier serves in the .NET runtime first. If you are already familiar with this feel free to skip to <a href="https://medium.com/@maoni0/write-barrier-optimization-in-regions-984bde6c0ffc#121b">the section</a> that describes the 7.0 optimization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*U_CvGsNg7rVaQ1eJaQscdQ.jpeg" /></figure><p><strong>What is a write barrier?</strong></p><p>A write barrier is a piece of code that gets invoked when you do this -</p><pre>obj_x.field0 = obj_y;</pre><p>when <em>obj_x.field0</em> is a reference field, in other words, <em>obj_y</em> is an object on the GC heap. We refer to <em>obj_x.field0</em> as &quot;location&quot; from here on, as it&#39;s the location we are assigning to, and <em>obj_y</em> as the child object. And I’ll use the WB acronym in place of Write Barrier.</p><p>When .NET was started, the WB’s only purpose was to facilitate the GC being generational. A generational GC means we don&#39;t collect the full heap all the time - in fact we do that pretty rarely. <strong>When we are only collecting part of the heap, we need to know if the rest of the heap is pointing into the portion that we are collecting.</strong> For example, if we are doing a gen1 GC, it means we are only collecting gen1 and its younger generations (which in this case is just one generation - gen0) so the rest of the heap - gen2, LOH and POH (POH did not exist till .NET 5) are not collected. If those objects are pointing to any gen1 or gen0 objects, that needs to be discovered in some way so GC can keep these gen0 or gen1 objects alive if they are not already considered live by other means such as being pointed to by a stack root. In the WB we indicated this by setting a byte in a data structure called the card table. The reason we set a byte instead of a bit, even though a card is a single bit, is because you can&#39;t set a bit atomically.</p><p>When we needed to make the runtime run on Linux, <strong>we added another functionality in the WB which is to record heap modifications</strong> in our own data structures instead of using the OS&#39;s write watch on Windows which did not exist on Linux.</p><p><strong>The most common function that gets called is <em>JIT_WriteBarrier</em>.</strong> When this is called, it knows that the location is on the GC heap. When the JIT can&#39;t decide this is the case, it generates calls to one of the 2 other versions - <em>JIT_CheckedWriteBarrier</em> and <em>JIT_ByRefWriteBarrier</em>. These need to check if the location is on the GC heap first and exit right away if that&#39;s not the case. <em>JIT_ByRefWriteBarrier</em> is a special case to handle copying a range of reference fields in a struct (which is why rsi/rdi need to be incremented by a ptr size).</p><p>For Workstation GC in the segments case, we know that there&#39;s a single ephemeral range because there&#39;s only one ephemeral segment. In fact, where there is only one segment, instead of a range, we just need the lower limit, i.e., anything higher than where generation 1 starts is an ephemeral object. So we filter the child object by checking if it&#39;s in the ephemeral range by doing either one or two comparisons. Unfortunately this isn&#39;t the case for Server GC which has as many ephemeral ranges as there are heaps. So for Server GC we set the cards unconditionally (if they are not set yet).</p><h4><strong>More precise WB in 7.0</strong></h4><p>In .NET 7.0 we made the WB much more precise for the regions case, especially for Server GC. <a href="https://github.com/PeterSolMS">Peter</a> did the <a href="https://github.com/dotnet/runtime/pull/67389">work</a>. <strong>One of the major differences between regions and segments is a region can only belong to one generation, unlike segments where the ephemeral segment can contain gen0, gen1 and potentially gen2 objects.</strong> We maintain a compact generation info map for regions (so it&#39;s likely in the cache) and pass this info over to the WB side. Then in WB we can get the generation for any address that&#39;s on the GC heap. We do the following in the WB -</p><ol><li>Get the generation number for the location.</li><li>If this is in gen0, nothing to do; else get the generation of the child object.</li><li>If the generation of the child object is &gt;= generation of location, nothing to do; else we need to set the card, if it hasn&#39;t been set.</li></ol><p>By only setting cards when location is in an older generation than the child object, we cut down the number of false positively set cards by a whole lot. For example, it&#39;s very common that you&#39;d allocate a new objects and initialize their fields with another new object you just allocated. So no cards are set for that case anymore. This greatly reduces the number of cards GC has to look at during an ephemeral GC which makes the mark phase faster.</p><p>Initially my plan was to still set the byte in the card table like we used to for segments. But Peter experimented with setting just a bit and found that the perf was still acceptable (because we don&#39;t tend to interlock on the same bit when we update). And of course this reduced the number of false positive cards noticeably further, so we decided to set this as the default.</p><p>We actually do still maintain an ephemeral range but this is no longer precise like in the segments case for Workstation GC. This is simply the beginning of lowest ephemeral region to the end of the highest ephemeral region. And there may be gen2/LOH/POH regions in between. We do use this in the GC, however we don&#39;t currently use this range on the WB side because updating WB when this range is updated added what we considered an unacceptable amount of overhead. However, we plan on doing more optimizations in the future to make this range much more stable. So today even though we leave the code of comparing with this range in WB, this range is just set to the GC heap range (so we don&#39;t need to update the WB code because this range is updated).</p><p>Note that the data structure that allows us to get the generation number for an address is quite compact (it’s a lot smaller than the card table) so we expect that this is often a cache hit.</p><p>Here&#39;s a comparison of 3 10-min runs from a 1st party environment with real time forked production traffic. Note that the result for the Region Byte WB is doing fewer GCs because it happened to hit a tuning decision for budget adjustment differently from the other 2 runs (one of the things I&#39;m hoping to fix in the upcoming releases to make this more robust). The last 2 columns are very important — they are what we use to measure <strong>GC&#39;s efficiency, ie, how long does it take to promote the same number of bytes?</strong> For GC the number of promoted bytes is the indication of how much work GC needs to when comparing the same workload. So higher is better. With the Byte WB version of regions gen0 speed is 18% higher. And with the Bit WB version of regions it&#39;s 25% higher than segments. Note that the improvement in regions does include more than just the WB changes but you can clearly see the difference between bit and byte WB.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49cbbdee59b233a2c05718b3ff525bca/href">https://medium.com/media/49cbbdee59b233a2c05718b3ff525bca/href</a></iframe><p>Of course we do pay a price in the WB code because it now does more work -</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22208790ee1e7e42e0b27b7e34788b5a/href">https://medium.com/media/22208790ee1e7e42e0b27b7e34788b5a/href</a></iframe><p>For Bit WB regions, there were 25,556 samples in <em>JIT_WriteBarrier</em> which is 0.7% of the total CPU samples. So even though it does have 15% more samples than segments, it&#39;s still a small cost. I&#39;ve rarely seen <em>JIT_WriteBarrier</em> take more than 1% of CPU.</p><p>I do plan to change the WB implementation in the future releases in .NET to move most of the work we have to do in the WB code path elsewhere so it does not block the thread that executes the WB as long.</p><p><strong>How to know if I hit a regression due to this?</strong></p><p>We did see regression from running tiny microbenchmarks for this change. As I mentioned above, usually <em>JIT_WriteBarrier</em> takes very little CPU. But if you have a microbenchmark that spends a lot of time in the WB, you could certainly see a regression. Let&#39;s say you have a small test that just assigns an existing object to different locations, it can easily spend a significant percentage of CPU in WB. But since this is not the typical behavior real workloads exhibit, we view it as acceptable regression. However, we do provide a way to revert back to the old Server GC behavior if you do run into the very rare cases where the WB cost is actually significant in your app. In src\coreclr\gc\gcconfig.h, we define the following values you can specify for the DOTNET_GCWriteBarrier environment variable -</p><pre>enum WriteBarrierFlavor <br> { <br>     WRITE_BARRIER_DEFAULT = 0, <br>     WRITE_BARRIER_REGION_BIT = 1, <br>     WRITE_BARRIER_REGION_BYTE = 2, <br>     WRITE_BARRIER_SERVER = 3, <br> };</pre><p>Default is the Bit WB so 0 and 1 are handled the same way, both using the Bit WB. Setting the config to 3 means to use the old Server GC WB which is to unconditionally set the card (byte). And if you want a middle ground, you can set it to 2 which means to use the Byte WB. There is no equivalent of the old Workstation GC WB. However we observed that in our Workstation GC testing, even though WB costs a bit more, the end result is the total execution time is usually noticeably less.</p><p>In .NET 7 we also implemented <a href="https://github.com/dotnet/runtime/issues/50391">Write XOR Execute</a>. And the page that the WB code lives on is mapped twice and the address of the code is on the writable page that’s not part of coreclr.dll. Without W^X you can set a bp on coreclr!JIT_WriteBarrier and see it getting hit -</p><pre>ModLoad: 00007fff`28420000 00007fff`2890a000   C:\runtime-segments\artifacts\tests\coreclr\windows.x64.Release\Tests\Core_Root\coreclr.dll</pre><pre>Breakpoint 1 hit  <br>coreclr!JIT_WriteBarrier:  <br>00007fff`2857e040 488911          mov     qword ptr [rcx],rdx ds:00000240`80ad46c8=0000000000000000</pre><p><em>00007fff2857e040</em> is in the coreclr.dll module.</p><p>But with W^X you will only hit coreclr!JIT_WriteBarrier_Callable which will then jump to the WB code -</p><pre>ModLoad: 00007fff`26150000 00007fff`2663a000   C:\runtime-segments\artifacts\tests\coreclr\windows.x64.Release\Tests\Core_Root\coreclr.dll<br><br>Breakpoint 1 hit  <br>coreclr!JIT_WriteBarrier_Callable:  <br>00007fff`262ae240 ff25baed3600    jmp     qword ptr [coreclr!JIT_WriteBarrier_Loc (00007fff`2661d000)] ds:00007fff`2661d000={CORINFO_HELP_ASSIGN_REF (00007ffe`c65d0010)}  <br>0:017&gt; t  <br>CORINFO_HELP_ASSIGN_REF:  <br>00007ffe`c65d0010 488911          mov     qword ptr [rcx],rdx ds:0000029c`2c86b7e8=0000000000000000</pre><p><em>00007ffec65d0010 </em>is not part of coreclr.dll so when you look at this in a profiler like PerfView, it does not know how to map this address to a symbol. This can be made work by firing a rundown event to help with the decoding. But for now, you can turn W^X off by setting the DOTNET_EnableWriteXorExecute env var to 0.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=984bde6c0ffc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How segments and regions differ in decommitting memory in the .NET 7 GC]]></title>
            <link>https://itnext.io/how-segments-and-regions-differ-in-decommitting-memory-in-the-net-7-gc-68c58465ab5a?source=rss-df86aab12c0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/68c58465ab5a</guid>
            <category><![CDATA[memory-management]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[dotnet]]></category>
            <dc:creator><![CDATA[Maoni0]]></dc:creator>
            <pubDate>Thu, 24 Mar 2022 23:25:28 GMT</pubDate>
            <atom:updated>2022-04-08T23:17:24.082Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>How segments and regions differ in decommitting memory in the .NET 7 GC</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W6B79JGWHuJyCWjkvaVUSA.jpeg" /></figure><p>We just snapped the source with regions in GC <em>on by default</em> for the next .NET 7.0 preview build this past Monday (and the build will be release in April). So I wanted to explain <strong>how the regions feature differs in its policy for decommitting memory from segments</strong> so you can evaluate whether you are seeing something by design or not if you decide to try it out. <strong>Note that the regions feature is only available in 64-bit and for now we disable it for macOS.</strong> So everything in this article will assume 64-bit. Also we are still in the middle of the development cycle for .NET 7.0 so I’ll mention things that might change later.</p><p>As I’ve always said we often have to make tradeoffs in our work. Regions provide more flexibility, which was why I wanted to do this in the first place. And in <a href="https://devblogs.microsoft.com/dotnet/put-a-dpad-on-that-gc/">my last blog entry related to regions</a> I mentioned some of these tradeoffs. Today I’m going to talk specifically about decommitting memory as how much memory the GC heap takes is one of the most important metrics our customers care about. Since I don’t want this to get too long I’ll <strong>limit the discussion to Server GC for this blog entry</strong> because apps that run with Server GC usually have much bigger heaps and much longer running so when memory gets decommitted is especially important. As usual, since I know many of my readers are also interested in the inner works of the GC I will touch on relevant design/implementation details.</p><p>Committing memory is pretty straightforward — GC just commits memory as needed — it doesn’t commit one page at a time which would have a lot of overhead. It generally commits 16 pages at a time (if you are allocating an object that’s larger than 16 pages of course it would need to commit more because each object always occupies a contiguous memory space). This works the same in the .NET 6.0 GC and the .NET 7.0 GC.</p><p>Decommit, OTOH, is a very different story. You don’t want to decommit memory that you know you will use pretty soon, because then you’d have to pay the cost to decommit and to commit it again. But you also don’t want to leave too much committed if that space doesn’t get used. So we have policies in place that specify how much we decommit and when. Note that <strong>free spaces are by definition always committed memory </strong>as they are counted as part of what’s in use on the heap (it’s just currently used by objects of type Free and will be used to accommodate allocations into that generation). Could we decommit free spaces? Technically yes but that requires yet another layer of non trivial memory management on regions/segments. So <strong>we only decommit the end of a segment or a region after the last live object</strong>, but never memory occupied by free objects between the first and last object on that segment/region.</p><p>If I had a dollar every time I heard someone say “GC never decommits memory” or “GC never decommits memory unless it’s running out of memory”, I’d be able to afford a pretty nice espresso machine (I just tried out a bunch of espresso machines on the weekend…that’s how I know). Those are simply not the policies we use. To start off, let’s talk about how segments decommit memory.</p><h3><strong>Details of decommitting for segments and regions</strong></h3><p><strong>Segments are much larger in size than regions.</strong> For SOH segments (they are always the same size), in Server GC, it’s 1GB if you have &gt;8 heaps which is very often the case for Server workloads. UOH segments (meaning LOH and POH) are a lot smaller (256mb) to start with but can be larger if needed to be. What this means is it’s very possible that you only get one segment per heap. Let’s say you have a 20GB heap, all in SOH, and your app is running with 48 heaps, you will only get one segment per heap. If that’s the case, for SOH it means we only ever decommit on this segment. And since gen0 always lives on the higher address side on this segment (that we call the ephemeral segment for this heap), it means our policy just needs to define how the end of segment space is decommitted for the ephemeral segment. More details on how segments are organized is described <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#physical-representation-of-the-gc-heap">here</a>.</p><p><strong>For the ephemeral segment our policy says after the last live object at the end of a blocking GC, we will only keep (full gen0 budget + partial gen1 budget) committed and decommit the rest, if any.</strong> Fig. 1 below illustrates how the committed changes on the ephemeral segment before and after the very first GC. We keep the (full gen0 budget + partial gen1 budget) after the last live object committed, and decommit the rest. In Fig. 1 I’m illustrating a scenario where gen0 is empty and gen1 becomes bigger because it now holds the survivors from gen0.</p><p>Fig. 1 On entry and exit of the 1st GC</p><figure><img alt="On entry and exit of the 1st GC" src="https://cdn-images-1.medium.com/max/1024/1*foxMotbMwDo8bZb3PMtfNw.jpeg" /></figure><p>Okay, so that’s not exactly accurate for Server GC — starting in .NET 5, we no longer do the actual decommit work during a blocking GC so all we do is remember on the segment the address we want to decommit after (ie, the portion of the segment higher than that address will be decommitted). And the decommit work is run on one of the heaps’ Server GC thread concurrently with the user threads. We make sure to have at least full gen0 budget amount of space committed because we wanted to avoid having to introduce synchronization between the GC thread and the allocating threads that might need to commit more on that same segment. As we will see below this is one of the limitations that we no longer have with regions.</p><p>For gen2 and UOH segments, the policy is pretty straightforward — we just decommit after the end of the last live object. Of course there’s nothing to decommit unless we are actually doing a gen2 GC because if we don’t collect a generation, that generation can’t get smaller. For BGC, this happens concurrently (we chose not to do this concurrently for blocking gen2 GCs because we often do blocking gen2 when the memory is already tight so we’d like to decommit ASAP).</p><p>I have a very detailed explanation on commit and decommit with segments in <a href="https://www.youtube.com/watch?v=ImeiUzbdMzc">my talk</a> about diagnosing managed memory leak. I would recommend to watch it if you want more info/illustrations how the decommit happens as <em>different generations of GCs</em> happen. I cannot stress this point enough — our GC is a generational GC therefore the generational aspect is very important in perf analysis.</p><p><strong>Regions are a lot smaller — 4MB by default (for SOH; UOH regions are &gt;=32MB by default)</strong> We keep a free region pool that holds regions that are completely empty. This allows us to exchange memory freely between generations. So if gen0 frees up a region and puts it on the free region list, and if gen1 or gen2 needs a new region, the region gen0 freed can be used to satisfy such a request.</p><p>Greater flexibility means greater complexity. Now we have more management to do for these smaller memory units. Along the same lines with segments we also don’t want to keep too many free regions around but we also don’t want to keep too few which would cause us to get into the “decommit then immediately having to recommit” situation. But with regions we have additional aspects that we need to consider. Below are a couple of examples -</p><ul><li>Which region to pick when we need one? Do we pick one with more committed or less committed? Think about how a region is used — if we get a region that’s only partially committed, let’s say 2mb is committed, and we needed more than 2mb. We’d be committed more on this region. If we had another region on the free list that’s fully committed (ie, 4mb), we’d want to use that one instead. But you might say “what if I needed a region that’s only 2mb committed, wouldn’t I exactly want that partially committed region?”. Yes but when would you need a 2mb region? If the gen0 allocation budget is 10mb, then you would want 2 fully committed regions + 1 with 2mb committed. And we know that one would be at the end of the region list for gen0. Because the region size is small, it means we’d only need the partial committed region for the tail of the region list in a generation. So our policy is we always pick the most committed and decommit the tail region if needed.</li><li>How many total regions to keep? As with segments we want to keep the allocation budget worth of regions committed because we’d want to allocate that much. However, sometimes GCs happen not due to allocation budget being exceeded. In other words, we could want to elevate to collection a generation even when its allocation budget hasn’t been exceeded. So we do use the budgets to decide how many regions we want to decommit but we also want to notice if a free region keeps not getting used. And for those regions we do want to decommit them.</li></ul><p>We do keep at least one region per generation so for example, after a GC, even if we find gen0 empty (which is very likely if you have no pins), we will keep one region in gen0. And if gen0 isn’t empty we keep all the non empty regions on the gen0 region list and return the rest to the free region list. The reason why we keep at least one region in a generation is mostly for historical reason — in segments we always have a segment for a generation (multiple generations could share a segment but if you enumerate the segment for a generation you’d always get at least one) — we don’t strictly have to but having the invariant of at least one region per generation makes the implementation easier. I actually experimented with getting rid of this invariant but didn’t actually check it in because it wasn’t worth the complexity.</p><p>Fig. 2 illustrates what happens during the first GC assuming the gen0 budget started as occupying 2.x regions. In 2.a we have 2.x regions committed on entry of the first GC. Note that the relative size between a segment and a region is obviously not meant to be accurate.</p><p>Fig. 2.a On entry of the 1st GC</p><figure><img alt="On entry of the 1st GC" src="https://cdn-images-1.medium.com/max/955/1*IgARz68GT1jifc-h-uTHUQ.jpeg" /></figure><p>Fig. 2.b On exit of the 1st GC</p><figure><img alt="On exit of the 1st GC" src="https://cdn-images-1.medium.com/max/1024/1*79hSr06jNLspCc-WSNh_sw.jpeg" /></figure><p>In 2.b we see the region free list now contains 2 regions. <strong>If there are too many regions on the free list, we will remove some from the free list and add them to the decommit list which will then be decommitted</strong> (the decommit list is not shown in Fig. 2). “too many”, as mentioned above, is calculated by 2 factors — 1) estimating the difference between the budget and what regions we already have in that generation. For example, if gen0 budge is 10MB, which means we’ll need 3 regions. And we already have one region in gen0, it means we will need 2 more regions for gen0. We do this with all generations. 2) if we observe a region hasn’t been used for a while, ie, their ages are too high, we will decommit it. For now we just set it to 20 GCs but we will dynamically tune this based on the memory situation.</p><p>Right now <strong>there’s a separate free list for all of the generations in SOH and UOH free lists are separate. We haven’t done this yet but will be repurposing regions on the UOH free lists for SOH if we need to.</strong></p><p>For gen0 and gen1 we also decommit the end of the tail region based on budget calculation, if needed. Normally gen0 would have more than one region but gen1 could be very small. So after we do a gen1 GC if most objects on that region died, we would decommit end of region space. So if a gen1 region has 4MB committed with currently 500KB in use, and its budget is only 1MB, it means we will decommit (4MB — 500KB — 1MB) = 2.5MB at the end of the region (if there’s free space in gen1 we also estimate part of that can be used for gen1 allocations). When you have many heaps with a benchmark that uses very little memory (which unfortunately is often the case for these little benchmarks) this can make a big difference. As with segments we also do this decommit work concurrently but we only need to synchronization with the allocating threads if we need to decommit the gen0 tail region. Most of the decommit work would be handled by the regions that are on the decommit list (which requires no synchronization at all) so the decommit work for that one region is small anyway.</p><h3><strong>Difference in decommitting between regions and segments</strong></h3><p>Now we can summarize what the difference is based on the above — <strong>for regions on the free list, we do not decommit end of region space right now.</strong> <strong>This would make up the majority of the difference in the committed size between segments and regions for steady state.</strong> When we calculate how many regions we’d need based on the allocation budget, we round that up to a region. So if the budget says we need to keep 9mb in free regions, we’d round it up to 12mb and keep 3 free regions for it, which means if these are 3 fully committed regions, we could decommit 3mb in one of them. Currently we don’t decommit end of region space for these regions — I expect us to, especially for large regions. So for SOH you could have 3 such regions in the free list. For UOH you could have 2 such regions (LOH and POH). This is of course assuming you do have such regions on the free list. If say you don’t make much LOH/POH allocation at all, you’d just have the one region on LOH/POH and wouldn’t have any large regions on the region free list at all. If your heap is pretty sizable, unless you happen to be in a situation where you have such free regions with the extra commit, this shouldn’t affect you much. But if it does,<strong> please do let us know! You could either file an issue in our GH repo, or post a comment on this article.</strong></p><h3><strong>Committed in different stages for an application</strong></h3><ul><li><strong>Peak committed</strong></li></ul><p>Peak committed usually occurs during startup stage, which really refers to when you allocate a lot of long lived data which makes the GC adjust the budgets up. When the budgets are at their largest, the committed size is at its peak. For short running, small benchmarks, the normal pattern is they will allocate static data at the beginning which means GC will observe high survival rates and make the budgets for the relevant generations high so GC will trigger only after it’s allocated quite a bit. And that will be the peak. Later the survival rate will drop which means the budget will be smaller and that means less memory will be allocated before GCs trigger. So the committed size will be smaller.</p><p>Our policy of adjust the budget is the same for segments and regions so in theory the peak should be the same. However, in practice you may observe peak volatility just because the budget could be adjusted quite dramatically. So from run to run the peak may vary. But that volatility applies to both segments and regions. Also GC tuning has changed, even though the budget calculation is the same, we might still be triggering different GCs just because from segments to regions, some of the factors for deciding what generation of GCs to do have changed. For example, we are no longer limited by the “end of segment” factor — for segments, if we are close the end of the ephemeral segment we’d trigger a gen1 GC. That’s no longer a factor for regions as we can just get another region if the budget says we should. To analyze this kind of differences, the best way is to <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#how-to-collect-top-level-gc-metrics">capture a top level GC trace</a>.</p><ul><li><strong>Steady state committed</strong></li></ul><p>Steady state committed for small benchmarks could be noticeably higher with regions because we do not decommit end of region space for regions on the free list. This means you could get maximum almost 1 regions per generation (since the budget is rounded to region size). Of course for small benchmarks because they likely have just one region each for gen1/gen2/LOH/POH anyway, in practice this means just almost 1 region for gen0. Note that this is per heap so if you have many heaps in Server GC, yet your total heap size is small, this factor can add up to a noticeable amount of increase from segments. However, a small heap size with a large number of heaps is not a normal setup.</p><h3><strong>Committed for workloads of different sizes</strong></h3><ul><li><strong>Larger workload</strong></li></ul><p>For large workload these differences become relatively much smaller. If the whole heap is 10GB, it means if you happen to have almost 1 region worth of space per heap that could be decommitted but is not, it’s only a small percentage of the whole heap (eg, if you have 48 heaps, 4MB*48/10GB=2%). If we have 3 such regions per heap (ie, for gen0, gen1 and gen2) which would be 6%. I do expect us to do more on the UOH side as so far we haven’t done much perf analysis for UOH (especially LOH). As you can see this “almost 1 region” factor could be very significant if it happens to be a large region.</p><p>I also expect us to do more work to handle high memory load situations. For starters we’d want to age regions out much faster.</p><ul><li><strong>What to do for small benchmarks with many heaps</strong></li></ul><p>For small benchmarks with low memory usage, you could reduce the region size by setting the DOTNET_GCRegionSize env var (those runtime env vars used to be called COMPlus_X, now we recommend you to rename them to DOTNET_X). It must be power of 2. We recommend to set it to 1MB especially if you have many heaps -</p><p>set DOTNET_GCRegionSize=100000</p><p>Note that we will be doing perf experiments with more workloads in .NET 7 and the default region size might change. The reason why we don’t want the region size to be too small is it increases the cost of searching/sorting.</p><h3><strong>What kind of workloads will have their memory usage improved by regions as it’s implemented now?</strong></h3><p>Obviously we will continue working on taking advantage of the flexibility regions give us throughout .NET 7.0. As of now, the most obvious place that you will notice an improvement in memory usage is if you have very large free spaces on the heap, if they are larger than a region’s size, it means they will naturally be released into the free list and be decommitted after they were not used for a while. So the heap size will naturally be smaller. This situation can (and has been observed to) happen when you are mostly doing only BGCs for gen2 GCs. Since BGC does not compact, its job is to build up free space in gen2 to accommodate gen1 survivors. But if your allocating/surviving pattern changes and at one point your app was consuming a lot more memory, you’ll likely observe that you have increasing large free spaces in gen2/UOH. And regions will definitely help there. I’ve seen folks having free spaces as large as 10s or 100s of MB. Those will be returned to the region free lists and eventually if they are not used anymore, they will be decommitted. In Fig. 3 I’m illustrating an example where we have large free spaces in gen2 and gen0 (it’s very common to see large free spaces in gen0 due to long lived pins — for network scenario you’ll often see in PerfView’s GCStats view that gen0 is pretty sizable and the frag ratio is 99%).</p><p>Fig. 3 An example of the SOH portion of the heap with segments</p><figure><img alt="SOH of a heap with large free spaces" src="https://cdn-images-1.medium.com/max/1024/1*duaGWPDjF5rGFDmG_O2rug.jpeg" /></figure><p>The 1mb free space will be there but most of the other free spaces will be released to free regions and will be decommitted if they are not used. This means your heap’s committed size will be a lot smaller.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*It2c0fl1r_RcTRTcOoBQEQ.jpeg" /></figure><p>If you are running with .NET 6.0 and want to try the .NET 7.0 GC, but don’t want to upgrade to .NET 7.0 (e.g., maybe there’s a problem with the preview build that affects you), you could build a clrgc.dll yourself from the 7.0 branch and drop that into your .NET 6.0 installation next to coreclr.dll. And now you can use the 7.0 GC by setting this environment variable -</p><p>set COMPlus_GCName=clrgc.dll</p><p>The added bonus of doing this is that you are not affected by tons of changes that happened outside of the GC so whatever effect you are seeing is purely from GC changes.</p><p>The relevant defines for building clrgc.dll is in <a href="https://github.com/dotnet/runtime/blob/main/src/coreclr/gc/gcpriv.h">gcpriv.h</a>. If you want to build a clrgc.dll that uses regions, you just need to flip the current !defined (BUILD_AS_STANDALONE) around #define USE_REGIONS to defined (BUILD_AS_STANDALONE) then clrgc.dll would use regions and coreclr.dll would use segments.</p><p>Going forward I’d like to ship clrgc.dll (maybe even 2 versions, one with regions and the other segments) with the .NET SDK (currently it’s not shipped at all which is why you’d need to build it yourself).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68c58465ab5a" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/how-segments-and-regions-differ-in-decommitting-memory-in-the-net-7-gc-68c58465ab5a">How segments and regions differ in decommitting memory in the .NET 7 GC</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>