<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Christophe Nasarre on Medium]]></title>
        <description><![CDATA[Stories by Christophe Nasarre on Medium]]></description>
        <link>https://medium.com/@chnasarre?source=rss-a7db8195a2bf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BIGI7sbOfJJGLivxaNi4aA.jpeg</url>
            <title>Stories by Christophe Nasarre on Medium</title>
            <link>https://medium.com/@chnasarre?source=rss-a7db8195a2bf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:41:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chnasarre/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I’m moving to chrisnas.github.io]]></title>
            <link>https://chnasarre.medium.com/im-moving-to-chrisnas-github-io-aa972d9ff6c6?source=rss-a7db8195a2bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa972d9ff6c6</guid>
            <category><![CDATA[github-pages]]></category>
            <category><![CDATA[githubio]]></category>
            <category><![CDATA[blog]]></category>
            <dc:creator><![CDATA[Christophe Nasarre]]></dc:creator>
            <pubDate>Tue, 21 Apr 2026 11:03:55 GMT</pubDate>
            <atom:updated>2026-04-21T11:03:55.282Z</atom:updated>
            <content:encoded><![CDATA[<p>For easier access, I’ve decided to move my blog to github pages at <a href="https://chrisnas.github.io/">https://chrisnas.github.io/</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k_7tHDm7t66oqJyXZUWiYg.png" /></figure><p>You will be able to easily navigate through the archive, search by text or by tag.</p><p>All existing Medium posts have been migrated but if you find any issue, please feel free to let me know.</p><p>I will try to go back in time and recover older posts from Medium or even older from <a href="https://codenasarre.wordpress.com/">https://codenasarre.wordpress.com/</a> to have everything available at the same place.</p><p>Happy reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa972d9ff6c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to support .NET Framework PDB format and source line with ISymUnmanagedReader]]></title>
            <link>https://chnasarre.medium.com/how-to-support-net-framework-pdb-format-and-source-line-with-isymunmanagedreader-7663f9043b6b?source=rss-a7db8195a2bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/7663f9043b6b</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[dotnet]]></category>
            <category><![CDATA[c]]></category>
            <category><![CDATA[cplusplus]]></category>
            <dc:creator><![CDATA[Christophe Nasarre]]></dc:creator>
            <pubDate>Wed, 11 Feb 2026 09:16:11 GMT</pubDate>
            <atom:updated>2026-02-11T09:16:11.720Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to support .NET Framework PDB format and source line with ISymUnmanagedReader</h3><p>In my previous posts, I explained how to use <a href="https://chnasarre.medium.com/how-to-dump-function-symbols-from-a-pdb-file-9826fd4f251e?source=friends_link&amp;sk=7cc1eec7319633d537f37eec2469e598">DIA</a> and <a href="https://chnasarre.medium.com/but-where-is-my-method-code-dbghelp-comes-to-the-rescue-4188f862211d?source=friends_link&amp;sk=b00736e5a9cf1d31a07528a860043f59">DbgHelp</a> to map a method to its line in source code. I forgot to mention that it was correct for .NET Core but not for the “old” .NET Framework Windows PDB format. Instead of encoding the method token in the name, the symbol file contains the name of the methods. So, how to do the mapping for .NET Framework assemblies? You will find the answer (plus some tricks) in this article.</p><p>When I started to work on the support of the old Windows PDB format, I looked at what existed to parse the <a href="https://github.com/microsoft/microsoft-pdb/tree/master">raw format</a> and… I decided to try DbgHelp instead. With this <a href="https://chnasarre.medium.com/but-where-is-my-method-code-dbghelp-comes-to-the-rescue-4188f862211d?source=friends_link&amp;sk=b00736e5a9cf1d31a07528a860043f59">first implementation</a>, I realized that some token where missing and source code information was not retrieved for most of the methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*ppZFw6o-Ygx7BzpX3jyWIw.png" /></figure><p>So, I looked for another API to use and I found <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/diagnostics/isymunmanagedreader-interface?WT.mc_id=DT-MVP-5003325">ISymUnmanagedReader</a>. The usage philosophy is totally different from DIA or DbgHelp.</p><h3>A little bit of magic</h3><p>This interface is implemented in diasymreader.dll that comes with every .NET Framework installation. But you need to do COM magic to get it. After having called <a href="https://learn.microsoft.com/en-us/windows/win32/api/objbase/nf-objbase-coinitialize?WT.mc_id=DT-MVP-5003325"><strong>CoInitialize</strong></a> to setup COM, you ask for an instance of <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/diagnostics/isymunmanagedbinder-interface?WT.mc_id=DT-MVP-5003325"><strong>ISymUnmanagedBinder</strong></a> from <strong>CLSID_CorSymBinder_SxS</strong>:</p><pre>CComPtr&lt;ISymUnmanagedBinder&gt; pBinder;<br>hr = CoCreateInstance(<br>    CLSID_CorSymBinder_SxS,<br>    NULL,<br>    CLSCTX_INPROC_SERVER,<br>    IID_ISymUnmanagedBinder,<br>    (void**)&amp;pBinder<br>);</pre><p>From the binder, you can get the <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/diagnostics/isymunmanagedreader-interface?WT.mc_id=DT-MVP-5003325"><strong>ISymUnmanagedReader</strong> interface</a> corresponding to the assembly you are interested in with <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/diagnostics/isymunmanagedbinder-getreaderforfile-method?WT.mc_id=DT-MVP-5003325"><strong>GetReaderForFile</strong></a>. However, there are two tiny details to consider.</p><p>First, one parameter expects the path to the assembly, not to the .pdb file. That symbol file has to be stored in the same folder but note that the documentation states that you could have more flexible search with <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/diagnostics/isymunmanagedbinder2-getreaderforfile2-method?WT.mc_id=DT-MVP-5003325"><strong>ISymUnmanagedBinder2::GetReaderForFile2</strong></a> but I did not test it.</p><p>The second detail is the first parameter: an instance of <strong>IMetaDataImport</strong> for the same assembly. The steps to get it are… complicated.</p><h3>Hosting the CLR</h3><p>The idea is to host the .NET Framework and get the corresponding <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/hosting/iclrmetahost-interface?WT.mc_id=DT-MVP-5003325">ICLRMetaHost</a> interface:</p><pre>CComPtr&lt;ICLRMetaHost&gt; pMetaHost;<br>HRESULT hr = CLRCreateInstance(CLSID_CLRMetaHost, IID_ICLRMetaHost, (void**)&amp;pMetaHost);</pre><p>Calling the <a href="https://medium.com/CComPtr%3cICLRMetaHost%3e%20pMetaHost;"><strong>CLRCreateInstance</strong> API</a> allows you to get an instance of <strong>ICLRMetaHost</strong> from which you could <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/hosting/iclrmetahost-enumerateinstalledruntimes-method?WT.mc_id=DT-MVP-5003325">enumerate installed version</a> of .NET Framework. In my case, I know which version I want:</p><pre>// Get the installed .NET Framework runtime (v4.0+)<br>CComPtr&lt;ICLRRuntimeInfo&gt; pRuntimeInfo;<br>hr = pMetaHost-&gt;GetRuntime(L&quot;v4.0.30319&quot;, IID_ICLRRuntimeInfo, (void**)&amp;pRuntimeInfo);</pre><p>The <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/hosting/iclrruntimeinfo-interface?WT.mc_id=DT-MVP-5003325">ICLRRuntimeInfo interface</a> allows you to get access to runtime services via <strong>GetInterface</strong>:</p><pre>CComPtr&lt;IMetaDataDispenser&gt; pDispenser;<br>hr = pRuntimeInfo-&gt;GetInterface(CLSID_CorMetaDataDispenser, IID_IMetaDataDispenser, (void**)&amp;pDispenser);</pre><p>The service I’m interested in is the <a href="https://learn.microsoft.com/en-us/windows/win32/api/rometadataapi/nn-rometadataapi-imetadatadispenser?WT.mc_id=DT-MVP-5003325"><strong>IMetadataDispenser</strong> interface</a> that allows you to “open a scope” on the assembly you are interested in:</p><pre>hr = pDispenser-&gt;OpenScope(<br>    wModulePath.c_str(),<br>    ofRead,<br>    IID_IMetaDataImport,<br>    (IUnknown**)&amp;_pMetaDataImport<br>);</pre><p>Note that the first parameter is the path to the assembly not to the .pdb file. The scope is abstracted by an <strong>IMetadataImport</strong> interface <a href="https://chnasarre.medium.com/dealing-with-modules-assemblies-and-types-with-clr-profiling-apis-a7522a5abaa9?source=friends_link&amp;sk=3e010ab991456db0394d4cca29cb8cb2">I have already described</a> and that is needed to call <strong>GetReaderForFile</strong>: and get the <strong>ISymUnmanagedReader</strong>:</p><p>hr = pBinder-&gt;GetReaderForFile(_pMetaDataImport, wModulePath.c_str(), nullptr, &amp;_pReader);</p><h3>The road to get symbol details for a method</h3><p>The <strong>ISymUnmanagedReader</strong> interface implements <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/diagnostics/isymunmanagedreader-getmethod-method?WT.mc_id=DT-MVP-5003325"><strong>GetMethod</strong></a> to get details about a given method token via an <strong>ISymUnmanagedMethod</strong> interface. So, the next question is how to get these tokens. If you remember <a href="https://chnasarre.medium.com/but-where-is-my-method-code-dbghelp-comes-to-the-rescue-4188f862211d?source=friends_link&amp;sk=b00736e5a9cf1d31a07528a860043f59">the previous article</a>, these tokens are from the 06 MethodDef table in the assembly metadata; starting from <strong>06000001</strong> to the last one.</p><p>This means that you could write a simple loop starting from 1 up to a hardcoded maximum value, call <strong>TokenFromRid(index, mdtMethodDef)</strong> to get the corresponding token. However, since you are a professional developer, you would search for the exact number of tokens from <a href="https://learn.microsoft.com/en-us/dotnet/core/unmanaged-api/metadata/interfaces/imetadatatables-interface?WT.mc_id=DT-MVP-5003325"><strong>IMetadataTables</strong></a> retrieved from <strong>IMetadataImport</strong>:</p><pre>ULONG cRows = 0;<br><br>// Get IMetaDataTables interface to query the MethodDef table<br>CComPtr&lt;IMetaDataTables&gt; pTables;<br>hr = _pMetaDataImport-&gt;QueryInterface(IID_IMetaDataTables, (void**)&amp;pTables);<br>if (FAILED(hr) || pTables == nullptr)<br>{<br>    cRows = LAST_METHODDEF_TOKEN;<br>}<br>else<br>{<br>    // Get the number of rows in the MethodDef table (table index 0x06 = Method)<br>    hr = pTables-&gt;GetTableInfo(<br>        0x06,           // MethodDef table<br>        NULL,           // cbRow (not needed)<br>        &amp;cRows,         // pcRows (number of methods)<br>        NULL,           // pcCols (not needed)<br>        NULL,           // piKey (not needed)<br>        NULL            // ppName (not needed)<br>    );<br><br>    if (FAILED(hr))<br>    {<br>        cRows = LAST_METHODDEF_TOKEN;<br>    }<br>}</pre><p>Now that you have the number of rows (i.e. number of methods defined in the metadata), it is easy and safe to get method information from symbols:</p><pre>for (uint32_t i = 1; i &lt;= cRows; i++)<br>{<br>    mdMethodDef token = TokenFromRid(i, mdtMethodDef);<br><br>    CComPtr&lt;ISymUnmanagedMethod&gt; pMethod;<br>    hr = _pReader-&gt;GetMethod(token, &amp;pMethod);<br>    if (SUCCEEDED(hr) &amp;&amp; pMethod != nullptr)<br>    {<br>        MethodInfo info;<br>        if (GetMethodInfoFromSymbol(pMethod, info))<br>        {<br>            _methods.push_back(info);<br>        }<br>    }<br>}</pre><p>Note that <strong>GetMethod</strong> might fail (returning <strong>E_FAIL</strong>) for P/Invoked functions, abstract methods, or methods decorated with <strong>DebuggerHidden</strong> attribute.</p><h3>Give me line and source code!</h3><p>For the other methods with symbol information, you can get its token via the <strong>GetToken</strong> method. The <strong>ISymUnmanagedMethod</strong> interface allows low level access to line/column mapping that is beyond the scope of this article. At a high level, positions in source file are named <em>sequence points</em>. Call <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/diagnostics/isymunmanagedmethod-getsequencepointcount-method?WT.mc_id=DT-MVP-5003325"><strong>GetSequencePointCount</strong></a> to get… the number of sequence points for a given method.</p><p>The next step is to call <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/diagnostics/isymunmanagedmethod-getsequencepoints-method?WT.mc_id=DT-MVP-5003325">GetSequencePoints</a> with the number of points you want and the corresponding arrays of offsets, lines, columns, end lines, end columns and <strong>ISymUnmanagedDocument</strong>. In my case, I’m only interested in where the method starts so the first sequence point is good enough:</p><pre>// Get sequence points (source line information)<br>ULONG32 cPoints = 0;<br>hr = pMethod-&gt;GetSequencePointCount(&amp;cPoints);<br>if (SUCCEEDED(hr) &amp;&amp; (cPoints &gt; 0))<br>{<br>    cPoints = 1; // We only need the first sequence point for start line<br>    std::vector&lt;ULONG32&gt; offsets(cPoints);<br>    std::vector&lt;ULONG32&gt; lines(cPoints);<br>    std::vector&lt;ULONG32&gt; columns(cPoints);<br>    std::vector&lt;ULONG32&gt; endLines(cPoints);<br>    std::vector&lt;ULONG32&gt; endColumns(cPoints);<br>    std::vector&lt;ISymUnmanagedDocument*&gt; documents(cPoints);<br><br>    ULONG32 actualCount = 0;<br>    hr = pMethod-&gt;GetSequencePoints(<br>        cPoints,<br>        &amp;actualCount,<br>        &amp;offsets[0],<br>        &amp;documents[0],<br>        &amp;lines[0],<br>        &amp;columns[0],<br>        &amp;endLines[0],<br>        &amp;endColumns[0]<br>    );<br><br>    if (SUCCEEDED(hr) &amp;&amp; (actualCount &gt; 0))<br>    {</pre><p>The source file is described by <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/diagnostics/isymunmanageddocument-interface?WT.mc_id=DT-MVP-5003325">ISymUnmanagedDocument</a> that provides its name when <a href="https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/diagnostics/isymunmanageddocument-geturl-method?WT.mc_id=DT-MVP-5003325"><strong>GetURL</strong></a> is called:</p><pre>        // Get the first sequence point&#39;s document and line<br>        ISymUnmanagedDocument* pDoc = documents[0];<br>        if (pDoc != nullptr)<br>        {<br>            // Get document URL (file path)<br>            ULONG32 urlLen = 0;<br>            hr = pDoc-&gt;GetURL(0, &amp;urlLen, NULL);<br>            if (SUCCEEDED(hr) &amp;&amp; (urlLen &gt; 0))<br>            {<br>                std::vector&lt;WCHAR&gt; url(https://medium.com/urlLen);<br>                hr = pDoc-&gt;GetURL(urlLen, &amp;urlLen, &amp;url[0]);<br>                if (SUCCEEDED(hr))<br>                {<br>                    // Convert wide string to narrow string<br>                    int len = WideCharToMultiByte(CP_UTF8, 0, &amp;url[0], urlLen, NULL, 0, NULL, NULL);<br>                    std::string narrowUrl(len, &#39;\0&#39;);<br>                    WideCharToMultiByte(CP_UTF8, 0, &amp;url[0], urlLen, &amp;narrowUrl[0], len, NULL, NULL);<br>                    info.sourceFile = narrowUrl;<br>                }<br>            }<br><br>            // NOTE: 0xFEEFEE is a special value indicating hidden lines<br>            info.lineNumber = lines[0];<br>            pDoc-&gt;Release();<br>        }<br>    }<br>}</pre><p>The final interesting trick is that the line number might have the special <strong>0xFEEFEE</strong> value. It means that the line is hidden. I have seen it for methods generated by the C# compiler such as <strong>MoveNext</strong> for async state machines or anonymous methods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*ZBXLm1oWWKp4pacMcX3g5Q.png" /></figure><p>The source code is available from <a href="https://github.com/chrisnas/DumpManagedMethodInfoFromSymbols">my Github repository</a>.</p><p>Happy coding!</p><h3>References</h3><ul><li><a href="https://github.com/microsoft/microsoft-pdb/tree/master">Archived Microsoft-pdb repository</a></li><li><a href="https://chnasarre.medium.com/how-to-dump-function-symbols-from-a-pdb-file-9826fd4f251e?source=friends_link&amp;sk=7cc1eec7319633d537f37eec2469e598">DIA implementation article</a></li><li><a href="https://chnasarre.medium.com/but-where-is-my-method-code-dbghelp-comes-to-the-rescue-4188f862211d?source=friends_link&amp;sk=b00736e5a9cf1d31a07528a860043f59">DbgHelp implementation article</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7663f9043b6b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[But where is my method code? DbgHelp comes to the rescue]]></title>
            <link>https://chnasarre.medium.com/but-where-is-my-method-code-dbghelp-comes-to-the-rescue-4188f862211d?source=rss-a7db8195a2bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/4188f862211d</guid>
            <category><![CDATA[pdb]]></category>
            <category><![CDATA[dotnet]]></category>
            <category><![CDATA[dbghelp]]></category>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[cplusplus]]></category>
            <dc:creator><![CDATA[Christophe Nasarre]]></dc:creator>
            <pubDate>Fri, 16 Jan 2026 08:21:30 GMT</pubDate>
            <atom:updated>2026-01-16T08:21:30.576Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In our Datatog continuous .NET profiler implementation, we collect the call stack of a thread when something interesting happens such as an exception is thrown for example. In addition to the method name we would like to figure out at what line in which source code file this method is implemented.</p><p>This information is usually stored in the <em>program database</em> (.pdb) file that is generated by the compiler when the assembly is generated from the source code. The type and the name of the method are stored in the metadata of the assembly itself but <a href="https://chnasarre.medium.com/dealing-with-modules-assemblies-and-types-with-clr-profiling-apis-a7522a5abaa9?source=friends_link&amp;sk=3e010ab991456db0394d4cca29cb8cb2">I already told this story before</a>. The .NET compilers support two formats of .pdb: the Portable format for .NET Core and the Windows format for .NET Framework.</p><p>I’ve explained <a href="https://chnasarre.medium.com/how-to-dump-function-symbols-from-a-pdb-file-9826fd4f251e?source=friends_link&amp;sk=7cc1eec7319633d537f37eec2469e598">how to use the DIA API</a> and it is now time to show how to leverage the <strong>DbgHelp</strong> API that is available on all Windows machines (even though <a href="https://learn.microsoft.com/en-us/windows/win32/debug/dbghelp-versions?WT.mc_id=DT-MVP-5003325">it is recommended</a> to always install the latest release via the Debugging Tools for Windows.</p><p>This time, my goal is to extract from a Windows .pdb file the source code and line information for a given managed method. You can find the corresponding source code of this DumpLine tool in <a href="https://github.com/chrisnas/DumpManagedMethodInfoFromSymbols">my Github repository</a>.</p><h3>Starting with DbgHelp</h3><p>There are two major ways to get access to symbols with DbgHelp: either from a running process (i.e. to map the currently loaded .dll to their associated .pdb files) or from a tool that would explicitly load a .dll or a .pdb file.</p><p>Before anything, you tell DbgHelp which options you want by calling <a href="https://learn.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symsetoptions?WT.mc_id=DT-MVP-5003325">SymSetOptions</a>:</p><pre>    DWORD options = SymGetOptions();<br>    options |= SYMOPT_DEBUG;<br>    options |= SYMOPT_LOAD_LINES;           // Load line number information<br>    options |= SYMOPT_UNDNAME;              // Undecorate symbol names<br>    //options |= SYMOPT_DEFERRED_LOADS;       // Defer symbol loading<br>    options |= SYMOPT_EXACT_SYMBOLS;        // Require exact symbol match<br>    options |= SYMOPT_FAIL_CRITICAL_ERRORS; // Don&#39;t show error dialogs<br>    SymSetOptions(options);</pre><p>This is where I ask that line number information should be collected.</p><p>The next step is to call <a href="https://learn.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-syminitialize?WT.mc_id=DT-MVP-5003325">SymInitialize</a> to setup DbgHelp environment. The first parameter expects a process handle (returned by <a href="https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-getcurrentprocess?WT.mc_id=DT-MVP-5003325">GetCurrentProcess</a> in my case). You could pass a path where to find the .pdb files for your dlls as a second parameter. In my case, since I will provide a .pdb file path, I don’t need it, and NULL will be passed. It means that, if needed, DbgHelp will use the current folder and the path set in _NT_SYMBOL_PATH and _NT_ALTERNATE_SYMBOL_PATH environment variables.</p><p>The last boolean parameter tells DbgHelp if you want that <a href="https://learn.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symloadmodule?WT.mc_id=DT-MVP-5003325">SymLoadModule64</a> to be called for each and every loaded .dll in the given process. Definitively not what I want so I’m passing FALSE.</p><pre>    _hProcess = GetCurrentProcess();<br>    if (!SymInitialize(_hProcess, NULL, FALSE))<br>    {<br>        _hProcess = NULL;<br>    }</pre><p>At that point, I’m ready to load a .pdb file.</p><h3>Loading a .pdb file</h3><p>The API is straightforward: just call <a href="https://learn.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symloadmoduleex?WT.mc_id=DT-MVP-5003325">SymLoadModuleEx</a> :</p><pre>    _baseAddress = SymLoadModuleEx(<br>        _hProcess,<br>        NULL,<br>        pdbFilePath.c_str(),<br>        NULL,<br>        0x10000000, // arbitrary base address<br>        0,<br>        NULL,<br>        0<br>    );<br><br>    if (_baseAddress == 0)<br>    {<br>        return false;<br>    }</pre><p>The important parameters are the process handle (same as for <strong>SymInitialize</strong>) and the path of the .pdb file. I’ve lost some time trying to understand why my code was not working due to a weird behavior of this function. You know that it succeeds when the returned address is not 0. Well… This is not 100% correct. If the path you provide does not exist, you won’t get 0 but the base address that you also provide. Even worth, when you call the functions I’ll detail later on, no error will happen but nothing will work as expected. So, I simply check that the file exists:</p><pre>    // BUG? : dbghelp does not fail if the .pdb file does not exist...<br>    if (GetFileAttributesA(pdbFilePath.c_str()) == INVALID_FILE_ATTRIBUTES)<br>    {<br>        return false;<br>    }</pre><p>Note that it is possible to unload the symbols of a given loaded module by calling <a href="https://learn.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symunloadmodule64?WT.mc_id=DT-MVP-5003325">SymUnloadModule</a> with the same process handle and its base address: this will reduce the memory consumption if you don’t need the symbols anymore.</p><p>In case of deferred load symbols, it is needed to call <a href="https://learn.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symgetmoduleinfo64?WT.mc_id=DT-MVP-5003325">SymGetModuleInfo64</a> before trying to access the symbols:</p><pre>    IMAGEHLP_MODULE64 moduleInfo = { 0 };<br>    moduleInfo.SizeOfStruct = sizeof(IMAGEHLP_MODULE64);<br>    if (!SymGetModuleInfo64(_hProcess, _baseAddress, &amp;moduleInfo))<br>    {<br>        return false;<br>    }</pre><p>In addition, this will fill up an <a href="https://learn.microsoft.com/en-us/windows/win32/api/dbghelp/ns-dbghelp-imagehlp_module?WT.mc_id=DT-MVP-5003325">IMAGEHLP_MODULE64 structure</a> with possibly interesting details:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SUaczO2zsWrnmCvQ5Y30OA.png" /></figure><p>The .pdb signature and age could be useful to build urls to communicate with symbol servers; but this is another story:</p><pre>    _age = moduleInfo.PdbAge;<br>    GUID guid = moduleInfo.PdbSig70;<br>    char strGUID[80];<br>    sprintf_s(strGUID, 80, &quot;%08x%04x%04x%02x%02x%02x%02x%02x%02x%02x%02x&quot;,<br>        guid.Data1, guid.Data2, guid.Data3,<br>        guid.Data4[0], guid.Data4[1], guid.Data4[2], guid.Data4[3],<br>        guid.Data4[4], guid.Data4[5], guid.Data4[6], guid.Data4[7]<br>        );<br>    _guid = strGUID;</pre><p>You also know if line numbers are available or not thanks to the <strong>LineNumbers</strong> field.</p><p>Note that if you asked for deferred symbols option, you won’t get any interesting details:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X4Y-pHdVa77r58SrppUR3Q.png" /></figure><p>Only the module name and path are provided but nothing else.</p><h3>Enumerating the methods</h3><p>It is now time to iterate on the symbols in the loaded .pdb thanks to <a href="https://learn.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symenumsymbols?WT.mc_id=DT-MVP-5003325">SymEnumSymbols</a>:</p><pre>    if (!SymEnumSymbols(<br>            _hProcess,<br>            _baseAddress,<br>            &quot;*!*&quot;,  // Mask (all symbols)<br>            EnumMethodSymbolsCallback,<br>            this    // User context to store the methods in _methods instance field<br>    ))<br>    {<br>        return false;<br>    }</pre><p>In addition to the obvious parameters, this function expects a callback function that will be called for each symbol in the module specified by the process handle and the base address. Note that you can pass any context as the last parameter. In my case, the instance of my <strong>DbgHelpParser</strong> class is passed to be able to store the methods in a dedicated <strong>_methods</strong> field:</p><pre>struct MethodInfo<br>{<br>    std::string name;<br>    uint64_t address;<br>    uint32_t size;<br>    std::string sourceFile;<br>    uint32_t lineNumber;<br>};<br>std::vector&lt;MethodInfo&gt; _methods;</pre><p>The “<strong>*!*</strong>” mask tells DbgHelp to look for symbols in all modules. This might sound counter intuitive, but the syntax is similar to what you find in WinDBG or Visual Studio: <strong>&lt;module&gt;!&lt;symbol&gt;</strong>. This could be useful if you load more than one .pdb.</p><p>The job of the callback function is to detect the symbols you are interested in from the <a href="https://learn.microsoft.com/en-us/windows/win32/api/DbgHelp/ns-dbghelp-symbol_info?WT.mc_id=DT-MVP-5003325">SYMBOL_INFO structure</a> passed for each matching symbol:</p><pre>BOOL CALLBACK DbgHelpParser::EnumMethodSymbolsCallback(PSYMBOL_INFO pSymInfo, ULONG SymbolSize, PVOID UserContext)<br>{<br>    DbgHelpParser* parser = reinterpret_cast&lt;DbgHelpParser*&gt;(UserContext);<br><br>    if (<br>        (pSymInfo-&gt;Tag == SymTagFunction) &amp;&amp;<br>        ((pSymInfo-&gt;Flags &amp; (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA)) == (SYMFLAG_CLR_TOKEN | SYMFLAG_METADATA))<br>        )<br>    {</pre><p>The <strong>Tag</strong> field contains a value from <a href="https://learn.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2010/bkedss5f(v=vs.100)?WT.mc_id=DT-MVP-5003325">SymTagEnum</a> but, for a managed .pdb file, you will only get <strong>SymTagFunction</strong>. Also, the <strong>Flags</strong> field should contain SYMFLAG_CLR_TOKEN and SYMFLAG_METADATA because we are only interested in managed methods.</p><p>Next, you get the name, address and size from other fields before looking for the source file and line details by calling <a href="https://learn.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symgetlinefromaddr64?WT.mc_id=DT-MVP-5003325">SymGetLineFromAddr64</a>:</p><pre>        MethodInfo info;<br>        info.name = pSymInfo-&gt;Name;<br>        info.address = pSymInfo-&gt;Address;<br>        info.size = pSymInfo-&gt;Size;<br><br>        // Try to get source file and line information<br>        IMAGEHLP_LINE64 line = { 0 };<br>        line.SizeOfStruct = sizeof(IMAGEHLP_LINE64);<br>        DWORD displacement = 0;<br><br>        if (SymGetLineFromAddr64(parser-&gt;_hProcess, pSymInfo-&gt;Address, &amp;displacement, &amp;line))<br>        {<br>            info.sourceFile = line.FileName ? line.FileName : &quot;&quot;;<br>            info.lineNumber = line.LineNumber;<br>        }<br>        else {<br>            info.sourceFile = &quot;&quot;;<br>            info.lineNumber = 0;<br>        }<br><br>        parser-&gt;_methods.push_back(info);<br>    }<br><br>    return TRUE; // Continue enumeration<br>}</pre><p>The callback returns TRUE to continue the enumeration. You could return FALSE if you would look for specific symbols and wanted to speed up the processing.</p><h3>The managed side of the story</h3><p>When the tool is run on a managed assembly, you get the following kind of output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KW21HTdZwwDwwKI69wlvUg.png" /></figure><p>The name of the method does not match at all with the names in my test assembly! Why do all methods have this generic <strong>Method.#&lt;number&gt;</strong> format?</p><p>Well… the number corresponds to the RID of the corresponding method in the metadata of the assembly. Let’s have a look at what the <strong>MethodDef</strong> metadata table of this assembly looks like in ILSpy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S_vIz4k8maK3smcE2MNdVw.png" /></figure><p>The RID column corresponds to the number in the name in the tool output. So, the <strong>Method.#3</strong> is the <strong>get_Records</strong> property getter in PDBFormatReaderTest.cs:28. And this is exactly what I can see in the test source code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*WZWHUMS4jfWwzomHlrN9pw.png" /></figure><p>You can also check that the lines look correct compared to what is listed by the tool:</p><ul><li><strong>FilePath</strong> getter at line 19</li><li><strong>FilePath</strong> setter at line 20</li><li><strong>Records</strong> getter at line 28</li><li><strong>Records</strong> setter at line 29</li></ul><p>Feel free to look at the source code from <a href="https://github.com/chrisnas/DumpManagedMethodInfoFromSymbols">my Github repository</a>.</p><p>DbgHelp provides many more services to look into symbols but that’s all for today!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4188f862211d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to dump function symbols from a .pdb file]]></title>
            <link>https://chnasarre.medium.com/how-to-dump-function-symbols-from-a-pdb-file-9826fd4f251e?source=rss-a7db8195a2bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/9826fd4f251e</guid>
            <category><![CDATA[cplusplus]]></category>
            <category><![CDATA[cursor]]></category>
            <category><![CDATA[debugging]]></category>
            <dc:creator><![CDATA[Christophe Nasarre]]></dc:creator>
            <pubDate>Mon, 08 Dec 2025 10:12:20 GMT</pubDate>
            <atom:updated>2025-12-08T10:12:20.786Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to dump function symbols from a .pdb file</h3><p>During this R&amp;D week at Datadog, I wanted to implement a tool accepting a .pdb file and generate a .sym file listing functions symbols with their address, size, name with signature and if they are public or private. This post dig into the implementation details of using <a href="https://learn.microsoft.com/en-us/visualstudio/debugger/debug-interface-access/getting-started-debug-interface-access-sdk??WT.mc_id=DT-MVP-5003325">Microsoft Debug Interface Access (DIA) COM API</a> to achieve these objectives. If you want to see what my vibe coding experience in Cursor was, read <a href="https://chnasarre.medium.com/f957106a3e9f?source=friends_link&amp;sk=26ddf8b78e4afc69d171650c082029ae">this other post</a> instead.</p><h3>One self-contained tool please!</h3><p>I would like the tool to be self-contained but since DIA is based on a COM server, it would require registering msdia40.dll on the machine. Not a good idea. In case the dll is in the same folder as the tool, one could “emulate” the magic done by <strong>CoCreateInstance</strong> to get an instance of <strong>IDiaDataSource</strong> (more on this interface soon) by:</p><ol><li>Call <strong>LoadLibrary</strong> to load the dll in memory</li><li>Call <strong>GetProcAddress</strong> to get the <strong>DllGetClassObject</strong> implementation</li><li>Call this function to get the <strong>IClassFactory</strong> implementation</li><li>Call its <strong>CreateInstance</strong> method to get an object implementing <strong>IDiaDataSource</strong></li></ol><p>Here is the corresponding code (without error checking for readability)</p><pre>// Create DIA data source without registration using DLL loading<br>HRESULT PdbSymbolExtractor::NoRegCoCreate(const std::wstring&amp; dllPath, REFCLSID rclsid, REFIID riid, void** ppv) {<br>    HMODULE hDll = LoadLibraryW(dllPath.c_str());<br><br>    typedef HRESULT(__stdcall* DllGetClassObjectFunc)(REFCLSID, REFIID, LPVOID*);<br>    DllGetClassObjectFunc pDllGetClassObject = (DllGetClassObjectFunc)GetProcAddress(hDll, &quot;DllGetClassObject&quot;);<br><br>    CComPtr&lt;IClassFactory&gt; pClassFactory;<br>    HRESULT hr = pDllGetClassObject(rclsid, IID_IClassFactory, (void**)&amp;pClassFactory);<br><br>    hr = pClassFactory-&gt;CreateInstance(NULL, riid, ppv);<br><br>    // Note: We intentionally don&#39;t call FreeLibrary here because the DLL needs to stay loaded<br>    // The COM object references will keep it alive<br>    return S_OK;</pre><p>This function is called with the path of msdia40.dll and the UUID of the expected <strong>IDiaDataSource</strong>:</p><pre>// Create DIA data source without registration<br>hr = NoRegCoCreate(dllPath, CLSID_DiaSource, __uuidof(IDiaDataSource), (void**)&amp;_pDiaDataSource);</pre><p>But still, I don’t want to have two binaries!</p><p>The trick is to embed the msdia40.dll inside the tool as a Windows resource. In the .rc file, add an RCDATA entry that points to the dll:</p><pre>IDR_MSDIA_DLL      RCDATA      &quot;x64\\Release\\msdia140.dll&quot;</pre><p>You should see it in the Resource View in Visual Studio:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*-GYj2ovADruJxXqRMF930A.png" /></figure><p>Here is the code that extracts it as a file on disk is straightforward (error checking has been removed for readability):</p><pre>// Extract embedded msdia140.dll from resources<br>bool PdbSymbolExtractor::ExtractEmbeddedDll(const std::wstring&amp; outputPath) {<br>    // Find the resource<br>    HMODULE hModule = GetModuleHandle(NULL);<br>    HRSRC hResource = FindResource(hModule, MAKEINTRESOURCE(IDR_MSDIA_DLL), RT_RCDATA);<br><br>    // Load the resource<br>    HGLOBAL hLoadedResource = LoadResource(hModule, hResource);<br><br>    // Lock the resource to get a pointer to the data<br>    LPVOID pResourceData = LockResource(hLoadedResource);<br><br>    // Get the size of the resource<br>    DWORD resourceSize = SizeofResource(hModule, hResource);<br><br>    // Write the DLL to disk<br>    std::ofstream outFile(outputPath, std::ios::binary);<br>    outFile.write(static_cast&lt;const char*&gt;(pResourceData), resourceSize);<br>    outFile.close();<br><br>    return true;<br>}</pre><h3>Where are my function symbols?</h3><p>After calling <strong>NoRegCoCreate()</strong>, _<strong>pDiaDataSource</strong> stores a reference to the entry point into the DIA APIs. Here are the steps to follow before being able to list the symbols:</p><pre>HRESULT PdbSymbolExtractor::ExtractSymbolsFromPdb(const std::wstring&amp; pdbPath, std::vector&lt;FunctionSymbol&gt;&amp; symbols) <br>{<br>    // Load the PDB file<br>    HRESULT hr = _pDiaDataSource-&gt;loadDataFromPdb(pdbPath.c_str());<br><br>    // Open a session<br>    CComPtr&lt;IDiaSession&gt; pSession;<br>    hr = _pDiaDataSource-&gt;openSession(&amp;pSession);<br><br>    // Get the global scope<br>    CComPtr&lt;IDiaSymbol&gt; pGlobal;<br>    hr = pSession-&gt;get_globalScope(&amp;pGlobal);</pre><p>Now, you have the global scope of the symbols, you can ask for an enumerator for the type of symbols you are interested in; <strong>SymTagFunction</strong> in my case:</p><pre>    // Enumerate all function symbols<br>    CComPtr&lt;IDiaEnumSymbols&gt; pEnumSymbols;<br>    hr = pGlobal-&gt;findChildren(SymTagFunction, NULL, nsNone, &amp;pEnumSymbols);<br><br>    LONG count = 0;<br>    pEnumSymbols-&gt;get_Count(&amp;count);<br>    std::wcout &lt;&lt; L&quot;Found &quot; &lt;&lt; count &lt;&lt; L&quot; function symbols&quot; &lt;&lt; std::endl;</pre><p>The <strong>pEnumSymbols</strong> iterator allows you to loop on each <strong>SymTagFunction</strong> symbol and get its name:</p><pre>    while (SUCCEEDED(pEnumSymbols-&gt;Next(1, &amp;pSymbol, &amp;celt)) &amp;&amp; celt == 1) {<br>        FunctionSymbol func;<br><br>        // Get function name<br>        BSTR bstrName;<br>        if (pSymbol-&gt;get_name(&amp;bstrName) == S_OK) {<br>            func.name = bstrName;<br>            SysFreeString(bstrName);<br>        }</pre><p>Note that each symbol details are stored in a <strong>FunctionSymbol</strong> instance:</p><pre>struct FunctionSymbol {<br>    std::wstring name;<br>    ...<br>    std::wstring signature; // Function signature (parameters only, no return type)<br>    DWORD rva;              // Relative Virtual Address<br>    ULONGLONG length;<br>    bool isPublic;<br>};</pre><p>with the rest of the code in the <strong>while()</strong> loop:</p><pre>    // Get function signature (parameters only, no return type)<br>    func.signature = ExtractFunctionSignature(pSymbol);<br><br>    // Get relative virtual address<br>    DWORD rva;<br>    if (pSymbol-&gt;get_relativeVirtualAddress(&amp;rva) == S_OK) {<br>        func.rva = rva;<br>    }<br><br>    // Get function length<br>    ULONGLONG length;<br>    if (pSymbol-&gt;get_length(&amp;length) == S_OK) {<br>        func.length = length;<br>    }<br><br>    // Determine if function is public or private<br>    // Check access level - default to private<br>    func.isPublic = false;<br>    DWORD access;<br>    if (pSymbol-&gt;get_access(&amp;access) == S_OK) {<br>        func.isPublic = (access == CV_public);<br>    }<br><br>    pSymbol.Release();</pre><p>I did not have the time to do more trial for private/public state, but I should have tried by enumerating <strong>SymTagPublicSymbol</strong> or <strong>SymTagExport</strong> that could be considered as public.</p><h3>Better with a signature</h3><p>The final step is to figure out the signature of each function. This is where the genericity of DIA could be confusing because so many things are represented by <strong>IDiaSymbol</strong>: a symbol, a function, the type of a function, or the type of a parameter…</p><p>So, the type of the function is retrieved as an <strong>IDiaSymbol</strong> by calling <strong>getType()</strong> on the function symbol. From that <strong>IDiaSymbol</strong>, <strong>findChildren()</strong> lets you iterate on the parameters:</p><pre>// Extract function signature (parameters only, no return type)<br>std::wstring PdbSymbolExtractor::ExtractFunctionSignature(IDiaSymbol* pSymbol) {<br>    if (!pSymbol) {<br>        return L&quot;()&quot;;<br>    }<br><br>    // Get function type<br>    CComPtr&lt;IDiaSymbol&gt; pFunctionType;<br>    if (pSymbol-&gt;get_type(&amp;pFunctionType) != S_OK || !pFunctionType) {<br>        return L&quot;()&quot;;<br>    }<br><br>    // Enumerate function arguments<br>    CComPtr&lt;IDiaEnumSymbols&gt; pEnumArgs;<br>    if (FAILED(pFunctionType-&gt;findChildren(SymTagFunctionArgType, NULL, nsNone, &amp;pEnumArgs))) {<br>        return L&quot;()&quot;;<br>    }<br><br>    LONG argCount = 0;<br>    pEnumArgs-&gt;get_Count(&amp;argCount);<br><br>    if (argCount == 0) {<br>        return L&quot;()&quot;;<br>    }</pre><p>Now, the same <strong>Next()</strong> method is called on the enumerator to iterate on each parameter:</p><pre>    // Build signature string<br>    std::wstring signature = L&quot;(&quot;;<br>    CComPtr&lt;IDiaSymbol&gt; pArg;<br>    ULONG argCelt = 0;<br>    bool first = true;<br><br>    while (SUCCEEDED(pEnumArgs-&gt;Next(1, &amp;pArg, &amp;argCelt)) &amp;&amp; argCelt == 1) {<br>        if (!first) {<br>            signature += L&quot;, &quot;;<br>        }<br>        first = false;<br><br>        // Get the argument type<br>        CComPtr&lt;IDiaSymbol&gt; pArgType;<br>        if (pArg-&gt;get_type(&amp;pArgType) == S_OK &amp;&amp; pArgType) {<br>            signature += GetTypeName(pArgType);<br>        } else {<br>            signature += L&quot;?&quot;;<br>        }<br><br>        pArg.Release();<br>    }<br><br>    signature += L&quot;)&quot;;<br>    return signature;<br>}</pre><p>The final step is to get the name of the type from the <strong>IDiaSymbol</strong> returned by <strong>get_type()</strong>. If it is a custom type, call <strong>get_name()</strong> like any other symbol. Otherwise, for basic types, call <strong>get_baseType()</strong> and <strong>get_length()</strong> as shown by the code below:</p><pre>std::wstring PdbSymbolExtractor::GetTypeName(IDiaSymbol* pType) {<br>    if (!pType) {<br>        return L&quot;?&quot;;<br>    }<br><br>    // Try to get type name directly<br>    BSTR bstrTypeName;<br>    if (pType-&gt;get_name(&amp;bstrTypeName) == S_OK &amp;&amp; bstrTypeName &amp;&amp; wcslen(bstrTypeName) &gt; 0) {<br>        std::wstring typeName = bstrTypeName;<br>        SysFreeString(bstrTypeName);<br>        return typeName;<br>    }<br><br>    // For basic types or unnamed types, try getting basic type info<br>    DWORD baseType = 0;<br>    ULONGLONG length = 0;<br><br>    if (pType-&gt;get_baseType(&amp;baseType) == S_OK) {<br>        pType-&gt;get_length(&amp;length);<br><br>        // Map basic types to names<br>        switch (baseType) {<br>            case btVoid: return L&quot;void&quot;;<br>            case btChar: return L&quot;char&quot;;<br>            case btWChar: return L&quot;wchar_t&quot;;<br>            case btBool: return L&quot;bool&quot;;<br><br>            case btInt:<br>            case btLong:<br>                if (length == 1) return L&quot;char&quot;;<br>                else if (length == 2) return L&quot;short&quot;;<br>                else if (length == 4) return L&quot;int&quot;;<br>                else if (length == 8) return L&quot;__int64&quot;;<br>                else return L&quot;int&quot; + std::to_wstring(length * 8);<br><br>            case btUInt:<br>            case btULong:<br>                if (length == 1) return L&quot;unsigned char&quot;;<br>                else if (length == 2) return L&quot;unsigned short&quot;;<br>                else if (length == 4) return L&quot;unsigned int&quot;;<br>                else if (length == 8) return L&quot;unsigned __int64&quot;;<br>                else return L&quot;uint&quot; + std::to_wstring(length * 8);<br><br>            case btFloat:<br>                if (length == 4) return L&quot;float&quot;;<br>                else if (length == 8) return L&quot;double&quot;;<br>                else return L&quot;float&quot; + std::to_wstring(length * 8);<br><br>            default:<br>                return L&quot;?&quot;;<br>        }<br>    }<br><br>    return L&quot;?&quot;;<br>}</pre><p>This is a “simple” implementation that does not take pointers, addresses, arrays, and more into account. For a more complete solution, I would recommend looking at the <strong>PrintType()</strong> implementation in the DIA2Dump code sample that is installed with Visual Studio.</p><p>I hope this will get your foot in the door of symbol parsing and make you want to dig further into DIA.</p><h3>References</h3><ul><li>Corresponding source code is available in <a href="https://github.com/chrisnas/VibeCoding">my github repository</a>.</li><li>Archived Microsoft <a href="https://github.com/microsoft/microsoft-pdb/tree/master">documentation/implementation of .pdb format</a> including a <a href="https://github.com/microsoft/microsoft-pdb/tree/master/cvdump">symbol dumper</a> code.</li><li><a href="https://learn.microsoft.com/en-us/visualstudio/debugger/debug-interface-access/dia2dump-sample">DIA2Dump</a> Visual Studio code sample.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9826fd4f251e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vibe coding a .pdb dumper or how I became a Product Manager]]></title>
            <link>https://chnasarre.medium.com/vibe-coding-a-pdb-dumper-or-how-i-became-a-product-manager-f957106a3e9f?source=rss-a7db8195a2bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/f957106a3e9f</guid>
            <category><![CDATA[cursor]]></category>
            <category><![CDATA[cplusplus]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[debugging]]></category>
            <dc:creator><![CDATA[Christophe Nasarre]]></dc:creator>
            <pubDate>Mon, 08 Dec 2025 10:12:16 GMT</pubDate>
            <atom:updated>2025-12-08T10:12:16.860Z</atom:updated>
            <content:encoded><![CDATA[<h3>Vibe coding a .pdb dumper or how I became a Product Manager</h3><p>Every quarter at Datadog, we have the opportunity to dig into a specific technical problem during our R&amp;D Week. This time, I needed to list the function symbols from a .pdb file and, for each, figure out if it is public or private, its relative virtual address, its name and its signature plus an obfuscated name. Instead of directly opening my Visual Studio, I decided to really vibe code such a project from scratch.</p><p>This post will detail my journey in vibe coding but, if you are more interested in the technical implementation of dumping these symbols, look at <a href="https://chnasarre.medium.com/9826fd4f251e?source=friends_link&amp;sk=7cc1eec7319633d537f37eec2469e598"><strong>this other post</strong></a> instead.</p><h3>The Setup</h3><p>I started my vibe journey by creating a blank solution and C++ console project with Visual Studio because I had in mind to test how vibe coding helps with refactoring. Next, I simply <a href="https://cursor.com/download">installed Cursor</a> and opened the project as a folder.</p><p>No special setup, no MCP configuration: a plain empty IDE. And that’s it.</p><p>Next, I focused on what I wanted even if I had some ideas. For example, the first implementation I had in mind would have been based on DbgHelp but I simply asked for “<strong>a console application that accepts a .pdb file as input and a .sym file as output where function symbols would be listed line by line with PUBLIC or PRIVATE, followed by its address, its size and its name</strong>”.</p><p>And my adventure started.</p><h3>My first steps</h3><p>It was easy to see what the agent understood (<em>Thought</em> section) and what it would try to do based on its understanding of my prompt:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*5IX8nd4ytz5uc-KNujLfEw.png" /></figure><p>The different steps to achieving this goal then appeared before a final very detailed <em>Summary</em> section:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*JClAZw_DkUW0PBvSwscTMA.png" /></figure><p>This was followed by even more detailed information about the format, usage and how to build:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*xyJ3nFpfeODuF8uQ3ZbQmw.png" /></figure><p>Looks good… before I read the <em>Important Notes</em> that followed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*2clWZkHwniKgz-aqiHXsDw.png" /></figure><p>I was thinking about using DbgHelp API but DIA is even better except that, as mentioned, the corresponding COM object needs to be registered on the machine for the tool to be used. And I did not want that limitation.</p><p>Finally, I had to decide, after reviewing, to keep or discard what has been generated:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*5HMDgMMAw6__KWFTVglF8w.png" /></figure><p>Note that the agent took the initiative to create a readme markdown file with great details and update the vcxproj to use DIA. I did not want to go back to Visual Studio in order to build the project, so I kindly asked: <strong>please let me build the project from the Terminal panel</strong></p><p>I have to admit that my request was not very specific, but the agent got the idea:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*6D9ouW7LS66mKstKofrboQ.png" /></figure><p>It was interesting to follow the different “ideas” that were tried. First, based on directory content (I’m not sure if the solution file was found in the parent directory):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*Nb-0Mgu5N32srEpsPChEnA.png" /></figure><p>When a new command needed to be executed, I preferred to add it to the <strong>AllowList</strong> so that my confirmation won’t be needed again and again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*QYNUgAbDUfRBHcw5RuQQBw.png" /></figure><p>Unfortunately, the command failed but… no problem, another idea was tried:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*IW_d9HcjwKoxn1mHidD1UQ.png" /></figure><p>Long story short, after different errors and trials that were interesting to follow, the build was successful. Fixing a compilation issue is typically a task that you don’t want to waste your time on and it was done for me. In addition, I got a nice summary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*0AbjwXiuh919E8sYKI6_Kg.png" /></figure><p>This is where I started to be impressed: the agent not only decided to create both build.ps1/.bat and clean.ps1/.bat scripts but also to run them to ensure that they were working and update the readme to keep the documentation in sync with the changes! I tried to build by myself in the Terminal (“trust does not exclude control” as my father always said) and everything went well!</p><p>I was confident enough to now accept all changes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*tye2KLlngwbFAnaQdTGQww.png" /></figure><h3>It is testing time!</h3><p>But wait… I had no clue if the generated code was doing what I asked. So, I kindly asked “<strong>Would you please add a project containing functions, struct and classes where methods with different signatures and overloads would be present. It would be used to validate the output of ObfSymbols</strong>”</p><p>After validating the creation of the new project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*B1ggOm6vOWdW-oHfNWyHfg.png" /></figure><p>And accepting that a few files could be updated, I realized that I forgot the DIA COM object registration problem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*VQBzssazlLm-d5MjLbWPtg.png" /></figure><p>So, I asked “<strong>Wait… I realized that I don’t want to force the registration of the DIA COM object: would you please get the corresponding .dll from Visual Studio installation, embed it as a resource and extract it at runtime</strong>”</p><p>Note that I like when Cursor tells how smart I am :^)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*paTlbxrc_kyTK0RoeDKtAA.png" /></figure><p>And then, after the code was modified, I was able to follow the different steps that I would have probably followed myself: get the DIA dll, update and build, run the tool, and test the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*OR0ywNqLwamAqBAlkGulUA.png" /></figure><p>At the end, I got a very detailed summary in the chat, a new IMPLEMENTATION_NOTES.md file was added without forgetting that the readme was also updated with great additional details.</p><p>When looking at the documented project structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*sSP8l9m3GJ4PDA3FPRywww.png" /></figure><p>I realized that the Visual Studio solution in the parent folder was probably forgotten so I asked “<strong>Did you notice that a Visual Studio solution file exists in the parent folder? Would you please update based on all the changes?</strong>”</p><p>And with this vague request, the actions to follow were clearly identified:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*75n1zztKQHfXuQsP2HRz_A.png" /></figure><p>Yes: I did not tell it explicitly but what I had in mind was to include the new testing project!</p><p>And a .sln file was also created for “better compatibility” but even better, new build-all scripts and solution-level readme files were also created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*tpCLWZyuoQmxEKwvgjvY3Q.png" /></figure><p>In less than a couple of hours, I got what I asked without writing a single line of code… myself!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*xcL5qRtN7SLES2YVhATfiQ.png" /></figure><h3>Ask for refactor with no fear</h3><p>First implementations often need to be improved. For example, I wanted to ensure that the symbol dumping code was working for both .exe and .dll. So, I asked to “<strong>update the tests so that symbols from both .dll and .exe are validated</strong>” and a new .dll project was created but also all related files (solution, project, markdown, scripts) were updated accordingly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*Q7OKJ5j-dOcwc3a3bme0Fg.png" /></figure><p>before being automatically tested (i.e. build + run scripts)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*LacmZ2juPIGIJ487jPXRbg.png" /></figure><p>I can’t imagine how much time it would have taken for me to do that!</p><p>These kinds of changes give me confidence in asking for more refactoring. Each developer has her own code and architecture habits. In my case, I like to split responsibility between small classes instead of having a big chunk of code with different responsibilities. Easier to understand, easier to code for objects lifetime, easier to debut and test. The initial implementations for symbol retrieval and output file generation were in the ObfSymbols.cpp file. So, I asked to “<strong>split the code so that symbol listing is implemented by a class and the code that generates the output file stays in ObfSymbols.cpp</strong>”.</p><p>Again, I got a nice summary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*UN9Dfxj2mIRFofvrJxuHvg.png" /></figure><p>But also, key benefits that I was not really interested in:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*yT7SP8e2gMkAdTEG66CpDA.png" /></figure><p>Maybe I could change the global context to express the technical level I’m expecting to avoid this kind of noisy feedback in the chat.</p><p>The <em>Technical Highlights</em> are also listed (including issues the agent had to fix) with the validation results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*i7U0osuIJ3bsQg-9IGbTWg.png" /></figure><p>In case of large code changes, this should help focusing on the areas to review and give me more confidence because build and validations are always run before presenting these results.</p><p>However, you should always take the time to review what was generated because it might not map exactly what you would expect based on your prompt. It was the case for me because the code was still too complicated in the symbol extractor due to both the COM initialization and the parsing code were in the same method. So, I asked “<strong>refactor the PdbSymbolExtractor class to load the DIA COM object and keep the IDiaDataSource as a field in the constructor and use it in LoadPdbAndExtractSymbols(). This should make the code cleaner with easier error handling</strong>”</p><p>I noticed the class fields were not prefixed with _</p><pre>private:<br>    bool initialized;               // True if DIA SDK was loaded successfully<br>    std::wstring lastError;         // Last error message<br>    IDiaDataSource* pDiaDataSource; // DIA data source (initialized in constructor)</pre><p>so, I asked, “<strong>as a coding style, I would like the class fields to be prefixed with _</strong>” to get the expected result:</p><pre>private:<br>    bool _initialized;               // True if DIA SDK was loaded successfully<br>    std::wstring _lastError;         // Last error message<br>    IDiaDataSource* _pDiaDataSource; // DIA data source (initialized in constructor)</pre><p>Sometimes, it could be faster to do this kind of change myself, but I got used to the easiness with the triple checks + automagical markdown updates.</p><h3>Iterate, iterate, and iterate again</h3><p>As I gained more and more confidence in what was generated by Cursor, I decided to concentrate on the missing features or possible improvements.</p><ul><li>When I looked at the generated .sym file, I realized that it would be great to have the functions signature, so I asked to “<strong>add the function signatures after their name but without their return type</strong>”. Since the implementation was inserted in the extraction code, I asked to “<strong>refactor the signature code inside a dedicated method for better readability</strong>”</li><li>In fact, the initial goal of this tool was to generate 2 files. The first one with obfuscated names and the second one for the mapping with the real names. So, I decided to operate in 2 steps:<br> <br>First, I asked to “<strong>change the format by inserting an obfuscated name BETWEEN the size and the real name</strong>” and I needed to be more explicit because the implementation was too simple (i.e. using <em>sub_address</em> as obfuscated name) by asking “<strong>find a better obfuscation implementation and prefix the name with obf_ instead of sub_</strong>”<br> <br>Second, I asked to “<strong>generate a second output file without the real name(signature) to be able to do the mapping between obfuscated and non obfuscated symbols</strong>”. When checking the new obfuscated file, I noticed that the signature was still there, so I asked “<strong>it seems that the signatures are still there in the obfuscated file, please remove them</strong>” and it worked. <br>Note that the final summary contains a section about size reduction<br>that does not really apply to the action that was requested. This is part of the “noise” that appear from time to time in the chat:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*hxT0LwNSs3yw02_0UkAAjQ.png" /></figure><ul><li>When checking a last time before making everything available in my github repository, I realized that the msdia40.dll was not embedded in the tool binary as I asked in my very early prompt. So, I made it more explicit by asking to “<strong>embed the msdia40.dll binary inside the Windows resources and extract it at runtime if it does not exist in the same folder as the tool before loading it dynamically</strong>” and that was it:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*T5-KV3_FEiMHYetVik9ahw.png" /></figure><p>As you can see, leveraging the code generation does not force you to trust and ship. Check and validate both the implementation and the outputs. However, focusing on what I got compared to what I really wanted allowed me to converge on an optimal solution so much more rapidly than if I wrote everything myself, and with automatic validation and documentation!</p><p>Note: once a coding session is over, I recommend exporting the chat so you can review your prompts and the results of your interactions. Click the … button at the top right of the chat panel, click <strong>Export Chat</strong> to select the folder where Cursor will save the whole discussion as markdown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*bbrCQ2Gu0mbyD1Kfp5EC3w.png" /></figure><h3>Conclusion: unexpectedly good initiatives!</h3><p>I’m mostly using different AIs (ChatGPT, DeepSeek and Mistral) as an advanced search engine / code generator for niche APIs usage around profiling. And most of the time, I’m facing too many hallucinations issues that make their usage that useful. The second usage of AI is the Copilot integration in Visual Studio that provides very efficient auto-completion of sometimes large chunks of code such as switch/case or even simple function implementation.</p><p>So, I must admit that I did not expect the experience to be so smooth and the results so correct. When an error happens, the search for a fix starts immediately. Beyond a perfect understanding of what I meant in my prompts, it is interesting to see how the agent is automatically adding documentation and scripts always in synch with the code. It is also possible to provide best practices even in terms of formatting such as no training space characters.</p><p>Talking about the code, its quality is far from being bad in terms of objects lifetime with smart pointers, error management and logs. When I was not satisfied with the implementation, I simply asked for another solution such as with the COM registration issue. I did not ask for a better solution at that time so the agent might have found the same resource embedding trick; who knows? However, the generated code will need more polish before going into production. For example, I did not focus on performance hot path or take edge cases into account such as OMAP support and address collisions. This is where your expertise will let you detect anti-patterns or missing checks.</p><p>Time taken by the agents to implement a feature gave me more time to think about the choices taken by the AI and propose improvements or changes. Also, during the processing of planned steps, I was able to search for other topics. It was like, after one technical decision was taken, letting a not lazy clone of myself deal with the uninteresting and error prone stuff on the side while I was exploring the next improvement step. Being able to focus on each improvement is a real plus in terms of productivity and quality of each feature.</p><p>In the prompts, I never felt the necessity to provide the “how” but just focus on the “what” with as many details as possible. For example, I did not mention how each column of the output .sym file should be computed and it was simply “inferred” from their name. When I detected a problem such as COM registration, I asked for the solution I had in mind. I’m not even sure that I had to be that explicit. Again, focusing on the “what” instead of the “how”. This is exactly what, as a developer, I’m expecting from a product manager! Maybe I’m still more of a developer and not a product manager yet ;^)</p><h3>References</h3><ul><li><a href="https://cursor.com/download">Download Cursor</a></li><li>Corresponding source code and chat history is available in <a href="https://github.com/chrisnas/VibeCoding">my github repository</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f957106a3e9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to monitor .NET applications startup]]></title>
            <link>https://chnasarre.medium.com/how-to-monitor-net-applications-startup-ff3721dc3faf?source=rss-a7db8195a2bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff3721dc3faf</guid>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[dotnet]]></category>
            <dc:creator><![CDATA[Christophe Nasarre]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 10:50:55 GMT</pubDate>
            <atom:updated>2025-03-13T10:50:55.810Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to monitor .NET applications startup</h3><p>In <a href="https://chnasarre.medium.com/measuring-the-impact-of-locks-and-waits-on-latency-in-your-net-apps-a05cd199021d?source=friends_link&amp;sk=033a69ba22254b72a302a6f6a7b28a01">the previous article</a>, I presented what is needed (i.e. listen to <strong>WaitHandleWait</strong> events) to compute lock/wait durations and call stacks for <strong>Mutex</strong>, <strong>Semaphore</strong>, <strong>SemaphoreSlim</strong>, <strong>Manual</strong>/<strong>AutoResetEvent</strong>, <strong>ManualResetEventSlim</strong>, <strong>ReaderWriterLockSlim</strong> .NET synchronization constructs for a running process.</p><p>However, since the application is already running, some JIT-related events are missing, and some frames of the call stacks cannot be symbolized. Also, it would be great to monitor an application’s startup to see if it could be faster.</p><p>This post will detail how to monitor a .NET application since the very beginning of its life and the issues you might face.</p><h3>Preparing a new .NET process to be monitored</h3><p>From .NET 5, the <strong>dotnet-trace</strong> CLI tool allows you to <a href="https://github.com/dotnet/diagnostics/blob/main/documentation/dotnet-trace-instructions.md#using-dotnet-trace-to-launch-a-child-process-and-trace-it-from-startup">pass a command line to execute and trace it from startup</a>. In a <a href="https://medium.com/@ocoanet/tracing-allocations-with-eventpipe-part-3-tracing-without-dotnet-trace-7244bdb86e03">very interesting article</a>, Olivier Coanet presented the gory details about how to tell the .NET runtime to start an application in a pseudo-suspended mode as shown in the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9J_2x2R0n0nkuRTl6MDzZw.png" /></figure><p>The first step is to create a <strong>ReverseDiagnosticsServer</strong> instance with a specific port (i.e. <strong>dotnet-wait_1234</strong> in the diagram). Next, the process to monitor is spawned with the <strong>DOTNET_DiagnosticPorts</strong> environment variable set to the same port (i.e. <strong>dotnet-wait_1234</strong>). Look at the <a href="https://github.com/dotnet/diagnostics/blob/main/documentation/design-docs/ipc-protocol.md#diagnostic-ports">Diagnostics documentation</a> of the Diagnostic Ports with <strong>DOTNET_DiagnosticPorts</strong> environment variable for more details. The .NET runtime is the new process will listen to this port and… wait.</p><p>When the tool is ready, it sends a resume command via a <strong>DiagnosticsClient</strong>: from that point in time, the CLR executes the normal flow of actions to run the application and… you will receive all events without missing one!</p><h3>Get my command line please</h3><p>Following the <a href="https://github.com/dotnet/diagnostics/blob/main/documentation/dotnet-trace-instructions.md#using-dotnet-trace-to-launch-a-child-process-and-trace-it-from-startup"><strong>dotnet-trace</strong> example</a>, my <a href="https://www.nuget.org/packages/dotnet-wait">dotnet-wait</a> tool accepts the command line of the child process in its final arguments that follow the<strong> — </strong>trigger. For example, dotnet-wait — dotnet foo.dll will start the program in foo.dll by using dotnet.exe. I’m reusing <a href="https://github.com/dotnet/diagnostics/blob/main/src/Tools/Common/ReversedServerHelpers/ReversedServerHelpers.cs#L37">the code in ReversedServerHelper.cs</a> to deal with arguments containing spaces:</p><pre>else if (current == &quot;--&quot;)  // this is supposed to be the last one<br>{<br>    i++;<br>    if (i &lt; args.Length)<br>    {<br>        parameters.pathName = args[i];<br><br>        // use the remaining arguments as the arguments for the child app to spawn<br>        i++;<br>        if (i &lt; args.Length)<br>        {<br>            parameters.arguments = &quot;&quot;;<br>            for (int j = i; j &lt; args.Length; j++)<br>            {<br>                if (args[j].Contains(&#39; &#39;))<br>                {<br>                    parameters.arguments += $&quot;\&quot;{args[j].Replace(&quot;\&quot;&quot;, &quot;\\\&quot;&quot;)}\&quot;&quot;;<br>                }<br>                else<br>                {<br>                    parameters.arguments += args[j];<br>                }<br><br>                if (j != args.Length)<br>                {<br>                    parameters.arguments += &quot; &quot;;<br>                }<br>            }<br>        }<br><br>        // no need to look for more arguments<br>        break;<br>    }<br>    else<br>    {<br>        throw new InvalidOperationException($&quot;Missing path name value...&quot;);<br>    }<br>}</pre><p>The code to spawn the child process is simple:</p><pre>// start the monitored app<br>var psi = new ProcessStartInfo(pathName);<br>if (!string.IsNullOrEmpty(arguments))<br>{<br>    psi.Arguments = arguments;<br>}<br>psi.EnvironmentVariables[&quot;DOTNET_DiagnosticPorts&quot;] = port;<br>psi.UseShellExecute = false;<br>var process = System.Diagnostics.Process.Start(psi);</pre><p>Here is an example with the following prompt:</p><pre>-- dotnet &quot;C:\CommandLineTest.dll&quot; one two &quot;t h r e e&quot; four &#39;five six&#39;</pre><p>that generates the output (the test application is just listing its arguments):</p><pre>dotnet-wait v1.0.0.0 - List wait duration<br>by Christophe Nasarre<br><br>Press ENTER to exit...<br>6 arguments<br>   1 | one<br>   2 | two<br>   3 | t h r e e<br>   4 | four<br>   5 | &#39;five<br>   6 | six&#39;</pre><p>This test reminded me to never use simple quotes in prompts :^)</p><h3>It’s my console!</h3><p>Once I implemented these steps, I immediately faced a very simple problem: my <strong>dotnet-wait</strong> tool and the test application are console applications. It means that they will share the same console for both input and output. For example, both are waiting for the RETURN key to (1) stop for the tool and (2) start for the test application: too bad for me because the tool will stop as soon the application starts…</p><p>Going back in time in my Windows memories, I remembered that the Win32 <a href="https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw?WT.mc_id=DT-MVP-5003325">CreateProcess</a> API accepts <a href="https://learn.microsoft.com/en-us/windows/win32/procthread/process-creation-flags?WT.mc_id=DT-MVP-5003325"><strong>CREATE_NEW_CONSOLE</strong> as creation flag</a> to automagically start the child process into its own new console. Unfortunately, it is not possible to pass this flag in .NET; maybe a limitation due to Linux support.</p><p>One simple solution could be to redirect the output of the tool or the application to a file: that would avoid mixing them in the console. Note that, by default, <strong>dotnet-trace</strong> discards output from the child process (by setting <strong>RedirectStandardOutput</strong>, <strong>RedirectStandardError</strong> and <strong>RedirectStandardInput</strong> <a href="https://github.com/dotnet/diagnostics/blob/main/src/Tools/Common/ReversedServerHelpers/ReversedServerHelpers.cs#L95">to false</a> and by <a href="https://github.com/dotnet/diagnostics/blob/main/src/Tools/Common/ReversedServerHelpers/ReversedServerHelpers.cs#L69">ignoring the error and output streams</a>) except if you pass <strong>— show-child-io</strong> on the command line. In this case, no output for <strong>dotnet-trace</strong>.</p><p>I decided to do the opposite for <strong>dotnet-wait</strong>: by default, you also get the child output but you can redirect the output of the tool to a file with <strong>-o &lt;output path name&gt;</strong>. Still, this does not solve the input problem in case of common expected keys.</p><p>If you remember the interactions between the tool and the monitored application, the latter is suspended until <strong>DiagnosticsClient::ResumeRuntime</strong> is called. So, why not starting the tool that spawns the application in one console and another instance of the tool in a new console that will resume the application? This is exactly what my friend <a href="https://x.com/KooKiz">Kevin Gosse</a> imagined and how <strong>dotnet-wait</strong> works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u9iVEg9L5Z_DVpn6jtcM2w.png" /></figure><p>After the timeout that you give to <strong>diagnosticsServer.AcceptAsync(cancellation.Token)</strong> has elapsed, the runtime in the child process will display the following message:</p><pre>The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.<br>DOTNET_DiagnosticPorts=&quot;dotnet-wait_34296&quot;<br>DOTNET_DefaultDiagnosticPortSuspend=0</pre><p>And this is exactly what the <strong>-r 34296</strong> parameter will do!</p><p>You can now install <strong>dotnet wait</strong> and monitor the lock and wait contentions of your .NET9+ applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff3721dc3faf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Measuring the impact of locks and waits on latency in your .NET apps]]></title>
            <link>https://chnasarre.medium.com/measuring-the-impact-of-locks-and-waits-on-latency-in-your-net-apps-a05cd199021d?source=rss-a7db8195a2bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/a05cd199021d</guid>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[latency]]></category>
            <category><![CDATA[dotnet]]></category>
            <category><![CDATA[dotnet-core]]></category>
            <dc:creator><![CDATA[Christophe Nasarre]]></dc:creator>
            <pubDate>Mon, 13 Jan 2025 15:31:03 GMT</pubDate>
            <atom:updated>2025-01-13T15:31:03.206Z</atom:updated>
            <content:encoded><![CDATA[<h3>Measuring the impact of locks and waits on latency in your .NET apps</h3><h3>Introduction</h3><p>In an <a href="https://labs.criteo.com/2018/09/monitor-finalizers-contention-and-threads-in-your-application/">old post</a>, I detailed how to use <strong>ContentionStart</strong> and <strong>ContentionStop</strong> events to measure the lock contentions duration for a .NET application. In a <a href="https://github.com/DataDog/dd-trace-dotnet/issues/5814">.NET 9 pull request</a>, a former Criteo’s colleague <a href="https://www.linkedin.com/in/gregoire-verdier">Grégoire Verdier</a> has added new events to be notified when wait time similar to lock contention is happening for Mutex, Semaphore, Manual/AutoResetEvent. Read <a href="https://techblog.criteo.com/a-perfview-alternative-in-webassembly-f6833820b699">his post</a> for more details about what he was trying to investigate.</p><p>With asynchronous and multi-threaded algorithms, it is essential to detect unexpected wait/locks in our applications. This post shows you how to leverage these events to measure the duration of these waits and get the call stack when the wait started:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*OTO7qWO5aYvNXprhaPvRoA.png" /></figure><h3>New WaitHandleWait events</h3><p>These new events are emitted by the <strong>Microsoft-Windows-DotNETRuntime</strong> CLR provider when you enable the <strong>WaitHandle</strong> (= 0x40000000000) keyword with <strong>Verbose</strong> verbosity. Each time <strong>WaitOne</strong> is called on a waitable object and this object is already owned, a <strong>WaitHandleWaitStart</strong> event is emitted. When the object is released, a <strong>WaitHandleWaitStop</strong> event is emitted.</p><p>For example, the following code:</p><pre>static Mutex mutex = new Mutex();<br><br>static void Main()<br>{<br>    var owningThread = new Thread(OwningThread);<br>    owningThread.Start();<br>    var mutexThread = new Thread(MutexThread);<br>    mutexThread.Start();<br><br>    owningThread.Join();<br>    mutexThread.Join();<br>}<br><br>static void OwningThread()<br>{<br>    Console.WriteLine($&quot;    [{GetCurrentThreadId(), 8}] Start to hold resources&quot;);<br>    Console.WriteLine(&quot;___________________________________________&quot;);<br>    mutex.WaitOne();<br><br>    Thread.Sleep(3000);  // the wait should last ~3 seconds<br><br>    Console.WriteLine(&quot;    Release resources&quot;);<br>    mutex.ReleaseMutex();<br>}<br><br>static void MutexThread()<br>{<br>    Console.WriteLine($&quot;    [{GetCurrentThreadId(), 8}] waiting for Mutex...&quot;);<br>    mutex.WaitOne();  // events are emitted in the implementation when a contention happens<br>    mutex.ReleaseMutex();<br>    Console.WriteLine(&quot;    &lt;-- Mutex&quot;);<br>}</pre><p>generates a Start and Stop events pair:</p><pre>125980 | 00000000-0000-0000-0000-000000000000 &gt; event 301 __ [ 1| Start] WaitHandleWait/Start<br>125980 | 00000000-0000-0000-0000-000000000000 &gt; event 302 __ [ 2|  Stop] WaitHandleWait/Stop</pre><p>There is no associated activity ID so you rely on the fact that the same waiter thread (125980 in the previous example) is emitting for both events.</p><h3>Listening to the new Wait events</h3><p><a href="https://chnasarre.medium.com/implementing-dotnet-http-to-monitor-your-http-requests-04d79422b98d?source=friends_link&amp;sk=470d78e97ec89c1dcfbccaf72a3d4138">As usual</a>, you should rely on the <a href="https://www.nuget.org/packages/Microsoft.Diagnostics.Tracing.TraceEvent/">TraceEvent nuget</a> to start an EventPipe session with an already running .NET application. The last version already contains the definition of the keyword:</p><pre>keywords |= ClrTraceEventParser.Keywords.WaitHandle; // .NET 9 WaitHandle kind of contention</pre><p>and the C# events for Start and Stop:</p><pre>source.Clr.WaitHandleWaitStart += OnWaitHandleWaitStart;<br>source.Clr.WaitHandleWaitStop += OnWaitHandleWaitStop;</pre><p>The handler’s implementation is straightforward. The start of the wait is recorded for the current thread:</p><pre>private void OnWaitHandleWaitStart(WaitHandleWaitStartTraceData data)<br>{<br>    // get the contention info for the current thread<br>    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);<br>    if (info == null)<br>        return;<br><br>    // keep track of the wait start<br>    info.ContentionStartRelativeMSec = data.TimeStampRelativeMSec;<br>}</pre><p>When the wait ends, the duration is computed based on the recorded wait start because it is not provided in the payload <a href="https://github.com/dotnet/runtime/blob/main/src/coreclr/vm/ClrEtwAll.man#L1788">like for ContentionStop</a>:</p><pre>private void OnWaitHandleWaitStop(WaitHandleWaitStopTraceData data)<br>{<br>    ContentionInfo info = _contentionStore.GetContentionInfo(data.ProcessID, data.ThreadID);<br>    if (info == null)<br>        return;<br>    // unlucky case when we start to listen just after the WaitHandleStart event<br>    if (info.ContentionStartRelativeMSec == 0)<br>    {<br>        return;<br>    }<br><br>    // Too bad the duration is not provided in the payload like in ContentionStop...<br>    var contentionDurationMSec = data.TimeStampRelativeMSec - info.ContentionStartRelativeMSec;<br>    info.ContentionStartRelativeMSec = 0;<br>    var duration = TimeSpan.FromMilliseconds(contentionDurationMSec);<br>    Console.WriteLine($&quot;{e.ThreadId,7} | {e.Duration.TotalMilliseconds} ms&quot;);<br>}</pre><p>This is nice but it would be more useful if we could get the call stack of long waits.</p><h3>Call stacks with EventPipe</h3><p>In a <a href="https://techblog.criteo.com/build-your-own-net-memory-profiler-in-c-call-stacks-2-2-1-f67b440a8cc">previous post</a>, I explained that it is possible to get the call stack when an event is emitted thanks to the <a href="https://learn.microsoft.com/en-us/dotnet/framework/performance/stack-etw-event?WT.mc_id=DT-MVP-5003325"><strong>ClrStackWalk</strong> event</a> that follows the event you are interested in. Unfortunately, this is not more the case for .NET 5+ that is using EventPipe instead of ETW.</p><p>As <a href="https://x.com/ocoanet">Olivier Coanet</a> presents in his <a href="https://medium.com/@ocoanet/tracing-allocations-with-eventpipe-part-2-reading-call-stacks-without-tracelog-4b0bfe4592aa">post</a>, you can get the call stack as an array of addresses from the hidden event record that is mapped by the <strong>TraceEvent</strong> parameter passed to each event handlers. This <a href="https://learn.microsoft.com/en-us/windows/win32/api/evntcons/ns-evntcons-event_record?WT.mc_id=DT-MVP-5003325"><strong>EVENT_RECORD</strong></a> structure contains a <strong>ExtendedData</strong> field that is an array of <a href="https://learn.microsoft.com/en-us/windows/win32/api/evntcons/ns-evntcons-event_header_extended_data_item?WT.mc_id=DT-MVP-5003325"><strong>EVENT_HEADER_EXTENDED_DATA_ITEM</strong></a>:</p><pre>public struct EVENT_HEADER_EXTENDED_DATA_ITEM<br>{<br>    public ushort Reserved1;<br>    public ushort ExtType;<br>    public ushort Reserved2;<br>    public ushort DataSize;<br>    public ulong DataPtr;<br>}</pre><p>If the <strong>ExtType</strong> value is <strong>EVENT_HEADER_EXT_TYPE_STACK_TRACE64</strong> (=6) then <strong>DataPtr</strong> points to a <a href="https://learn.microsoft.com/en-us/windows/win32/api/evntcons/ns-evntcons-event_extended_item_stack_trace64?%3FWT.mc_id=DT-MVP-5003325"><strong>EVENT_EXTENDED_ITEM_STACK_TRACE64</strong></a> structure:</p><pre>public struct EVENT_EXTENDED_ITEM_STACK_TRACE64<br>{<br>    public ulong MatchId;<br>    public unsafe fixed ulong Address[1];<br>}</pre><p>that contains an array of 64-bit addresses. The size of this array is given by <strong>DataSize — sizeof(ulong)</strong>.</p><p>For 32-bit applications, you will get <strong>EVENT_HEADER_EXT_TYPE_STACK_TRACE32</strong> (=5) as <strong>ExtType</strong> value and DataPtr will point to <a href="https://learn.microsoft.com/en-us/windows/win32/api/evntcons/ns-evntcons-event_extended_item_stack_trace32?WT.mc_id=DT-MVP-5003325"><strong>EVENT_EXTENDED_ITEM_STACK_TRACE32</strong></a>:</p><pre>public struct EVENT_EXTENDED_ITEM_STACK_TRACE32<br>{<br>    public ulong MatchId;<br>    public unsafe fixed uint Address[1];<br>}</pre><p>that stores an array of 32-bit addresses.</p><p>Knowing that makes writing the code to get the call stacks as an array of 64-bit addresses (same with 32-bit applications for simplicity sake) pretty straightforward:</p><pre>public static EventPipeUnresolvedStack ReadStackUsingUnsafeAccessor(TraceEvent traceEvent)<br>{<br>    return GetFromEventRecord(traceEvent.eventRecord);<br>}<br><br>private static EventPipeUnresolvedStack GetFromEventRecord(TraceEventNativeMethods.EVENT_RECORD* eventRecord)<br>{<br>    if (eventRecord == null)<br>        return null;<br><br>    var extendedDataCount = eventRecord-&gt;ExtendedDataCount;<br><br>    for (var dataIndex = 0; dataIndex &lt; extendedDataCount; dataIndex++)<br>    {<br>        var extendedData = eventRecord-&gt;ExtendedData[dataIndex];<br>        if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE64)<br>        {<br>            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE64*)extendedData.DataPtr;<br>            var addresses = &amp;stackRecord-&gt;Address[0];<br>            var addressCount = (extendedData.DataSize - sizeof(UInt64)) / sizeof(UInt64);<br>            if (addressCount == 0)<br>                return null;<br><br>            var callStackAddresses = new ulong[addressCount];<br>            for (var index = 0; index &lt; addressCount; index++)<br>            {<br>                callStackAddresses[index] = addresses[index];<br>            }<br>            return new EventPipeUnresolvedStack(callStackAddresses);<br>        }<br>        else if (extendedData.ExtType == TraceEventNativeMethods.EVENT_HEADER_EXT_TYPE_STACK_TRACE32)<br>        {<br>            var stackRecord = (TraceEventNativeMethods.EVENT_EXTENDED_ITEM_STACK_TRACE32*)extendedData.DataPtr;<br>            var addresses = &amp;stackRecord-&gt;Address[0];<br>            var addressCount = (extendedData.DataSize - sizeof(UInt32)) / sizeof(UInt32);<br>            if (addressCount == 0)<br>                return null;<br><br>            var callStackAddresses = new ulong[addressCount];  // store the 32 addresses as 64 bit addresses<br>            for (var index = 0; index &lt; addressCount; index++)<br>            {<br>                callStackAddresses[index] = addresses[index];<br>            }<br><br>            return new EventPipeUnresolvedStack(callStackAddresses);<br>        }<br>    }<br><br>    return null;<br>}</pre><p>Note that the last version of TraceEvent nuget provides a public access to the <strong>eventRecord</strong> field so it is no more needed to use the <strong>UnsafeAccessor</strong> attribute used by Olivier.</p><h3>Symbolize the call stack addresses</h3><p>Address is good but the corresponding method name is better. I won’t repeat what I’ve already detailed in <a href="https://techblog.criteo.com/build-your-own-net-memory-profiler-in-c-call-stacks-2-2-2-ec9657eb17f9?source=friends_link&amp;sk=b34465f583867cb7dcf5bad6395bf151">an older post</a> that shows how to get the name of a native and managed name from an instruction pointer address. Instead, I want to pinpoint a big limitation of this solution to listen to CLR provider <strong>MethodLoadVerbose</strong>/<strong>MethodDCStartVerboseV2</strong> events. If the methods you are interested in are jitted BEFORE your tool attaches to the application, you will never get these events.</p><p>You could get the same mapping address span/method name via the other “<em>Microsoft-Windows-DotNETRuntimeRundown</em>” provider and its <strong>MethodDCEndVerbose</strong> event that contains the expected <strong>MethodStartAddress</strong>, <strong>MethodSize</strong> and <strong>MethodName</strong> in its <a href="https://github.com/dotnet/runtime/blob/d897415e02340a13dc1c5078c09937bdf7ec8a56/src/coreclr/vm/ClrEtwAll.man#L4864">payload</a>. But I need this information before the end of the application…</p><p>Looking at <a href="https://github.com/dotnet/docs/blob/main/docs/framework/performance/clr-etw-keywords-and-levels.md#keyword-combinations-for-symbol-resolution-for-the-rundown-provider">the documentation</a>, it seems that the rundown provider accepts the <strong>StartRundownKeyword</strong> value to emit the DCStart events when the provider is enabled! <a href="https://github.com/dotnet/runtime/issues/42378">Since .NET 9</a>, it is possible to pass the keywords you want (before, the default value did not contain <strong>StartRundownKeyword</strong>) when creating the EventPipe session</p><pre>//                V-- this is the default rundown keyword<br>rundownKeywords = 0x80020139 | (long)ClrTraceEventParser.Keywords.StartEnumeration;<br>var config = new EventPipeSessionConfiguration(GetProviders(), 256, rundownKeywords, true);<br>using (var session = client.StartEventPipeSession(config))<br>{<br>    var source = new EventPipeEventSource(session.EventStream);<br>    RegisterListeners(source);<br><br>    // this is a blocking call<br>    source.Process();<br>}</pre><p>Note that you should not add the rundown provider to the list passed as parameter.</p><p>Unfortunately, there is <a href="https://github.com/dotnet/runtime/issues/42378">currently an issue in the runtime since September 2020</a> that pinpoints this exact problem. I even tried to create and close a session to get the DCStop events before recreating a new one, but I failed.</p><p>The next episode will talk about how it is possible to start a .NET application and get the events since its startup… with the problems that are happening.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a05cd199021d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitor HTTP redirects to reduce unexpected latency]]></title>
            <link>https://chnasarre.medium.com/monitor-http-redirects-to-reduce-unexpected-latency-56e13d7c5899?source=rss-a7db8195a2bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/56e13d7c5899</guid>
            <category><![CDATA[http-request]]></category>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[dotnet]]></category>
            <dc:creator><![CDATA[Christophe Nasarre]]></dc:creator>
            <pubDate>Fri, 13 Dec 2024 14:21:41 GMT</pubDate>
            <atom:updated>2024-12-13T14:21:41.590Z</atom:updated>
            <content:encoded><![CDATA[<p>In the <a href="https://chnasarre.medium.com/implementing-dotnet-http-to-monitor-your-http-requests-04d79422b98d?source=friends_link&amp;sk=470d78e97ec89c1dcfbccaf72a3d4138">previous post</a>, I detailed how I used the undocumented events from the BCL to create the <a href="https://www.nuget.org/packages/dotnet-http">dotnet-http CLI tool</a> to monitor your outgoing HTTP requests. After testing with older versions of .NET, I realized that the code needed to be updated and I’m sharing my findings in this post.</p><p>The main point is that url redirections could have a major impact on requests latency:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w_2f8F9E6hGmoiGLBCMqfw.png" /></figure><h3>Always test supported versions…</h3><p>When I wrote the initial version of dotnet-http, I only tested it with .NET 8 and .NET 9 with limited formats of urls. Unfortunately, things went bad when I tried to monitor applications running on .NET 5 and .NET 6: no events are emitted by these versions of the BCL.</p><p>So, the next step was to test .NET 7 and the result was simple: crash! After investigating, I realized that some events I looked at in .NET 8 source code did not have the same payload in .NET 7; even no payload at all:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2HJxVmyxQfzhCr1i7_dpWw.png" /></figure><p>Even though there is no version field in the events payload, it is easy to check its size such as the following:</p><pre>private void OnConnectionEstablished(<br>     DateTime timestamp,<br>     int threadId,<br>     Guid activityId,<br>     Guid relatedActivityId,<br>     byte[] eventData<br>     )<br> {<br>     ...<br>     EventSourcePayload payload = new EventSourcePayload(eventData);<br>     var versionMajor = payload.GetByte();<br>     var versionMinor = payload.GetByte();<br><br>     // in .NET 7, nothing else is available<br>     Int64 connectionId = 0;<br>     var scheme = &quot;&quot;;<br>     var host = &quot;&quot;;<br>     UInt32 port = 0;<br>     var path = &quot;&quot;;<br><br>     if (eventData.Length &gt; 2)<br>     {<br>         connectionId = payload.GetInt64();<br>         scheme = payload.GetString();<br>         host = payload.GetString();<br>         port = payload.GetUInt32();<br>         path = payload.GetString();<br>     }<br>     ...</pre><p>Another difference is that one even is not even emitted in .NET 7:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Iwpe5YVHJwyBnRFvHMTiQw.png" /></figure><h3>Explain what a redirection is please!</h3><p>Because I based the code on this Redirect event, I needed to find another way to support .NET 7 even though I would not have a redirected url to display. But first, let’s see what I’m talking about in terms of HTTP communication.</p><p>When you try to get the content / status code behind a url, the code is following different phases with the related events:</p><ol><li><strong>Start<br></strong>RequestStart</li><li><strong>DNS resolution<br></strong>ResolutionStart<br>ResolutionStop/Fail</li><li><strong>Socket connection<br></strong>ConnectStart<br>ConnectStop</li><li><strong>Security hand check (HTTPS only)<br></strong>HandshakeStart<br>HandshakeStop/Failed</li><li><strong>Request/response<br></strong>RequestHeadersStart<br>RequestHeadersStop<br>ResponseHeadersStart<br>ResponseHeadersStop<br>Redirect (.NET 8+)<br>ResponseContentStart<br>ResponseContentStop</li><li><strong>Request stop<br></strong>RequestStop/Failed</li></ol><p>Based on the received url, a server can decide to answer that another url should be used instead. For example, if you call github with <strong>http://</strong> instead of <a href="https://,/"><strong>https://</strong>,</a> such a redirection will happen. Without invasive tools such as Wireshark, these redirections are impossible to detect and could cause unnecessary delay.</p><p>From the client perspective, this can be detected in the <strong>ResponseHeadersStop</strong> event payload that provides a status code. If its value is 301, then it is a redirection. The other effect of a redirection is that the BCL code will change the flow of events because it needs to start over with the new url from step 2. to step 6. As you can see, instead of paying the cost of just one request, two are actually emitted and processed.</p><h3>Impact on the implementation</h3><p>In addition to the payload size checks addition, my initial implementation was not properly handling the redirection because the values (timestamps and durations) where overridden by the events related to the second redirected url.</p><p>The new implementation is splitting the request details into two classes. A base class that contains common fields to both parts of the request in case of redirection:</p><pre>   private class HttpRequestInfoBase<br>   {<br>       public HttpRequestInfoBase(DateTime timestamp, string scheme, string host, uint port, string path)<br>       {<br>           StartTime = timestamp;<br>           if (scheme == string.Empty)<br>           {<br>               Url = string.Empty;<br>           }<br>           else<br>           {<br>               if (port != 0)<br>               {<br>                   Url = $&quot;{scheme}://{host}:{port}{path}&quot;;<br>               }<br>               else<br>               {<br>                   Url = $&quot;{scheme}://{host}:{path}&quot;;<br>               }<br>           }<br>       }<br><br>       public string Url { get; set; }<br>       public DateTime StartTime { get; set; }<br>       public DateTime ReqRespStartTime { get; set; }<br>       public double ReqRespDuration { get; set; }<br><br>       // DNS<br>       public double DnsWait { get; set; }<br>       public DateTime DnsStartTime { get; set; }<br>       public double DnsDuration { get; set; }<br><br>       // HTTPS<br>       public double HandshakeWait { get; set; }<br>       public DateTime HandshakeStartTime { get; set; }<br>       public double HandshakeDuration { get; set; }<br><br>       // socket connection<br>       public DateTime SocketConnectionStartTime { get; set; }<br>       public double SocketWait { get; set; }<br>       public double SocketDuration { get; set; }<br><br><br>       public DateTime QueueuingEndTime { get; set; }<br>       public double QueueingDuration { get; set; }<br>   }</pre><p>The second one inherits from the base class and contains addition details; including the details of the redirected url if any:</p><pre>   private class HttpRequestInfo : HttpRequestInfoBase<br>   {<br>       public HttpRequestInfo(DateTime timestamp, string scheme, string host, uint port, string path)<br>           :<br>           base(timestamp, scheme, host, port, path)<br>       {<br>       }<br><br>       public HttpRequestInfoBase Redirect { get; set; }<br><br>       public UInt32 StatusCode { get; set; }<br><br>       // HTTPS<br>       public string HandshakeErrorMessage { get; set; }<br><br>       public string Error { get; set; }<br>   }</pre><p>A new instance of <strong>HttpRequestInfoBase</strong> is created when a 301 status code is received in <strong>HttpResponseHeaderStop</strong> handler:</p><pre>    private void OnHttpResponseHeaderStop(object sender, HttpRequestStatusEventArgs e)<br>    {<br>        // used to detect redirection in .NET 8+<br>        if (e.StatusCode != 301)<br>        {<br>            return;<br>        }<br><br>        // create a new request info for the redirected request<br>        // because .NET 7 does not emit a Redirect event, we need to create a new request info here<br>        // --&gt; it means that the redirect url will be empty in .NET 7<br>        var root = GetRoot(e.ActivityId);<br>        if (_requests.TryGetValue(root, out HttpRequestInfo info))<br>        {<br>            info.Redirect = new HttpRequestInfoBase(e.Timestamp, &quot;&quot;, &quot;&quot;, 0, &quot;&quot;);<br><br>            // if you really want to have the duration of both original request + redirected request,<br>            // then do the following:<br>            //    info.ReqRespDuration = (e.Timestamp - info.ReqRespStartTime).TotalMilliseconds;<br>            // However, I prefer to show the duration of the redirected request only to more easily<br>            // compute the cost of the initial redirected request = total duration - other durations<br>        }<br>    }</pre><p>For .NET 8+, the redirected url is provided to the <strong>Redirect</strong> handler and stored in the Url field of the instance created in the previous handler:</p><pre>    private void OnHttpRedirect(object sender, HttpRedirectEventArgs e)<br>    {<br>        // since this is an Info event, the activityID is the root<br>        var root = ActivityHelpers.ActivityPathString(e.ActivityId);<br>        if (_requests.TryGetValue(root, out HttpRequestInfo info))<br>        {<br>            info.Redirect.Url = e.RedirectUrl;<br>        }<br>    }</pre><p>In each handler of events that could be received for both initial and redirected requests,</p><pre>   private void OnHttpResponseContentStop(object sender, EventPipeBaseArgs e)<br>   {<br>       var root = GetRoot(e.ActivityId);<br>       if (!_requests.TryGetValue(root, out HttpRequestInfo info))<br>       {<br>           return;<br>       }<br><br>       if (info.Redirect == null)<br>       {<br>           info.ReqRespDuration = (e.Timestamp - info.ReqRespStartTime).TotalMilliseconds;<br>       }<br>       else<br>       {<br>           info.Redirect.ReqRespDuration = (e.Timestamp - info.Redirect.ReqRespStartTime).TotalMilliseconds;<br>       }<br>   }</pre><p>The wait time and durations are now computed as the events are received and aggregated at the end of the request by adding the value of both parts (initial and redirected if any:</p><pre>    double dnsDuration = info.DnsDuration + ((info.Redirect != null) ? info.Redirect.DnsDuration : 0);<br>    if (dnsDuration &gt; 0)<br>    {<br>        double dnsWait = info.DnsWait + ((info.Redirect != null) ? info.Redirect.DnsWait : 0);<br>        Console.Write($&quot;{dnsWait,9:F3} | {dnsDuration,9:F3} | &quot;);<br>    }<br>    else<br>    {<br>        Console.Write($&quot;          |           | &quot;);<br>    }</pre><p>As a conclusion, you should try to monitor these redirections by using my <a href="https://www.nuget.org/packages/dotnet-http">dotnet-http</a> CLI tool. Feel free to download it or install it with the following command line: <strong>dotnet tool install -g dotnet-http </strong>or update to the latest version with<strong>: dotnet tool update -g dotnet-http</strong></p><p>You could also integrate some event listening code into your framework that simply handles the <strong>ResponseHeadersStop</strong>/<strong>Redirect</strong> events.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56e13d7c5899" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing dotnet-http to monitor your HTTP requests]]></title>
            <link>https://chnasarre.medium.com/implementing-dotnet-http-to-monitor-your-http-requests-04d79422b98d?source=rss-a7db8195a2bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/04d79422b98d</guid>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[http-request]]></category>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[dotnet]]></category>
            <dc:creator><![CDATA[Christophe Nasarre]]></dc:creator>
            <pubDate>Wed, 13 Nov 2024 13:18:37 GMT</pubDate>
            <atom:updated>2024-11-25T08:44:47.495Z</atom:updated>
            <content:encoded><![CDATA[<p>The <a href="https://chnasarre.medium.com/digging-into-the-undocumented-net-events-from-the-bcl-4e836989ade9?source=friends_link&amp;sk=b1d7bb35fc0b83e6d620383f905acb45">previous episode</a> detailed how to find the events dealing with network requests that are emitted by the BCL classes with their undocumented payload. It is now time to see how to listen to them and extract valuable insights such as what is happening when an HTTP request is sent to a server as an example. This is how my new <a href="https://www.nuget.org/packages/dotnet-http">dotnet-http CLI tool</a> is implemented.</p><p>You are now able to see the cost of DNS, socket connection, security and redirection as shown in the following screenshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xx8OW9oP34V6fVKKdxqZQA.png" /></figure><p>As you can see, the cost of an unexpected redirection could be high and hides security handshakes. In this example, using <a href="https://github.com/Maoni0"><strong>https</strong>://github.com/Maoni0</a> instead of <a href="http://github.com/Maoni0"><strong>http</strong>://github.com/Maoni0</a> divides by 2 the request duration!</p><p>Once the DNS checks are done, the socket connections are established, and the security handshakes are validated, the corresponding phases are no more needed for the next requests.</p><h3>Listen to custom EventSource events</h3><p>As explained in previous posts, it is easy to listen to events emitted by .NET application thanks to the Microsoft TraceEvent nuget. You also need to use <strong>EventPipeClient</strong> from the <a href="https://www.nuget.org/packages/Microsoft.Diagnostics.NETCore.Client">Microsoft.Diagnostics.NETCore.Client</a> nuget to connect to the running application. In my case, I’m relying on an older version that supports both ETW and EventPipe:</p><pre>var configuration = new SessionConfiguration(<br>    circularBufferSizeMB: 2000,<br>    format: EventPipeSerializationFormat.NetTrace,<br>    providers: GetProviders()<br>);<br>var binaryReader = EventPipeClient.CollectTracing(_processId, configuration, out var sessionId);<br>EventPipeEventSource source = new EventPipeEventSource(binaryReader);</pre><p>The configuration contains a list of providers that are emitting the events you are interested in with the right keyword and verbosity. Here is what is needed for the HTTP requests monitoring :</p><pre>private static IReadOnlyCollection&lt;Provider&gt; GetProviders()<br>{<br>    var providers = new Provider[]<br>    {<br>        new Provider(<br>            name: &quot;System.Net.Http&quot;,<br>            keywords: (ulong)(1),<br>            eventLevel: EventLevel.Verbose),<br><br>        new Provider(<br>            name: &quot;System.Net.Sockets&quot;,<br>            keywords: (ulong)(0xFFFFFFFF),<br>            eventLevel: EventLevel.Verbose),<br><br>        new Provider(<br>            name: &quot;System.Net.NameResolution&quot;,<br>            keywords: (ulong)(0xFFFFFFFF),<br>            eventLevel: EventLevel.Verbose),<br><br>        new Provider(<br>            name: &quot;System.Net.Security&quot;,<br>            keywords: (ulong)(0xFFFFFFFF),<br>            eventLevel: EventLevel.Verbose),<br><br>        new Provider(<br>            name: &quot;System.Threading.Tasks.TplEventSource&quot;,<br>            keywords: (ulong)(0x80),<br>            eventLevel: EventLevel.Verbose),<br><br>    };<br><br>    return providers;<br>}</pre><p>You might be surprised by the presence of the <strong>TplEventSource</strong> provider but it is required to get a correct <strong>ActivityID</strong>. This major subject is detailed later.</p><p>There are so many events to listen to that it is easier to listen to the source <strong>AllEvents</strong> C# event:</p><pre>source.AllEvents += OnEvents;</pre><p>This global handler passes the events to <strong>ParseEvent</strong> that forwards the important data to handlers for each provider:</p><pre>private static Guid NetSecurityEventSourceProviderGuid = Guid.Parse(&quot;7beee6b1-e3fa-5ddb-34be-1404ad0e2520&quot;);<br>private static Guid DnsEventSourceProviderGuid = Guid.Parse(&quot;4b326142-bfb5-5ed3-8585-7714181d14b0&quot;);<br>private static Guid SocketEventSourceProviderGuid = Guid.Parse(&quot;d5b2e7d4-b6ec-50ae-7cde-af89427ad21f&quot;);<br>private static Guid HttpEventSourceProviderGuid = Guid.Parse(&quot;d30b5633-7ef1-5485-b4e0-94979b102068&quot;);<br><br>private void ParseEvent(<br>    DateTime timestamp,<br>    int threadId,<br>    Guid activityId,<br>    Guid relatedActivityId,<br>    Guid providerGuid,<br>    string taskName,<br>    Int64 keywords,<br>    UInt16 id,<br>    byte[] eventData<br>    )<br>{<br>    if (providerGuid == NetSecurityEventSourceProviderGuid)<br>    {<br>        HandleNetSecurityEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);<br>    }<br>    else<br>    if (providerGuid == DnsEventSourceProviderGuid)<br>    {<br>        HandleDnsEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);<br>    }<br>    else<br>    if (providerGuid == SocketEventSourceProviderGuid)<br>    {<br>        HandleSocketEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);<br>    }<br>    else<br>    if (providerGuid == HttpEventSourceProviderGuid)<br>    {<br>        HandleHttpEvent(timestamp, threadId, activityId, relatedActivityId, id, taskName, eventData);<br>    }<br>    else<br>    {<br>        WriteLogLine();<br>    }<br>}</pre><p>The next step is to handle each event based on its id:</p><pre>private void HandleNetSecurityEvent(<br>    DateTime timestamp,<br>    int threadId,<br>    Guid activityId,<br>    Guid relatedActivityId,<br>    ushort id,<br>    string taskName,<br>    byte[] eventData<br>    )<br>{<br>    switch (id)<br>    {<br>        case 1: // HandshakeStart<br>            OnHandshakeStart(timestamp, threadId, activityId, relatedActivityId, eventData);<br>            break;<br>        case 2: // HandshakeStop<br>            OnHandshakeStop(timestamp, threadId, activityId, relatedActivityId, eventData);<br>            break;<br>        case 3: // HandshakeFailed<br>            OnHandshakeFailed(timestamp, threadId, activityId, relatedActivityId, eventData);<br>            break;<br>        default:<br>            WriteLogLine();<br>            break;<br>    }<br>}</pre><h3>Extract information from an event payload</h3><p>The payload of each event has been detailed in <a href="https://chnasarre.medium.com/digging-into-the-undocumented-net-events-from-the-bcl-4e836989ade9?source=friends_link&amp;sk=b1d7bb35fc0b83e6d620383f905acb45">the previous post</a>. For example, I need to extract the following fields from the <strong>RequestStart</strong> event payload:</p><pre>private void OnRequestStart(<br>    DateTime timestamp,<br>    int threadId,<br>    Guid activityId,<br>    Guid relatedActivityId,<br>    byte[] eventData<br>    )<br>{<br>    ...<br>    // string scheme<br>    // string host<br>    // int port<br>    // string path<br>    // byte versionMajor<br>    // byte versionMinor<br>    // enum HttpVersionPolicy</pre><p>I’ve implemented the <strong>EventSourcePayload</strong> class that provides strongly typed helpers to get the different fields one after the other:</p><pre>public class EventSourcePayload<br>{<br>    private byte[] _payload;<br>    private int _pos = 0;<br><br>    public EventSourcePayload(byte[] payload)<br>    {<br>        _payload = payload;<br>    }</pre><p>It accepts the payload as the array of bytes received by each handler.</p><p>A string information is serialized as a list of UTF-16 Unicode characters; each one stored in 2 bytes:</p><pre>    public string GetString()<br>    {<br>        StringBuilder builder = new StringBuilder();<br>        while (_pos &lt; _payload.Length)<br>        {<br>            var characters = UnicodeEncoding.Unicode.GetString(_payload, _pos, 2);<br>            _pos += 2;<br><br>            if (characters == &quot;\0&quot;)<br>            {<br>                break;<br>            }<br>            builder.Append(characters);<br>        }<br><br>        return builder.ToString();<br>    }</pre><p>The current position in the array is incremented character by character up to the final ‘\0’.</p><p>The other helpers implementation is straightforward thanks to the <a href="https://learn.microsoft.com/en-us/dotnet/api/system.bitconverter?WT.mc_id=DT-MVP-5003325"><strong>BitConverter</strong> class</a>:</p><pre>    public byte GetByte()<br>    {<br>        return _payload[_pos++];<br>    }<br><br>    public UInt16 GetUnit16()<br>    {<br>        UInt16 value = BitConverter.ToUInt16(_payload, _pos);<br>        _pos += sizeof(UInt16);<br>        return value;<br>    }<br><br>    public UInt32 GetUInt32()<br>    {<br>        UInt32 value = BitConverter.ToUInt32(_payload, _pos);<br>        _pos += sizeof(UInt32);<br>        return value;<br>    }<br><br>    public UInt64 GetUInt64()<br>    {<br>        UInt64 value = BitConverter.ToUInt64(_payload, _pos);<br>        _pos += sizeof(UInt64);<br>        return value;<br>    }<br><br>    public Int64 GetInt64()<br>    {<br>        Int64 value = BitConverter.ToInt64(_payload, _pos);<br>        _pos += sizeof(UInt64);<br>        return value;<br>    }<br><br>    public double GetDouble()<br>    {<br>        double value = BitConverter.ToDouble(_payload, _pos);<br>        _pos += sizeof(double);<br>        return value;<br>    }<br>}</pre><p>Again, the position in the array is incremented to reflect the size of the field being read.</p><p>If you look at the rest of the <strong>OnRequestStart</strong> handler, you will see how each field is extracted:</p><pre>    EventSourcePayload payload = new EventSourcePayload(eventData);<br>    var scheme = payload.GetString();<br>    var host = payload.GetString();<br>    var port = payload.GetUInt32();<br>    var path = payload.GetString();<br>    var versionMajor = payload.GetByte();<br>    var versionMinor = payload.GetByte();</pre><h3>My activity or not my activity: that is the question</h3><p>In the previous episode, I forgot (on purpose) to mention that the <strong>EventSource</strong> is keeping track of “activities” when emitting events:</p><pre>protected unsafe void WriteEventCore(int eventId, int eventDataCount, EventData* data)<br>{<br>    WriteEventWithRelatedActivityIdCore(eventId, null, eventDataCount, data);<br>}</pre><p>The <a href="https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/EventSource.cs#L1367"><strong>WriteEventWithRelatedActivityIdCore</strong></a> method looks at the event metadata and if its opcode is <strong>Start</strong> then a new activity is created; if it is <strong>Stop</strong> then the current activity ends:</p><pre>if (opcode == EventOpcode.Start)<br>{<br>    m_activityTracker.OnStart(m_name, metadata.Name, metadata.Descriptor.Task, ref activityId, ref relActivityId, metadata.ActivityOptions);<br>}<br>else if (opcode == EventOpcode.Stop)<br>{<br>    m_activityTracker.OnStop(m_name, metadata.Name, metadata.Descriptor.Task, ref activityId);<br>}</pre><p>These <a href="https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/ActivityTracker.cs#L45"><strong>OnStart</strong></a> and <a href="https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/ActivityTracker.cs#L127"><strong>OnStop</strong></a> methods are doing nothing if the <strong>TplEventSource</strong> is not enabled with <strong>Keywords.TasksFlowActivityIds</strong> (= 0x80) set. This explains the code in <strong>GetProviders</strong> listed earlier where this non-HTTP provider is enabled.</p><p>When a request is created, the current global count managed by the <strong>ActivityTracker</strong> is incremented and it becomes the id of the current activity. Note that there is a “root” identifier before any activity gets created corresponding to the current AppDomain ID; starting at 1. If you think of an HTTP request, after a <strong>RequestStart</strong> event, each phase starts a new activity with, for example, <strong>ResolutionStart</strong> or <strong>ConnectStart</strong>. Informative events are emitted with the current request activity such as <strong>Redirect</strong> or <strong>ConnectionEstablished</strong>.</p><p>Here is a simplified view of the events emitted for an HTTP request (without DNS nor security events):</p><pre>Thread +-- Path &gt;------- ID ---- Opcode -- Event ----------------------------------<br> 78568 |    1/1 &gt;  event  1 __ [ 1| Start] RequestStart<br>       |<br> 32388 |  1/1/1 &gt;  event  1 __ [ 1| Start] ResolutionStart<br>       |<br> 32388 |  1/1/1 &gt;  event  2 __ [ 2|  Stop] ResolutionStop<br>       |<br> 32388 |  1/1/2 &gt;  event  1 __ [ 1| Start] ConnectStart<br>       |<br> 32388 |  1/1/2 &gt;  event  2 __ [ 2|  Stop] ConnectStop<br>       |<br> 32388 |    1/1 &gt;  event  4 __ [ 0|  Info] ConnectionEstablished<br>       |<br> 53324 |    1/1 &gt;  event  6 __ [ 0|  Info] RequestLeftQueue<br>       |<br> 53324 |  1/1/3 &gt;  event  7 __ [ 1| Start] RequestHeadersStart<br>       |<br> 53324 |  1/1/3 &gt;  event  8 __ [ 2|  Stop] RequestHeadersStop<br>       |<br> 68024 |  1/1/4 &gt;  event 11 __ [ 1| Start] ResponseHeadersStart<br>       |<br> 68024 |  1/1/4 &gt;  event 12 __ [ 2|  Stop] ResponseHeadersStop<br>       |<br> 68024 |  1/1/5 &gt;  event 13 __ [ 1| Start] ResponseContentStart<br>       |<br> 68024 |  1/1/5 &gt;  event 14 __ [ 2|  Stop] ResponseContentStop<br>       |<br> 68024 |    1/1 &gt;  event  2 __ [ 2|  Stop] RequestStop<br>  200 &lt;|</pre><p>As you can see, different threads are emitting events associated to the same request. At the <strong>ActivityTracker</strong> level, an <a href="https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/ActivityTracker.cs#L285">ActivityInfo</a> is stored in an async local so that each thread has its own storage that will be propagated by the methods of the Task Parallel Library (a.k.a. TPL) from task to task. This is why the very asynchronous code of the HTTP client implementation can go back to the current activity from different threads.</p><p>The activities are encoded into the 16 bytes of a GUID. In fact, only the first 12 bytes are used, and the final 4 bytes contain a checksum that includes the current process ID. Since the activity identifiers in the path do, most of the time, have a small value, the encoding is using 4 bits, also known as a nibble, to encode each of them. There <a href="https://github.com/dotnet/runtime/issues/109078">was a bug</a> in <a href="https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/ActivityTracker.cs#L466">ActivityTracker.AddIdToGuid</a> that will be fixed in .NET 10. Unfortunately, the decoding code in Perfview <a href="https://github.com/microsoft/perfview/issues/2122">needs to take it into account</a> so the last activity is not lost.</p><h3>Computing the different durations</h3><p>With this infrastructure in place, it is now possible to extract the “root” activity path corresponding to an HTTP request during each phase and update the corresponding state that is used to output the details when a request ends:</p><pre>private class HttpRequestInfo<br>{<br>    public HttpRequestInfo(DateTime timestamp, Guid activityId, string scheme, string host, uint port, string path)<br>    {<br>        Root = ActivityHelpers.ActivityPathString(activityId);<br>        if (port != 0)<br>        {<br>            Url = $&quot;{scheme}://{host}:{port}{path}&quot;;<br>        }<br>        else<br>        {<br>            Url = $&quot;{scheme}://{host}:{path}&quot;;<br>        }<br>        StartTime = timestamp;<br>    }<br><br>    public string Root { get; set; }<br>    public string Url { get; set; }<br>    public string RedirectUrl { get; set; }<br>    public DateTime StartTime { get; set; }<br>    public DateTime ReqRespStartTime { get; set; }<br>    public double ReqRespDuration { get; set; }<br><br>    public UInt32 StatusCode { get; set; }<br><br>    // DNS<br>    public DateTime DnsStartTime { get; set; }<br>    public double DnsDuration { get; set; }<br><br>    // HTTPS<br>    public DateTime HandshakeStartTime { get; set; }<br>    public double HandshakeDuration { get; set; }<br>    public string HandshakeErrorMessage { get; set; }<br><br>    // socket connection<br>    public DateTime SocketConnectionStartTime { get; set; }<br>    public double SocketDuration { get; set; }<br>}</pre><p>As you have seen earlier, the different phases of a request may be processed by different threads. It means that an available thread needs to be found in order to execute them. If the thread pool is busy, you can expect some wait time. This is shown in the different __wait__ sections between the phases. They are easily computed using the timestamp of each event. Having long wait durations might be reduced by increasing the number of threads in the thread pool.</p><p>Feel free to use the new <strong>dotnet-http</strong> CLI tool available from nuget.org or via <strong>dotnet tool install -g dotnet-http</strong>.</p><p>The corresponding sources are available from my <a href="https://github.com/chrisnas/ClrEvents/tree/master/Events/dotnet-http">github repository</a> in case you would like to integrate this kind of analysis directly in your code or your monitoring pipeline.</p><p>Happy monitoring!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04d79422b98d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digging into the undocumented .NET events from the BCL]]></title>
            <link>https://chnasarre.medium.com/digging-into-the-undocumented-net-events-from-the-bcl-4e836989ade9?source=rss-a7db8195a2bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e836989ade9</guid>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[dotnet]]></category>
            <category><![CDATA[http-request]]></category>
            <dc:creator><![CDATA[Christophe Nasarre]]></dc:creator>
            <pubDate>Sun, 13 Oct 2024 12:12:37 GMT</pubDate>
            <atom:updated>2024-12-06T15:45:30.811Z</atom:updated>
            <content:encoded><![CDATA[<h3>Digging into the undocumented .NET events from the BCL</h3><p>I’ve presented in depth the events emitted by the CLR <a href="https://github.com/chrisnas/ClrEvents">in many posts</a> to get insightful details about how the .NET runtime is working (lock contention, GC, allocations, …). Some .NET features are not implemented at the runtime level but at the Base Class Library (a.k.a. BCL) level. For example, if you are using <a href="https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?WT.mc_id=DT-MVP-5003325"><strong>HttpClient</strong></a>, you might want to measure how long it takes to get the response to your HTTP requests.</p><p>In this new series, I will describe how the BCL is using <a href="https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.tracing.eventsource?WT.mc_id=DT-MVP-5003325"><strong>EventSource</strong></a> to emit the events, how you can listen to them with TraceEvent; focusing on HTTP requests.</p><h3>Where do these events come from?</h3><p>When you look for these BCL events in the documentation, you end up to this <a href="https://learn.microsoft.com/en-us/dotnet/core/diagnostics/well-known-event-providers#systemnethttp-provider?WT.mc_id=DT-MVP-5003325">well-known event provides in .NET</a> page and in the <a href="https://learn.microsoft.com/en-us/dotnet/core/diagnostics/well-known-event-providers#framework-libraries?WT.mc_id=DT-MVP-5003325">Framework libraries section</a>. It lists the providers with their name and the emitted events with their keyword and verbosity. However, there is no detail about their payload! For example, for the “System.Net.Http” provider, you know that the <strong>RequestStart</strong> informal event is emitted each time “an HTTP request has started”. But you don’t know the corresponding url… Let me tell you how to get these tiny details.</p><p>Within the BCL, the classes responsible for emitting events derive from <a href="https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.tracing.eventsource?WT.mc_id=DT-MVP-5003325"><strong>EventSource</strong></a> with the “<em>Telemetry</em>” suffix as a naming convention. They are decorated with an <strong>EventSource</strong> attribute to define their name (used as provider name by a listener) and their Guid that will be provided with each event. For example, here is <a href="https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/HttpTelemetry.cs">the declaration of the class</a> responsible for events related to sending HTTP requests:</p><pre>[EventSource(Name = &quot;System.Net.Http&quot;)]<br>internal sealed partial class HttpTelemetry : EventSource<br>{</pre><p>If a Guid is not provided, one is automatically computed (see the <a href="https://github.com/microsoft/perfview/blob/main/src/TraceEvent/TraceEventSession.cs#L2854">corresponding code in Perfview</a>).</p><p>Next, public helper methods decorated with a <a href="https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.tracing.noneventattribute?view=net-8.0&amp;WT.mc_id=DT-MVP-5003325"><strong>NonEvent</strong> attribute</a> are provided to be used in the BCL code when it is needed to emit events. These methods are calling private methods decorated with an <a href="https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.tracing.eventattribute?WT.mc_id=DT-MVP-5003325"><strong>Event</strong> attribute</a> to define their unique ID and their verbosity level:</p><pre>[NonEvent]<br>public void RequestStart(HttpRequestMessage request)<br>{<br>    ...<br>    RequestStart(<br>        request.RequestUri.Scheme,<br>        request.RequestUri.IdnHost,<br>        request.RequestUri.Port,<br>        request.RequestUri.PathAndQuery,<br>        (byte)request.Version.Major,<br>        (byte)request.Version.Minor,<br>        request.VersionPolicy);<br>} <br><br>[Event(1, Level = EventLevel.Informational)]<br>private void RequestStart(string scheme, string host, int port, string pathAndQuery, byte versionMajor, byte versionMinor, HttpVersionPolicy versionPolicy)<br>{<br>    ...<br>    WriteEvent(eventId: 1, scheme, host, port, pathAndQuery, versionMajor, versionMinor, versionPolicy);<br>}</pre><p>The different <strong>WriteEvent</strong> overloads are responsible for filling an array of <a href="https://github.com/dotnet/runtime/blob/main/src/libraries/System.Diagnostics.Tracing/ref/System.Diagnostics.Tracing.cs#L255"><strong>EventData</strong> elements</a> (each one contains a pointer to the data and its size) that is passed to <a href="https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/EventSource.cs#L1332"><strong>EventSource.WriteEventCore</strong></a>. This helper methods dispatches the event to EventPipe/ETW pipelines.</p><h3>Document the undocumented</h3><p>Unlike CLR events, their payload is not explicitly described in a text file but, instead, you have to look at the implementation of the different <strong>WriteEvent</strong> overloads. The rest of this section provides the details of the sources I’ve looked at and the corresponding events payload.</p><h3>HTTP</h3><p>Sources: <a href="https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/HttpTelemetry.cs">https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/HttpTelemetry.cs</a><br>Name: System.Net.Http<br>Guid: d30b5633–7ef1–5485-b4e0–94979b102068</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3M-B2oQSv9q67xj-wkb_HA.png" /></figure><h3>Sockets</h3><p>Sources: <a href="https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketsTelemetry.cs">https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketsTelemetry.cs</a></p><p>Name: System.Net.Sockets<br>Guid: d5b2e7d4-b6ec-50ae-7cde-af89427ad21f</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*9NOqFY_5ZXR6toGqRE3IPg.png" /></figure><h3>DNS</h3><p>Sources: <a href="https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.NameResolution/src/System/Net/NameResolutionTelemetry.cs">https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.NameResolution/src/System/Net/NameResolutionTelemetry.cs</a></p><p>Name: System.Net.NameResolution<br>Guid: 4b326142-bfb5–5ed3–8585–7714181d14b0</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*ZcftMvnViwYebVCW_q1nXg.png" /></figure><h3>Network Security</h3><p>Sources: <a href="https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Security/src/System/Net/Security/NetSecurityTelemetry.cs">https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Security/src/System/Net/Security/NetSecurityTelemetry.cs</a></p><p>Name: System.Net.Security<br> Guid: 7beee6b1-e3fa-5ddb-34be-1404ad0e2520</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNAxTUuPK3kNRcCbUnwN9A.png" /></figure><p>The next episode will describe how to listen to these events and extract their payload in C#.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e836989ade9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>