<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Emanuele Fumagalli on Medium]]></title>
        <description><![CDATA[Stories by Emanuele Fumagalli on Medium]]></description>
        <link>https://medium.com/@emafuma?source=rss-89f70b816ca0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*Ff4_SSTrgmQc5DNwaszKEA.png</url>
            <title>Stories by Emanuele Fumagalli on Medium</title>
            <link>https://medium.com/@emafuma?source=rss-89f70b816ca0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:09:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@emafuma/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[GitHub Daily Stars Explorer: Hourly Insights, a New UI, and Much More]]></title>
            <link>https://itnext.io/github-daily-stars-explorer-hourly-insights-a-new-ui-and-much-more-96b4b971f6c7?source=rss-89f70b816ca0------2</link>
            <guid isPermaLink="false">https://medium.com/p/96b4b971f6c7</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Emanuele Fumagalli]]></dc:creator>
            <pubDate>Sat, 14 Mar 2026 16:43:06 GMT</pubDate>
            <atom:updated>2026-03-14T19:55:48.989Z</atom:updated>
            <content:encoded><![CDATA[<h4>From hourly star granularity to mobile support, pinned repos, and a fully redesigned interface here’s everything new in the Daily Stars Explorer</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U2mpsPRENL0oOLuCa2jWpA.png" /></figure><p>This article follows up on my <a href="https://medium.com/@emafuma/github-daily-stars-explorer-new-features-and-user-requested-improvements-f2805ac98cfd">previous one</a> about the <a href="https://emanuelef.github.io/daily-stars-explorer">Daily Stars Explorer tool</a>. <br>Since then, the project has continued to grow, driven by user feedback and some features I have wanted to add for a while. This post covers everything new across several releases, from v0.0.5 up to the latest v0.1.0.</p><p>The tool can be self hosted but there’s a hosted version available <a href="https://emanuelef.github.io/daily-stars-explorer">here</a>, the code is on <a href="https://github.com/emanuelef/daily-stars-explorer">GitHub</a>.</p><p>GitHub stars are a somewhat controversial metric. Many great projects have few stars, and a high star count does not automatically mean high quality. That said, stars remain one of the most visible signals of community interest, most projects actively ask for them, and they are increasingly used as a concrete threshold in real decisions. A recent example is Anthropic’s <a href="https://claude.com/contact-sales/claude-for-oss">Claude for Open Source program</a>, which offers six months of free Claude Max to maintainers of repositories with 5k or more GitHub stars. Whether you find that reasonable or not, it shows that stars have real consequences.</p><h3>Beyond the 40k Star Limit</h3><p>As covered in the <a href="https://medium.com/@emafuma/how-to-get-full-history-of-github-stars-f03cc93183a7">original article</a>, the GitHub REST API caps star history at 40k entries. Services that rely on it show a flat, linear approximation beyond that point, which makes it impossible to see what actually happened.</p><p>The Daily Stars Explorer uses the GitHub GraphQL API to retrieve the full history, and this matters enormously for any repository that crosses the 40k threshold. A great example is <a href="https://emanuelef.github.io/daily-stars-explorer/#/openclaw/openclaw"><strong>openclaw/openclaw</strong></a>, which surged past 300k stars following a massive viral spike at the end of January 2026. The chart shows the exact shape of that spike, how quickly activity peaked, and how it has settled since. No other tool can show you that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*739TJfBeXMoTMDQAcyROCg.png" /></figure><h3>Hourly Stars Chart</h3><p>The biggest addition in recent releases is the <strong>Stars by Hour</strong> chart. Daily granularity is useful, but it does not tell the whole story, especially when a repository gets featured on Hacker News or goes viral on social media. With hourly data, you can see exactly <em>when</em> the spike happened.</p><p>The chart overlays hourly star counts (as bars) alongside the cumulative total (as a line), giving you both the moment by moment activity and the bigger picture at a glance. The dashboard also automatically calculates:</p><ul><li><strong>Peak Hour (UTC):</strong> the single hour with the most activity</li><li><strong>Peak Day:</strong> the specific date with the highest engagement</li><li><strong>Best Hour(s):</strong> an averaged view to identify consistently high-engagement windows</li></ul><p>To support very large repositories, the timeout for fetching hourly data has been extended from 10 to 20 minutes, along with improvements to API stability and context timeout handling.</p><p>You can select your <strong>preferred timezone</strong> directly in the hourly view, so you are always looking at star activity in your own local time context. This setting persists across sessions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k95xNMGx3_PtEVcrcqxpUA.png" /></figure><h3>GitHub Mentions Feed</h3><p>Building on the existing Hacker News and Reddit feeds, the tool now also tracks <strong>GitHub mentions</strong>, instances where other repositories reference your project in issues, pull requests, or comments.</p><p>These mentions are plotted as markers directly on the star timeline, colour coded by type:</p><ul><li>Issues (orange)</li><li>Pull Requests (purple)</li><li>Comments (blue)</li></ul><p>This makes it much easier to correlate spikes in star activity with organic community interest. Hovering over a marker shows the details, and clicking it takes you directly to the relevant discussion on GitHub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F5WQC2IJAbheFIzF7KWN9A.png" /></figure><h3>Pin Favourite Repositories</h3><p>If you frequently track the same set of repositories, you no longer have to retype them every visit. You can now <strong>pin repositories</strong> directly from the search interface, and they will be saved across sessions. A quick access bar shows your top pinned repos so you can jump straight in. Pinning is available in both the mobile and hourly views.</p><h3>Dedicated Mobile View</h3><p>The tool now has a proper <strong>dedicated mobile experience</strong> for smartphones and tablets, optimised for touch interactions. It focuses on the most essential data, daily stars and hourly views, in a simplified layout that works well on smaller screens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*GH2qoE0_x2V9v-GmucbYaw.png" /></figure><h3>Light Mode</h3><p>The interface now supports a <strong>light theme</strong> alongside the existing dark mode. It auto-detects your system preference by default, and you can toggle manually at any time. Transitions between modes are smooth, and the styling is consistent across all views.</p><h3>Redesigned UI</h3><p>Version 0.1.0 brought a comprehensive visual refresh across the whole application. The interface has been modernised with improved visual hierarchy and consistent styling throughout. Charts are easier to read thanks to thicker lines and cleaner formatting. Large numbers are now displayed in a compact format (for example 1.2k or 45.3k) which makes them much easier to scan. The overall feel is noticeably more polished, while keeping the same data-first philosophy.</p><h3>Other Additions</h3><p>A few more improvements worth mentioning:</p><ul><li><strong>Contributors timeline</strong> (v0.0.5): visualise the history of contributors to a repository over time</li><li><strong>Stars counter for selection</strong>: see an aggregate star count for any selected time range</li><li><strong>GH New Public Repos chart</strong>: a new visualisation tracking the trend of new public repositories created on GitHub over time</li><li><strong>Show Last 30 Days</strong> button: quickly zoom into the most recent activity without manually adjusting the date range</li><li><strong>Live current-day star count</strong>: the in-progress count for today updates live each time you request a repo</li></ul><h3>Conclusions</h3><p>The <a href="https://github.com/emanuelef/daily-stars-explorer">Daily Stars Explorer</a> has continued to evolve well beyond its original scope. The full history capability, now paired with hourly granularity, makes it possible not just to see that a spike happened but to understand exactly when it started, how fast it grew, and what likely caused it.</p><p>If you would like to try it out, the tool is available at <a href="https://emanuelef.github.io/daily-stars-explorer">https://emanuelef.github.io/daily-stars-explorer</a>.</p><p>As always, feedback and feature requests are welcome in the <a href="https://github.com/emanuelef/daily-stars-explorer/discussions">GitHub discussions</a>, and contributions via pull request are very much appreciated.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96b4b971f6c7" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/github-daily-stars-explorer-hourly-insights-a-new-ui-and-much-more-96b4b971f6c7">GitHub Daily Stars Explorer: Hourly Insights, a New UI, and Much More</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How reusing exception objects can affect Python memory]]></title>
            <link>https://itnext.io/how-reusing-exception-objects-can-affect-python-memory-0521be2ec9e7?source=rss-89f70b816ca0------2</link>
            <guid isPermaLink="false">https://medium.com/p/0521be2ec9e7</guid>
            <category><![CDATA[memory-improvement]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[exception-handling]]></category>
            <category><![CDATA[memory-leak]]></category>
            <dc:creator><![CDATA[Emanuele Fumagalli]]></dc:creator>
            <pubDate>Fri, 26 Sep 2025 18:11:51 GMT</pubDate>
            <atom:updated>2025-09-27T16:34:51.985Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why Singleton Exception Patterns cause hidden memory leaks and how to fix them</h4><h4><strong>TL;DR</strong></h4><p><strong>Be cautious with reusing exception objects in Python.</strong> Every time an exception is raised, Python attaches a new traceback to it without cleaning up old ones. This can create a hidden memory leak in long-running applications that grows over time. Using factory methods to create fresh exception instances can help prevent this issue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zfBhU-nSaW9ZAUZK03e9cA.png" /><figcaption><em>Memory usage comparison: Singleton pattern (red) vs Factory pattern (green)</em></figcaption></figure><h3>A Pattern Worth Reconsidering</h3><p>As Python developers, we often strive to write clean, efficient code. One common pattern that seems elegant but may lead to unexpected memory behaviour involves creating reusable exception objects:</p><pre># Pattern that can lead to memory leaks:<br>class Errors:<br>    user_not_found = CustomException(&quot;User not found&quot;, code=404)<br>    invalid_token = CustomException(&quot;Invalid token&quot;, code=401)<br><br>def process_request(request):<br>    # ...<br>    if not user:<br>        raise Errors.user_not_found  # Reusing the same exception instance<br>    # ...</pre><p>This pattern appears in some Python codebases, especially in web services that define standardised error responses. The intention is good: create a single set of consistent exception objects that can be reused throughout the application.</p><h3><strong>The Hidden Leak</strong></h3><p>When an exception is raised in Python, the interpreter attaches a traceback object to it. This traceback contains references to:</p><ul><li>The entire call stack at the point where the exception was raised</li><li>All local variables in each frame of the stack</li><li>The context where the exception occurred</li></ul><p>The critical issue is that <strong>when you reuse an exception object, Python attaches a new traceback each time without cleaning up previous ones</strong>. Since each traceback contains references to potentially large objects, memory usage grows unbounded as the exception is raised repeatedly.</p><pre># Pseudocode showing what happens internally<br>singleton_exception = CustomException(&quot;Some error&quot;)<br><br># First raise<br>singleton_exception.__traceback__ = current_traceback  # First traceback added<br><br># Second raise (later)<br>singleton_exception.__traceback__ = current_traceback  # New traceback added<br># But the previous traceback is still referenced internally!</pre><h3><strong>Visualising the Problem</strong></h3><p>To demonstrate this issue, I created a test that raises the same exception 1,000 times with a simulated context of ~500KB for each traceback:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9RDRA5n7YpMBFS9EdCpN7Q.png" /><figcaption><em>Memory growth with the singleton exception pattern</em></figcaption></figure><p>The results are striking:</p><ul><li>Memory usage increased by 1.44 MB after just 1,000 iterations</li><li>Memory growth rate: 8.543 MB/s</li><li>The trend is linear and unbounded</li></ul><p>In a long-running application like a web server handling thousands of requests per minute, this leak could quickly consume gigabytes of memory, eventually leading to crashes or degraded performance.</p><h3><strong>The Solution: Factory Methods</strong></h3><p>The fix is simple but requires a shift in your exception handling pattern. Instead of reusing exception objects, create a fresh instance each time:</p><pre># Recommended approach: Factory methods for exceptions<br>class Errors:<br>    @staticmethod<br>    def user_not_found():<br>        return CustomException(&quot;User not found&quot;, code=404)<br>    <br>    @staticmethod<br>    def invalid_token():<br>        return CustomException(&quot;Invalid token&quot;, code=401)<br><br>def process_request(request):<br>    # ...<br>    if not user:<br>        raise Errors.user_not_found()  # Creates a new instance each time<br>    # ...</pre><p>With this pattern, each exception has its own traceback that gets properly garbage collected when the exception is handled. The memory profile is drastically different:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JJgVgjxXZYq7ZWB8m3_6oA.png" /><figcaption><em>Stable memory usage with the factory exception pattern</em></figcaption></figure><p>The results show:</p><ul><li>Zero measurable memory growth after 1,000 iterations</li><li>Memory growth rate: 0.000 MB/s</li><li>Perfect stability regardless of the number of exceptions raised</li></ul><h3><strong>Real-World Impact</strong></h3><p>This memory behaviour is particularly challenging to identify because:</p><p>1. <strong>It’s silent: </strong>There are no errors or warnings, just gradually increasing memory usage</p><p>2. <strong>It can be difficult to detect:</strong> The issue only appears when exceptions are raised, which might be rare in testing but more common in production</p><p>3. <strong>It scales with complexity:</strong> The more complex your call stack and the larger your local variables, the more significant the memory impact</p><p>This pattern can be especially relevant in certain types of systems, particularly:</p><ul><li>Web services handling thousands of requests with standardised error responses</li><li>Background workers processing large batches of data with retry mechanisms</li><li>Long-running applications where exceptions are used for control flow</li></ul><h3><strong>Conclusion</strong></h3><p>The pattern of reusing exception objects in Python is an interesting example of how a seemingly helpful optimisation can sometimes lead to unexpected issues. By adopting factory methods that create fresh exception instances, you can prevent this hidden memory leak while still maintaining clean, readable code.</p><p>Consider these best practices:</p><ul><li>Creating fresh exception instances when raising exceptions</li><li>Using factory methods for standardised exceptions</li><li>Monitoring memory usage in long-running applications</li><li>Being mindful of how Python manages exception objects internally</li><li>Load test even non-happy path scenarios and track memory usage</li></ul><p>This small change in your exception handling pattern can prevent serious memory issues in production systems and ensure your Python applications run reliably over extended periods.</p><p>Want to see the full code for this demonstration? Check out the <a href="https://github.com/emanuelef/python-mem-leak-exception">GitHub repository</a> with complete examples and visualisation tools.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0521be2ec9e7" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/how-reusing-exception-objects-can-affect-python-memory-0521be2ec9e7">How reusing exception objects can affect Python memory</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GitHub Daily Stars Explorer: New Features and User-Requested Improvements]]></title>
            <link>https://medium.com/@emafuma/github-daily-stars-explorer-new-features-and-user-requested-improvements-f2805ac98cfd?source=rss-89f70b816ca0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2805ac98cfd</guid>
            <category><![CDATA[react]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[repositories]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Emanuele Fumagalli]]></dc:creator>
            <pubDate>Sat, 24 Aug 2024 20:49:08 GMT</pubDate>
            <atom:updated>2024-08-24T21:21:17.567Z</atom:updated>
            <content:encoded><![CDATA[<h4>New Updates and Enhancements Since the Initial Release</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WD3XM5mg78JuEQJ51eIsYA.png" /></figure><h3>Introduction</h3><p>This article follows up on <a href="https://medium.com/@emafuma/how-to-get-full-history-of-github-stars-f03cc93183a7">my previous one</a> about the <a href="https://emanuelef.github.io/daily-stars-explorer">Daily Stars Explorer tool</a>. Since then, new features have been added, some based on user requests.</p><h3>New functionalities</h3><h4>Trend</h4><p>Observing the graph of daily stars can sometimes make it challenging to see the overall trend. The Trend option uses <a href="https://facebook.github.io/prophet/">Facebook Prophet</a> to analyse the daily trend and also provides a forecast for the next 60 days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vu9ErwVuCcUdaf2Sig_DvA.png" /><figcaption><a href="https://emanuelef.github.io/daily-stars-explorer/#/facebook/react">Facebook React</a> standard timeline visualisation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NyGLiMGQupXn5g4e4Mcdaw.png" /><figcaption><a href="https://emanuelef.github.io/daily-stars-explorer/#/facebook/react">Facebook React</a> using log scale on the Y-axis for daily stars</figcaption></figure><h4>Binning with aggregation</h4><p>The binning feature allows you to group data by year, month, or week. For each grouping option, you can choose different aggregation methods, such as minimum, maximum, mean, or total. All aggregation methods, except total, represent the number of stars in a single day.<br>For example, if you set the binning to monthly with the minimum aggregation, it will show the lowest number of new stars recorded in a single day for each month.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zFsS6xrsGiBfzhvIjzG7qg.png" /><figcaption>Example of binning for <a href="https://emanuelef.github.io/daily-stars-explorer/#/ollama/ollama">ollama</a></figcaption></figure><h4>Log scale</h4><p>In some cases, large spikes in the data can obscure the trend during periods of normal values. Switching to a log scale can help reveal variations more clearly across different scales.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8zJyIESp5LJJwv14oxSW6A.png" /><figcaption>Graph with big spikes with normal scale (<a href="https://emanuelef.github.io/daily-stars-explorer/#/opentofu/opentofu">Opentofu</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2AKhqjqNoHSF9RhadCfA2w.png" /><figcaption>and using log scale</figcaption></figure><h4>Feeds</h4><p>When looking at the daily stars graph, it can be interesting to guess the causes behind sudden spikes. One user suggested adding a feature for this, and I’ve now included an option to search various sources (such as Hacker News, Reddit, and YouTube) and add relevant timeframes to the timeline. Sometimes, these results align with the spikes, but even when they don’t, they can provide valuable articles or videos for context.<br>This feature is still experimental, as the search results need refining, and there can be many false positives, especially for repositories with common names.<br>When hovering over the time markers, the title of the article and some key metrics are displayed. Clicking on a time marker will open the related article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WD3XM5mg78JuEQJ51eIsYA.png" /><figcaption>Hacker News feed</figcaption></figure><h4>Comparison of two repos</h4><p>Keep in mind that the number of stars on a repository can be influenced by various factors, and comparing two repositories based solely on star count might not accurately reflect their quality. <br>As I wrote in the <a href="https://emanuelef.github.io/daily-stars-explorer/#/info">info section</a> there are multiple reasons behind the number of stars for a repository.<br>This comparison aims to highlight trends rather than quality, as factors like community support can significantly impact the star count. Therefore, it’s essential to interpret these comparisons with that context in mind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K0CCSqsXmoOmzp0YO-mKpQ.png" /></figure><h4>Commits / PRs / Issues / Forks History</h4><p>Some users suggested to provide the history of other metrics of a GitHub repo. The following have been added:</p><ul><li>Commits: daily commits on the default branch</li><li>Issues: Daily Opened, Daily Closed and Total Opened issues</li><li>PRs: Daily Opened, Daily Closed, Daily Merged and Total opened PRs</li><li>Forks: Daily new Forks created</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rVLlTyrGqlwUnDVEBKeuaw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qgC-tCIG5R1sDsHShVDV-Q.png" /><figcaption>commits on default branch and issues</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XuyD1lBWDZrJcsWZZ-wbJw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dSE5YuvSEi96qlvCfiu-wA.png" /><figcaption>PRs and forks</figcaption></figure><p>Please note that there is no progress update available for these, and they take longer because, unlike the stars fetching process, there’s only a single fetch running from start to finish. In contrast, the stars fetching runs two processes in parallel.</p><h3>Conclusions</h3><p>This project has grown beyond what I originally expected when I first started.<br>Some people have begun using it and provided invaluable feedback and feature requests.<br>While I still believe that some of these features could eventually be implemented directly by GitHub, it has been enjoyable to work on.<br>If anyone is interested in contributing feel free to open a PR on the <a href="https://github.com/emanuelef/daily-stars-explorer">repo</a>, there’s also a <a href="https://github.com/emanuelef/daily-stars-explorer/discussions">discussion section</a> in case there are questions or feature requests.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2805ac98cfd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Debugging Microservices Locally with mirrord]]></title>
            <link>https://blog.stackademic.com/mastering-local-microservices-debugging-with-mirrord-0a99443c1544?source=rss-89f70b816ca0------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a99443c1544</guid>
            <category><![CDATA[mirrord]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[telepresence]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Emanuele Fumagalli]]></dc:creator>
            <pubDate>Mon, 26 Feb 2024 16:55:40 GMT</pubDate>
            <atom:updated>2024-04-01T21:38:25.933Z</atom:updated>
            <content:encoded><![CDATA[<h4>Unlock the Power to Debug Microservices Locally with Full Access to Network, Environment Variables and Files as running a pod in the Kubernetes cluster</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-TR6bV46bfgs92YcVTa6Xg.png" /><figcaption>Debugging locally step-by-step a Rust app with full access to the cluster resources</figcaption></figure><h3>Introduction</h3><p>On some occasions we would love to have the ability to debug code running in pods in a Kubernetes cluster. This need might arise while developing new functionalities, creating new microservices, or troubleshooting existing systems.</p><p>In my experience, troubleshooting sometimes involves attempting to replicate the issue, and resorting to local debugging when other strategies prove to be ineffective.<br>While observable solutions are great and offer insights through distributed traces and logs, there are some scenarios where a finer level of granularity is needed to detect the root cause.</p><p>Furthermore, code debugging is equally important during development phases. When building new functionalities or creating entirely new microservices, developers frequently leverage debugging tools to inspect code behavior, identify potential issues, and ensure that the newly introduced features work as intended.</p><p>When the need arises to debug pods, simulating a Kubernetes cluster environment offers various options.<br>One approach involves aligning the development environment precisely with Staging and Production resources. Alternatively, developers can maintain personalized environments, incorporating mocks for databases or resources like <a href="https://github.com/localstack/localstack">Localstack</a> for AWS. While using kubectl port-forward is a viable method to access services from locally running code, it becomes challenging in more complex scenarios. Situations requiring multiple port forwards or the replication of the environment locally with ENV vars or additional mock databases present difficulties.</p><p>To enable local execution and debugging with the same experience as being within the cluster, several Local Kubernetes Development tools are available. The most widely used ones include <a href="https://www.telepresence.io/">Telepresence</a>, <a href="https://gefyra.dev/">Gefyra</a>, and <a href="https://mirrord.dev/">mirrord</a>.</p><p>In this article, I’m going to explain how I use mirrord. For a detailed comparison of these tools, refer to this <a href="https://kubernetes.io/blog/2023/09/12/local-k8s-development-tools/">article</a>.<br>Additionally, you can find a comparison between Telepresence and mirrord in this <a href="https://metalbear.co/blog/mirrord-as-an-alternative-to-telepresence">blog post</a>.</p><p>In my experience, mirrord stands out as my preferred choice. One key advantage is that it doesn’t disrupt traffic by default, ensuring the cluster remains fully functional. This distinction becomes crucial, as I’ve encountered instances where a Telepresence session left open in the Dev cluster made it unusable.<br>mirrord also stands out because is easy to set up and generally works out of the box.</p><h3>mirrord</h3><p><a href="https://mirrord.dev/">mirrord</a> integrates with the local binary by utilizing LD_PRELOAD on Linux or DYLD_INSERT_LIBRARIES on macOS. It overrides libc function calls and proxies them through a temporary agent running in the cluster. For instance, when the local process attempts to read a file, mirrord intercepts the call, directs it to the agent, and retrieves the file from the remote pod. This approach enables mirrord to handle all process inputs and outputs, including network and file access, as well as environment variables.</p><p>There’s an interesting page explaining why the team decided to use <a href="https://metalbear.co/blog/carcinisation-of-mirrord-or-why-we-use-rust/">Rust</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I7scgop9QBXsWdnVbEOSFw.png" /><figcaption>Taken from <a href="https://mirrord.dev/docs/overview/introduction/#how-it-works">https://mirrord.dev/docs/overview/introduction/#how-it-works</a></figcaption></figure><h4>Requirements</h4><p><strong>Languages of the code to debug</strong>: <a href="https://metalbear.co/blog/hooking-go-from-rust-hitchhikers-guide-to-the-go-laxy/">Go</a>, Rust, Node, Python, Java, Kotlin, Ruby, and other languages using libc.<br><strong>OS</strong><em> </em>: macOS, Linux and Windows (using WSL)<br><strong>IDEs</strong>: <a href="https://mirrord.dev/docs/overview/quick-start/#vs-code-extension">VSCode</a> and <a href="https://mirrord.dev/docs/overview/quick-start/#intellij-plugin">IntelliJ</a> or from the <a href="https://mirrord.dev/docs/overview/quick-start/#cli-tool">CLI</a><br><strong>Service mesh</strong>: <a href="https://mirrord.dev/docs/faq/limitations/#does-mirrord-support-clusters-with-a-service-mesh-like-istio-or-linkerd">Istio or Linkerd</a> are supported but only with steal mode, so no chance to mirror the traffic, and that means disrupting the cluster.</p><p>I’ll provide a brief overview of the functionalities I used. However, there’s more to explore, and I encourage you to check the well-written and comprehensive <a href="https://mirrord.dev/docs/using-mirrord/">documentation</a>.</p><h4>Primary Functionality</h4><p>A key capability of mirrord is its capacity to mirror incoming traffic from the impersonated pod directly into the locally running code.<br>In the default mode, this ensures that the cluster remains undisturbed, with the local code receiving a copy of the requests while the remote pod continues to function normally. <br>Moreover, all other resources such as remote files or environment variables within the pod, along with access to cluster services, become accessible from the locally running code.</p><h4>Targetless</h4><p>This option makes it possible to run a completely new app in the context of the cluster, this can be useful if you are implementing a new microservice and you need to have access to the services or resources available inside the cluster.</p><h4>Pause Queues</h4><p>If one of the microservices is reading from a queue or a message broker you might want to use the <a href="https://mirrord.dev/docs/using-mirrord/pause/">pause feature</a>.</p><h4>mirrord for Teams</h4><p>mirrord also offers a premium version that utilizes a Kubernetes operator installed within the cluster. The biggest advantages to me are:</p><ul><li>ability to use mirrord on the same pod, this is needed if multiple developers are accessing that pod at the same time</li><li>deployments with more than one pod can be mirrored/intercepted in their entirety</li><li>RBAC</li></ul><p>For my needs I didn’t need to buy mirrord for Teams but I’d be interested in trying that.</p><h4>Gotchas</h4><p>As you start using mirrord, here are some key challenges to keep in mind. Being aware of these in advance will save you valuable time and potential headaches.</p><p><strong>Handling .env Files</strong><br>When working with mirrord, be cautious about .env files. If you have a .env file configuring environment variables for local code execution (not using mirrord), these variables will override those in the cluster. To address this, consider emptying the .env file and ensuring that no environment variables are set. For more insights and potential improvements, you can explore the discussion in the GitHub issue I raised: <a href="https://github.com/metalbear-co/mirrord/issues/2225">https://github.com/metalbear-co/mirrord/issues/2225</a>.</p><p><strong>Relative paths</strong><br>Another one was the access to files using relative paths, in that case the files will be taken from local filesystem. This scenario may arise when mounting files from Configmap. So the solution would be to have the files locally in sync with the ones in the cluster, or an ENV var that specifies the absolute path (there will be one used locally different than when running in the cluster).</p><p><strong>Duplicated operations<br></strong>When mirroring the traffic (the default) the remote pod proceeds to fulfill the request being debugged locally, the risk there is that you will run some operations twice (one in the remote and the other still on the remote cluster but triggered locally), something that is stateful like adding a new entry in a DB. To mitigate this, consider disabling outgoing traffic. Further details on this can be found in a note in the <a href="https://mirrord.dev/docs/reference/traffic/#outgoing">documentation</a>.</p><p><strong>Kernel &lt; 4.20</strong><br>In the uncommon scenario of encountering a Kernel version below 4.20, as observed in a VM running <a href="https://github.com/kubernetes-sigs/kind">Kind</a>, the mirroring feature may be disrupted. In such cases, the only viable option for utilizing mirrord is to enable the steal mode.<br>An interesting aspect of identifying this issue is the need to inspect the logs within the mirrord agent pod installed in the cluster.</p><p><strong>Rust</strong><br>When utilizing the debug option in VSCode, ensure that the application has been built with the latest code changes. Failure to do so may result in disappearing breakpoints when running in Debug mode. This consideration applies similarly to running the code locally without the use of mirrord.</p><h3>Example</h3><p>In order to test a real example I set up a <a href="https://github.com/emanuelef/k8s-local-debugging">repository</a> with some simple µservices that run inside a K8s cluster.<br>You need to have access to a cluster, I tested on the Kubernetes that is integrated in Docker Desktop, but also had the chance to use mirrord on a <a href="https://github.com/kubernetes-sigs/kind">Kind</a> cluster on a remote VM and a real development AWS EKS cluster.</p><p>The example is deliberately kept very simple, only interacting with other services in the cluster through exposed HTTP APIs.<br>The need and the power of mirrord is more evident on more complex architecture with multiple services/resources.<br>The example is also a bit opinionated but that will make it work out of the box if you have macOS and VSCode.</p><p>To use the code in the repo you need:</p><ul><li>A K8s cluster (<a href="https://github.com/kubernetes-sigs/kind">Kind</a>, EKS, <a href="https://github.com/kubernetes/minikube">Minikube</a>, Docker Desktop Kubernetes…) make sure Linux Kernel version is 4.20+</li><li>kubectl to be configured on the local machine (So you are able to run kubectl locally to the cluster of interest)</li></ul><p>I experimented with diverse microservices within a real cloud solution (on a Dev environment on EKS), including those interfacing with AWS resources like S3, DynamoDB, Aurora Postgres, as well as clusters running queues (RabbitMQ) and services exposing TCP and leveraging gRPC. Across these cases, I successfully accessed them from locally executed code, mirroring the cluster environment.</p><h4>Install mirrord</h4><p>You can follow the <a href="https://mirrord.dev/docs/overview/quick-start/">Quick Start</a> section in mirrord documentation to install what is required, you don’t need the operator which is only needed for mirrord for Teams.</p><p>In my case I installed the mirrord client with:</p><pre>brew install metalbear-co/mirrord/mirrord</pre><p>And the <a href="https://marketplace.visualstudio.com/items?itemName=MetalBear.mirrord">VSCode extension</a> as any other extension.<br>After that you have everything needed to start using mirrord, you can try already with your cluster and your code but you can follow along with the example I created.</p><h4>Deploy services and pods</h4><p>Once you’ve cloned the <a href="https://github.com/emanuelef/k8s-local-debugging">repository</a>, deploying the Kubernetes manifests to your cluster is straightforward. Simply execute the provided script.</p><pre>./start.sh</pre><p>This script deploys the deployment manifests to the Kubernetes cluster specified by the default kubeconfig file.<br>If you need to deploy to a different cluster, you can add the kubeconfig and context arguments in the kubectl commands before running the script.<br>something like:</p><pre>kubectl apply -f deployment-service.yaml --kubeconfig &lt;path/to/your/kubeconfig&gt; --context &lt;context-name&gt;</pre><p>You will see the new pods coming up in the default namespace:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W20ier3fYfTrBVcJZoVz-w.png" /><figcaption><a href="https://k9scli.io/">k9s</a> showing pods in the cluster created by the <a href="https://github.com/emanuelef/k8s-local-debugging">k8s-local-debugging</a></figcaption></figure><p>Four pods, written in different languages (Go, Rust, and Python), are running in the cluster. These pods interact by calling each other through exposed Kubernetes services.</p><h4>Start debugging locally with mirrord</h4><p>This is the <a href="https://github.com/emanuelef/k8s-local-debugging/blob/main/.mirrord/mirrord.json">configuration file</a> if you want to mirror the traffic to your app running locally. As in the example this needs to be located in the .mirrord folder in the root of the codebase.</p><pre>{<br>  &quot;feature&quot;: {<br>    &quot;env&quot;: true,<br>    &quot;fs&quot;: {<br>      &quot;mode&quot;: &quot;read&quot;<br>    },<br>    &quot;network&quot;: true<br>  },<br>  &quot;target&quot;: {<br>    &quot;namespace&quot;: &quot;default&quot;<br>  }<br>}</pre><p>This is using the default kubeconfig, same as when running kubectl so you might need to add the path to a different <a href="https://mirrord.dev/docs/reference/configuration/#root-kubeconfig">kubeconfig</a> file and maybe also the <a href="https://mirrord.dev/docs/reference/configuration/#root-kube_context">kube-context</a>.<br>So in case the config file will be something like:</p><pre>{<br>  &quot;feature&quot;: {<br>    &quot;env&quot;: true,<br>    &quot;fs&quot;: {<br>      &quot;mode&quot;: &quot;read&quot;<br>    },<br>    &quot;network&quot;: true<br>  },<br>  &quot;target&quot;: {<br>    &quot;namespace&quot;: &quot;default&quot;<br>  },<br>  &quot;kubeconfig&quot;: &quot;&lt;path/to/your/kubeconfig&gt;&quot;,<br>  &quot;kube_context&quot;: &quot;&lt;context-name&gt;&quot;<br>}</pre><p>Refers to the official <a href="https://mirrord.dev/docs/reference/configuration/">configuration documentation</a> for additional settings.</p><p>Make sure mirrord debugging in enabled by checking mirrord in the VSCode status bar:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I3VyYyhLzU_wf3GkW2Kv6g.png" /></figure><p>Create a breakpoint in the handle method of <a href="https://github.com/emanuelef/k8s-local-debugging/blob/70edd9f9190bcec73664b7577e274862dc370932/python-app/app.py#L15">python-app/app.py</a>.</p><p>Then run from Run and Debug menu the app you are interested to debug (in the repo those are defined in the .vscode/launch.json file).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*8EStRYE5CIBzbseQ_YgZtw.png" /><figcaption>launch the debug on the app of interest</figcaption></figure><p>If all is set up correctly and mirrord can reach the cluster you should be able to select the deployment (or pod) of interest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*thvQeXDx6gXHj6wAFT13Lg.png" /><figcaption>deployment selection when using mirrord for debug</figcaption></figure><p>After that just wait few seconds and you should hit the line with the breakpoint. That’s because the go-app keeps calling the python-app service (every 15s).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ebTHRMPF0hF5jFeaXP_6OA.png" /></figure><p>You can then debug like a normal app in VSCode but with the ability to interact with all the resources accessible from the pod in the cluster (other services, AWS resources, ENV vars, configmap mounted…).</p><p>Since we are using default mode the traffic is mirrored and the single replica pod in the cluster will keep responding to all requests even if we are debugging locally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6lBjHUv0nXyH_-fteL4Qzg.png" /><figcaption>Pod logs showing the requests keep getting answered even when debugging locally</figcaption></figure><p>If you proceed with the debugging and get to the line where the ENV var is read you will see that this is coming from the ENV var set in the pod in the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-OFQrhAk1zrTwk-xB1E0ZA.png" /><figcaption>Inspecting ENV vars</figcaption></figure><p>And you will also see that the requests coming from the local code will indeed reach the python-app-second pod.</p><p>Similar experiments can be done with the go-app and rust-app.</p><p>To test the other modes you can change the .mirrord file with one of the others in the <a href="https://github.com/emanuelef/k8s-local-debugging/tree/main/.mirrord">.mirrord folder</a>.</p><ul><li><a href="https://github.com/emanuelef/k8s-local-debugging/blob/main/.mirrord/mirrord-steal.json">mirrord-steal.json</a>: Uses the steal mode, if repeating the steps previously described you will see that the remote pod will stop answering the requests coming from go-app. Only the code running locally will receive them.</li><li><a href="https://github.com/emanuelef/k8s-local-debugging/blob/main/.mirrord/mirrord-off.json">mirrord-off.json</a>: This configuration will stop forwarding locally the incoming requests to the remote pod.</li></ul><p>If you inspect the cluster while debugging you will see that a job is added that creates a mirrord-agent pod, this will be automatically cleaned up after stopping debugging locally.</p><blockquote><strong>Note on multiple replicas</strong></blockquote><blockquote>When using multiple replicas mirrord will impersonate the first pod, so if you are expecting some calls to be triggered when you run locally it might not be always the case. Of course you can scale the replicas to 1 manually, or use mirrord for Teams.</blockquote><p>To clean up the cluster:</p><pre>./stop.sh</pre><p>If you encounter any issues while running the example in the repository, please don’t hesitate to reach out by opening an issue.</p><h3>Conclusion</h3><p>The addition of mirrord to my toolkit has proven to be a delightful and valuable asset, especially when developing new features for existing microservices or creating new ones.<br>It has also proven instrumental in replicating and troubleshooting issues observed in production, the mirroring functionality in our development and staging environments guarantees minimal or no disruption to the cluster.<br>Additionally, for our scaled-down personal VM cluster (using mostly Kind and <a href="https://github.com/localstack/localstack">Localstack</a>), mirrord excels in avoiding the recreation of local mocks and providing access to the simulated AWS resources.</p><p>While I wouldn’t necessarily advocate for using mirrord in a production environment for troubleshooting, it is technically feasible. This assumes you have machine access to the cluster (which is often unlikely), and you need to exercise caution due to the potential disruptions it may cause.</p><p>The mirrord community on the <a href="https://discord.gg/metalbear">Discord Channel</a> is responsive and always ready to provide assistance and answers to inquiries.</p><p>I’m curious about your debugging strategies and tool preferences.<br>Are there other tools you find effective? Share your insights in the comments.</p><h3>Stackademic 🎓</h3><p>Thank you for reading until the end. Before you go:</p><ul><li>Please consider <strong>clapping</strong> and <strong>following</strong> the writer! 👏</li><li>Follow us <a href="https://twitter.com/stackademichq"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/stackademic"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/c/stackademic"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a></li><li>Visit our other platforms: <a href="https://plainenglish.io/"><strong>In Plain English</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a><strong> | </strong><a href="https://blog.cubed.run/"><strong>Cubed</strong></a></li><li>More content at <a href="https://stackademic.com/"><strong>Stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a99443c1544" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/mastering-local-microservices-debugging-with-mirrord-0a99443c1544">Debugging Microservices Locally with mirrord</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Cost-Free, Always-On Personal Project Stack]]></title>
            <link>https://blog.stackademic.com/building-a-cost-free-always-on-personal-project-stack-3eaa02ac16b6?source=rss-89f70b816ca0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3eaa02ac16b6</guid>
            <category><![CDATA[oracle-cloud]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[free]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Emanuele Fumagalli]]></dc:creator>
            <pubDate>Mon, 05 Feb 2024 15:51:02 GMT</pubDate>
            <atom:updated>2024-02-07T05:13:22.473Z</atom:updated>
            <content:encoded><![CDATA[<h4>Behind the Scenes of the “Daily Stars Explorer”: Unveiling the Free Backend &amp; Frontend Tech Stack</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gOELmetFgzHQB1Sbrxn6OQ.png" /></figure><h3>Introduction</h3><p>Starting a <a href="https://emanuelef.github.io/daily-stars-explorer">personal project</a>, I needed an always-online and publicly accessible backend and frontend. To keep it budget-friendly, I checked out what options were available. My main requirements were:</p><ul><li>Zero cost: No recurring or hidden fees throughout the project’s lifecycle.</li><li>Unlimited free tier: Avoid limitations that expire after a specific period or credit usage.</li><li>GitOps deployment: Streamline deployments and configuration management through Git-based workflows.</li><li>Focus on functionality: Basic functionality is sufficient, without the need for high availability or scaling at this stage.</li></ul><p>Bonus:</p><ul><li>ARM compatibility: Leverage the efficiency and affordability of ARM-based instances.</li><li>Free observability: Gain insights into system health and performance without additional costs.</li></ul><p>Here are my picks, and I’ll dive deeper into each in this article:</p><ul><li>Cloud Provider: <a href="https://www.oracle.com/cloud/">Oracle Cloud Infrastructure</a> (generous unlimited free tier, ARM-compatible instances)</li><li>Docker Registry: <a href="https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry">GitHub Packages</a></li><li>Automatic redeploy: <a href="https://containrrr.dev/watchtower/">Watchtower</a></li><li>Frontend Hosting: <a href="https://pages.github.com/">GitHub Pages</a> with a <a href="https://react.dev/">React</a> frontend powered by <a href="https://vitejs.dev/">Vite</a></li><li>Observability: <a href="https://www.honeycomb.io/">Honeycomb</a></li><li>TLS Certificates: <a href="https://zerossl.com/">ZeroSSL</a> (free for 90 days for 3 times) or <a href="https://letsencrypt.org/">Let’s Encrypt</a></li></ul><p>Explore a project leveraging this technology stack at <a href="https://github.com/emanuelef/daily-stars-explorer">https://github.com/emanuelef/daily-stars-explorer</a>, which is detailed in this accompanying <a href="https://medium.com/@emafuma/how-to-get-full-history-of-github-stars-f03cc93183a7">article</a>.</p><p><em>Note: The selections mentioned here align with my current criteria of being free; however, keep in mind that pricing models and offers may evolve in the future.</em></p><h3>Backend</h3><h4>OCI</h4><p>Among the different cloud providers one that has an unlimited free tier is <a href="https://www.oracle.com/uk/cloud/">Oracle Cloud Infrastracture</a> (choose the region you prefer). Their <a href="https://www.oracle.com/uk/cloud/free/#free-cloud-trial">fee tier</a> is very generous:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PA3ubWjU1UlTRq2Xn6bp0g.png" /></figure><p>In my case I set up a single ARM instance where I installed Docker, but the free tier will allow you to use 2 AMD VMs and 4 ARM instances. <br>If you want to install Kubernetes there’s the <a href="https://github.com/jpetazzo/ampernetacle">ampernetacle</a> repo that has the Terraform scripts to deploy a 4-node cluster using ARM machines. This is not meant for use in production but it can be a good and free way to have your own K8s always available.</p><p>As for many Cloud services the registration requires to use a valid email and add a payment verification method. <br>With the free tier you shouldn’t be charged unless you upgrade it to paid account, and even if you do that it’s easy to set a maximum budget just to be sure there will be no additional costs when staying inside free tier limits.<br>More details on the free tier can be found in the <a href="https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier_topic-Always_Free_Resources.htm">Always Free Resources documentation</a>.</p><p>I won’t go into the registration details here, but you can find helpful tutorials on Medium and in the <a href="https://docs.oracle.com/en-us/iaas/Content/home.htm">OCI documentation</a> to guide you through the setup based on your preferences.</p><h4>ARM instance</h4><ul><li><strong>OS:</strong> Canonical-Ubuntu-22.04-aarch64–2023.04.19–0</li><li><strong>Shape:</strong> VM.Standard.A1.Flex</li><li><strong>OCPU count:</strong> 4</li><li><strong>Network bandwidth (Gbps):</strong> 4</li><li><strong>Memory (GB):</strong> 24</li></ul><p>This is the VM where I installed Docker and run <a href="https://containrrr.dev/watchtower/">Watchtower</a>, more on that in the Application section.<br>Note: This VM has kept the same public IP for months, you can read more about public IPs on <a href="https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/managingpublicIPs.htm#overview__res_eph">OCI Documentation</a>.</p><h4>Load Balancer</h4><p>Client HTTP requests are directed to a Load Balancer. The Load Balancer falls under the free tier and has maintained a <a href="https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/managingpublicIPs.htm#overview__Resource">constant public IP</a>.</p><p>Here, I handle TLS termination, and for certificate generation, I opted for <a href="https://zerossl.com/">ZeroSSL</a>. It’s essential to note that ZeroSSL certificates are free for 90 days and can be renewed up to three times.<br>This enables the React app running in the browser to seamlessly call the backend APIs, including Server-Sent Events (SSE), without triggering any browser warnings.</p><p>As an alternative, I’ve begun experimenting with <a href="https://letsencrypt.org/">Let’s Encrypt</a>. Unlike ZeroSSL where I issued a certificate based on IP, Let’s Encrypt appears to necessitate DNS, adding an extra layer of consideration for certificate issuance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kjrAymUBgr7NdMNHy1y-Zw.png" /><figcaption>LB metrics in OCI</figcaption></figure><h4>Backend Application</h4><p>In my project, the backend is a singular web app written in Go, designed to handle specific HTTP requests. The primary function revolves around <a href="https://143.47.235.108:8090/allStars?repo=cue-lang/cue">fetching a repository’s star history</a>, providing daily and cumulative star counts for each day.</p><p>The web framework of choice is <a href="https://github.com/gofiber/fiber">Fiber</a>, which not only facilitates web app development in Go but also integrates middleware.<br>This middleware not only compresses responses for efficient data transfer but also incorporates OpenTelemetry instrumentation — a feature to which I’ve made <a href="https://github.com/gofiber/contrib/commits?author=efumagal">contributions</a>.</p><p>To enable real-time updates to the frontend, I used <a href="https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events">Server Sent Events</a>. In my context, these events play a crucial role in updating the client regarding the progress of star history retrieval, especially for repositories with a substantial number of stars. This feature ensures that clients stay informed during lengthier processes.</p><p>It’s essential to note that the <a href="https://github.com/emanuelef/daily-stars-explorer/blob/main/main.go">code</a>, while serving its purpose in implementing the concept, is not production-ready. The implementation is consolidated almost entirely into a single file, and unit tests are yet to be added. The primary focus here was on bringing the idea to life rather than achieving production-level robustness.</p><h4>Continuos Deploy</h4><p>Each time there are changes to the codebase, a <a href="https://github.com/emanuelef/daily-stars-explorer/blob/main/.github/workflows/ghcr-build-push.yml">GitHub Action</a> is initiated, orchestrating the construction of a Docker image (arm64 and amd64) and its subsequent push to the GitHub registry. The image is tagged with ‘latest’ and a more precise identifier, incorporating the day and build number.</p><p>To manage updates on the OCI Arm VM, I configured Watchtower to monitor image modifications. When an update is detected, <a href="https://containrrr.dev/watchtower/">Watchtower</a> automatically downloads the latest Docker image and initiates the container.</p><p>While this method might not adhere to best practices for production environments, it proved to be a practical and efficient solution for my particular needs, guaranteeing smooth backend updates for the application. In a professional setting tools like ArgoCD or Flux are commonly used to manage deployments on Kubernetes.</p><h3>Frontend</h3><p>The <a href="https://github.com/emanuelef/daily-stars-explorer/tree/main/website">frontend</a> consists of a <a href="https://react.dev/">React</a> application built with <a href="https://vitejs.dev/">Vite</a> for development and deployed on <a href="https://pages.github.com/">GitHub Pages</a>.<br>A free alternative for hosting I use in other projects is <a href="https://www.netlify.com/pricing/">Netlify</a>.</p><p>Whenever changes occur in the React code, this <a href="https://github.com/emanuelef/daily-stars-explorer/blob/main/.github/workflows/publish-website.yml">GitHub action</a> is triggered, which packages the website and publishes it on the GitHub Pages associated with the same repository.</p><p>As for the backend the code is far from production quality, even there the focus was mostly on fast prototyping and choosing the right tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Llx5zsEL6Xq8WvIfmn8Izg.png" /><figcaption>Main page of the website</figcaption></figure><p>Among the various libraries used, it’s worth highlighting <a href="https://mui.com/material-ui/getting-started/">Material-UI</a>, <a href="https://github.com/azouaoui-med/react-pro-sidebar">React-Pro-Sidebar</a>, and <a href="https://www.fusioncharts.com/fusiontime">Fusiontime</a>.</p><p>For complete dependency management, including Docker and GitHub Actions updates, the project utilizes this <a href="https://github.com/emanuelef/daily-stars-explorer/blob/main/.github/dependabot.yml">dependabot configuration</a>.</p><h3>Observability</h3><p>In my previous article <a href="https://medium.com/@emafuma/start-using-opentelemetry-with-go-gin-web-framework-9bebca5abadc">Start Using OpenTelemetry with Go Gin Web Framework</a>, I discussed using Honecomb.io. Their free<a href="https://www.honeycomb.io/pricing"> pricing model</a> is perfect for small personal projects, and the registration process doesn’t require adding a payment method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zcxlHbUtEMYQssNVZYRCWQ.png" /><figcaption>Honeycomb.io</figcaption></figure><h3>Conclusions</h3><p>In summary, the choices I implemented have demonstrated their efficacy, enabling the continuous operation of my project over the past few months without incurring any associated costs.<br>I’m keen to explore alternative options that might meet similar requirements. There also could be free Serverless alternatives that I haven’t yet explored.<br>I invite you to share your suggestions or insights in the comments section. Additionally, I’m more than happy to address any questions and provide detailed explanations of the various steps taken.</p><h3>Stackademic</h3><p>Thank you for reading until the end. Before you go:</p><ul><li>Please consider <strong>clapping</strong> and <strong>following</strong> the writer! 👏</li><li>Follow us <a href="https://twitter.com/stackademichq"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/stackademic"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/c/stackademic"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a></li><li>Visit our other platforms: <a href="https://plainenglish.io/"><strong>In Plain English</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3eaa02ac16b6" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/building-a-cost-free-always-on-personal-project-stack-3eaa02ac16b6">Building a Cost-Free, Always-On Personal Project Stack</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the Rust Ecosystem: A Guide to awesome-rust Repositories]]></title>
            <link>https://itnext.io/navigating-the-rust-ecosystem-a-guide-to-awesome-rust-repositories-813abd2b1393?source=rss-89f70b816ca0------2</link>
            <guid isPermaLink="false">https://medium.com/p/813abd2b1393</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[awesome]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Emanuele Fumagalli]]></dc:creator>
            <pubDate>Sun, 21 Jan 2024 16:54:50 GMT</pubDate>
            <atom:updated>2024-03-04T08:29:09.673Z</atom:updated>
            <content:encoded><![CDATA[<h4>Uncover Daily Statistics and Organise Repositories Based on Metrics</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0QKjKJzbdvSfFytZSuw2YA.png" /></figure><h3>TL;TR</h3><p>Just go straight to the <a href="https://emanuelef.github.io/awesome-rust-repo-stats">website</a> to explore the various graphs. You can interact with most of them by sorting with different metrics or applying filters. Tooltips provide additional information.</p><p>Kindly be aware that certain metrics focus on stars; while they can aid in understanding trends, it’s important not to regard stars as the sole metric for defining a repository’s significance.<br>Enjoy exploring!</p><h3>Introduction</h3><p>If you’re a Rust developer looking for the latest libraries and tools, you might find <a href="https://github.com/rust-unofficial/awesome-rust">awesome-rust</a> helpful. It’s a well-organized list of top-notch Rust resources, regularly updated with contributions from the community. Strict criteria ensure only the best make the cut, so you can trust its quality:</p><blockquote>the entry needs to either have at least 50 stars on Github, 2000 downloads on crates.io, or an equivalent level of other popularity metrics (which should be specified in the PR)</blockquote><p>Predating the Rust hype by a decade, the impressive growth of the awesome-rust repository reflects the language’s current popularity. It currently boasts over 40,000 stars, with daily star growth averaging 30, showcasing sustained momentum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jOD5sVfKdSk9rAe2Fa0eSA.png" /><figcaption><a href="https://emanuelef.github.io/daily-stars-explorer/#/rust-unofficial/awesome-rust">https://emanuelef.github.io/daily-stars-explorer/#/rust-unofficial/awesome-rust</a></figcaption></figure><h3>Awesome Rust stats tool</h3><p>Whether you are looking for a specific tool or library, or more interested in seeing what new GitHub repositories using Rust are trending you might find useful the <a href="https://emanuelef.github.io/awesome-rust-repo-stats/">tool</a> I created to easily visualise and order the Github repos present in the list by different metrics.<br>The website maintains its freshness through a daily process of fetching the awesome-rust list’s GitHub repositories and their associated metrics using efficient GraphQL APIs.</p><h3>Views</h3><h4>Grid</h4><p>The metrics are displayed in a grid, allowing you to sort them based on your preference. It’s also possible to filter by category or subcategory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rdv5XCA9OW9OPFcFYhi2NA.png" /></figure><p><em>The collected metrics include:</em></p><p><em>stars: Total stars up to the latest update<br>new-stars-last-30d: Number of new stars in the last 30 days<br>new-stars-last-7d: Number of new stars in the last 7 days<br>stars-per-mille-30d: (New stars in the last 30 days / Total Stars) * 1000<br>days-last-star: Number of days since the last star, a value of 0 indicates that the repository received at least one star within the last day</em></p><p><em>days-last-commit: Number of days since the last commit, 0 indicates at least one commit in the last day<br>unique-contributors: Number of unique contributors to the commits in the last 30 days</em><strong><em><br></em></strong><em>new-commits-last-30d: Number of commits in the last 30 days</em></p><p><em>age: Days passed since the project was created on GitHub<br>mentionable-users: Mentionable users returned by the GitHub GraphQL Api<br>archived: boolean to indicate if the repo has been archived</em></p><h4>Bubble chart</h4><p>Inspired by Hans Rosling’s Gapminder, this interactive chart takes your data exploration to the next level. Choose custom metrics for axes, bubble size to unlock hidden insights. Hover for detailed stats, click for tailored actions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0QKjKJzbdvSfFytZSuw2YA.png" /></figure><h4>Dependencies</h4><p>In this views are collected the dependencies used by the repos in the list, with metrics including the number of repos using them and crates downloads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*0xsBH7LBkn4WKX_2FjoavA.png" /></figure><h4>Last 30 days star history</h4><p>In this view, you can explore the star history of the last 30 days, showcasing both the daily influx of new stars and the cumulative total.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7UF7w3UavmAQ4YY7SLMolQ.png" /></figure><p>In case interested to the full star history you can follow the link to another tool I’ve developed: <a href="https://emanuelef.github.io/gh-repo-stats-server/#/juspay/hyperswitch">https://emanuelef.github.io/gh-repo-stats-server/#/juspay/hyperswitch</a></p><h3>Conclusions</h3><p>I created this tool for fellow Rust explorers like myself. It fuels my curiosity for new libraries and innovative projects, sparking ideas and keeping me constantly inspired. Now, I want to share that power with you, to unlock your own journey of discovery within the Rust ecosystem.</p><p>I’m curious to hear if others find this tool helpful as well, let me know in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=813abd2b1393" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/navigating-the-rust-ecosystem-a-guide-to-awesome-rust-repositories-813abd2b1393">Navigating the Rust Ecosystem: A Guide to awesome-rust Repositories</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring CNCF Repos: Daily Stats and Language Distribution]]></title>
            <link>https://itnext.io/exploring-cncf-repos-daily-stats-and-language-distribution-055305faf420?source=rss-89f70b816ca0------2</link>
            <guid isPermaLink="false">https://medium.com/p/055305faf420</guid>
            <category><![CDATA[computer-language]]></category>
            <category><![CDATA[cncf]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[visualization]]></category>
            <dc:creator><![CDATA[Emanuele Fumagalli]]></dc:creator>
            <pubDate>Mon, 04 Dec 2023 21:47:52 GMT</pubDate>
            <atom:updated>2024-03-13T16:23:02.137Z</atom:updated>
            <content:encoded><![CDATA[<h4>Discover Daily Statistics for CNCF Repositories — Sort by Metrics and Visualise Language Distribution Charts</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ftapUVymTYMNT0AgVnPQdQ.png" /></figure><h3>Introduction</h3><p>This <a href="https://github.com/emanuelef/cncf-repos-stats">project</a> is designed to offer a daily snapshot of <a href="https://www.cncf.io/">CNCF</a> projects on GitHub, showcasing comprehensive statistics through various metrics presented in insightful graphs.<br>This project is meant for people who are curious, enjoy staying updated, and just want a straightforward way to track trends in CNCF.<br>For the statistics maintained by the CNCF community you can look at the Grafana dashboards in <a href="https://devstats.cncf.io/">Cloud Native Devstats</a> or the <a href="https://cncf.landscape2.io/stats">CNCF Landscape v2 Stats</a>.</p><h3>TL;TR</h3><p>Just go straight to the <a href="https://emanuelef.github.io/cncf-repos-stats/">website</a> to explore the various graphs. You can interact with most of them by sorting with different metrics or applying filters. Tooltips provide additional information.</p><p>Kindly be aware that certain metrics focus on stars; while they can aid in understanding trends, it’s important not to regard stars as the sole metric for defining a repository’s significance.<br>Enjoy exploring!</p><h3>Metrics collected</h3><p>A GitHub action fetches daily repositories from this <a href="https://github.com/cncf/landscape/blob/master/landscape.yml">YAML file</a>, generating a <a href="https://github.com/emanuelef/cncf-repos-stats/blob/main/analysis-latest.csv">CSV file</a> with metrics for GitHub repositories only.</p><p>The collected metrics include:</p><p>stars: Total stars up to the latest update<br>new-stars-last-30d: Number of new stars in the last 30 days<br>new-stars-last-7d: Number of new stars in the last 7 days<br>stars-per-mille-30d: (New stars in the last 30 days / Total Stars) * 1000<br>days-last-star: Number of days since the last star, a value of 0 indicates that the repository received at least one star within the last day</p><p>days-last-commit: Number of days since the last commit, 0 indicates at least one commit in the last day<br>unique-contributors: Number of unique contributors to the commits in the last 30 days<strong><br></strong>new-commits-last-30d: Number of commits in the last 30 days</p><p>days-since-creation: Days passed since the project was created on GitHub<br>mentionable-users: Mentionable users returned by the GitHub GraphQL Api<br>language: main language used in the repo<br>archived: boolean to indicate if the repo has been archived</p><p>CNCF status: Graduated, Sandbox, Incubating, Archived</p><p>Beyond that the action is fetching the daily <a href="https://github.com/emanuelef/cncf-repos-stats/blob/main/stars-history-30d.json">stars</a> and <a href="https://github.com/emanuelef/cncf-repos-stats/blob/main/commits-history-30d.json">commits</a> for the last 30 days in JSON files.</p><h3>Views</h3><h4>Grid</h4><p>The metrics are displayed in a grid, allowing you to sort them based on your preference. It’s also possible to filter by category or subcategory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*prKtvfFEn0j9dVhm2lzWpQ.png" /></figure><p>For each repo there’s a link to the project on GitHub and an internal link to show the stars/commits in the last 30 days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OVtr1w1JqxaXJhs9b4dv6g.png" /></figure><h4>Languages</h4><p>This stacked bar chart illustrates the primary languages used, grouped by CNCF Status. You can interact with the legend to toggle the visibility of each language. The languages are color-coded based on GitHub’s designated language colors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pcSUZvvQjRGCrCdZeUpdeQ.png" /></figure><h4>Bubble chart</h4><p>This design draws inspiration from Hans Rosling’s gapminder bubble chart. You have the flexibility to choose different metrics for the axes, as well as for bubble size and color. Hovering over a bubble reveals an informative tooltip displaying values for select metrics. Additionally, the click action for the bubbles can be customized according to your preference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*98fMyXQeQl8P6YaBmJgi7g.png" /><figcaption>Colour by CNCF Status</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tGYy1zBgwQv88ONvHnCdow.png" /><figcaption>Colour by liveness, a metric to show how active a repo is</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tejI_ngb5Zhq44bPzSKbpQ.png" /><figcaption>Repos coloured by main language used</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fprIhUqolZa3a4ZdlAb1eA.png" /><figcaption>Change of axis and size based on different metrics</figcaption></figure><p>When choosing the “Full Star History” as a click option, a link opens to another project providing the complete history of stars. As mentioned earlier, stars can offer some indication of interest in a repository but may be influenced by other factors. Use your judgment. Personally, I find more interest in the trend of daily stars. If I observe growth and consistency, it sparks my curiosity to learn more about that project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wi1M7VKCpWZngzQ6kemyRQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k8wa8oxE2Nzyw4LMPsPKaQ.png" /></figure><p>Some example of links to full stars history:<a href="http://Cilium full stars history"><br>https://emanuelef.github.io/gh-repo-stats-server/#/cilium/cilium</a><br><a href="https://emanuelef.github.io/gh-repo-stats-server/#/kubernetes/kubernetes">https://emanuelef.github.io/gh-repo-stats-server/#/kubernetes/kubernetes</a></p><h4>Treemap</h4><p>Explore a treemap featuring key metrics. Hover over a repository to display its name, and click to navigate to the GitHub project page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XuvIQ5opIOElnhbffw6SqA.png" /><figcaption>Treemap by unique authors in the last 30 days commits</figcaption></figure><h3>Conclusions</h3><p>Working on this project has been a rewarding experience for me. I frequently visit the website and, on occasion, discover new repositories that are exhibiting growth. It has proven to be a delightful and effective way for me to stay up to date with the latest developments in CNCF projects.</p><p>I’m curious to hear if others find this tool helpful as well, let me know in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=055305faf420" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/exploring-cncf-repos-daily-stats-and-language-distribution-055305faf420">Exploring CNCF Repos: Daily Stats and Language Distribution</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to get full history of GitHub stars]]></title>
            <link>https://medium.com/@emafuma/how-to-get-full-history-of-github-stars-f03cc93183a7?source=rss-89f70b816ca0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f03cc93183a7</guid>
            <category><![CDATA[repositories]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Emanuele Fumagalli]]></dc:creator>
            <pubDate>Thu, 12 Oct 2023 22:30:21 GMT</pubDate>
            <atom:updated>2025-03-14T08:11:40.790Z</atom:updated>
            <content:encoded><![CDATA[<h3>Unmasking the GitHub Star History: Track Daily Trends &amp; Break the 40k Limit</h3><h4>Gain deeper insights into your favourite open-source repositories. Explore star trends showing daily stars and go beyond the 40k-star limit.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3f8KWVXiopybMPTxCDQi6g.png" /></figure><p>In this article, I’m excited to introduce a <a href="https://emanuelef.github.io/daily-stars-explorer">service to retrieve the full star history</a> of a GitHub repo. As of now, GitHub lacks built-in functionality for showing full star history (I suppose it will be added at some point), and one workaround is to rely on external services like <a href="https://star-history.com/">https://star-history.com/</a>.</p><p>The webserver is developed using a Go library I created that can be accessed on GitHub at the following link: <a href="https://github.com/emanuelef/github-repo-activity-stats">https://github.com/emanuelef/github-repo-activity-stats</a>. While documentation is currently pending, I have plans to provide comprehensive documentation in the near future.</p><p>The website showcases the capabilities of this library and serves as a practical tool to obtain comprehensive star history data for any public GitHub repository. The website is available here: <a href="https://emanuelef.github.io/daily-stars-explorer">https://emanuelef.github.io/daily-stars-explorer</a>.<br>In case interested on the server side and the React app the code is available here: <a href="https://github.com/emanuelef/daily-stars-explorer">https://github.com/emanuelef/daily-stars-explorer</a>.</p><p>This project primarily serves as a response to inquiries regarding trends within GitHub repositories. The accompanying website is more of an experimental concept and may not undergo significant further development.<br>While there are certainly other factors to consider when evaluating the quality of a GitHub repository, I believe that tracking stars can provide valuable insights into trends and the repository’s popularity over time.</p><p>One pivotal motivation behind creating this website is to provide a means for accessing the complete star history, including the daily count of individual stars. This in-depth data representation offers a more accurate depiction of trending patterns.</p><p>It’s worth noting that using REST APIs, such as those employed by services like star-history.com, comes with certain limitations. There’s a restriction on fetching a maximum of 400 pages, each containing up to 100 items. As a result, this approach enables access to a maximum of 40k stars. As you can observe in the graph, this limitation results in a linear progression from 40k stars up to the current total star count.</p><pre>https://api.github.com/repos/kubernetes/kubernetes/stargazers?page=400&amp;per_page=100</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*Gd1_ezWDZTQgIQMxcORNwQ.png" /></figure><p>The same repository will generate the following graph, offering the added functionality of zooming in on specific time periods and providing tooltips for each day since the repository’s inception.<br>The URL for the next graph is <a href="https://emanuelef.github.io/daily-stars-explorer/#/kubernetes/kubernetes">https://emanuelef.github.io/daily-stars-explorer/#/kubernetes/kubernetes</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V1YLcQcCp6MtI1U4NicUuw.png" /></figure><p>In the website I’ve developed, you can also export a CSV file that contains the daily and total star counts for each day. This is in contrast to stars-history.com, where the CSV only includes data for specific days, as shown in the example I downloaded:</p><pre>kubernetes/kubernetes,Tue Jun 10 2014 18:13:03 GMT+0100 (British Summer Time),0<br>kubernetes/kubernetes,Tue Aug 12 2014 15:07:03 GMT+0100 (British Summer Time),2610<br>kubernetes/kubernetes,Wed Jan 28 2015 22:44:25 GMT+0000 (Greenwich Mean Time),5280<br>kubernetes/kubernetes,Tue Jul 14 2015 03:00:22 GMT+0100 (British Summer Time),7950<br>kubernetes/kubernetes,Tue Dec 15 2015 05:02:03 GMT+0000 (Greenwich Mean Time),10620<br>kubernetes/kubernetes,Thu May 26 2016 00:07:10 GMT+0100 (British Summer Time),13290<br>kubernetes/kubernetes,Tue Oct 11 2016 12:55:50 GMT+0100 (British Summer Time),15960<br>kubernetes/kubernetes,Thu Feb 02 2017 19:29:45 GMT+0000 (Greenwich Mean Time),18630<br>kubernetes/kubernetes,Thu May 11 2017 09:27:32 GMT+0100 (British Summer Time),21270<br>kubernetes/kubernetes,Tue Aug 15 2017 19:22:31 GMT+0100 (British Summer Time),23940<br>kubernetes/kubernetes,Fri Nov 10 2017 00:15:48 GMT+0000 (Greenwich Mean Time),26610<br>kubernetes/kubernetes,Sun Jan 21 2018 12:54:42 GMT+0000 (Greenwich Mean Time),29280<br>kubernetes/kubernetes,Wed Mar 28 2018 07:24:01 GMT+0100 (British Summer Time),31950<br>kubernetes/kubernetes,Sat Jun 02 2018 17:37:56 GMT+0100 (British Summer Time),34620<br>kubernetes/kubernetes,Thu Aug 09 2018 20:07:02 GMT+0100 (British Summer Time),37290<br>kubernetes/kubernetes,Tue Oct 16 2018 11:31:30 GMT+0100 (British Summer Time),39960<br>kubernetes/kubernetes,Fri Oct 06 2023 13:34:09 GMT+0100 (British Summer Time),102106</pre><p>And the following is an example of a CSV you can download in the website I created:</p><pre>date,day-stars,total-stars<br>06-06-2014,0,0<br>07-06-2014,0,0<br>08-06-2014,0,0<br>09-06-2014,317,317<br>10-06-2014,441,758<br>11-06-2014,158,916<br>12-06-2014,96,1012<br>13-06-2014,31,1043<br>14-06-2014,37,1080<br>15-06-2014,79,1159<br>16-06-2014,63,1222<br>17-06-2014,55,1277<br>18-06-2014,34,1311<br>19-06-2014,26,1337<br>20-06-2014,12,1349<br>21-06-2014,10,1359<br>22-06-2014,18,1377<br>23-06-2014,10,1387<br>24-06-2014,10,1397<br>...</pre><p>Currently, my existing solution demonstrates how to retrieve the complete star history, but it has some limitations. One major drawback is its caching capability, which is limited. Additionally, a significant disadvantage is that the GraphQL APIs employ a cursor-based approach, making it impossible to parallelise requests effectively. The only potential workaround I’ve considered involves initiating requests for the first half from the beginning and the second half from the end. However, this would only result in a partial reduction in the total time required. Please feel free to share your insights in the comments if you have any suggestions on how to parallelise GitHub requests for paging when using GraphQL.<br>As an example, fetching the entire star history for a repository with 100k stars will approximately takes around 5 minutes. The obtained result is subsequently cached for a few days. Additionally, there’s an option to force a refetch of the data up to the current day.<br>Please note that I’m using my Personal Access Token (not shared in the repo with the server) and so there’s a limit of 5k API calls per hours, that means you need to wait if that limit is reached.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CGq968k1Vhz8mRWCeak44A.png" /></figure><p>Possible future improvements:<br>- query both forward and backward to reduce total time [DONE]<br>- add persistency for cache<br>- analyse time series to find peaks or trends<br>- compare multiple repos in the same graph [DONE]</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f03cc93183a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Start using OpenTelemetry with Go Gin Web Framework]]></title>
            <link>https://itnext.io/start-using-opentelemetry-with-go-gin-web-framework-9bebca5abadc?source=rss-89f70b816ca0------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bebca5abadc</guid>
            <category><![CDATA[go]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[gin]]></category>
            <category><![CDATA[instrumentation]]></category>
            <dc:creator><![CDATA[Emanuele Fumagalli]]></dc:creator>
            <pubDate>Wed, 23 Aug 2023 21:07:47 GMT</pubDate>
            <atom:updated>2024-03-29T19:40:40.427Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to instrument a Go web server using Gin and export traces to Honeycomb.io</h4><p>This is an adaptation of my previous article <a href="https://blog.stackademic.com/start-using-opentelemetry-with-go-fiber-1005697d841f">https://blog.stackademic.com/start-using-opentelemetry-with-go-fiber-1005697d841f</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBzJh40ohr0kP_k-rECROQ.png" /></figure><p><a href="https://blog.stackademic.com/start-using-opentelemetry-with-go-fiber-1005697d841f">Start using OpenTelemetry with Go Fiber</a></p><h3>Introduction</h3><p>This is a tutorial on how to instrument Go code with OpenTelemetry. <br>This particular article is a bit more opinionated, and it focuses on how to instrument an HTTP webserver written in Go and using <a href="https://github.com/gin-gonic/gin">Gin</a>, and run HTTP and gRPC requests from there.<br>It also shows how to export the traces to <a href="https://www.honeycomb.io/">Honeycomb.io</a>, a popular observability solution that offers a free plan with unlimited time and doesn’t require a credit card for registering.<br>On the accompanying repo <a href="https://github.com/emanuelef/go-fiber-honeycomb/blob/main/README.md">README</a> there’s a one minute video that shows how to set up everything and start seeing the spans.</p><p>I tried to collect here the most common questions when starting to instrument a Go app with OpenTelemetry, .<br>The only signal discussed here are the <a href="https://opentelemetry.io/docs/concepts/signals/traces/">traces</a>, I’m not going to present the <a href="https://opentelemetry.io/docs/concepts/signals/metrics/">OTeL metrics</a> or <a href="https://opentelemetry.io/docs/concepts/signals/logs/">logs</a> and I’ll probably write another article related to them.</p><p>This is definitely not a guide on <a href="https://opentelemetry.io/docs/">OpenTelemetry</a> or Observability but a very practical way to start seeing distributed tracing on a popular observability solution.</p><p><em>Disclaimer: The code is for showing how to do the instrumentation and not an example on how to write clean code. There are no unit tests and no abstractions.</em></p><p>The repo is available on GitHub:</p><p><a href="https://github.com/emanuelef/go-gin-honeycomb">GitHub - emanuelef/go-gin-honeycomb: Example of instrumenting a Go Gin app and export traces to Honeycomb.io</a></p><p>To run the example code on your local machine the requirements are:</p><ul><li>Go (&gt; 1.19)</li><li>Docker (if you want to run Docker compose or build the images)</li></ul><h3>TL;TR</h3><p>Just create an account in Honeycomb.io, clone the repo <a href="https://github.com/emanuelef/go-gin-honeycomb">https://github.com/emanuelef/go-gin-honeycomb</a> (or run from GitHub Codespaces, docker compose works there as well) create the .env files with the API key (manually or running set_token.sh) and from the root folder run:</p><pre>./set_token.sh</pre><pre>docker compose up</pre><p>Call the GET endpoints and watch the traces appear in Honeycomb.io.</p><ul><li>curl <a href="http://localhost:8080/hello-grpc">http://localhost:8080/hello</a></li><li>curl <a href="http://localhost:8080/hello">http://localhost:8080/hello-resty</a></li><li>curl <a href="http://localhost:8080/hello-grpc">http://localhost:8080/hello-grpc</a></li></ul><p>If you are using <a href="https://www.postman.com/">Postman</a> there’s a <a href="https://github.com/emanuelef/go-fiber-honeycomb/tree/main/postman-collection">collection</a> in the repo.<br>Or you can just run the script that calls all of the endpoints:</p><pre>./run_http_requests.sh</pre><h3>Create an account on Honeycomb.io</h3><p>Any observability solution could be used with the code in the repo, and the only changes needed to export the traces to a service different than Honeycomb.io will be in the env variables.</p><p>Honeycomb.io, as of today, offers a very generous <a href="https://www.honeycomb.io/pricing">free plan</a> with no need to add a credit card to create the account.<br>All is needed is an email from where you will get the activation link.</p><p>Sign up <a href="https://ui.honeycomb.io/signup">here</a>, confirm the email and create a new team (any team name is fine and all the steps for the creation of the team are presented automatically after signing up).</p><p>The home page will contain instructions on how to do the instrumentation and the API key that will be needed in the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1b6hvHarM05rJcp1.png" /></figure><p>You can retrieve the API key even later by going into Account (at the bottom on the left hand side) and Settings.</p><p>The Home page will start showing the traces after instrumenting and running the code in the repo.</p><h3>Instrumentation</h3><h3>Preparation</h3><p>Get the code in the repo <a href="https://github.com/emanuelef/go-gin-honeycomb">https://github.com/emanuelef/go-gin-honeycomb</a>, either clone it or get the zip.</p><p>Alternatively you can use the GitHub Codespaces (I tried that for this example and it works perfectly).</p><p>Run set_token script and paste the API Key from the Honeycomb website.</p><pre>./set_token.sh<br>Honeycomb API Key: *********</pre><p>This will just create the .env files needed for every app (they contain the API keys so better not put them in the repos), should that not work just copy the .env.example files in each folder (you might just want to start with the one in the root folder, /secondary and/grpc-server are needed later) to .env and replace your_key_here with the API key.</p><p>At this point we can check if everything is set up correctly and the traces are exported.<br>In a terminal from the root folder run :</p><pre>go run main.go</pre><p>From another terminal:</p><pre>curl http://127.0.0.1:8080/hello</pre><p>Go to the Honeycomb.io website and after a short time you should see the exported trace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WGB9ojlIvHJFiNf5.png" /></figure><p>Click on the button on the left and you will see the trace with the single span.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6ObCu9_-vhdsaqtr.png" /></figure><p>Span created for a request to /hello</p><h3>Set up trace provider and exporter</h3><p>First step is to create a trace provider with an exporter.</p><p>In this article and in the repo the traces are exported directly to the public endpoint provided by Honeycomb and specified in the OTEL_EXPORTER_OTLP_ENDPOINT (<a href="https://api.honeycomb.io/">https://api.honeycomb.io:443</a>) env variable. <br>Using a <a href="https://opentelemetry.io/docs/collector/">Collector</a> is probably more likely to happen in production.</p><p>In <a href="https://github.com/emanuelef/go-fiber-honeycomb/blob/main/otel_instrumentation/opentelemetry_setup.go">opentelemetry_setup.go</a> the function InitializeGlobalTracerProvider creates a Trace Provider and an Exporter. The Go OpenTelemetry SDK will use the env variables defined in the .env file, the only ones needed for this examples are:</p><pre>OTEL_SERVICE_NAME=GoGinExample<br>OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io:443<br>OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=your_key_here</pre><p>During the creation of the Trace Provider is possible to add custom <a href="https://opentelemetry.io/docs/instrumentation/go/resources/">resources</a> that will be added to each span as attributes, resources can be added with the OTEL_RESOURCE_ATTRIBUTES<em> </em>env variable as well.</p><p>Check the <a href="https://opentelemetry.io/docs/specs/otel/trace/semantic_conventions/">Semantic Conventions</a> to see if the kind of resources you want to add are already defined.</p><p>This is where is possible to add the resources in code:</p><pre>resource, rErr := resource.Merge(<br> resource.Default(),<br> resource.NewWithAttributes(<br>  semconv.SchemaURL,<br>  attribute.String(&quot;environment&quot;, &quot;test&quot;),<br>  attribute.String(&quot;stack&quot;, &quot;001&quot;),<br> ),<br>)</pre><h3>Instrument Gin webserver with Otelgin</h3><p>Gin can be instrumented with <a href="https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/github.com/gin-gonic/gin/otelgin">Otelgin</a> middleware.</p><pre>r := gin.New()</pre><pre>r := gin.New()<br><br>r.Use(gin.Recovery())<br>r.Use(otelgin.Middleware(&quot;my-server&quot;))<br><br>r.GET(&quot;/hello&quot;, func(c *gin.Context) {<br> c.JSON(http.StatusNoContent, gin.H{})<br>})</pre><p>By adding the otelgin Middleware a span is created automatically for every HTTP request received.</p><p>Should you want for some reason to exclude some endpoints from generating spans you can use the otelgin.WithFilter option, if the passed function returns true the otelgin Middleware will skip the OTeL instrumentation.</p><pre><br>var notToLogEndpoints = []string{&quot;/health&quot;, &quot;/metrics&quot;}<br><br>func FilterTraces(req *http.Request) bool {<br> return slices.Index(notToLogEndpoints, req.URL.Path) == -1<br>}<br><br>...<br><br>r.Use(otelgin.Middleware(&quot;my-server&quot;, otelgin.WithFilter(FilterTraces)))</pre><p>As showed in the Preparation a call to GET /hello will create the span.</p><h3>Add child span</h3><p>Quite likely the work done by the HTTP request will involve additional spans.<br>In order to create a child span, or a new root span, you need to have a tracer, there will probably be one for the entire app or one for every package.</p><pre>var tracer trace.Tracer</pre><pre>func init() {<br> // this seems to work even if the init happens before setting up the trace provider<br> tracer = otel.Tracer(&quot;github.com/emanuelef/go-gin-honeycomb&quot;)<br>}</pre><p>Then in the code inside a gin handler</p><pre>ctx, childSpan := tracer.Start(c.Request.Context(), &quot;custom-child-span&quot;)<br>time.Sleep(10 * time.Millisecond) // simulate some work<br>childSpan.End()</pre><p>c.Request.Context() retrieves the context from Gin Context with the opentelemetry infos (trace id, span id).</p><p>GET /hello-child will generate the following spans:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eVULjrh7vzMSqzNL.png" /></figure><p>If the ctx passed to the tracer.Start contains a span then the newly-created span will be a child of that span, otherwise it will be a root span.<br>tracer.Start also accepts SpanStartOption so it is possible to add attributes or the span kind if needed.</p><h3>Create a new span</h3><p>If a new root span is needed (not child of any other) the process is similar to create a child span but passing a context that doesn’t contain a span already.</p><pre>ctx, span := tracer.Start(context.Background(), &quot;timed-operation&quot;)<br>// Do something<br>span.End()</pre><p>In the repo main app code there’s a time tick that creates a new root span every minute.</p><h3>Instrument outgoing HTTP requests</h3><p>I’ll present three ways to add the spans for each outgoing HTTP request (and also propagate the trace to the web server called).<br>In the end they all build the client with the standard http.Client.</p><p><strong>Otelhttp package</strong></p><p>The simplest way is to use the package otelhttp defined in go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp, this wraps the http requests adding a span around.</p><p>The methods currently implemented there are:</p><ul><li>Get</li><li>Head</li><li>Post</li><li>PostForm</li></ul><p>Inside a Gin handler the context to pass is c.Request.Context(), otherwise the HTTP request span will be a root one.</p><pre>externalURL := &quot;https://pokeapi.co/api/v2/pokemon/ditto&quot;<br>resp, err := otelhttp.Get(c.Request.Context(), externalURL)<br>_, _ = io.ReadAll(resp.Body) // This is needed to close the span</pre><p>A little gotcha here, the io.ReadAll is actually required to close the span, even if the response is not needed.</p><p>Otelhttp will also take care of the Transport injecting the span context into the outbound request headers.</p><p>In the repo a call to GET <em>/hello-otelhttp</em> will create the following spans, this will require the secondary app to run and is used to demonstrate that the traceparent (a single HTTP header containing trace_id and span_id, among version and trace_flags) is carried in the HTTP header allowing the distributed tracing across multiple apps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t82L6Nik-oiR2sfR.png" /></figure><p><strong>http.Client</strong></p><p>To have more control (and use methods not implemented in otelhttp yet, like PATCH) you can use the http.Client directly.</p><pre>externalURL := &quot;https://pokeapi.co/api/v2/pokemon/ditto&quot;<br>client := http.Client{<br> Transport: otelhttp.NewTransport(<br> http.DefaultTransport,<br>   ),<br> }</pre><pre>req, err := http.NewRequestWithContext(c.Request.Context(), &quot;GET&quot;, externalURL, nil)<br>if err != nil {<br> return err<br>}<br>resp, _ := client.Do(req)<br>_, _ = io.ReadAll(resp.Body)</pre><p>Passing the context with an existing span to http.NewRequestWithContext<em> (</em>if inside a Gin handler will be c.Request.Context()) will make the outgoing HTTP call to be a child span of the incoming HTTP call served by Gin.</p><p>By adding the otelhttp transport (using otelhttp.NewTransport) to the http.Client the outgoing request will have injected in the HTTP header (traceparent) all the information needed for enabling the distributed tracing.<br>Alternatively, the traceparent header can be injected manually in the http.Request with:</p><pre>otel.GetTextMapPropagator().Inject(c.Request.Context(), propagation.HeaderCarrier(req.Header))</pre><p>To make it more interesting in the GET <em>/hello-http-client</em> implemented in the accompaning repo the Transport has an option to show the details of the HTTP request (htttp.getconn, htt.headers…)</p><pre>client := http.Client{<br>   Transport: otelhttp.NewTransport(<br>    http.DefaultTransport,<br>    otelhttp.WithClientTrace(func(ctx context.Context) *httptrace.ClientTrace {<br>     return otelhttptrace.NewClientTrace(ctx)<br>    })),<br>  }</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QnisdLJ0sf8cD-kL.png" /></figure><p><strong>Resty</strong></p><p>As the last example on how to have outgoing HTTP calls intrumented we will use the <a href="https://github.com/go-resty/resty">Resty</a> popular library.<br>As of today Resty doesn’t support OTeL instrumentation directly but it is possible by passing the http.Client<em> </em>in the resty.NewWithClient</p><pre>externalURL := &quot;https://pokeapi.co/api/v2/pokemon/ditto&quot;</pre><pre>client := resty.NewWithClient(<br> &amp;http.Client{<br>  Transport: otelhttp.NewTransport(http.DefaultTransport,<br>   otelhttp.WithClientTrace(func(ctx context.Context) *httptrace.ClientTrace {<br>    return otelhttptrace.NewClientTrace(ctx)<br>   })),<br> },<br>)<br>restyReq := client.R()<br>restyReq.SetContext(c.Request.Context())<br>resp, _ := restyReq.Get(externalURL)</pre><p>Make sure to set the context with the existing span (if you want to create a child span)</p><p>As for the previous example if you don’t want to set the Transport, or not having the opportunity in another library to pass an http.Client with the transport, you can inject the traceparent header directly with:</p><pre>otel.GetTextMapPropagator().Inject(c.Request.Context(), propagation.HeaderCarrier(restyReq.Header))</pre><p>the GET <em>/hello-resty</em> in the repo will produce something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p3Q4S5CU6pNe-dkS.png" /></figure><h3>Add span attributes</h3><p>To add additional attributes to an existing span:</p><pre>span := trace.SpanFromContext(c.Request.Context())<br>span.SetAttributes(attribute.Bool(&quot;isTrue&quot;, true), attribute.String(&quot;test.text&quot;, &quot;Ciao&quot;))</pre><h3>Add span events</h3><p>Events can be added to a span to define something relevant happening in a point in time inside the span, they can have attributes as well.<br>Events can also be used as a form of log doing something like:</p><pre>// simple event<br>span.AddEvent(&quot;Done second fake long running task&quot;)</pre><pre>// event with attributes to have a log linked to the span<br>span.AddEvent(&quot;log&quot;, trace.WithAttributes(<br> attribute.String(&quot;log.severity&quot;, &quot;warning&quot;),<br> attribute.String(&quot;log.message&quot;, &quot;Example log&quot;),<br>))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*HR9aeEEwByzW9mNR.png" /></figure><h3>Instrument gRPC calls</h3><p>In the <a href="https://github.com/emanuelef/go-gin-honeycomb">https://github.com/emanuelef/go-gin-honeycomb</a> there is also an example to see traces when using gRPC.</p><p>In order to instrument the gRPC calls.</p><p>On the client:</p><pre>conn, err := grpc.Dial(grpcTarget,<br> grpc.WithTransportCredentials(insecure.NewCredentials()),<br> grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()))</pre><pre>defer conn.Close()<br>cli := protos.NewGreeterClient(conn)<br>r, err := cli.SayHello(c.Request.Context(), &amp;protos.HelloRequest{Greeting: &quot;ciao&quot;})</pre><p>On the server:</p><pre>grpcServer := grpc.NewServer(grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()))</pre><p>call GET <a href="http://localhost:8080/hello-grpc">http://localhost:8080/hello-grpc</a> to see it in action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*XMJlJ0cUlTJ4dOPO.png" /></figure><h3>Conclusion</h3><p>I hope this article and the related <a href="https://github.com/emanuelef/go-gin-honeycomb">repo</a> will help in easily starting to instrument your Go code with OpenTelemetry. <br>We have been using OTeL in production and having the distributed tracing (and all the other amazing features for analyzing all the OTeL signals) has helped understand our platform performance and reduce troubleshooting time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bebca5abadc" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/start-using-opentelemetry-with-go-gin-web-framework-9bebca5abadc">Start using OpenTelemetry with Go Gin Web Framework</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Start using OpenTelemetry with Go Fiber]]></title>
            <link>https://blog.stackademic.com/start-using-opentelemetry-with-go-fiber-1005697d841f?source=rss-89f70b816ca0------2</link>
            <guid isPermaLink="false">https://medium.com/p/1005697d841f</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[fiber]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <dc:creator><![CDATA[Emanuele Fumagalli]]></dc:creator>
            <pubDate>Thu, 17 Aug 2023 15:47:27 GMT</pubDate>
            <atom:updated>2023-08-24T05:02:35.989Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to instrument a Go web server using Fiber and export traces to Honeycomb.io</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GCOrikZ5ZKHUSKa0vD_tdQ.png" /></figure><h3>Introduction</h3><p>This is a tutorial on how to instrument Go code with OpenTelemetry. <br>This particular article is a bit more opinionated and it focuses on how to instrument an HTTP webserver written in Go and using <a href="https://gofiber.io/">Fiber</a> (if you are using Gin there’s an adaptation of this article <a href="https://medium.com/@emafuma/start-using-opentelemetry-with-go-gin-web-framework-9bebca5abadc">here</a>), and run HTTP and gRPC requests from there.<br>It also shows how to export the traces to <a href="https://www.honeycomb.io/">Honeycomb.io</a>, a popular observability solution that offers a free plan with unlimited time and doesn’t require a credit card for registering.<br>On the accompanying repo <a href="https://github.com/emanuelef/go-fiber-honeycomb/blob/main/README.md">README</a> there’s a one minute video that shows how to set up everything and start seeing the spans.</p><p>I tried to collect here the most common questions when starting to instrument a Go app with OpenTelemetry, .<br>The only signal discussed here are the <a href="https://opentelemetry.io/docs/concepts/signals/traces/">traces</a>, I’m not going to present the <a href="https://opentelemetry.io/docs/concepts/signals/metrics/">OTeL metrics</a> or <a href="https://opentelemetry.io/docs/concepts/signals/logs/">logs</a> and I’ll probably write another article related to them.</p><p>This is definitely not a guide on <a href="https://opentelemetry.io/docs/">OpenTelemetry</a> or Observability but a very practical way to start seeing distributed tracing on a popular observability solution.</p><p><em>Disclaimer: The code is for showing how to do the instrumentation and not an example on how to write clean code. There are no unit tests and no abstractions.</em></p><p>The repo is available on GitHub:</p><p><a href="https://github.com/emanuelef/go-fiber-honeycomb">GitHub - emanuelef/go-fiber-honeycomb: Example of instrumenting a Go Fiber app and export to Honeycomb.io</a></p><p>To run the example code on your local machine the requirements are:</p><ul><li>Go (&gt; 1.19)</li><li>Docker (if you want to run Docker compose or build the images)</li></ul><h3>TL;TR</h3><p>Just create an account in Honeycomb.io, clone the repo <a href="https://github.com/emanuelef/go-fiber-honeycomb/tree/main">emanuelef/go-fiber-honeycomb</a> (or run from GitHub Codespaces, docker compose works there as well) create the .env files with the API key (manually or running set_token.sh) and from the root folder run:</p><pre>./set_token.sh<br><br>docker compose up</pre><p>Call the GET endpoints and watch the traces appear in Honeycomb.io.</p><ul><li>curl <a href="http://localhost:8080/hello-grpc">http://localhost:8080/hello</a></li><li>curl <a href="http://localhost:8080/hello">http://localhost:8080/hello-resty</a></li><li>curl <a href="http://localhost:8080/hello-grpc">http://localhost:8080/hello-grpc</a></li></ul><p>If you are using <a href="https://www.postman.com/">Postman</a> there’s a <a href="https://github.com/emanuelef/go-fiber-honeycomb/tree/main/postman-collection">collection</a> in the repo.<br>Or you can just run the script that calls all of the endpoints:</p><pre>./run_http_requests.sh</pre><h3>Create an account on Honeycomb.io</h3><p>Any observability solution could be used with the code in the repo, and the only changes needed to export the traces to a service different than Honeycomb.io will be in the env variables.</p><p>Honeycomb.io, as of today, offers a very generous <a href="https://www.honeycomb.io/pricing">free plan</a> with no need to add a credit card to create the account.<br>All is needed is an email from where you will get the activation link.</p><p>Sign up <a href="https://ui.honeycomb.io/signup">here</a>, confirm the email and create a new team (any team name is fine and all the steps for the creation of the team are presented automatically after signing up).</p><p>The home page will contain instructions on how to do the instrumentation and the API key that will be needed in the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*icJaJV3_iYDm-xw5Vl1MVQ.png" /></figure><p>You can retrieve the API key even later by going into Account (at the bottom on the left hand side) and Settings.</p><p>The Home page will start showing the traces after instrumenting and running the code in the repo.</p><h3>Instrumentation</h3><h4>Preparation</h4><p>Get the code in the repo <a href="https://github.com/emanuelef/go-fiber-honeycomb/tree/main">emanuelef/go-fiber-honeycomb</a>, either clone it or get the zip.</p><p>Alternatively you can use the GitHub Codespaces (I tried that for this example and it works perfectly).</p><p>Run set_token script and paste the API Key from the Honeycomb website.</p><pre>./set_token.sh<br>Honeycomb API Key: *********</pre><p>This will just create the .env files needed for every app (they contain the API keys so better not put them in the repos), should that not work just copy the .env.example files in each folder (you might just want to start with the one in the root folder, /secondary and/grpc-server are needed later) to .env and replace your_key_here with the API key.</p><p>At this point we can check if everything is set up correctly and the traces are exported.<br>In a terminal from the root folder run :</p><pre>go run main.go<br><br> ┌───────────────────────────────────────────────────┐ <br> │                   Fiber v2.48.0                   │ <br> │               http://127.0.0.1:8080               │ <br> │                                                   │ <br> │ Handlers ............ 16  Processes ........... 1 │ <br> │ Prefork ....... Disabled  PID ............. 84354 │ <br> └───────────────────────────────────────────────────┘ </pre><p>From another terminal:</p><pre>curl http://127.0.0.1:8080/hello</pre><p>Go to the Honeycomb.io website and after a short time you should see the exported trace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OpvJpd-aAU8nRtP3XhQXaA.png" /></figure><p>Click on the button on the left and you will see the trace with the single span.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Umi7yAsEViB6TizXpsnGdA.png" /><figcaption>Span created for a request to /hello</figcaption></figure><h4>Set up trace provider and exporter</h4><p>First step is to create a trace provider with an exporter.</p><p>In this article and in the repo the traces are exported directly to the public endpoint provided by Honeycomb and specified in the OTEL_EXPORTER_OTLP_ENDPOINT (<a href="https://api.honeycomb.io/">https://api.honeycomb.io:443</a>) env variable. <br>Using a <a href="https://opentelemetry.io/docs/collector/">Collector</a> is probably more likely to happen in production.</p><p>In <a href="https://github.com/emanuelef/go-fiber-honeycomb/blob/main/otel_instrumentation/opentelemetry_setup.go">opentelemetry_setup.go</a> the function InitializeGlobalTracerProvider creates a Trace Provider and an Exporter. The Go OpenTelemetry SDK will use the env variables defined in the .env file, the only ones needed for this examples are:</p><pre>OTEL_SERVICE_NAME=GoFiberExample<br>OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io:443<br>OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=your_key_here</pre><p>During the creation of the Trace Provider is possible to add custom <a href="https://opentelemetry.io/docs/instrumentation/go/resources/">resources</a> that will be added to each span as attributes, resources can be added with the OTEL_RESOURCE_ATTRIBUTES<em> </em>env variable as well.</p><p>Check the <a href="https://opentelemetry.io/docs/specs/otel/trace/semantic_conventions/">Semantic Conventions</a> to see if the kind of resources you want to add are already defined.</p><p>This is where is possible to add the resources in code:</p><pre>resource, rErr := resource.Merge(<br> resource.Default(),<br> resource.NewWithAttributes(<br>  semconv.SchemaURL,<br>  attribute.String(&quot;environment&quot;, &quot;test&quot;),<br>  attribute.String(&quot;stack&quot;, &quot;001&quot;),<br> ),<br>)</pre><h4>Instrument Fiber webserver with Otelfiber</h4><p><a href="https://gofiber.io/">Fiber</a> can be instrumented with <a href="https://github.com/gofiber/contrib/tree/main/otelfiber">Otelfiber</a> middleware.</p><pre>app := fiber.New()<br><br>app.Use(otelfiber.Middleware())<br><br>app.Get(&quot;/hello&quot;, func(c *fiber.Ctx) error {<br> return c.Send(nil)<br>})</pre><p>By adding the otelfiber Middleware a span is created automatically for every HTTP request received.</p><p>Should you want for some reason to exclude some endpoints from generating spans you can use the otelfiber.WithNext option, if the passed function returns true the otelfiber Middleware will skip the OTeL instrumentation.</p><pre> app.Use(otelfiber.Middleware(otelfiber.WithNext(func(c *fiber.Ctx) bool {<br>  return c.Path() == &quot;/health&quot;<br> })))</pre><p>As showed in the Preparation a call to GET /hello will create the span.</p><h4>Add child span</h4><p>Quite likely the work done by the HTTP request will involve additional spans.<br>In order to create a child span, or a new root span, you need to have a tracer, there will probably be one for the entire app or one for every package.</p><pre>var tracer trace.Tracer<br><br>func init() {<br> // this seems to work even if the init happens before setting up the trace provider<br> tracer = otel.Tracer(&quot;github.com/emanuelef/go-fiber-honeycomb&quot;)<br>}</pre><p>Then in the code inside a fiber handler</p><pre>  ctx, childSpan := tracer.Start(c.UserContext(), &quot;custom-child-span&quot;)<br>  time.Sleep(10 * time.Millisecond) // simulate some work<br>  childSpan.End()</pre><p>c.UserContext() retrieves the context from Fiber Context with the opentelemetry infos (trace id, span id).</p><p>GET /hello-child will generate the following spans:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SOqB6JgvSHXJApr69dRK1w.png" /></figure><p>If the ctx passed to the tracer.Start contains a span then the newly-created span will be a child of that span, otherwise it will be a root span.<br>tracer.Start also accepts SpanStartOption so it is possible to add attributes or the span kind if needed.</p><h4>Create a new span</h4><p>If a new root span is needed (not child of any other) the process is similar to create a child span but passing a context that doesn’t contain a span already.</p><pre>ctx, span := tracer.Start(context.Background(), &quot;timed-operation&quot;)<br>// Do something<br>span.End()</pre><p>In the repo main app code there’s a time tick that creates a new root span every minute.</p><h4>Instrument outgoing HTTP requests</h4><p>I’ll present three ways to add the spans for each outgoing HTTP request (and also propagate the trace to the web server called).<br>In the end they all build the client with the standard http.Client.</p><p><strong>Otelhttp package</strong></p><p>The simplest way is to use the package otelhttp defined in go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp, this wraps the http requests adding a span around.</p><p>The methods currently implemented there are:</p><ul><li>Get</li><li>Head</li><li>Post</li><li>PostForm</li></ul><p>Inside a Fiber handler the context to pass is c.UserContext(), otherwise the HTTP request span will be a root one.</p><pre>externalURL := &quot;https://pokeapi.co/api/v2/pokemon/ditto&quot;<br>resp, err := otelhttp.Get(c.UserContext(), externalURL)<br>_, _ = io.ReadAll(resp.Body) // This is needed to close the span</pre><p>A little gotcha here, the io.ReadAll is actually required to close the span, even if the response is not needed.</p><p>Otelhttp will also take care of the Transport injecting the span context into the outbound request headers.</p><p>In the repo a call to GET <em>/hello-otelhttp</em> will create the following spans, this will require the secondary app to run and is used to demonstrate that the traceparent (a single HTTP header containing trace_id and span_id, among version and trace_flags) is carried in the HTTP header allowing the distributed tracing across multiple apps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tyERvEURXzj209hkd0sr3A.png" /></figure><p><strong>http.Client</strong></p><p>To have more control (and use methods not implemented in otelhttp yet, like PATCH) you can use the http.Client directly.</p><pre>externalURL := &quot;https://pokeapi.co/api/v2/pokemon/ditto&quot;<br>client := http.Client{<br> Transport: otelhttp.NewTransport(<br> http.DefaultTransport,<br>   ),<br> }<br><br>req, err := http.NewRequestWithContext(c.UserContext(), &quot;GET&quot;, externalURL, nil)<br>if err != nil {<br> return err<br>}<br><br>resp, _ := client.Do(req)<br>_, _ = io.ReadAll(resp.Body)</pre><p>Passing the context with an existing span to http.NewRequestWithContext<em> (</em>if inside a Fiber handler will be c.UserContext()) will make the outgoing HTTP call to be a child span of the incoming HTTP call served by Fiber.</p><p>By adding the otelhttp transport (using otelhttp.NewTransport) to the http.Client the outgoing request will have injected in the HTTP header (traceparent) all the information needed for enabling the distributed tracing.<br>Alternatively, the traceparent header can be injected manually in the http.Request with:</p><pre>otel.GetTextMapPropagator().Inject(c.UserContext(), propagation.HeaderCarrier(req.Header))</pre><p>To make it more interesting in the GET <em>/hello-http-client</em> implemented in the accompaning repo the Transport has an option to show the details of the HTTP request (htttp.getconn, htt.headers…)</p><pre>  client := http.Client{<br>   Transport: otelhttp.NewTransport(<br>    http.DefaultTransport,<br>    otelhttp.WithClientTrace(func(ctx context.Context) *httptrace.ClientTrace {<br>     return otelhttptrace.NewClientTrace(ctx)<br>    })),<br>  }</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m3mgY8Eg9-v-SnSoJNBJUQ.png" /></figure><p><strong>Resty</strong></p><p>As the last example on how to have outgoing HTTP calls intrumented we will use the <a href="https://github.com/go-resty/resty">Resty</a> popular library.<br>As of today Resty doesn’t support OTeL instrumentation directly but it is possible by passing the http.Client<em> </em>in the resty.NewWithClient</p><pre>externalURL := &quot;https://pokeapi.co/api/v2/pokemon/ditto&quot;<br><br>client := resty.NewWithClient(<br> &amp;http.Client{<br>  Transport: otelhttp.NewTransport(http.DefaultTransport,<br>   otelhttp.WithClientTrace(func(ctx context.Context) *httptrace.ClientTrace {<br>    return otelhttptrace.NewClientTrace(ctx)<br>   })),<br> },<br>)<br><br>restyReq := client.R()<br>restyReq.SetContext(c.UserContext())<br><br>resp, _ := restyReq.Get(externalURL)</pre><p>Make sure to set the context with the existing span (if you want to create a child span)</p><p>As for the previous example if you don’t want to set the Transport, or not having the opportunity in another library to pass an http.Client with the transport, you can inject the traceparent header directly with:</p><pre>otel.GetTextMapPropagator().Inject(c.UserContext(), propagation.HeaderCarrier(restyReq.Header))</pre><p>the GET <em>/hello-resty</em> in the repo will produce something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cOCfmMY6JyMRC4GiqWZfmg.png" /></figure><h4>Add span attributes</h4><p>To add additional attributes to an existing span:</p><pre>span := trace.SpanFromContext(c.UserContext())<br>span.SetAttributes(attribute.Bool(&quot;isTrue&quot;, true), attribute.String(&quot;test.text&quot;, &quot;Ciao&quot;))</pre><h4>Add span events</h4><p>Events can be added to a span to define something relevant happening in a point in time inside the span, they can have attributes as well.<br>Events can also be used as a form of log doing something like:</p><pre>// simple event<br>span.AddEvent(&quot;Done second fake long running task&quot;)<br><br>// event with attributes to have a log linked to the span<br>span.AddEvent(&quot;log&quot;, trace.WithAttributes(<br> attribute.String(&quot;log.severity&quot;, &quot;warning&quot;),<br> attribute.String(&quot;log.message&quot;, &quot;Example log&quot;),<br>))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*92upv852tSdPY0Ez2h3R9Q.png" /></figure><h4>Instrument gRPC calls</h4><p>In the <a href="https://github.com/emanuelef/go-fiber-honeycomb">go-fiber-honeycomb</a> there is also an example to see traces when using gRPC.</p><p>In order to instrument the gRPC calls.</p><p>On the client:</p><pre>conn, err := grpc.Dial(grpcTarget,<br> grpc.WithTransportCredentials(insecure.NewCredentials()),<br> grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()))<br><br>defer conn.Close()<br>cli := protos.NewGreeterClient(conn)<br>r, err := cli.SayHello(c.UserContext(), &amp;protos.HelloRequest{Greeting: &quot;ciao&quot;})</pre><p>On the server:</p><pre>grpcServer := grpc.NewServer(grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()))</pre><p>call GET <a href="http://localhost:8080/hello-grpc">http://localhost:8080/hello-grpc</a> to see it in action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*qXI4_oVkImpiOelz65gd2g.png" /></figure><h3>Conclusion</h3><p>I hope this article and the related <a href="https://github.com/emanuelef/go-fiber-honeycomb">repo</a> will help in easily starting to instrument your Go code with OpenTelemetry. <br>We have been using OTeL in production and having the distributed tracing (and all the other amazing features for analyzing all the OTeL signals) has helped understand our platform performance and reduce troubleshooting time.</p><p><em>Thank you for reading until the end. Please consider following the writer and this publication. Visit </em><a href="https://stackademic.com/"><em>Stackademic</em></a><em> to find out more about how we are democratizing free programming education around the world.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1005697d841f" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/start-using-opentelemetry-with-go-fiber-1005697d841f">Start using OpenTelemetry with Go Fiber</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>