<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Glen Keane on Medium]]></title>
        <description><![CDATA[Stories by Glen Keane on Medium]]></description>
        <link>https://medium.com/@glentiki?source=rss-1732a77f9888------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*LrJyEsBnqeU-QxgL.jpg</url>
            <title>Stories by Glen Keane on Medium</title>
            <link>https://medium.com/@glentiki?source=rss-1732a77f9888------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 09:20:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@glentiki/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Writing an Impressively Fast HTTP Load Testing Tool in Golang for fun and learning!]]></title>
            <link>https://medium.com/@glentiki/writing-an-impressively-fast-http-load-testing-tool-in-golang-for-fun-and-learning-9043b164bd99?source=rss-1732a77f9888------2</link>
            <guid isPermaLink="false">https://medium.com/p/9043b164bd99</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Glen Keane]]></dc:creator>
            <pubDate>Mon, 10 Feb 2020 14:35:14 GMT</pubDate>
            <atom:updated>2020-02-10T14:39:07.312Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GHFfExpkE85zYe37" /><figcaption>Photo by <a href="https://unsplash.com/@crisovalle?utm_source=medium&amp;utm_medium=referral">Cris Ovalle</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Recently I took some time to reinvent the wheel to understand Golang on a deeper level. To do this, I rewrote software I previously worked on and help to maintain: <a href="https://github.com/mcollina/autocannon">Autocannon</a>. The repo for this experiment is available <a href="https://github.com/GlenTiki/autocannon-go">here</a>.</p><p>Autocannon is a <em>fast</em> HTTP/S 1.1 benchmarking tool written with Node.js, intending to be familiar for those who have already used wrk or wrk2. It is a cross-platform tool that can be run via the command line or using a programmable interface. It is simple to use, understand and extend.</p><p>Hitting all of these features for a personal investigation in my spare time would be impossible, so I focused on the following key goals:</p><ul><li>Speed; I primarily wanted to compare speed between Node.js and Golang.</li><li>Simple to Use (Command Line); I have never really needed the programmable interface of Autocannon, so to keep it simple, I decided to build this for command line usage only.</li><li>Reduced Feature Set: There is no need to enable HTTPS support out of the box for a simple thought experiment, so that entire functionality can be scrapped, and many of the features around controlling the flow of requests in Autocannon aren’t needed — we just want to run as fast as possible.</li><li>Cross-Platform; I wanted to investigate Golang’s ability to deliver cross-platform tools.</li></ul><h3>How Autocannon became fast.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*XRejwqWhGasK-726L5ohzQ.gif" /><figcaption><a href="https://media3.giphy.com/media/ULhy3hP2EqRDa/giphy.gif">https://media3.giphy.com/media/ULhy3hP2EqRDa/giphy.gif</a></figcaption></figure><p><em>If you’re interested in Node.js, you may want to read the following paragraphs to understand how this speed is achieved, otherwise, feel free to skip to the next section.</em></p><p>As previously mentioned, Autocannon is <em>fast. </em>50K+ Requests per second <em>fast</em>. While it is built on Node.js, It has been observed to be faster than <a href="https://github.com/wg/wrk">wrk</a>, a benchmarking tool built with C. The reason for this is many choice decisions in the implementation of Autocannon for optimisation. Autocannon itself is built on a custom HTTP/S client, which is primarily where these optimisations exist. It is believed that the only route to optimise Autocannon further is working within the underlying networking stack, within the OS.</p><p>One of the key optimisations for Autocannon is that everything is built within the JavaScript domain, meaning there are no native dependencies. While native dependencies can be very powerful for heavy compute tasks, they come with a non-negligible overhead when the runtime must deviate from JavaScript land into native dependency land. This overhead can then stack up significantly in an application where there are many calls between JavaScript to native code. Additionally, by working only within the JavaScript domain, it enabled the code paths to be optimised by the JS engine optimising compiler. <a href="https://stackoverflow.com/a/42874867/2178267">This StackOverflow</a> answer by a V8 developer covers this quite well. When Autocannon was created, there were some primary places where native dependencies could be a slowdown due to the number of calls between the JavaScript to the native domain. These were identified to be in the HTTP client/parser and within the library used for tracking the histogram of benchmarked values.</p><p>The HTTP parser was an issue because for every request made, a response needed to be transferred to the native dependency, parsed, and then the parsed values must be returned/exposed to the JavaScript land. The <a href="https://github.com/nodejs/http-parser">HTTP parser</a> that Node.js uses in the native ‘http’ library is a dependency created specifically for that use case. While this is pretty powerful, it was decided to avoid using this for the reason mentioned, which meant writing a custom HTTP client with another JS-based HTTP Parser. The HTTP parser in use is <a href="https://github.com/creationix/http-parser-js">http-parser-js</a>, which is a JS library specifically created for the use-case as laid out above.</p><p>The histogram tracking library was not a major issue, but it was still a consideration due to the number of calls being made for tracking benchmarked data within it. Initially, Autocannon was built on top of <a href="https://github.com/mcollina/native-hdr-histogram">native-hdr-histogram</a> which exposed the C library bindings to Node.js. After some time of using this, the HDR Histogram community built and released a TypeScript based version which would expose only JavaScript code, so Autocannon was quickly migrated to <a href="https://github.com/HdrHistogram/HdrHistogramJS">HdrHistogramJS</a>.</p><p>Another key optimisation of Autocannon is that it made use of <a href="https://en.wikipedia.org/wiki/HTTP_pipelining">HTTP Pipelining</a>. This enabled the reuse of connections for multiple concurrent requests, so connections being dropped and reopened is not an issue like it could be otherwise. Think HTTP <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Keep-Alive">keep-alive</a>, but for multiple requests and responses in parallel.</p><h3>Building this with Golang.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*A-HK8lhPFu-_C6vsFy2NPg.png" /><figcaption>Taken from <a href="https://ednsquare.com/story/marshal-and-unmarshal-of-struct-to-json-in-golang------CG5PTJ">https://ednsquare.com/story/marshal-and-unmarshal-of-struct-to-json-in-golang------CG5PTJ</a></figcaption></figure><p>Because of the understanding I had of the optimisations of Autocannon, I knew what would be needed to get this Golang Version up to a comparable speed.</p><p>The primary way this speed could be achieved is by making use of an HTTP Client that enabled pipelining. In Autocannon, we needed to build this Client ourselves because nothing out of the box enabled it the way we wanted. After some googling for a Golang library for this purpose, I quickly discovered <a href="http://github.com/valyala/fasthttp">github.com/valyala/fasthttp</a>, which had <a href="https://pkg.go.dev/github.com/valyala/fasthttp?tab=doc#PipelineClient">HTTP Pipelining</a> support out of the box. This simplified things for my implementation significantly, as this Client did most of the heavy lifting. However, as a note, this library is only able to give a rough approximation of the throughput vs Autocannon’s true representation. This is a limitation in how response sizes are calculated. See <a href="https://github.com/valyala/fasthttp/issues/734">this issue</a> for details.</p><p>The other key component in this tool is the actual tracking of the benchmarked data. In Autocannon, we used HDR Histogram for this and with some quick searching, I discovered a native version for Golang: <a href="http://github.com/codahale/hdrhistogram">github.com/codahale/hdrhistogram</a>. However, while trying to use this I discovered a minor issue in comparison to the native or JS versions, and due to it being archived on Github I needed to fork it for a tiny update here: <a href="http://github.com/glentiki/hdrhistogram">github.com/glentiki/hdrhistogram</a>.</p><p><em>Sidenote: As a responsible OpenSource contributor I’ve reached out to the repo owner on Twitter to ask for it be un-archived and I’ll happily maintain it, if possible :)</em></p><p>Once I had these bits figured out, glueing these together and making it user-friendly in Golang was pretty easy. However, I had some struggles finding useful Command-Line formatting tools, such as an ASCII table, etc. Eventually, after trying a few I discovered <a href="http://github.com/olekukonko/tablewriter">github.com/olekukonko/tablewriter</a> works best to meet the autocannon experience.</p><p>Building for multiple platforms and doing releases that could be comparable to Node.js was a slightly more intense endeavour and required some investigation, which I’ve written about <a href="https://medium.com/threefold-digital/creating-automated-multi-platform-golang-executable-binary-releases-with-xgo-and-github-actions-8877f7b72740">here</a>. <em>TL;DR</em><strong>: </strong>To build a simple release pipeline, I made use of Github Actions, publishing the code I had pushed under git tag. This code was built using xgo for multiple platforms and published using a custom action I created for the purpose: <a href="https://github.com/GlenTiki/xbin-release-action">glentiki/xbin-release-action</a>.</p><blockquote>Overall, re-implementing the key functionality from Autocannon took me 2–3 evenings, with a lot of that time just spent googling to find the right tools for the job. <strong>The code was the easy part</strong>.</blockquote><p>Overall, I got the key functionality implemented with 250 Lines of Code (LoC), where pretty much 150 LoC is purely for command line input/output formatting. While I can’t stress enough that the following is an unfair comparison and I understand LoC is a troublesome metric, I believe 100 LoC for a Golang tool to minimally replicate what was done with many hundreds more LoC in Autocannon is impressive in many ways. It shows that the utility of the language, the ecosystem of tooling available and the overall experience is very positive and noteworthy.</p><h3>Conclusion and comparing output and benchmark speeds.</h3><p>The following screenshot shows the output from running Autocannon-go, and then Autocannon against a simple “Hello World” Fastify server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*azf_mAvSfGtK-X_rlyFZFg.png" /></figure><p>In the above image, you can see that the Autocannon-go version is making roughly 59K requests per second where Autocannon is making roughly 56K requests per second.</p><p>Both of these are impressive numbers, but again, it strikes me as noteworthy that I could achieve all of this with the goals that I set out with only 250 LoC and a few dependencies.</p><p>I would also like to note that part of this personal work was done as validation to see if it could be fair to compare a Node.js program to a compiled (Golang) one and I believe that, yes, this comparison is fair and valid and Node.js performs quite well within the comparison, too. I also want to give a quick shoutout to <a href="https://medium.com/u/bc6e8b811bba">Matteo Collina</a> for building a strong base and mentoring me in building out the awesome Node.js tool that is Autocannon.</p><p>To finish I will say that I have found my recent endeavours with Golang quite positive and something I plan on pursuing it further. There is also some great tooling available out of the box and from the module EcoSystem, and as someone who appreciates all of this in Node.js, I can appreciate having this available to me in Golang.</p><blockquote>If you found this interesting I normally keep my experiments documented on <a href="https://twitter.com/GlenTiki">my Twitter</a>, so give me a follow if you’re into this sort of thing. The repo for this experiment is also available <a href="https://github.com/GlenTiki/autocannon-go">here</a> if you want to check out the code, and if you think it’s worth it, a star on GitHub would be appreciated 🚀✨</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9043b164bd99" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating automated multi-platform Golang executable binary releases with xgo and Github actions.]]></title>
            <link>https://medium.com/@glentiki/creating-automated-multi-platform-golang-executable-binary-releases-with-xgo-and-github-actions-8877f7b72740?source=rss-1732a77f9888------2</link>
            <guid isPermaLink="false">https://medium.com/p/8877f7b72740</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Glen Keane]]></dc:creator>
            <pubDate>Fri, 24 Jan 2020 12:56:11 GMT</pubDate>
            <atom:updated>2020-02-07T16:39:22.091Z</atom:updated>
            <content:encoded><![CDATA[<h3>Creating automated multi-platform Golang executable binary releases with xgo and Github Actions.</h3><figure><img alt="A hipster gopher, wearing a github shirt and holding a wrench" src="https://cdn-images-1.medium.com/max/1024/1*qNhyeNiJZNusTbgyqHkjEA.png" /></figure><p><strong>TL/DR</strong>: If you’re not interested in the journey it took for me to get here, and details on the implementation, jump to the end of this post to see what the actions pipeline looks like. If you want to use it, you will need to take the configuration and adapt it to your use-case.</p><h3>The Backstory</h3><p>I’m pretty much a beginner with the golang language and to help learn it I’ve been working on a proof of concept of a http1.1 benchmarking tool. I have previously worked on and help to maintain another tool for the same purpose: <a href="https://github.com/mcollina/autocannon">autocannon</a>. I used this as a fun exercise to learn something with a complex task that I already understand deeply.</p><p>The power of autocannon is that it is a cross-platform compatible tool out of the box because it is a node.js based tool. Once you had node.js on your system all you ever had to do to use autocannon was run `<strong>npm i -g autocannon</strong>` and you could use it without needing any native dependencies. <em>It</em> <em>just worked</em>.</p><h4>Shipping something for end-users to easily get started</h4><figure><img alt="Code, build and deploy pipeline image" src="https://cdn-images-1.medium.com/max/1024/1*0GPE7IlJgSrU9oudZWykgA.jpeg" /></figure><p>The primary goal of any software developer should be to make life better for their users. That means their software should be as easy to use as possible. The installation step is the first interaction anyone has with any software. This step <strong>needs</strong> to be a familiar and straightforward process for users to understand and trust your software.</p><p>I did know there were patterns already set such as how compiled CLI tools, such as kubectl, get shipped and installed so I aimed to imitate that. Being someone who likes shipping software with the free tools easily available to me, my goal was to replicate this known pattern by making use of built-in Git and Github features, using tagging, Github releases with assets and then to tie it all together with <a href="https://github.com/features/actions">Github Actions</a>.</p><p>Using these tools, I wanted to enable the ability to develop the code/test/verify locally, commit and push as needed, and tag for a release. On a tag push, I would like there to be a new release created with all the binaries attached, after all of the tests passed successfully.</p><h4>Lessons Learned Using Github Actions</h4><figure><img alt="Github actions mockup" src="https://cdn-images-1.medium.com/max/1024/1*zyTQKIbCfws75mpA9mpZGA.jpeg" /></figure><p>I had very briefly looked at Github Actions and knew at a high level how they worked. I knew that only running on a tag and then creating a tag release action would be straightforward, but I didn’t know the exact tooling available to me.</p><p>As I investigated I quickly discovered <a href="https://github.com/actions/upload-release-asset">actions/upload-release-asset</a>, and the simple pipeline it provides allows developers to upload a <strong>single</strong> asset to a release. I’m sure using the matrix feature of Github actions you could potentially upload more, but that turned out to be a rabbit hole I very quickly got frustrated with.</p><p>I then discovered <a href="https://github.com/crazy-max/ghaction-xgo">crazy-max/ghaction-xgo</a>, which is an awesome tool that has been written to use <a href="https://github.com/karalabe/xgo">xgo</a> as an action. This would allow you to compile a go executable binary in your action pipeline for multiple systems. These compiled executable binaries are then stored in a configured sub-directory in the repo. This super useful tool is what I needed to get multiple platform binaries <em>built and ready</em> for uploading for a tagged release, but this didn’t solve the problem of actually <em>uploading</em> them.</p><p>I then tested wiring the xgo action up to the upload-release-asset action, but it fell over very quickly. I couldn’t use that release asset action multiple times in an action pipeline for the several binaries created, so I needed to do something different. At this point, I started searching for something to allow me to do this, but I turned up nothing. I knew what I needed to do was somehow take the directory of built binaries and upload them to a release.</p><p>While upload-release-asset only worked on a single file, I wanted something that worked on a single directory.</p><h4>Building My Tools</h4><p>To achieve my goal, I started to look into writing my own Github Action and I quickly found that the getting started guide uses simple node.js code with some simple YAML configuration. With the knowledge that it is very easy to just write with node.js code, I took a peek at the existing upload-release-asset action. I found it easy to understand with only~40 lines in the implementation that is very straightforward.</p><p>Using this existing code as the base I was able to very quickly build, push and debug my action. This action now lives at <a href="https://github.com/GlenTiki/xbin-release-action">glentiki/xbin-release-action</a>. This action allows me to just point the action at a directory and the files within it are uploaded to the release. They are auto-configured to be downloaded as binary files for the users.</p><p>Some limitations exist but they still allow the delivery of the binaries in a pattern similar to kubectl. For example, you may have to chmod the binary on macOS before using it to make it executable, etc.</p><h3>The Action Configuration</h3><p>So for the important bit. I had the details figured out and decided to put it all together in an easy to use/re-use action pipeline. The below outlines the configuration needed, and the documentation I provided for users to install and use the binaries. 🚀</p><p>The below action runs on a tag push and tests the code and then creates a Github Release. It cross-compiles the go binary with xgo and uploads the generated directory of binaries to the Release as assets. These can be downloaded and used by users, much like kubectl can be downloaded and used.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3d72dad2b55eb77eb88acdf2a6d28f6/href">https://medium.com/media/a3d72dad2b55eb77eb88acdf2a6d28f6/href</a></iframe><p>The above configuration provides the following pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jd0AAlpa_wTH3QohpzTWfw.png" /></figure><p>Which then provides the following in your repository releases:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VhjnemahWZCQhzLjtYSKLg.png" /></figure><h4>Installing published binaries</h4><p>At this stage, I had binaries that existed somewhere, but I needed to supply the details for users to install them. The following paragraphs are an example of information I could put within my readme for users.</p><p>Binaries are published in the format: <a href="https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-$SYSTEM-$PLAFORM"><em>https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-$SYSTEM-$PLAFORM</em></a></p><p>To install on macOS, you would need to run the following and then add to your path:</p><pre>curl -LO &quot;https://github.com/GlenTiki/autocannon-go/releases/download/v0.0.1/autocannon-go-darwin-10.6-amd64&quot; &amp;&amp; chmod +x autocannon-go-darwin-10.6-amd64</pre><p>To install on Windows you could install from <a href="https://github.com/GlenTiki/autocannon-go/releases/download/v0.0.1/autocannon-go-windows-4.0-amd64.exe">this link</a> or if you have curl installed then run the following and then add to your path:</p><pre>curl -LO &quot;https://github.com/GlenTiki/autocannon-go/releases/download/v0.0.1/autocannon-go-windows-4.0-amd64.exe&quot;</pre><p>To install on Linux, you could curl the following and then add to your path:</p><pre>curl -LO &quot;https://github.com/GlenTiki/autocannon-go/releases/download/v0.0.1/autocannon-go-linux-amd64&quot; &amp;&amp; chmod +x autocannon-go-linux-amd64</pre><p>You can use the following URLs to retrieve autocannon-go as needed, substituting the $GIT_TAG for the released version tag. E.g. v0.0.1.</p><pre><a href="https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-darwin-10.6-amd64"><em>https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-darwin-10.6-amd64</em></a></pre><pre><em>https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-darwin-10.6–386</em></pre><pre><a href="https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-linux-amd64"><em>https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-linux-amd64</em></a></pre><pre><a href="https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-linux-386"><em>https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-linux-386</em></a></pre><pre><a href="https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-windows-4.0-amd64.exe"><em>https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-windows-4.0-amd64.exe</em></a></pre><pre><a href="https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-windows-4.0-386.exe"><em>https://github.com/glentiki/autocannon-go/releases/download/$GIT_TAG/autocannon-go-windows-4.0-386.exe</em></a></pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8877f7b72740" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>