<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gaius on Medium]]></title>
        <description><![CDATA[Stories by Gaius on Medium]]></description>
        <link>https://medium.com/@gaius4888?source=rss-ef3d443a0357------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*771rEaazIJzP4gW60BIXBw.jpeg</url>
            <title>Stories by Gaius on Medium</title>
            <link>https://medium.com/@gaius4888?source=rss-ef3d443a0357------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:20:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gaius4888/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Analyzing CI performance with CIAnalyzer, Spark, Athena and Superset]]></title>
            <link>https://medium.com/@gaius4888/analyzing-ci-performance-with-cianalyzer-spark-athena-and-superset-7a5bc04a6639?source=rss-ef3d443a0357------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a5bc04a6639</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[apache-superset]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Gaius]]></dc:creator>
            <pubDate>Fri, 06 May 2022 04:08:00 GMT</pubDate>
            <atom:updated>2022-05-06T04:08:00.023Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Leveraging data engineering tools to solve DevOps problems.</blockquote><p>Since one of the goals of DevOps practices is to save “engineering time”, it is crucial that our CI pipelines aren’t running for 30 minutes just to show a simple error. If a step in a pipeline has a very low success rate, chances are it should be reviewed.</p><p>Last year I came across an interesting OSS called CIAnalyzer which consolidates CI data from different platforms (e.g. GitHub Actions, Jenkins, Gitlab Runners, etc). This is crucial for analyzing CI performance and trends for DevOps teams. The creator even wrote a detailed blog post explaining the motivation and internals of creating CIAnalyzer.</p><ul><li><a href="https://github.com/Kesin11/CIAnalyzer">GitHub - Kesin11/CIAnalyzer: A tool collecting multi CI services build data and export it for creating self-hosting build dashboard.</a></li><li><a href="https://zenn.dev/kesin11/articles/cf08579949b8b0">CI/CDのデータを収集するCIAnalyzerの紹介</a></li></ul><p>The creator used BigQuery for analyzing the data, but I wanted to do the same on AWS with S3, Athena and use Apache Superset, and it turns out to be a bit of work.</p><p>The source code is <a href="https://github.com/gai6948/ci-analyzer-project">here</a></p><h3>Setting up the AWS environment</h3><p>We will need some AWS resources for the analysis workflow, to be specific:</p><ul><li>An S3 bucket which stores the raw data acquired from CIAnalyzer, ETL scripts, and processed data</li><li>Glue database, tables, crawlers and job that runs Apache Spark jobs (more details on this later)</li><li>Athena table and workgroup for our BI tools</li><li>Supporting IAM roles and policies</li><li>Glue trigger that runs crawler once the Spark job finishes (without it we may miss new partitions)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*cq-JvYTYycgkofuhpvnauw.png" /><figcaption>Overview of the architecture</figcaption></figure><p>The infrastructure part is not interesting, but you can find the Terraform code in the terraform folder</p><h3>Running CIAnalyzer</h3><p>You supply a ci_analyzer.yaml telling how CIAnalyzer where to get CI data, in the case of GitHub:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e60f843250812ce945ce219a6da5896/href">https://medium.com/media/1e60f843250812ce945ce219a6da5896/href</a></iframe><p>You may have noticed there is a tests section looking for JUnit XML files. CIAnalyzer can generate structured data from JUnit formatted files. For Python and Go there are tools that integrate with the languages’ unit test framework.</p><ul><li><a href="https://github.com/xmlrunner/unittest-xml-reporting">GitHub - xmlrunner/unittest-xml-reporting: unittest-based test runner with Ant/JUnit like XML reporting.</a></li><li><a href="https://github.com/jstemmer/go-junit-report">GitHub - jstemmer/go-junit-report: Convert Go test output to JUnit XML</a></li></ul><p>CIAnalyzer can be run as a cronjob using any workflow orchestrator (e.g. Argo Workflows, GitHub Actions, etc). Here is a simple example of running CIAnalyzer as a daily job on GitHub Actions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad95c68cd5e5ee7f95d2baa9bea5291f/href">https://medium.com/media/ad95c68cd5e5ee7f95d2baa9bea5291f/href</a></iframe><p>The job above also copies the CI data to S3 bucket, partitioned by year, month and day.</p><h3>Transforming the raw data with Spark</h3><p>Querying raw data is <strong>inefficient, </strong>and<strong> </strong>Athena charges by how much data scanned. So there are some transformations I want to do:</p><ul><li>Cast some date fields into timestamp type (to enable time-series charting)</li><li>Break nested JSON into flat structure</li><li>Convert JSON into Parquet format for smaller storage space and cost</li><li>Re-partition the data based on repository (if the data grows we may also need to further partition by year/month)</li></ul><p>In this example I used AWS Glue as a managed Apache Spark runtime. There is some Glue-specific code added to the ETL script in order to leverage Glue’s feature, for example job arguments.</p><p>Here is the complete script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94ea710e543de2f816b6c915c0415d73/href">https://medium.com/media/94ea710e543de2f816b6c915c0415d73/href</a></iframe><h3>Analyzing the CI data with Athena/Superset</h3><p>The data is now ready to be consumed using SQL or BI tools. You can use Tableau, Superset or anything. Here I use Superset with the free tier from <a href="https://preset.io/">preset.io</a>, but you can use self-hosted Superset as well. You will need to set up a database connection to Athena in Superset, with an AWS user that has IAM access to <strong>both Athena/Glue catalog and the underlying S3 buckets (source data and Athena query result).</strong></p><p>Here are some example dashboards I used:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vljcJ-dd-Rl57I7nhmaJfg.png" /><figcaption>Overview of workflows</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XkXwwmnA-uuIMVNrWChjlw.png" /><figcaption>Per workflow trends</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yDyNKbnYWUMCj22yftzXMQ.png" /><figcaption>Test case details</figcaption></figure><p>The code for preset dashboards are <a href="https://github.com/gai6948/ci-analyzer-project/tree/main/dashboards/dashboard_export_20220503T142932">here</a></p><p>Besides visualization, Superset also let you run SQL directly, but the dialect is SQLAlchemy’s instead of Athena’s Presto SQL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-rq1dONutOjoccJ_2r0yxQ.png" /><figcaption>An ad-hoc SQL query to calculate workflow success rate</figcaption></figure><h3>Wrap-up</h3><p>It was an interesting learning journey playing with Data Engineering tools to solve DevOps problems. However this is only the beginning, the data we have now only covers the <strong>CI</strong> part of a pipeline, so in the future I would want to investigate on:</p><ul><li>Enriching CI data with GitHub/GitLab commits / PR information</li><li>Extends CI data with CD services status (e.g. Spinnaker, ArgoCD, etc)</li><li>Correlates production issues with CI data (this is a bit challenging so I am still pathfinding)</li></ul><p>I think every engineering team should start looking at CI data and analyze the performance and efficiency of their CI pipelines, so you should check out <a href="https://github.com/Kesin11/CIAnalyzer">CIAnalyzer</a> today!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a5bc04a6639" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What I learnt from building a homelab]]></title>
            <link>https://medium.com/@gaius4888/what-i-learnt-from-building-a-homelab-21b20b65c3aa?source=rss-ef3d443a0357------2</link>
            <guid isPermaLink="false">https://medium.com/p/21b20b65c3aa</guid>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[homelab]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[Gaius]]></dc:creator>
            <pubDate>Sun, 13 Mar 2022 09:46:21 GMT</pubDate>
            <atom:updated>2022-03-13T09:46:21.938Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>It was in the middle of last year when I decided to build a “homelab”, or a fancy term for “bunch of cheap computers”. It’s been quite a while so I got enough pain to lament about.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ewIuaZ7Z5ZgONSleYQQ_6Q.jpeg" /><figcaption>Current setup (forgive me for not being a “cable engineer” lol)</figcaption></figure><h3>But why you build a homelab at the first place?</h3><p>There are many arguments for and against building a homelab, a search on reddit and medium can give you a swarm of other people’s setup and experience.</p><ul><li><a href="https://medium.com/@prevole/home-lab-or-not-home-lab-6165a3eb0f73">Home Lab or Not Home Lab</a></li><li><a href="https://www.reddit.com/r/homelab/">r/homelab</a></li></ul><p>But in my case, I have 3 compelling reasons:</p><h4>1. Having a “real” K8s cluster to play with</h4><p>Yes, you can spin up a local cluster with <a href="https://minikube.sigs.k8s.io/docs/">minikube</a> or <a href="https://k3d.io/">k3d</a>, but I see them mainly for CI or local testing kinds of use cases. To me, a “cluster” is supposed to have <strong>multiple nodes</strong>, connected over local network, and any one of them can fail at any time. <a href="https://k8s.af/">If it is too simple, you don’t call it K8s, right?</a></p><h4>2. Not breaking the bank when it comes to computing power</h4><p><strong>Cloud is not cheap. </strong>Yes cloud vendors always tell you how building your own data center can be 100x more expensive than signing a multi-year VM deal. But for hobby use cases like homelab, the total cost of running a handful of $50 Raspberry Pi 24/7 is equivalent to running EC2s of similar spec for… 2 or 3 months? Of course, if we use cloud we can spin up and tear down clusters on-demand (<strong>Terraform FTW</strong>!), freeing everything up when we’re done playing. But <strong>waiting for 30min to spin up a K8s cluster</strong> when you need one is just damn annoying 😅.</p><h4>3. Playing around with latest technologies</h4><p><strong>Work is supposed to be boring. </strong>I mean the <a href="https://mcfunley.com/choose-boring-technology">tech stacks</a> not the actual work, of course :). When it comes to cluster networking, who would choose the shiniest <a href="https://cilium.io/">eBPF-based solution</a> when you can just pick the “battle-tested” <a href="https://istio.io/">Istio</a>, right? Troubleshooting and operating eBPF-based solutions are non-trivial at this moment, but I believe they will mature at some point in the future, and that’s one of the reason why you want a homelab to experiment with latest techs.</p><h3>Some key learnings running a homelab</h3><h4>Many subtle details bubbled up</h4><p>As an engineer, we are supposed to pay attention to details when building systems, I think no one will ever disagree. However, when we are at work, we tend to focus on the business problem rather than the inner details of technologies. When your company is new to Kafka and needs to deploy a Kafka cluster, will they give you a week to study how Kafka internals work? Very likely they don’t, because “running Kafka” is not really part of the business, so anything that “works” is fine, and you move on to the next task.</p><p>When you are packing so many applications in a handful of Raspberry Pi, many issues can appear, <a href="https://medium.com/omio-engineering/cpu-limits-and-aggressive-throttling-in-kubernetes-c5b20bd8a718">CPU throttling</a> is just one of them. If you encounter this at work, probably your first intuition is to “add more workers” and call it a day. If you aren’t running at Netflix or Google’s scale, then <em>throwing money to the problem makes perfect sense</em>. But you don’t have this option at home, you don’t buy 1 more Pi to solve every single problem, and instead work on <strong>improving resource efficiency, dive deeper into kernel internals, and understand systems better</strong>. This is not something we can always afford at work, but in the process of running a homelab I find myself <strong>more exposed to subtle details I have never encountered at work.</strong></p><h4>Build multi-architecture cluster</h4><p>ARM is really becoming the industry trend during past 2 years, from <a href="https://aws.amazon.com/ec2/graviton/">AWS’ Graviton</a> to <a href="https://www.apple.com/newsroom/2020/11/apple-unleashes-m1/">Apple’s M1</a>. Using ARM-based processor is not only because they are “cheaper”, but they are also sometimes “faster” because of their simpler architecture.</p><p>However, <strong>not every application is “ARM-ready”. </strong>Yes you can build ARM container images yourself or even re-compile applications to support ARM64, but some applications contain native C++ code that is going to require a rewrite or re-factor (e.g. <a href="https://github.com/falcosecurity/falco">falco</a>, and some eBPF-based application).</p><p>While ARM64 should be the default (<strong>Raspberry Pi FTW!!</strong>), it would make sense to add 1–2 x86 SBC to the cluster, to run those applications without ARM64 support (for now). For me I used <a href="https://www.seeedstudio.com/ODYSSEY-X86J4125800-p-4915.html">Seeed Studio’s Odyssey SBC</a>, but basically any one with an Intel-based and power-friendly CPU can do. Having a multi-arch cluster gives you more flexibility, when a workload is ARM-ready, just run them on the Pi! (<strong>Hint: use </strong><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector"><strong>K8s node-selector</strong></a><strong>!</strong>), when it’s not, you still have the option of running it on one of your x86 node!</p><h4>Don’t use SD card as storage</h4><p>Raspberry Pi by default <em>uses SD card as main storage</em>, but they are pretty darn slow when it comes to running K8s. If you are “serious”, you should consider <strong>booting from SSD</strong> instead (It’s not hard, I will make a detailed post later).</p><h4>You don’t have to run everything</h4><p>Yes we build a homelab cluster to “run stuffs”, but sometimes <strong>using a SaaS is much better than wasting our precious computing capacity</strong>. For example, if you need a distributed tracing solution, you can run <a href="https://www.jaegertracing.io/">Jaeger</a> on your own of course, but that would mean setting up <a href="https://www.opensearch.org/">ElasticSearch</a> as well, which is quite resource-intensive, and perhaps a better solution is to use a SaaS like <a href="https://www.honeycomb.io/">honeycomb.io</a> (their free tier is quite generous!). Answering the question of “whether to run something” should follow your role and interest, <em>“Is it something I need to know in detail for my career?”</em>, perhaps as an DevOps engineer I am less concerned about “how to run Prometheus”, as long as I get monitoring done.</p><h4>Homelab doesn’t mean no cloud</h4><p>You are still gonna need the cloud even if you have a homelab. At the very least maybe you’ll need <strong>S3 for backup</strong>, running ML jobs with <strong>GPU </strong>instances on the cloud or even <strong>running another cluster</strong> on the cloud (<a href="https://argoproj.github.io/cd/">ArgoCD</a> for multi-cluster!). All those Terraform stuffs are still relevant! Of course, when using the cloud, <strong>pay attention to data transfer cost</strong>, design your cluster dataflow so that you <em>don’t often send gigabytes of data from the cloud to your home.</em></p><h4>So is it a time+money sink?</h4><p>Honestly speaking, it costs quite some investment in money and time. It’s not just about the hardware costs (the servers + SSDs, bunch of network equipment, etc), but also time to troubleshoot issues (I still remember spending nights fixing broken ingress 😭). But after all I think running a homelab <strong>solidifies my system knowledge</strong>, at least in interview I can tell people “how I fit xxxx into a bunch of Raspberry Pis”.</p><p>Ciao and good luck with your homelab journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21b20b65c3aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>