<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by infraheads on Medium]]></title>
        <description><![CDATA[Stories by infraheads on Medium]]></description>
        <link>https://medium.com/@info_98249?source=rss-16994648a82b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*cdG18JnFb23tYTfUaMnqPQ.png</url>
            <title>Stories by infraheads on Medium</title>
            <link>https://medium.com/@info_98249?source=rss-16994648a82b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:30:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@info_98249/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Let it break]]></title>
            <link>https://medium.com/@info_98249/let-it-break-0be936fc0878?source=rss-16994648a82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/0be936fc0878</guid>
            <category><![CDATA[developer-experience]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[self-service]]></category>
            <dc:creator><![CDATA[infraheads]]></dc:creator>
            <pubDate>Thu, 16 Jan 2025 09:47:10 GMT</pubDate>
            <atom:updated>2025-01-16T09:47:10.522Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction</strong><br>Sometimes, our engineers ask if they should implement a solution that gives developers too much freedom. The answer is always the same. Let&#39;s compare the possible scenarios.</p><p><strong>So-called toxic environments<br></strong>First of all, a toxic environment is an environment filled with toxic substances. That is a biochemical term and nothing more. Now, back to software development. Every downtime costs at least money; sometimes, it ends the business. “Break it” on the walls of Facebook’s offices is not just a fun, leftie statement. <strong>Everyone</strong> should do everything possible to avoid unplanned downtimes; people should understand that. But they also need to be sure that because of downtime, no one will be punished or fired. They need to know that the management will deeply analyze the situation before taking action. If that is not the case, people will do less; they will cover each other failures, blame each other, and start competing within the same circle of responsibilities. In short, people will stop cooperating, and everyone will eventually lose.</p><p><strong>Non-toxic environments<br></strong>The <strong>highest</strong> responsibility for a smooth business operation lies on the shoulders of management; the higher the level, the bigger the responsibility and the stricter the manager&#39;s hard and soft skills requirements. In the best organizations, no one blames engineers for production failures. All failures are planned, regardless of the ambiguity level. Every engineer is hired carefully, and every engineer&#39;s opinion matters. No technical decisions are made on the management level. Blameless <a href="https://sre.google/sre-book/postmortem-culture/">postmortem</a> is a genius idea that belongs to brilliant managers.</p><p><strong>Why engineers are asking this?!<br></strong>Self-service for developers depends on many things. In many cases, that is a cost management question, and DevOps engineers should not be the decision-makers, but in many cases, they are. In some organizations, everyone silently agrees to delegate the decision-making right to the DevOps engineer as the higher management has no technical background to make a proper decision, and the developers complain of high dependence on DevOps engineers. We don’t like it when people depend on us to add a lambda function to their AWS account; we love learning new cool stuff or playing with a new thing in the lovely sandbox the company provides. But we also don’t enjoy being blamed when a developer misuses a tool we implemented and breaks something. An intelligent developer is a DevOps engineer’s best friend.</p><p><strong>Complete control is not an option<br></strong>In many cases, management is ready to give the engineers too much control, more than necessary, to ensure a smooth SDLC. That is a bad sign. That usually means that they need someone to blame if something goes wrong. DevOps engineers become too restrictive and tend to keep control over things at any operational cost. Developers start blaming them for that, and the political game starts. “Ah, you want them to be able to roll out a new k8s cluster; ok, create a ticket with that requirement, please; we want to have that on paper”. No one enjoys this situation. However, when control seems to be the only option, the DevOps engineers make a huge mistake; they believe that. If DevOps engineers refuse to take more control than they need, they force proper decision-making. It can’t be smooth, things will fail, DevOps engineers will be blamed, the higher management may start looking for replacement, which is not the works tace scenario, the worst case scenario is not firing that kind of clients. On the other hand, things may improve due to giving up unnecessary control. Management starts sorting things out. People started to grow, and the company became much more mature. Low performers leave, high flyers feel happy, and juniors grow.</p><p><strong>Conclusion<br></strong>Keeping a client at any cost is a bad idea, even if your company&#39;s survival depends on that client. DevOps engineers should always try to generate independence wityhing the QA engineers and the developers, no one should depend on DevOps engineer if the cost of not depending on them is a few hours of learning every month. If a DevOps engineer tries to create dependency with complexity, infraheads stop that and let them go, no matter what. And everything starts with the hiring decision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0be936fc0878" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Naming conventions]]></title>
            <link>https://medium.com/@info_98249/naming-conventions-55589f36ca3b?source=rss-16994648a82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/55589f36ca3b</guid>
            <category><![CDATA[abbreviations]]></category>
            <category><![CDATA[naming-conventions]]></category>
            <category><![CDATA[documentation]]></category>
            <category><![CDATA[code-quality]]></category>
            <category><![CDATA[best-practices]]></category>
            <dc:creator><![CDATA[infraheads]]></dc:creator>
            <pubDate>Mon, 19 Aug 2024 09:15:10 GMT</pubDate>
            <atom:updated>2024-08-19T09:17:02.504Z</atom:updated>
            <content:encoded><![CDATA[<p>Getting onboarded as a DevOps engineer is hard because you depend on many people from different departments.<br>However, a significant portion of that complexity is created by people. Sometimes things need to be complex; there are complex things in nature, and as we know from the zen of Python, “<em>complex is better than complicated</em>.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZBu9DMHxJsSIK9LeVIq4Fw.jpeg" /></figure><p>Naming conventions have no semantic information; those are not devices for describing something. Those are just conventions. Thus, anything works unless it doesn’t. Of course, many code-related naming conventions should be taken into account. Asking the team or polling seems a bad idea; a better idea or a coin flipping can be used instead, but we are not insisting. However, coding is relatively easy, and there are some opinions about different battle-proven conventions that can be borrowed. After all, engineers are better at communicating complex things together, but things get worse when it comes to non-technical people. Once, we worked for a company with a vast amount of documentation about everything in the world, and everything was abbreviated; it was so bad that they had another page on their abbreviations. We bet you saw something similar. We are lucky to have <a href="https://youtu.be/IGIrhBz6tYU?t=672">ThePrimeTime</a> expressing our pain.<br>Infraheads&#39; DevOps agency has a rule: We expand all abbreviations, no matter how often they occur in the text. Naming conventions are about respecting others.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55589f36ca3b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[turnk8s — turnkey k8s deployment solution]]></title>
            <link>https://medium.com/@info_98249/turnk8s-turnkey-k8s-deployment-solution-0ff2f42c195e?source=rss-16994648a82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ff2f42c195e</guid>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[helm-argocd]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <dc:creator><![CDATA[infraheads]]></dc:creator>
            <pubDate>Fri, 24 May 2024 11:26:06 GMT</pubDate>
            <atom:updated>2024-06-04T07:51:59.264Z</atom:updated>
            <content:encoded><![CDATA[<h3>turnk8s — turnkey k8s deployment solution</h3><p><a href="https://github.com/infraheads/turnk8s">turnk8s</a> is a turnkey Kubernetes deployment solution that adheres to <a href="https://www.talos.dev/v1.7/learn-more/philosophy/#:~:text=A%20design%20goal%20of%20Talos,There%20is%20only%20a%20cluster.">Talos Linux philosophy</a>. It inherits all the great features of Talos Linux and builds a complete end-to-end deployment solution for On-Prem environments.<br><a href="https://www.netris.io/">Netris</a> is used for network management. Read the repository readme or watch our YouTube video to understand its use better.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fh0lVOe087go%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dh0lVOe087go&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fh0lVOe087go%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/93b41e0ee4bec1601a0455f7015dda4c/href">https://medium.com/media/93b41e0ee4bec1601a0455f7015dda4c/href</a></iframe><h3>Introduction</h3><p>This article is the first in a series dedicated to turnk8s; we will walk through the architecture and the tools used.</p><h3>The end goal</h3><p>GitOps is an excellent approach to managing infrastructure. Therefore, we follow the GitOps practices: Everything starts on a Git push event, and Git stores the desired state of your infrastructure. <br>turnk8s deploys a Kubernetes cluster and creates an Infra (manifest) repository, which stores the manifests of the apps developed by your users deployed on the cluster. The repository is created and destroyed with the cluster. We do not recommend sharing repositories between multiple clusters. After turnk8s pipeline execution, you get a ~/.kube/conf file and a repository with sample service manifests deployed on the cluster available to all the users in your local network. Your users will then delete the manifests of the sample service and push relevant ones. ArgoCD deploys the sample service within minutes. Please note that turnk8s is a work in progress.</p><h3>The toolset and workflow</h3><ol><li>Users push a cluster declarative cluster definitions to the clusters repo, where all the Terraform and helper scripts are located.</li><li>GitHub action workflow runs Terraform to ensure the alignment between the cluster definitions and the actual infrastructure. Hence, Git is the source of the truth for the existing Kubernetes clusters.</li><li>Terrafrom Proxmox provider reads the cluster definitions and communicates with the Proxmox server to create and/or destroy Talos Linux virtual machines.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BqUfJ7JcvuiInzd8-St_Vg.png" /></figure><p>4. The Talos configuration is generated and applied to the newly created Talos Linux virtual machines, forming the Kubernetes cluster using the Talos Terraform provider.</p><p>5–6. ArgoCD and Netris operators are deployed with Helm Terraform provider.</p><p>7. A GitHub repo with a sample service manifest is being created. This repo is strictly associated with the creation of the Kubernetes cluster. ArgoCD is configured to watch this repository to ensure synchronization between the service manifests in the repository and the Kubernetes cluster.</p><p>8. ArgoCD pulls the sample service manifest and deploys it on the Kubernetes cluster.</p><p>9. As the service type is LoadBalancer, the Netris operator creates a load balancer on the Netris Softgate to make it available for the user.</p><p>10. The user gets the service URL from the Netris Controller dashboard and accesses it.</p><h3>Infraheads working environment</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IrUXyIE5g1hFOlTYZ6zBdw.png" /></figure><p>We have two SER7 boxes with 32 GB of RAM and 16 vCPUs. The router and switch are depicted as separate devices; however, that is a single SOHO router. The Netgear router serves as an internet gateway and a DHCP server for our management network, and Softgate serves as a DHCP server for Kubernetes cluster nodes.</p><h3>In the next episode…</h3><p>In the next episode, we will discuss the Terraform codebase and other details. We will explain our choices and demonstrate the creation of a sample cluster to demystify our solution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ff2f42c195e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adopting AI in Microsoft Azure]]></title>
            <link>https://medium.com/@info_98249/adopting-ai-in-microsoft-azure-b9994e41914a?source=rss-16994648a82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9994e41914a</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[infrastructure-automation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[infraheads]]></dc:creator>
            <pubDate>Thu, 25 Apr 2024 11:37:18 GMT</pubDate>
            <atom:updated>2024-04-25T11:41:25.725Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, we will walk through the stages of AI adoption for Azure shops that have the data necessary to start training/serving models.</p><p>Surprisingly, the statistics show that many companies spend around a year from the decision to the live production deployment <a href="#a0b8">[1]</a>. Another evidence is that 79% of AI projects fail because of the infrastructure <a href="#a0b8">[2]</a>. MLOps is evolving to solve these problems, and the best practices are already getting some shape.</p><p>We want to share our experiance from an engineering and management standpoint to <a href="https://www.linkedin.com/feed/update/urn:li:activity:7064853812703805440?utm_source=share&amp;utm_medium=member_desktop">contribute</a> to our <a href="https://www.linkedin.com/posts/edmarkosyan_epamarmenia-ai-devops-activity-7122119957802274816-wlbg?utm_source=share&amp;utm_medium=member_desktop">community</a>. Although this article describes the tools used in Microsoft Azure, we want to mention our partners. DVC&#39;s tools are excellent for experiment tracking and model traceability. Netris is the only tool that brings VPC-like network management to OnPrem platforms.</p><figure><img alt="Data Science Lifecycle" src="https://cdn-images-1.medium.com/max/1024/1*hiUjYtjCLgLE1rZuX446dA.jpeg" /></figure><p><strong>First things first: Data<br></strong>Data is the new oil <a href="#a0b8">[3]</a>. A few years ago, there was still a debate about data and algorithm priorities. ChatGPT and all the other tools proved that data wins. The higher the number of quality data points, the better the resulting model. The algorithms and network architectures are still improving; however, the phase of initial fundamental ideas like backpropagation is already behind us. Most papers are LEGO-like approaches to enhancing applications, and that situation will prevail. However, some pioners are still developing strategies for improved model architectures that are non-data-hungry. <a href="http://a0b8/">[2]</a><br>A famous tweet from the author of <a href="#a0b8">[3]</a> says, &quot;Data scientists: the ones who need big data will be unemployed in 10 years. Machine learning increasingly shifts from data-intensive to learning-intensive.&quot; That turned out to be wrong.</p><figure><img alt="The Mater Algorithm cover" src="https://cdn-images-1.medium.com/max/254/1*X6vQneStN265cHquEUPVeA.jpeg" /></figure><p>The keyword in the previous paragraph is &quot;quality data.&quot; The companies with high cardinality and trustworthy out-of-the-box data are lucky, as they can benefit from that. Many clients have an AI team but need help to get quality data. Some dive into the promising field of synthetic data, which will never replace actual data.</p><p>Business starts with quality data, and due to the nature of your business and its applications, you may be lucky to have it out of the box.</p><p><strong>Data infrastructure<br></strong>Data is part of any infrastructure long before the company starts the AI journey. It is usually represented in CSV files, web server access logs, relational or NoSQL database files, etc. We call that raw data. Raw data is stored in the systems used as Data Lakes, hence the name. Azure has a service built on top of Azure Blob Storage called Azure Data Lake Storage Gen 2. It is not an independent service. It is an addition to Azure Blob Storage. You still use the Azure Blob Storage when you use the Azure Data Lake Storage; it is just an interface with some added business features. This feature allows Azure Data Lake Storage to integrate with Azure Synapse Analytics or Azure Databricks for data exploration.</p><figure><img alt="Azure Data Lake Storage Gen 2" src="https://cdn-images-1.medium.com/max/1024/1*Cdl04fc35AqVjYLG7v3UWw.jpeg" /></figure><p>The first step is always to define the problem to solve. That is achieved with the cooperation of the infrastructure engineer, data engineer, and data scientist <a href="#a0b8">[1]</a>. The infrastructure engineer supports the data scientist with access to the Data Lake and the tools for convenient data exploration. However, many data scientists are used to consuming Python libs in Jupyther Notebooks for that purpose; a more robust approach is using specialized tools that Azure Synapse Analytics, Azure Databricks, and other 3rd party vendors bring to your Azure environments. The data exploration results in the definition of the problems that can be solved with machine learning using the existing data for training the models. Typically, at that stage, the cooperation between data engineers and data scientists becomes super efficient as the data scientist knows what needs to be extracted from the data lake and how it should be transformed and loaded into the data warehouse. Precise task definitions for the data pipelines are decomposed at that stage. Data engineers start defining those pipelines using Azure Data Factory or similar tools available in Azure. Feature engineering starts.</p><figure><img alt="Azure Synapse Analytics" src="https://cdn-images-1.medium.com/max/873/1*pyxStZKgYqijO2WNnGZdNA.jpeg" /></figure><p>During the last 3–5 years, specialized Data Warehouse extensions have appeared. Feature stores are not precisely Data Warehouses; they are specialized extensions for Data warehouses and sometimes for Data Lakes. The primary purpose of Feature Stores is to separate the data source specifics from the AI model training code. Data loader portions of the training code consume only the SDK Feature Stores provide. SQL queries are removed from the code. AI engineers deal with features instead of various data source types. Data Warehouses are used for reporting and other analytical applications in the company; AI engineers are not the only consumers of Data Warehouse solutions like Azure Synapse Analytics or Azure SQL Database. On the other hand, only AI engineers consume Feature Stores, and data Marts are extensions of data warehouses <a href="#a0b8">[4]</a>.</p><p>Feature stores are the last stop for the data before the start of the training process.</p><p><strong>Training infrastructure<br></strong>Data transformations are memory and compute-heavy operations that can be highly parallelized; therefore, sometimes, GPUs are also used for data transformations. However, the main application of GPUs is training the AI model, which is the most expansive stage in the AI development lifecycle.<br>However, the training stage does not start immediately; before the training begins with the total amount of the training dataset, the hyperparameter tuning and experimentation phases are executed. Typically, AI engineers use 10% to 15% of the data to experiment with hyperparameters. That involves multiple executions of the training code on actual training infrastructure to discover the optimal set of parameters. If the team is relatively big and numerous people are working on the same problems, experiment tracking and sharing becomes a crucial organizational problem that is solved with tools like Azure ML Studio. It has built-in compute auto-deployment features. Azure Kubernetes Service is one of the options the Azure ML Studio uses to deploy a compute cluster for experiments and actual training. Azure ML Studio is a great tool. However, we prefer DVC Studio for experiment tracking, and the DVC itself is the best tool for practicing GitOps in your AI workflows. Some of our clients in finance and healthcare use that as it ensures encapsulation of the data used for training, the model instance file, and the values of the hyperparameters into a single Git commit <a href="#a0b8">[2]</a>, providing the model traceability. Azure ML Studio has just a subset of those features.</p><figure><img alt="Azure ML Studio" src="https://cdn-images-1.medium.com/max/1024/1*KvQ9ZWUbq8qIeiMx9yqkvg.jpeg" /></figure><p>Typically, when a company succeeds in AI adoption, the costs of GPU-enabled instances skyrocket, and the need to move to OnPrem evolves. Infraheads has an excellent automation framework for bare metal Kubernetes Cluster provisioning to support the needs of our clients. In such cases, Azure provides a wide range of networking services that help build hybrid infrastructures.</p><p><strong>Testing and inference infrastructure<br></strong>After the testing, the model instance is registered in the Azure ML Model Registry. The ML engineers are finalizing the inference service. That is, in general, an API server that reads HTTP Post Requests with the features in the payload and returns the result the model produces. That process is called inference. Azure ML Studio has excellent features to facilitate that process. For highly loaded systems, Azure ML Studio provides horizontal autoscaling features. However, before the production traffic is forwarded to the inference endpoints, the model is tested with the testing data set, another 10% to 15% subset of the actual data. The testing data set is used for inference, and the results are compared with the exact value to assess the model&#39;s qualitative performance. The AI engineers can add custom metrics based on the business requirements, the application, and the widely used metrics for that assessment.</p><p><strong>Monitoring and redeployment<br></strong>Once deployed, the model&#39;s performance starts decreasing. Therefore, it has to be monitored constantly, and automated retraining and continual deployment workflow should be implemented and tested before going live. Azure Monitor and Application Insights provide an incomplete set of monitoring features.</p><figure><img alt="Azure Monitor" src="https://cdn-images-1.medium.com/max/1024/1*_1N1CUVoMZYKfY5ipDvLxQ.jpeg" /></figure><p><strong>Data Mesh and microservices<br></strong>From the overall architecture standpoint, model services should be treated as microservices. SemVer 2.0.0 <a href="https://medium.com/@info_98249/monolith-splitting-stories-part-5-semver-2-0-0-in-practice-898ec97d213b?source=user_profile---------13----------------------------">is to be used</a> for service source code versioning. (Versioning of models is a separate topic that is intentionally skipped).</p><figure><img alt="Data Mesh in Azure" src="https://cdn-images-1.medium.com/max/1024/1*SGYWvRU-9jxhxgNAwhL5fQ.jpeg" /></figure><p>In analogy with the microservices architecture, a new architecture paradigm is evolving for data. Data Mesh suggests moving away from centralized data lakes and warehouses, creating the data infrastructure for each microservice separately, and delegating the responsibility of the so-called data products to the team that owns the service. We closely monitor the evolution of this concept. However, we still need to implement this kind of architecture.</p><p><strong>References</strong></p><p>[1] <a href="https://www.manning.com/books/effective-data-science-infrastructure">Effective Data Science Infrastructure</a><br>[2] <a href="https://www.oreilly.com/library/view/designing-machine-learning/9781098107956/">Designing Machine Learning Systems</a><br>[3] <a href="https://en.wikipedia.org/wiki/The_Master_Algorithm">The Master Algorithm</a><br>[4] <a href="https://web.archive.org/web/20110420134556/http://csis.bits-pilani.ac.in/faculty/goel/Data%20Warehousing/Articles/Data%20Marts/dataWarehouse_com%20%20Article_DM%20VS%20DW.htm">Data Mart Does Not Equal Data Warehouse</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9994e41914a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Infrahead’s perspective on growing talent]]></title>
            <link>https://medium.com/@info_98249/infraheads-perspective-on-growing-talent-f8f01147c0d1?source=rss-16994648a82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8f01147c0d1</guid>
            <category><![CDATA[staffing]]></category>
            <category><![CDATA[talent-management]]></category>
            <category><![CDATA[talent-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[talent-acquisition]]></category>
            <dc:creator><![CDATA[infraheads]]></dc:creator>
            <pubDate>Thu, 04 Apr 2024 10:50:43 GMT</pubDate>
            <atom:updated>2024-04-04T10:50:43.342Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/238/1*Kd1NfmdpSn9p9-HaYS7mSw.jpeg" /></figure><p>One of the two main reasons we don’t like remote or even hybrid working styles is blockers that introduce the growth of junior specialists. Although infraheads can interview DevOps engineers from the market and pick the best one, we prefer helping people start DevOps careers and organize their learning process. The first step of talent growth is choosing the right people. We’ve made many mistakes trying to analyze people’s behavior and tried different approaches. After hundreds of interviews, we realized that interviews are acting games and, therefore, not the best way to choose someone to bring in. We interview people and suggest they work on a task that seems super hard for them; in fact, many companies would benefit greatly from having someone who can solve our task. So, when we share the tasks, it is clear to everyone that they don’t have the hard skills and the knowledge to solve the problem, which is the key. We share our opinion on how long it should take but clarify that it is not a problem if it takes twice as long. And that is it; in 99% of cases, we don’t ask for updates, and we don’t even expect they will come back with the solution. The good thing about this approach is that some come back, and if it is much faster than we thought, we assume that we’ve made mistakes during the interview and suggest more complex improvements. Some don’t come back on that stage.</p><p>When they notify us that they’ve solved the problem, we suggest they prepare a presentation with a demo for our team. Senior engineers and other candidates are invited to this private event. Surprisingly, the seniors constantly complain about being too busy asking questions and advising the candidates. Then, we give learning advice to the candidates and let them go with no promises or expectations. Immediately after the event, we shared opinions on the candidates and forgot about them. After, as soon as an in-house engineer joins a new client or a new project, we contact the candidate the in-house engineer liked the most and propose an internship, and we propose free resources to the client to shadow our in-house engineers. Those juniors are often used for rubber duck debugging (humor and self-irony are the most significant parts of it), and we assign them to our internal projects and loosely manage the process. We try hard to ensure that they learn independently unless the in-house engineer is willing to chime in and support somewhere in in-house projects.</p><p>In about a month, when we see what we expect, we introduce our perspective on IT certifications to our interns, pay for the successful attempts, and hire them when they pass any professional-level certification exam. Typically, by that time, we have a position with a senior engineer in the team.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8f01147c0d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Triggering CI Pipelines: When and Why]]></title>
            <link>https://medium.com/@info_98249/triggering-ci-pipelines-when-and-why-7442d5af9a08?source=rss-16994648a82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7442d5af9a08</guid>
            <category><![CDATA[gitflow]]></category>
            <category><![CDATA[sdlc]]></category>
            <category><![CDATA[semver]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[ci-pipeline]]></category>
            <dc:creator><![CDATA[infraheads]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 15:37:11 GMT</pubDate>
            <atom:updated>2024-04-03T15:37:11.521Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, we will walk through the CI pipeline that builds a docker image for an API service that has some dependencies and services dependent on it. Let&#39;s count all the cases when an event should trigger the CI pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*04qTAhwrJeXAxU4XgWXzxA.jpeg" /></figure><ol><li>The pipeline should execute unit tests on the commit.</li><li>The pipeline should execute integration tests on the commit.</li><li>A commit should be deployed to pre-production.</li><li>A commit should be deployed to the production.</li></ol><h3><strong>Unit tests are blessings for Developers</strong></h3><p>As the developer creates the future branch and pushes the code, the CI pipeline should run the unit test to inform the developer of any failures as soon as possible. In most cases, you will not need to build any images; some other preparation is required. You&#39;ll need to install more unit testing libs on your CI worker; however, you can use prebuilt docker images for the worker. Unit tests are executed on a single machine; therefore, preparation is fast, and execution is more rapid. These are the most valuable tests from an ROI standpoint. Invest in writing as many unit tests as possible; unit tests are a blessing.</p><h3><strong>Only the developer knows the next version</strong></h3><p>When the developer thinks the code is good enough to be checked with the rest of the codebase, a PR to the integration branch (called development in many cases) should be created or converted from the draft(GitHub) to a regular PR. In this case, you must build a docker image and run the service on it, as it will serve requests and initiate others. In some cases, contract testing is used, which changes things slightly, but in general, you need to build an image to run a container. However, you only need to store it if it passes the integration tests. You can fully automate this process if you have a quality set of integration tests with no flakiness. As the pipeline sees all the integration tests passing, it can create a release candidate and push it to the container registry. But there is a problem here. Assigning a version tag before moving it to the container registry is a must. Suppose the current version is 0.1.0. What will the new version be? We know it should end with &quot;-rcN,&quot; but what should come before that? The answer is only people know. A bot can&#39;t know what kind of commit it is. Has it been introduced as a bug fix or a new feature? Therefore, you must think twice before building an image from the integration branch. The PR can have the semantics missing. That can be solved with conventional commits, forcing developers to be mindful of commits. And remember, most of them hate that. If no conventional commits are used, PR creation can trigger the integration tests, and if all the tests are passed, a human can create a release candidate tag and push that to the integration branch, which will trigger another pipeline to assign the tag to the image with some metadata(tag) about the commit(a commit sha) to ensure that the workflow assigns the release candidate tag to the proper image. There is no need to say that the image should be removed immediately if the tests are filed.</p><h3><strong>No images should be rebuilt</strong></h3><p>The image with a release candidate(pre-release) tag can be deployed in a testing environment for manual testing or the so-called UAT. As the QA gives the green light for the production deployment, a new tag with the pre-release portion removed should be assigned to the same image. That should be done with another pipeline execution, and the trigger for that should be in the event of a release tag push to the main branch. Of course, the tested pre-release tag should be merged with the main branch, and a release tag should be manually created on the main branch. In many compliance standards, a business process that ensures the deployment of the same image used for testing is required for production.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7442d5af9a08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud cost-cutting is more straightforward than you think]]></title>
            <link>https://medium.com/@info_98249/cloud-cost-cutting-is-more-straightforward-than-you-think-c0ae2a265854?source=rss-16994648a82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0ae2a265854</guid>
            <category><![CDATA[terragrunt]]></category>
            <category><![CDATA[cost-cutting]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[infraheads]]></dc:creator>
            <pubDate>Fri, 22 Mar 2024 12:38:35 GMT</pubDate>
            <atom:updated>2024-03-22T12:40:56.265Z</atom:updated>
            <content:encoded><![CDATA[<p>Do you know that the cloud providers practice oversubscription? They promise you the hardware that they still need to provision. They do so because they know you will not use the resources you pay for. It may not be about you, but most of their clients don’t. That is where their earnings come from.</p><p>We can assume that most companies misuse clouds. Change management is essential for cost-saving and mature engineering, and unfortunately, it is not available for most companies. The first consequence of that is financial waste. A company paying $1m monthly to Azure had too many provisioned MSSQL servers, and every new DBA could find an unused server earning ~$7000 monthly during the onboarding. Some of their engineers thought the cost could be reduced by at least 50%.</p><p>At some point, they assigned a tag to each team and made it mandatory to add the team tag for every resource, which was done for cost management. However, anyone could use an arbitrary value, basically any value. Most people were not abusing that. The policy was half done.</p><p>Inadequate resources may be provisioned due to inaccurate technical calculations, which is normal. People can overcommit and monitor to adjust shortly. But visibility can be improved when everyone is free to do anything on the web console, and people can prioritize their deadlines. Thanks to the Broken window theory, we can predict the future. Punishing for minor sins has proven to be an efficient approach, but determining the punishment is always a complex problem. Whatever it is, people need to accept the fairness of it. Good luck with figuring that out. What we propose is proper public cloud management practices.</p><p>Here are the recommendations of Infraheads:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*vNJnbuDMLrnEqs9LlquV7Q.jpeg" /></figure><ol><li><strong>Restrict write access to any resources to anyone: </strong>This may sound like an aggressive step, but people will complain, and the most common argument will be, “What if we need to change something when prod is down?!” This is a classical case of the Allegory of the Cave. Don’t be afraid of that resistance.</li><li><strong>Identify the roles and people</strong> who can modify different tiers of your cloud infrastructure and reflect that hierarchy in your Git configuration. If you use Terraform, it should be applied from the main branch. Beware of a gotcha there; the Terraform code can be “playable” but not “applicable.” Make sure to have a well-defined and trained validation and rollback process.</li><li><strong>Use GitOps:</strong> Make sure all the changes are reflected in Git. Tools like Atlantis can help you set up GitOps with Terraform for cloud management.</li><li><strong>Fragment your IaC:</strong> Terraform checks the resources in the state files during the plan. The longer the list of your resources in the plan, the longer it takes to plan. Remember, the engineers hate that. Waiting until something is being applied is the worst experience for your cloud engineers. It makes them super unproductive. Practice fragmented infrastructure and keep fragments simple and stupid.</li><li><strong>Don’t use Terragrunt: </strong>Terraform is not DRY by nature. Having a few repeating fragments of the code is fine for small infrastructures. The team behind Terraform is highly mature, so trust their decisions. If you want to use Terragrunt, you probably don’t understand the rationale behind the absence of the feature you need in Terraform. Gurntworks is also a team of brilliant engineers, but they look at the world through their prism of 300,000 infrastructure testing Go code. How many companies do you know that test their infra programmatically?</li><li><strong>Be prepared: </strong>Invest in your team’s knowledge and ensure it comes from hands-on practice. Periodically simulate downtimes and emergencies in the testing environment to prepare for the battlefield.</li></ol><p><strong>Summary:</strong></p><p>Instead of paying for the tools that scan your clouds to find unused resources, apply simple practices of proper cloud management. You can benefit even more from doing that from the very beginning.</p><p>At Infrahead, we have professionally certified AWS and Azure engineers who can fix your cloud management. Cost-cutting is just one of the benefits you’ll get.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0ae2a265854" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DevOps engineer’s perspective on software tests]]></title>
            <link>https://medium.com/@info_98249/devops-engineers-perspective-on-software-tests-7c3ca5592ce0?source=rss-16994648a82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c3ca5592ce0</guid>
            <category><![CDATA[testing-pyramid]]></category>
            <category><![CDATA[contract-testing]]></category>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[ci-pipeline]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <dc:creator><![CDATA[infraheads]]></dc:creator>
            <pubDate>Wed, 20 Mar 2024 12:09:55 GMT</pubDate>
            <atom:updated>2024-03-29T12:05:03.143Z</atom:updated>
            <content:encoded><![CDATA[<p>At Infraheads, we have worked with the QA engineers the most. Failing CI pipelines we implement mainly fail because of the flaky tests. In most cases, the CI pipeline users are not prepared to reflect on error messages shown in the pipeline logs, and we get the blame. That leads to many problems, especially in the teams that try to practice SCRUM. SCRUM masters assume they have a DevOps resource, but that resource is forced to do a reactive job most of the time instead of implementing robust tools for improved SDLC and IaC. That is why infraheads separate support from implementation.</p><p>Based on our observation, most software engineers underestimate the importance of quality tests, and most QA engineers need to be more skilled to implement a quality testing codebase. The lion’s share of flaky tests is due to implementation, not the infrastructure. But there are other problems as well. Ignoring improper reflection on the “Testing pyramid” is an even bigger problem.</p><p>One of our clients was in the middle of splitting a 10-year-old C# monolith, but they still had over 10000 tests, execution time was over 3 hours, and the monthly Azure bill was over $1m. Before production deployments, the pipeline had been executed multiple times until each test was passed at least once. They had a dedicated team for that, and it turned out that they were not unique at all. This situation results from a need for a common language and understanding of test types. It is easy to find a folder in the Git repo called unit tests with integration tests or vice versa. In many cases, the same mistake of letting the developer implement enterprise application tests, pipelines, and everything else shoots back.</p><p>There are multiple versions of the testing pyramid, but we prefer the simplest one shown in the image.</p><figure><img alt="Simplified testing pyramid with just 3 layers. E2E tests, Integration tests and Unit tests." src="https://cdn-images-1.medium.com/max/619/1*TYxwX80smwRyK0LR2KLBgw.jpeg" /><figcaption>Testing pyramid</figcaption></figure><p>TDD is an excellent idea. Although some 10x developers would argue how they want to use the tests and are right when they question any widely spread practices, for most companies, unit tests should be implemented before the feature implementation based on the document or formal requirements represented as tickets or something else. We are shy about saying documentation as Scrum questions the importance of documentation, and we need more confidence to argue. What we know for sure is that quality communication is essential.</p><p>The rule of thumb is that having as many unit tests as possible is a good sign, as unit tests are:</p><ol><li>Executed within seconds</li><li>Cost-effective</li><li>Keeping the infrastructure simple</li><li>Less flaky</li></ol><p>These are just a few benefits of unit tests.</p><p>Integration tests are expensive and make the codebase complicated. Mocks are used for cost-saving and efficiency, but those create solos and contribute to pushing the bugs down the line. One exciting idea is pactflow, which is an implementation of contract testing. Contracts are outstanding; one of the fundamental decisions made by AWS is having clear regulations between the services; contracts help to achieve that, but we remember that it is not about the tools; it is all about the people. Pactflow helps achieve integration testing without spinning up an entire infrastructure. It makes integration testing an asynchronous mechanism instead of the classical synchronous approach. This makes the integration testing much faster but less reliable, and it takes a skill to benefit from it.</p><p>End-to-end tests should be executed only before the release on the staging environments. If an organization runs those tests significantly more than the deployment pipeline, its processes probably need to be corrected.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c3ca5592ce0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Avoiding traps with IP Address management]]></title>
            <link>https://medium.com/@info_98249/avoiding-traps-with-ip-address-management-8b3ffffe4609?source=rss-16994648a82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b3ffffe4609</guid>
            <category><![CDATA[subnetting]]></category>
            <category><![CDATA[ipam]]></category>
            <category><![CDATA[cloud-networking]]></category>
            <category><![CDATA[infrastructure-management]]></category>
            <category><![CDATA[private-ip-address]]></category>
            <dc:creator><![CDATA[infraheads]]></dc:creator>
            <pubDate>Mon, 18 Mar 2024 14:11:24 GMT</pubDate>
            <atom:updated>2024-03-18T14:12:42.574Z</atom:updated>
            <content:encoded><![CDATA[<p>Nearly all of our clients that have infrastructure components in more than one cloud provider have the problem of overlapping private IP ranges. We believe that the best practices should not be postponed. Immediately after the product validation, building engineering practices should be started from scratch. Follow the best practices. The cost of fixing the infrastructure grows exponentially depending on the time of postponing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cU7muAXG1Ei47-nq1gWRpw.png" /></figure><p>IP Address Management or IPAM plays a crucial role in every infrastructure, and as all the cloud providers suggest using the same private IP ranges, most engineers don’t think twice before choosing a subnet, in fact, most of us don’t even change the default values. Unfortunately, the default values are often the same in many systems. The worst thing about this problem is that it becomes a blocker very late in the life of any startup. At that point, the startup typically solves high availability problems for the possible downtimes in the cloud provider services and most probably by that time absence of IPAM is a hidden problem for many other scenarios.</p><p>All the cloud providers suggest using their solutions for IPAM. However, if you stick to a single provider, you will not have the problem of overlapping private subnets because, in a single cloud provider, IPAM is done under the hood.</p><p>Netris has that feature out of the box, and we think it is the best tool to track the usage of your IP addresses because the Site Mesh feature it suggests is used in scenarios where you need to connect subnets of different cloud providers and On-Prem environments.</p><p>The alternatives are standalone services where your data is just data to be read by the engineers, while in Netris it is being actively updated and consumed by the other components of the engine. Deploying a standalone VM for a web application that your engineers use to track the IP addressing information is still a valid solution if you don’t use the other features of Netris, however, that becomes a burden of having yet another server to maintain.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b3ffffe4609" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Infraheads’ perspective on IT certifications: A balanced approach]]></title>
            <link>https://medium.com/@info_98249/infraheads-perspective-on-it-certifications-a-balanced-approach-2b029b33504d?source=rss-16994648a82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b029b33504d</guid>
            <category><![CDATA[certification-exam]]></category>
            <category><![CDATA[kubernetes-certification]]></category>
            <category><![CDATA[azure-certification]]></category>
            <category><![CDATA[devops-certification]]></category>
            <category><![CDATA[aws-certification]]></category>
            <dc:creator><![CDATA[infraheads]]></dc:creator>
            <pubDate>Mon, 18 Mar 2024 07:32:15 GMT</pubDate>
            <atom:updated>2024-09-04T09:40:06.787Z</atom:updated>
            <content:encoded><![CDATA[<p>In the dynamic realm of information technology, the debate around the value of IT certifications remains a hot topic. At Infraheads, we’ve embraced certifications, particularly in the Kubernetes domain, where each of us boasts at least one certification. Furthermore, for those embarking on a journey with us, holding a professional-level cloud provider certification is not just encouraged; it’s a requirement. But why emphasize certifications, especially when the tech community often debates their real-world applicability? Let’s dive deeper into our rationale and strategy, balancing the scales between certification and practical expertise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*wr2QEHPN9O5G1bFabhFWuA.png" /></figure><h3>The core of our belief: Certification as a benchmark</h3><p>Yes, the criticism is familiar to us all: the presence of certified professionals who lack the practical skills their credentials suggest. But here’s our take. Our goal-setting process, driven by Objectives and Key Results (OKRs), is client-centric, adopting a client development approach. Given our identity as a DevOps agency, we prioritize the skills most relevant to our clients’ needs. This client-first mindset necessitates a benchmark, a reliable means to ensure baseline competence among our engineers. This is where certifications play a crucial role.</p><h3>Certifications: Beyond the paper</h3><p>Certifications serve multiple purposes in our organization. On a large scale, where management thrives on data-driven decisions, certifications offer a tangible measure of an engineer’s foundational knowledge. This measurable aspect is particularly valued by large enterprises, forming a crucial part of their requirements for our proposals. However, our commitment to certification goes beyond client mandates; it’s a part of our culture, even when dealing with startups.</p><p>Numerous studies support the psychological aspect of learning, suggesting that clear goals significantly boost achievement rates. The structured path and clear objectives provided by certification programs cater to this human tendency. Many programs incorporate elements of gamification—a strategy excellently executed by companies like Cisco—so learning becomes not just a goal but an engaging journey. This approach fosters a healthy competitive spirit and encourages continuous skill development, albeit with a focus on practical application.</p><h3>Managing the downsides</h3><p>No system is without its flaws. Achieving a high-level certification, such as an AWS Certified DevOps Professional or Certified Kubernetes Administrator (CKA), can sometimes lead to misplaced expectations regarding compensation and recognition. The challenge lies in cultivating a culture where the value created precedes the badges earned. We view certifications within the framework of OKRs as indicators of effort and growth potential rather than direct qualifiers for advancement or pay raises.</p><p>Understanding and articulating this distinction is crucial. Acquiring a certification is commendable and indicative of expanded knowledge. Yet, the true measure of an engineer’s worth lies in their ability to apply this knowledge effectively — a principle we emphasize at Infraheads.</p><h3>A tailored approach to certification</h3><p>Our strategy in selecting certifications is far from arbitrary. We align our choices with our clients&#39; current and future needs, ensuring that the certifications our engineers pursue add tangible value to our projects and client relationships. This alignment also allows us to gauge an engineer’s growth through direct client feedback, creating a feedback loop that benefits all parties involved.</p><p>In conclusion, at Infraheads, certifications are more than just credentials; they are a part of our strategic fabric. They provide a framework for learning, a benchmark for skill assessment, and a catalyst for professional growth. However, the journey doesn’t end with certification. The real challenge, and our ultimate goal, is to continually apply and expand upon this knowledge, ensuring that our team grows in expertise and delivers unparalleled value to our clients.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b029b33504d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>