<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[DevOops Discourse - Medium]]></title>
        <description><![CDATA[A collection of commonly observed fumbles &amp; foibles in modern software engineering and thoughts on mitigation thereof. - Medium]]></description>
        <link>https://medium.com/devoops-discourse?source=rss----d31c962fddc2---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>DevOops Discourse - Medium</title>
            <link>https://medium.com/devoops-discourse?source=rss----d31c962fddc2---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:28:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/devoops-discourse" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Mismatching Team and System Structures]]></title>
            <link>https://medium.com/devoops-discourse/mismatching-team-and-system-structures-1d92c64b4cd9?source=rss----d31c962fddc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/1d92c64b4cd9</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Robert Glenn]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 22:00:38 GMT</pubDate>
            <atom:updated>2026-11-11T22:00:40.072Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article was a “deleted scene” from my upcoming book </em><a href="https://learning.oreilly.com/library/view/terraform-at-scale/9798341649705/"><em>Terraform at Scale</em></a><em> (O’Reilly, 2026) available for </em><a href="https://www.amazon.com/Terraform-Scale-Practices-Enterprise-Development/dp/B0GKF63FJ4/ref=sr_1_1"><em>pre-order now on Amazon</em></a><em>.</em></p><p>Professionally as a consultant, I’ve spent a lot of time working with different types and sizes of organizations and in some very different technical contexts, from developing applications to constructing the cloud infrastructure or platforms a customer-facing application runs upon. Certainly, these settings have their own requirements which may be filled by familiar technical roles or personas, such as <em>full-stack developer</em> or <em>site reliability expert</em>, but it’s just as certain that no two teams have exactly the same skills and experience found in exactly the same orientation of team members with exactly the same ethos, temperaments, and levels of commitment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ay4CknvIWXW_TBoh1Cno0g.png" /><figcaption>Generated with ChatGPT</figcaption></figure><p>As such, I’m often reminded of Conway’s Law which states “organizations which design systems…are constrained to produce designs which are copies of the communication structures of these organizations”. A common interpretation of this allows for an eventual consistency generally on the side of the system — i.e. the system itself will eventually align to the communication channels (both formal and informal) that the organization leverages at certain particular scopes. In a perfect world, there’s something of a feedback loop that helps to govern this alignment whereby collective goals encounter technical challenges, which are resolved by pairs or groups of individuals communicating and working in coordination, and which leads to more complex solutions that have additional challenges that require new coordinations among the same, or even new, sets of individuals, and so on.</p><p>However, many real world solution and architecture decisions in software development are based heavily on financial considerations rather than emerging from primarily technical ones. For example, whereas the choice to centralize or decentralize certain foundational cloud components is indeed largely a technical decision, a choice to heavily centralize the development and maintenance thereof may be made by regional or national brands to reduce the need for significant facilities in multiple jurisdictions or to take advantage of economic incentives only available in a certain locality, regardless of the technical implementation. Likewise, a choice to largely decentralize the operation of a singular, centralized cloud infrastructure ecosystem core might piggyback off a global firm’s existing facilities and be chosen as a way to enable round-the-clock support that also takes advantage of different job markets in different countries and regions.</p><p>In such cases, you may have to operate within a team model that seems to be at odds with Conway’s law: the organization’s systems do not (appear to) resemble the way in which it organizes its teams. However, Conway’s law is about communication channels, which, in today’s ever more connected world, need not include collocation. As such, cases in which the members of the team and the systems they operate do not match, it requires somewhat of a reverse of the paradigm: the communication structure of such teams may need to evolve to more closely resemble the systems they are intended to operate, but this shouldn’t happen through “brute force and ignorance” as a cherished teacher from my high school was wont to say.</p><h3>Colocated Teams + Decentralized Ecosystem</h3><p>In the case of a centralized team, such as one collocated at the same physical office, that manages several regionally-bounded portions of the ecosystem, the other teams that represent their direct customers may be scattered in different locations. This is especially common for a firm that has a wide portfolio of products that are regionally constrained, either due to legal regulations in a particular jurisdiction or localized brand popularity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lZrjMDNpB1JeSPyd3UGkPw.png" /><figcaption>Generated with ChatGPT</figcaption></figure><p>For an example in the case of legal regulations, consider a company that has Product A (PA) in Jurisdiction A (JA), and they want to market it in Jurisdiction B (JB), but JB has different requirements than JA, so they come up with Product B (PB) that is an exact facsimile of PA, except that it is distinct and configured so that it adheres to JB’s laws. This allows the firm to decentralize the coordination and decision making powers of the two products, enabling them to operate more efficiently than they could if decisions were centralized across the products. With regard to the popularity of a localized brand, this could simply be due to the firm acquiring similar products in different localities, and keeping the existing branding and product ownership intact. In this way, the firm’s upper management has significant control over their individual brands and can operate largely in isolation of the firm’s other brands.</p><p>Especially in cases where the organization largely operates as a portfolio of independent brands or business units, or if the company’s main money-making products are digital — which could require specialized software considerations for a particular product — it may be infeasible to centrally prescribe the exact implementation of the ecosystem, such as edge networking components and baseline IAM controls. At the same time, it may be deemed equally infeasible (or unnecessary) to scatter such a team, embedding them within the localities they service.</p><p>Despite their physical proximity, the disjoint nature of the projects can become its own siloing effect, creating a sense that since “they work on other things, they won’t understand the issues we face”. This can be unintentionally reinforced culturally by aligning the teams’ schedules, communication channels, or even nomenclature toward the region or brand they serve (e.g. referring to a team as “[Region Codename / Brand] Infrastructure”). The distance becomes psychological rather than physical.</p><h3>Scattered Teams + Centralized Ecosystem</h3><p>In today’s world, it’s increasingly common to have a significant number of remote team members. As long as you have a stable internet connection, you can contribute to cloud-based solutions. This means the choice to be fully colocated is somewhat arbitrary (although there is certainly something to be said for being in the same room when making decisions or providing updates). By the same token, many companies have increasingly invested in satellite offices, meaning even if everyone is working in an office, the team may be spread across two or more sites.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qF3mQ-pez2sz7jgnSsrw9w.png" /><figcaption>Generated with ChatGPT</figcaption></figure><p>This means that even though a team might be servicing a centralized ecosystem, it’s not unlikely that the team itself will be somewhat scattered, possibly even spanning multiple time zones. This creates friction both obvious and not-so-obvious. Certainly, it means there will likely be more formal meetings scheduled, which may be considered a negative to some, as it can take focus away from things like flow in one’s work.</p><p>There’s also a greater risk of miscommunication through a lack of awareness in terms of being able to see what people are working on or the conversations between coworkers. This might be mitigated if the teams cultivate an environment that promotes conversations in shared channels, but it doesn’t exactly disallow sidebar conversations, and if these happen in private chat channels, there’s no way for others to listen in — which, despite the privacy concerns, I can attest to multiple instances where I overheard something being discussed that was false or unnecessarily risky, and chiming in kept folks from going down a difficult or costly path.</p><h3>Gaining Alignment</h3><p>These scenarios highlight the fact that the decision to centralize especially foundational portions of the digital ecosystem is effectively disconnected from the decision to colocate the team members, themselves. The former may serve to enforce consistency and create central points through which network traffic travels or access is managed. The latter is often a combination of financial and cultural factors, such as the cost of office space, sensitivity to the personal wishes of top talent, and deference to Conway’s law.</p><p>However, sometimes, the decisions are, in truth, more arbitrary, perhaps an automatic choice because it’s what the decision maker or strongest voices have found success with in the past, and that they have concluded is universally correct. Or, they may have been sound when they were made, but become stale over time, and effectively kept alive through tradition rather than rationality. Whatever the reason, it may have become an invisible drag on productivity or a hidden risk to resilience or security. If this is the case, it can require a significant effort to better align the corporate environment or architecture to the true needs of the organization.</p><p>Although this might sound like a call for the dreaded “reorg”, any kind of abrupt change is unlikely to produce as quick a fix as might be suggested. Any big changes will undoubtedly induce negative reactions by personnel who have grown comfortable with the existing setup. At the same time, there might be processes that also need to change and that might not be obvious to the champions of the reorg, leading to systemic friction that, without sufficient awareness, could be construed as early failures of the effort, exacerbating any negative perspectives by facially proving their cynicism right. Instead, there are some smaller changes that can start the process more smoothly without introducing extreme, uncomfortable changes that might induce resistance.</p><p>For example, rather than changing entire teams, which is disruptive and fraught with friction, it can be lower risk to start small, by designating team liaisons or ambassadors, team champions, floating or flex team members who work on different teams based on bandwidth or skill needs, or even shared or loaned members who are farmed out from one team to another on a full or part-time basis for a specific amount of time. Team liaisons or ambassadors would be expected to interact with other teams, either directly or through other liaisons/ambassadors. A team champion acts as a spokesperson for their team, promoting the wins and needs of the team. Floating, flex, shared, and loaned team members can spread happenings among the teams with which they interact.</p><p>Perhaps the most important thing is to tie the designation to a mandated — and clearly communicated/published — responsibility, complete with performance expectations and validation. This would likely be in the form of a regular report, an internal blog/newsletter, or some other qualitative deliverable (a quantitative metric may invite gaming or nominalism that does nobody any good). The idea here is that it is A) taken seriously as a responsibility by the individual (not something that will be outright ignored, slowly devalued over time, or forgotten when a “higher priority” crops up), and B) said individual (because this is tied to their continued advancement, etc.) will in theory be an effective canary for when the approach needs to evolve either in terms of expanding the effort, or rethinking it altogether.</p><p>For fully co-located teams that work across decentralized stacks, this can keep the different stacks from making incompatible decisions, drifting further apart than necessary, or reinventing the proverbial wheel by slogging through solutions to difficult problems already experienced by other teams. In the case of a few scattered teams that collectively develop centralized solutions, keeping the news and ideas flowing among them can keep them moving in the same direction more consistently. If the team members are co-located within the teams, but the teams themselves are scattered across different sites, this might require the team member in question to travel or be temporarily relocated in order to perform their new role effectively. If the teams are already largely remote-friendly, there may not be a need for physical displacement.</p><p>Of course, it’s very likely that one or a few individuals embedded within infrastructure teams won’t dramatically change an organization — at least that cannot be counted on, long term, for a variety of reasons — so eventually a larger scale effort is likely to be required. However, in that event, it would be highly advantageous to incorporate the findings and insights from these individuals, rather than ignoring or overlooking the experience they’ve gained and blindly handing down directives from on high. Include them in whatever steering committee is making these cultural and process decisions (or at least have them prepare a report for such a decision-making body). Rather than seeing it as a temporary stopgap (or as an indefinite end state), use the effort as a stepping stone toward a more appropriate configuration of teams better suited to serve the resources and workloads they operate.</p><p>In the end, the goal is to formalize the natural tendency to orient around the communication channels described in Conway’s Law for cloud infrastructure teams within your organization. Taking conscious steps to shepherd the system itself or the team’s location and structure toward a more appropriate alignment will very often win out over attempting to maintain a status quo or imposing a drastic restructuring. The system will emerge one way or another.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d92c64b4cd9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devoops-discourse/mismatching-team-and-system-structures-1d92c64b4cd9">Mismatching Team and System Structures</a> was originally published in <a href="https://medium.com/devoops-discourse">DevOops Discourse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Terminal to Terraform, The Consistently Costly Side of Code]]></title>
            <link>https://medium.com/devoops-discourse/from-terminal-to-terraform-the-consistently-costly-side-of-code-0dd0623741e9?source=rss----d31c962fddc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/0dd0623741e9</guid>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Robert Glenn]]></dc:creator>
            <pubDate>Tue, 13 Oct 2026 04:06:02 GMT</pubDate>
            <atom:updated>2026-10-13T04:06:02.316Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Terminal to Terriform, The Consistently Costly Side of Code</h3><p><em>This article started as a “deleted scene” from my upcoming book </em><a href="https://learning.oreilly.com/library/view/terraform-at-scale/9798341649705/"><em>Terraform at Scale</em></a><em> (O’Reilly, 2026) available for </em><a href="https://www.amazon.com/Terraform-Scale-Practices-Enterprise-Development/dp/B0GKF63FJ4/ref=sr_1_1"><em>pre-order now on Amazon</em></a><em>.</em></p><p>A common refrain today is that “code is cheap”. The truth is, code, by and large, has been relatively cheap for some time now. After all, writing code has been a common task for a recent graduate of university for decades or, more recently, even that of a coding bootcamp (not to belittle such institutions or programs). The much more expensive — yes, in terms of cost rate, but also as a factor of hard-won experience — skills have long been system-level: architecture, system administration, and infrastructure. In today’s world, these disciplines increasingly intersect in a practice that looks deceptively like ordinary programming: infrastructure as code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eTOlsawnFxaGZtyrtPUOFA.png" /><figcaption>Generated with ChatGPT</figcaption></figure><p>Not so long ago as to evade memory, computers were generally thought of as physical things. Digital devices that one typically interacted with directly via physical peripherals like keyboards and monitors. These were tethered to a single computer through cables (gasp!), meaning you needed to be in very close proximity to said computer (swoon!). Remote procedure calls (RPC) and other remote interaction techniques were available since before the web and the secure shell (SSH) in the ’90s made remote access a true option, but even as the world moved workloads onto geographically distributed servers, updating them usually still meant accessing through a localized keyboard and monitor.</p><p>Then came the platforms that would eventually culminate with “the cloud”. In the earliest resemblant incarnations, well before the moniker was widely applied, these were private: companies would build platform layers over existing processes allowing administrators to perform their operations remotely. While this didn’t eliminate the need for physical access in all cases — machines still had to be physically plugged in, physically incorporated into the network, physically switched on, manually configured to incorporate them into the platform, and sometimes even physically restarted — it significantly reduced the need to transport the highly specialized administrators around constantly. Instead, the work could be segmented into centralized operations engineers and distributed resources that maintained the physical portions of platform integrity.</p><p>Unfortunately this was highly nonstandardized and costly, and thus effectively relegated to only the most sophisticated and/or resource-rich organizations. The use of virtualization through the virtual machine greatly improved the situation for the administrators, allowing a single physical computer to host multiple, isolated operating system instances (if desired). Perhaps more importantly, combining virtualization with machine imaging allowed administrators to create standardized templates containing a specific operating system and software that was already installed and configured, from which new instances could be provisioned quickly and consistently. Likewise, the popularization of configuration tools like Puppet and Chef (and the much older CFEngine) allowed entire fleets of computing machinery to be managed in a consistent way from a central platform. The machines were still bought in bulk every few years, but they could be much more easily repurposed to the present needs.</p><p>This all changed with the advent of public cloud service providers (CSPs) such as Amazon Web Services (AWS), whereby a robust cloud platform was offered on an on-demand, usage-oriented basis. Suddenly, compute power became highly commodified: rather than buying computers directly, and thus suffering from the dual challenges of navigating bulk discounting and significant depreciation (both of which requiring foresight and neither of which directly proportional to revenue), companies could rent the power they needed. This turned a significant capital expense largely into an operating expense, which meant much shorter timeframes to forecast and costs that were much more related to existing output (and thus, ideally, revenue).</p><p>While this brave new world made the lives of accounting-adjacent technical roles much easier, it didn’t, on its own, change the way in which the centralized platform administrators (or the fabled <em>sysadmins</em>, the true protagonists of this story) operated. In many cases, they still installed machine images and ran scripts on individual, virtualized machines. They just no longer were subject to long buying cycles or relied on remotely deployed colleagues to ensure the computers were working and available.</p><p>Scripts could fail, machine states could drift, and troubleshooting was extremely time consuming. The simultaneous gravitation toward the agile paradigm of software development, which heavily benefited from multiple (often not-perfectly-identical, by design) copies of a product’s technical configuration, or environments, put even more emphasis on both standardization and (almost paradoxically) customization. They needed a way to configure these environments efficiently and holistically.</p><p>Similar to the circumstances that made tools like Puppet and Chef valuable, these technology shifts created demand for declarative infrastructure as code (IaC). Rather than having multitude circuitous, brittle shell scripts that had to be applied carefully and constantly cleaned up after, there was now a way to build self-consistent specifications that could be applied in one go, and subsequent applications wouldn’t (ideally) undo portions of previous ones or create unwanted duplication. Entire collections of resources could be stood up, configured, and fully prepared for the workloads that would run on top of them. Even when things weren’t totally perfect, and dependent components weren’t well enough connected so as to get it right on the first run, subsequent runs could converge the collection to a desired state. This made keeping components and infrastructure stacks consistent far easier than before.</p><p>Terraform offered a highly attractive approach, combining provider-agnostic, declarative infrastructure provisioning with an agentless execution model. Rather than already needing existing infrastructure just to run it, administrators could create and manage infrastructure from their very own workstations. As an infrastructure stack evolved, they could rehome the state data and operate it consistently from anywhere they choose including a centralized platform or in a distributed fashion, sharing the responsibility across their team. And because it emphasizes idempotent operations, applying the same configuration repeatedly would (with some notable exceptions) converge the infrastructure toward its desired state.</p><p>With the natural extension to multi-cloud — whereby multiple CSPs are involved (for example, as a way to enhance disaster recovery capabilities, or to take advantage of competitive pricing across various offerings) — or hybrid-cloud — where a CSP is paired with private cloud infrastructure, perhaps built on platforms like Red Hat’s OpenStack or VMware’s vCloud — it becomes even more valuable. Although every cloud has similar goals and offerings in theory, in practice there are rarely perfectly symmetric resources and configurations. Context switching is always costly, but at the scale of a multi-environment infrastructure platform that undergirds your mission critical workloads, the stakes are raised significantly. IaC allows your configuration to be written once per CSP and repeated with confidence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dPfOZT_5y2Q12NlDcSoHvA.png" /><figcaption>Generated with ChatGPT</figcaption></figure><p>For better or for worse, and perhaps to the chagrin of many, this turned the venerated sysadmin role into yet another software engineer. However, instead of application software, these administrators now write infrastructure configuration in a domain-specific language (DSL) or a serializable format such as JSON or YAML. But compared to programming languages like C, Java, or Python, the HashiCorp Configuration Language — the DSL developed by HashiCorp (the authors of Terraform and thus the progenitor of OpenTofu) — is declarative, meaning it presents the desired infrastructure state rather than the sequence of operations necessary to achieve it. It does have a syntax, but it allows for considerable flexibility in how configurations are structured and organized. It also relies heavily on default parameter values in the case that explicit values are not specified. This means the exact same result can have wildly different formats and arrangements of code, leaving it largely up to the authors to impose quality and consistency measures in their codebase. Since it’s not uncommon for the responsible parties to come from the world of shell scripting, rather than application development, this can produce many valid dialects that can feel foreign from organization to organization.</p><p>Given the variability and fluidity of Terraform and OpenTofu development, implementing these frameworks is not, on its own, a silver bullet that can solve all infrastructure problems with little effort. Much effort is needed, especially as the solution scales. Oftentimes multiple teams (at maximum effort) still run up against frustration and seemingly impassable walls. Rarely is the answer as simple as “more engineers”, “more dashboards”, or “more automation”, without significantly more thorough analysis and thoughtful, systemic design choices.</p><p>This is also why simply implementing AI-powered coding agents, as everyone is wont to do, does not, itself, uncomplicate the situation. There are still choices to be made, and while there are theoretically perfect, mathematically optimal algorithms — which, especially for well-known solutions, agents are particularly good at invoking — there aren’t universally perfect choices in terms of infrastructure. Infrastructure is still hard, and it’s not cheap, at least not yet.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0dd0623741e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devoops-discourse/from-terminal-to-terraform-the-consistently-costly-side-of-code-0dd0623741e9">From Terminal to Terraform, The Consistently Costly Side of Code</a> was originally published in <a href="https://medium.com/devoops-discourse">DevOops Discourse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Arguably Irreplaceable]]></title>
            <link>https://medium.com/devoops-discourse/arguably-irreplaceable-ef3ec65da28d?source=rss----d31c962fddc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/ef3ec65da28d</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Robert Glenn]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 21:38:14 GMT</pubDate>
            <atom:updated>2026-09-14T21:43:05.696Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ie_9MTjbqxIfzlHQ8ylhg.png" /><figcaption>Generated with ChatGPT</figcaption></figure><p>Most folks in the software industry are nervous these days. It’s understandable: tools like Lovable and Replit can create a full working website or a mobile app in a matter of minutes with only a few descriptive statements. Coding agents like Anthropic’s Claude Code and OpenAI’s Codex can understand an entire repository and make a meaningful feature update or bug fix in the time it takes to pour a cup of coffee.</p><p>Coding agents can also write Terraform really well, producing a full environment for an application quicker than I can type out all the required resources I can think of (and they’ll remember a few of them for me that I forget). It’s almost always very high quality, complete with variable and output definitions. When’s the last time you can say you built all of them on the very first pass? I’d like to say it’s at least a pre-commit or pre-push requirement I make for myself, but I know that I’d be lying. Codex or Claude Code will do it without even explicitly asking for it.</p><p>The configuration looks solid. But it’s overly generic which means that you’re going to run into major issues probably sooner than you expect.</p><p>That’s mainly because unlike a language that produces an executable binary like Python, or one that is interpreted at runtime over and over again in a virtualized environment like JavaScript, Terraform/OpenTofu configuration is applied to a cloud environment, and essentially creates two additional <em>sources of truth</em>: the state that it stores in a configurable backend, and the live, running resources. Any time the configuration is updated, all of these components must be re-evaluated to ensure the results will all align.</p><p>Need a new feature or service for a NodeJS app? Deploy a new version of the workload that you vibe coded and boom, your users/consumers have access to it as soon as it’s live (up to DNS/CDN caching). Who cares what happens to the “old” version? It’s gone (assuming you’re not e.g. A/B testing or something).</p><p>That’s not how Terraform and OpenTofu work. They’re <em>idempotent</em> scripts and they essentially work as applied <em>changesets</em>, not wholesale replacements, meaning they (will try their very best to) leave unchanged resource instances alone, and only add/delete/update the instances that actually require it based on the new version of the configuration. That being said, because they work with Cloud Service Provider resources, they have to respect the rules of those platforms. If changing a particular attribute/field of a running resource instance isn’t allowed, Terraform/OpenTofu will have to destroy and recreate it. This may be fine for a replicated or clustered resource like a compute instance that hosts a web service, but it may be <em>really destructive</em> for a database or certain network components.</p><p>Suppose you vibe coded an RDS database in AWS, yolo-style, that turned out to be unnecessarily large for your needs (maybe it’s purely for reference and is never expected to grow). You can’t directly shrink the size, so you either have to continue to pay for the extra storage, or you have to destroy and recreate it with the appropriate size (making sure you snapshot it beforehand, and are able to account for the downtime). Likewise, if you accidentally created an Aurora database in the default VPC and need to move it to a newly minted custom VPC, you again have to snapshot the database and recreate. Perhaps this doesn’t seem like a big deal (it’s really not, if you know what you’re doing), but if you didn’t know enough before, are you really going to rely on the same coding agent that took you down the wrong path to now get you out of the trouble it led you into?</p><p>Good luck with that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PN7qwfCUyaqTj9ppvN5wjQ.png" /><figcaption>Generated with ChatGPT</figcaption></figure><p>Layer on the fact that every company is different and has different needs, so there is no one perfect configuration that will work for everyone. According to Conway’s Law, your organization is more than just your org chart: it’s communication channels, policies, and rituals many of which are not recorded anywhere, named, or even consciously recognized by the folks that actively engage in them. It could take many iterations to get a serviceable — let alone perfect — setup for your company. And that’s before even getting into lifecycles and jurisdictions, both of which can influence the need for multiple configuration projects (backed by distinct repositories), something that a coding agent is highly unlikely to proactively provide.</p><p>That’s where consultants often thrive: they may specialize in the very factors that are important to determine and that you may never have thought of before. It’s counterintuitive, but sometimes someone who has been embedded in an organization for many years is too close to see all the details (or problems). At the same time, they are the very people who have all the tribal knowledge that a third-party like a consultant simply wouldn’t know. The best engagements and outcomes see this dichotomy as complementary, rather than adversarial.</p><p>No doubt, a coding agent can make the necessary configuration changes to match your organization’s needs, but do you really want to build the infrastructure through trial and error? Or are you willing to wait to apply the configuration only when you know it’s “ready” (how do you really determine that, anyway, and can you even wait that long)?</p><p>What if you run out of tokens midway through?</p><p>What happens if you decide to build it and then later realize you made a mistake and have to make big changes, have to move it, or have to split it up across multiple regions and replicate certain parts? Or what happens if an instance that undergirds something mission critical needs to change and will be recreated? Do you even know what will happen?</p><p>What if it builds something you don’t understand or components you’re not even aware of? Are you sure you really need that resource?</p><p>Maybe your attitude is “well if it’s wrong, I’ll just scrap it and try again”. If it was generated in 2–3 minutes to begin with, what’s the danger in having to recreate it?</p><p>Around 5 years ago I was at a major international client building out their GCP account (they already had an AWS account where a good amount of their workloads were, and I think they even had some data analytics workloads in Azure; IIRC they wanted GCP because they had a ton of Google Analytics data and didn’t want to pay for the exfiltration, which is a common refrain). We were trying to convince them to let us build something before the final architecture was decided so we could validate some of our Terraform scripts. I said something like “it’ll only be temporary, and we’ll tear it down when we build the real thing”. One of the leaders at the client said something prescient to us that I’ll never forget: he said “Don’t tell me it’s temporary. Once things get built, they tend to stick around.” This is something to which many folks can surely relate: that resources, especially so-called “temporary” resources, have a tendency to be forgotten; they can even find a way to become valuable enough to someone with clout and be really hard to kill.</p><p>Fast forward to today, and the point still stands. That means something that a coding agent builds might be difficult or risky to take down. Especially if you don’t know what it is or exactly what it does (or didn’t even realize it was there). Something that appears meaningless might be the pillar to your whole setup. Or it might just be fluff, but it’s so embedded that you can’t cut it out without breaking something around it. Maybe it’s even the secret ingredient, despite seeming out of place and wasteful. Certainly you can query the model to find out what it does or to determine the consequences of removing it, but, again, you have to be aware of it.</p><p>That’s not to say I don’t use coding agents to generate infrastructure as code. I do. I just know when to segment across multiple repositories or wrap components in a module. I know what I need to nail down early versus what can be more ad hoc or transient. I know what questions to ask and I know that sometimes a question has to be posed three or four different ways (or to a few different people) before you get the actual answer.</p><p>Eventually, I will be replaced. It won’t be by an agent that is necessarily any better at coding than it is today, but by an agent that does a more effective job consulting, knowing all possible scenarios, correlations, and contingencies. Even then, I don’t believe we’re going to get to a place where one can simply prompt “build the infrastructure as code for my organization” and a coding agent will just do it, without asking a ton of questions that a single person may not even know all the answers to. Or maybe the agent will just hack all of the organization’s systems and gather this information on its own, as many a TikTok/Twitter doom-posts would suggest. In the meantime, don’t let your coding agent paint you into a corner with a generic configuration, no matter how polished and thorough it appears.</p><p><em>This is why I cofounded Scale The Vibe: to help coding-agent forward teams safely get their products into solid, scalable cloud setups. If you’re ready to take your product to the next level, check us out at </em><a href="https://scalethevibe.com/"><em>https://scalethevibe.com/</em></a><em>. We’re ready to make it happen.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef3ec65da28d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devoops-discourse/arguably-irreplaceable-ef3ec65da28d">Arguably Irreplaceable</a> was originally published in <a href="https://medium.com/devoops-discourse">DevOops Discourse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CAP Conjecture for SaaS]]></title>
            <link>https://medium.com/devoops-discourse/cap-conjecture-for-saas-72e8adfaf94a?source=rss----d31c962fddc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/72e8adfaf94a</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[thought-leadership]]></category>
            <dc:creator><![CDATA[Robert Glenn]]></dc:creator>
            <pubDate>Tue, 04 Mar 2025 19:05:04 GMT</pubDate>
            <atom:updated>2025-03-04T19:05:04.038Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>According to <a href="https://en.wikipedia.org/wiki/CAP_theorem">Wikipedia</a>:</p><blockquote>…the CAP theorem, also named Brewer’s theorem after computer scientist Eric Brewer, states that any distributed data store can provide only two of the following three guarantees: <strong>Consistency</strong> — Every read receives the most recent write or an error… <strong>Availability</strong> — Every request received by a non-failing node in the system must result in a response… <strong>Partition tolerance</strong> — The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes.</blockquote><p>This is a foundational property of all distributed databases and storage solutions, and requires careful evaluation of the trade offs associated with forgoing one of the three guarantees. Anyone who works with cloud storage and data solutions should be aware of this property, and data specialists make tool recommendations based on which guarantees need to be addressed, and how they are addressed. This leads to a variety of implementation details such as shards, clusters, partitions, replications, etc.</p><p>It’s also an easy way to remember a profound idea and a common way to self-identify with other professionals. Most importantly, it is useful to assert a sage-like sense of wisdom over the uninitiated. Why do we impose a short delay on the “real time” sales numbers? CAP theorem. Why don’t we allow simultaneous device login? CAP theorem. Why do we have a single virtual machine in Oregon with nearly two thousand cores and all of our data? Well, the workloads weren’t re-architected for the cloud, they were just lift-and-shift’ed…oh, and CAP theorem.</p><h3>Conjecture</h3><p>I submit there is an analogous balance to be found in Software as a Service (SaaS) products and propose the following conjecture:</p><p><em>Any functional, consistent, and continuing SaaS product can provide only two of the following three guarantees: </em><strong><em>Convenience</em></strong><em> — Operations have minimal friction and maximal usefulness; </em><strong><em>Accessibility</em></strong><em> — Services are essentially unrestricted across localities, hardware platforms, and language/assistance barriers; </em><strong><em>Privacy</em></strong><em> — Minimal sensitive data is recorded and end-user entities have full authority over their own data.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eGIAfKSRs95BbL8PsG8WeQ.png" /><figcaption>Image generated with Midjourney</figcaption></figure><p>I believe this is already generally well understood by many, so I don’t know that this is an especially groundbreaking assertion (assuming this is indeed found to be accurate). It also hinges upon the assumption that such a product is commercially viable (and not some contrived, yet effective, solution graciously gifted to the general population). In other words, I’m not suggesting that the three guarantees have hard, mathematical limitations that are verifiably not possible in totality. However, I do believe by considering SaaS products in these terms, and by respecting this hypothetical property, it simplifies certain technology selection decisions. Before I get to how one would use this property, I will present some (admittedly suggestive) evidence supporting the conjecture.</p><h3>Evidence</h3><h4>Convenience and Accessibility</h4><p>Consider a free service that is globally available across languages and assists users as necessary (Accessible), and that doesn’t regularly make you jump through hoops and moreover provides near seamless access to countless third-party tools (Convenient). How does this service continue to remain available? Who builds this service and how are they compensated for their efforts? In other words, what is the business value of such a service? The answer of course is the value comes from the actual usage of the product: the user activity, and the behavior, beliefs, and tastes inferred from that activity. Thus the billion dollar industry that is targeted digital marketing and the big tech companies’ continuous campaign to entrench everyone and everything into their platform.</p><h4>Accessibility and Privacy</h4><p>Now consider a service with the aforementioned accessibility (free and universally available) and with an approach that explicitly avoids the collection and preservation of sensitive information (Privacy). Certainly, there are a multitude of ways one might approach privacy: from data loss prevention mechanisms to fully air gapped solutions. Almost by design, this has very few integrations by default, considerably deteriorating the convenience of using said service — at least without considerable additional investment of effort or capital which would essentially trade convenience for accessibility and thereby invalidate the premise.</p><h4>Convenience and Privacy</h4><p>Finally, consider a service with the previously described convenience (easy and useful to the extreme) and the privacy as also described previously (effectively air-gapped). It is very wholly improbable that such a solution will be especially accessible to the masses. Most likely, it will have significant overhead associated with it, either through enterprise subscriptions or platform development. Any mitigation of these costs will likely jeopardize convenience or privacy, as vendors attempt to squeeze their quality, reliability, and maintenance budgets to conserve their cost of conversion.</p><h3>Implications</h3><p>If we subscribe to the above, it allows customers in all walks of life, from individual consumers making personal decisions to architects and directors dictating long-lasting and sweeping commitments, to make decisions about technology selection: whichever guarantee that is deemed least important can be forgone with the assurance that it isn’t possible to achieve all three. It’s not a matter of laziness or ignorance, it’s basically a matter of Game Theory: success and self-preservation dictate the “winning” strategies (i.e. remaining profitable).</p><p>It also offers a way to cut through marketing BS that suggests all three aspects are guaranteed, especially for “free” (or at suspiciously economical price point), in a capitalist society. No longer would customers risk being mesmerized by the dazzling showmanship of a sales rep, only later to be dissatisfied after trial, tribulation, and/or fine print. Instead, they would know, up front, what they are willing to sacrifice, and can frame vendor conversations with that in mind.</p><p>It even benefits the SaaS vendors — again, assuming it’s true and, in this case, that it’s well known — as it allows them to focus on whichever two features they do best, and not to have to try and spin their way into contracts that ultimately risk hurting their brand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*iVSg0b3oHhB-vu9sxFH0qw.png" /><figcaption>Image elements generated with Midjourney</figcaption></figure><h3>Confession Time</h3><p>This idea came to me while commenting on another piece. The piece was about privacy and, despite having a dubious premise and conclusion, highlighted some interesting happenings in the consumer technology space. As I stated there, I recognize I am going out on a limb making this claim. I will also reiterate that this is highly dependent on the assumption that there is an expectation of profit (or at least revenue) from such a product, and that I’m in no way suggesting there is some <em>theoretical</em> limitation preventing all three properties from existing in the same product (perhaps the real conjecture would be “CAPP” with the second P as profitability and the limit now three guarantees of the four).</p><p>I also recognize that the argument breaks down to some degree when the properties aren’t taken to the extremes that I used in the examples: there is almost certainly room for <em>some</em> convenience in a product that is both highly accessible and delivers best-in-class privacy. But we do live in a world where every SaaS vendor claims to be “the #1 solution for” their respective problem space, despite pretty much everyone in the industry knowing how numbers work. This is an attempt to pull back the curtain and reveal the false nature of such wizardry, but it admittedly has more entertainment value than it does rigor.</p><h3>Conclusion</h3><p>It’s always valuable to take familiar concepts and map them to new problems to see where there might be parallels. While the analogies rarely match perfectly, they give a quick way to consider intangible concepts and perform the proverbial “thought experiment”. In this case, I obviously mimic the structure of The CAP Theorem, but heavily rely on the mercurial “science” of economics. Nonetheless, I stand by the perspective that this serves as a quick mnemonic for making product assessments.</p><p>Nearly every SaaS product will miss out on at least one of best in class convenience, accessibility, or privacy, at least as these concepts are described (and to the degree at which they are presented) here. Surely there are simple services that could be argued to supply all three, just as there are certainly open source products hosted by non-profits and services donated by exceedingly “generous” (or both optics- and tax-conscious) corporations, but I would contend these exceptions are intentionally excluded. Don’t be fooled next time a vendor tries to sell you a product that purports to deliver all three. Every “no cost add-on” is “no cost” for a reason, and every available privacy setting comes at some price.</p><p>Try the hypothesis out for yourself and see if it holds true for you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72e8adfaf94a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devoops-discourse/cap-conjecture-for-saas-72e8adfaf94a">CAP Conjecture for SaaS</a> was originally published in <a href="https://medium.com/devoops-discourse">DevOops Discourse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mind the Overlap]]></title>
            <link>https://medium.com/devoops-discourse/mind-the-overlap-a863a7c507bc?source=rss----d31c962fddc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/a863a7c507bc</guid>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Robert Glenn]]></dc:creator>
            <pubDate>Mon, 13 Jan 2025 22:22:02 GMT</pubDate>
            <atom:updated>2025-01-13T22:22:42.981Z</atom:updated>
            <content:encoded><![CDATA[<h4>Dealing with Clashing CIDRs</h4><blockquote>Note — I am not by trade a “networking guy”. However, I’ve seen my fair share of network designs. Take this with a grain of salt, but also ignore this at your own risk.</blockquote><p>A free version of this article may be found here:</p><p><a href="https://cidrclash.com/blog/dealing-with-clashing-cidrs">CIDR Clash - Free Online CIDR Range Overlap Calculator</a></p><p>Sometimes, despite all efforts otherwise, it’s not possible to create perfectly segmented, non-overlapping networks (like if you have a need for 18M private IP addresses[1]). Other times, it is deemed convenient to reuse ranges (one thing that I’ve encountered more than once is the notion of reusing ones production network ranges as the dev/int/test/etc. environment ranges[2]). There’s even the strong possibility that there isn’t a single, centralized team making decisions regarding IP allocation. Rather than focusing on “Why it happened?”, we want to understand “How to proceed?”.</p><p>How do network architects approach integrating two private networks with overlapping CIDRs?</p><h3>Network Segmentation</h3><p>Before we go into how to integrate two private networks that have clashing IPs, we should first mention that, ideally, one should probably avoid needing to integrate such private networks in the first place. If you’d like some guidance on how to develop a network architecture that avoids such circumstances, check out the previous post <a href="https://medium.com/devoops-discourse/mitigating-network-overlap-70ef86c366c3">here</a>.</p><h3>Reusing Ranges</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cIic6lXsiACUc24d2gfc-Q.jpeg" /><figcaption>Image generated with Adobe Firefly</figcaption></figure><p>There are several options available and a reasonable architecture for a massive organization may include more than one. One clear and very common component to consider is to use network address translation (NAT; this is also sometimes how we refer to the device that performs this process, or “NAT gateway”, or “NAT appliance”, etc.). NATs are great for one way communication (or, perhaps more accurately, one way initiated communication): when you have a clear downstream/upstream relationship between components. The upstream contacts the downstream through the NAT gateway, and the upstream service’s IP address is translated to the NAT’s own IP address, for use by components on the downstream network. Return traffic is sent back to the NAT, which will translate the destination back to the original upstream address. The catch is that the originating component cannot use the IP address of the downstream component: they must instead leverage the component by host or domain name. An internal DNS solution helps to bridge the gap between two NAT-linked networks.</p><p>If mission critical traffic is expected to be over the public Internet, it is almost certainly necessary to encrypt it. This is where a virtual private network (VPN) becomes an attractive option. For our purposes, a VPN works much like a NAT. DNS would again be included to direct traffic through the VPN. In fact, DNS is a big part of any solution that connects two networks that have common IP addresses to avoid any confusion when attempting to communicate between them. DNS is the real hero of the story[3].</p><p>When two-way communication is needed, it is possible to simply use two NATs or VPNs[4] (and corresponding DNS), one for each network. Generally, I would say most solutions will include either NAT or VPN, or a variant of one of them, and DNS. Even hub-and-spoke or similar solutions that use “network peering” will typically need to include them (and many platforms/providers impose unique ranges across networks that are indirectly connected[5]).</p><h3>Two NICs Are Better Than One</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YVsqcT26hz5hyt-aYrxG7g.jpeg" /><figcaption>Image generated with Adobe Firefly</figcaption></figure><p>However, this does all presuppose the traffic between two networks is the same kind of traffic that the networks themselves were designed to support. Or another way to say it: it assumes the traffic needs to go over the same network interface controller (NIC). This is not always the case.</p><p>Depending on the nature of the traffic with which the two entities on each overlapping network need to communicate A) for their respective primary function, and B) with each other, it may be most prudent to simply use a third network[6], connected to each entity over a second NIC. If, for example, one entity is an application webserver for our corporate website, and the other is a logging sink for access logs (which itself has a frontend user interface, for site reliability engineers to access), then there’s no reason for them to communicate over the same network interface that serves traffic up the chain to end-users’ web browsers. In fact, that very example is likely to impose unwanted throughput constraints that grow proportionally with usage. This is an example of “east-west” traffic, and there is little drawback in architecting one’s network this way (beyond the cost of implementing and the headache the architecture diagrams can induce on those that behold them). However, there are still rules and best practices in introducing these “second NIC networks” (a perhaps obvious one being to not try to use the same IP for both NICs of the same machine), and some care must still be taken.</p><p>So far, I’ve only discussed IPv4, but another approach would be to use IPv6, instead. However, I mention it purely to be as holistic as possible. To be honest, I have never personally touched IPv6, and would defer any novel guidance to anyone else. I would say that this should be functionally equivalent to the “second NIC” approach above. I would also suggest that the continued lack of adoption for IPv6, despite it being considered a “standard” since 2017, may relegate it to very specific cases and thought experiments, until we begin to reach critical mass in the public Internet that we can’t find ways to work around.</p><h3>Public Internet Route</h3><p>A last, perhaps least gratifying, approach I’ll mention is simply exposing one or both networks to the public Internet and communicating as if the two sides were each third-party integrations. While this seems like a lazy approach, I think this is actually more common than we would like to admit. That being said, I don’t think it’s so especially pervasive or commonly systemic as to be a problem; I think it happens here and there and any impact is relatively negligible in many cases. This might be a “reasonable solution” in cases where:</p><ul><li>You already have a robust, secure, and accessible/extendable system for ingress and egress over the public Internet,</li><li>You do not yet have a convenient pattern with which to establish communication between two given, overlapping networks,</li><li>The phenomenon is determined to be “one-off” or to have an extremely low likelihood of recurrence, and</li><li>The traffic is neither sensitive in nature or in timeliness.</li></ul><h3>Conclusion</h3><p>As you can see, there are a few options to consider if it is necessary to integrate networks with colliding CIDRs. The most common components, DNS, NATs, and VPNs, most likely are already included at various points in one’s architecture, so the patterns should already be familiar and available. Other options like secondary NICs can provide performance benefits beyond simply enabling connectivity. When all else fails, there is even the good old public Internet. This commentary is by no means intended to be exhaustive, but merely illustrative of common approaches.</p><p>Check out <a href="https://cidrclash.com/">cidrclash.com</a> today and don’t let clashing networks ruin your day!</p><h3>Footnotes</h3><p>[1] Super common *wink wink*.</p><p>[2] I don’t know that this is inherently an anti-pattern, but it seems to coincide with hard coded addresses somewhere in the stack, which is never ok.</p><p>[3] Border Gateway Protocol (BGP) can also be used internally in conjunction with DNS for very large, complex networks. I will defer guidance on when and how to implement this.</p><p>[4] FWIU some VPNs can provide full two-way communication over the same physical link (where the upstream and downstream can compete for the total throughput of the link), but this is more of an internal implementation detail that is beyond both this discussion and my knowledge.</p><p>[5] DNS often provides a greater role in these solutions, in general, as most platforms/providers do not support transitive routing across network peers.</p><p>[6] Or collection of networks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a863a7c507bc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devoops-discourse/mind-the-overlap-a863a7c507bc">Mind the Overlap</a> was originally published in <a href="https://medium.com/devoops-discourse">DevOops Discourse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mitigating Network Overlap]]></title>
            <link>https://medium.com/devoops-discourse/mitigating-network-overlap-70ef86c366c3?source=rss----d31c962fddc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/70ef86c366c3</guid>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Robert Glenn]]></dc:creator>
            <pubDate>Mon, 13 Jan 2025 22:14:13 GMT</pubDate>
            <atom:updated>2025-01-13T22:14:13.429Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Note — I am not by trade a “networking guy”. However, I’ve seen my fair share of network designs. Take this with a grain of salt, but also ignore this at your own risk.</blockquote><p>A free version of this article may be found here:</p><p><a href="https://cidrclash.com/blog/mitigating-network-overlap">CIDR Clash - Free Online CIDR Range Overlap Calculator</a></p><p>How do network architects develop the private network schema (the CIDRs organized to not overlap nor waste IP space) for an organization of sufficient size? The answer is certainly qualified, especially since organizations and products can make large and rapid changes over time, whereas networks have a certain gravity to them that makes them difficult to modify without the potential of significant impact.</p><h3>Private Networks</h3><p>What are private networks? Generally, they fit into the following predefined CIDR ranges: 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. These can be used by any organization for its internal networks, as these are ignored by network switches in the public Internet. The challenge becomes how to further segment these predefined ranges (and in many cases, how to safely/efficiently reuse segments).</p><p>For more information on private networks, check out the <a href="https://en.wikipedia.org/wiki/Private_network">Wikipedia page</a>.</p><h3>Options for Segmenting</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9qU34QrAWBejqKmDg3Vbdw.jpeg" /><figcaption>Image generated with Adobe Firefly</figcaption></figure><p>One common way to use the three private network blocks is to designate one for each of three major groups of networks. The choice of how to group is often either functional (e.g. the purpose of the resources interfacing with the network such as virtual desktop instances or application virtual machines) or based on more physical aspects such as provider. It’s also not uncommon to segment by geographic region (e.g. 10.0.0.0/8 in AMER, 172.16.0.0/12 in EMEA, and 192.168.0.0/16 in APAC), but I would submit this is a poor choice as the sizes of the respective networks imposes unnecessarily incongruous scaling constraints on the regions; instead, regional segmentation should most likely be done within each private network, likely very near the top (e.g. 10.0.0.0/10 for AMER, 10.64.0.0/10 for EMEA, and 10.128.0.0/10 for APAC<strong>[1]</strong>).</p><p>Functional aspects to consider for network segmentation are departmental silos, specific layers of an architecture stack (e.g. web server, database, etc.), product applications, and customer type. Each of these have different implications depending on where they are in the segmentation for a particular organization. There is, to my knowledge, no perfect schema approach, but given a well understood organization (down to the personnel hierarchy), its products, its resources, and its third-party integrations, the choices where and how to split may become obvious. In other words, network architecture should largely match the organizational structure. This is nothing new or exciting, but it’s much easier said than done.</p><p>Often, the most typical (and perhaps “mathematically optimal”) approach is to attempt to select a segmentation that best balances the segments (like balancing a mathematical tree). If, for example, different department silos have wildly different network needs (i.e. one owns a massive cloud infrastructure, whereas the other manages virtual desktops for a small, embedded subcontractor team), then it may be more prudent to consider a segmentation that divides one or a few larger departments into chunks roughly the same order of magnitude as the smaller departments (or to consolidate the smaller departments, and further subdivide at a subsequent scope, etc.). It’s not necessary to make a fully symmetrical tree out of the networks, as long as the tree is roughly balanced, and the segments within are well utilized<strong>[2]</strong>.</p><h3>Activities That Impact Network Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3QuPf-FZt8Xs89fCD3ilDA.jpeg" /><figcaption>Image generated with Adobe Firefly</figcaption></figure><p>There are many types of business activities that impact an organization’s network architecture. Common examples include mergers &amp; acquisitions, the expansion to new data centers, the choice to build a new cloud environment, almost any kind of migration event, data federation, new products or lines of business (or greenfield replacements thereof), changes to workforce and facilities, changes to certain service providers, and third-party integrations. It is unlikely that a business of a certain size sees none of these activities in a single year.</p><p>Networks evolve over time and even optimal decisions made today may not even be appropriate a decade later. That means the network schema developed to implement the datacenter likely won’t see the end of life for that data center. It’s also nontrivial to decide what even “optimal” means. Is it “optimal” to ensure a tight packing of IP addresses, leaving nothing unallocated within large contiguous ranges? Perhaps for some organizations with highly segmented and extremely expansive network needs. But something like that, which may be obviously important to some organizations, may not be an operational concern for others if the goal is to introduce software abstractions of generic network configurations, rather than perfectly packed IPs. Even within an organization, it may be appropriate to develop a mix of approaches.</p><p>As mentioned above, one common and notable example is the decision to adopt (or as I like to say, occupy) a cloud provider. This could be with the intent to move one’s entire operation to said cloud provider or to enjoy whatever benefits to your existing operation are promised by the provider. Whatever the reason, this could have a significant impact on the virtual networks. Do we need to fully replicate the same network schema we have today, but in the new provider? If so, are we duplicating the network ranges (the CIDRs), or offsetting by a particular maskg.? E.g. if all of our virtual networks for application VMs are found to be contained within the 10.0.0.0/16 range, do we match them in their corresponding ranges in e.g. 10.1.0.0/16 or in 10.128.0.0/9 or even 10.128.0.0/10? The differences may seem arbitrary at the time of selection, but no decision, even with merit, is without its technical implications. This risk must be mitigated as the cost of making network changes after components are “in production” is much greater.</p><p>Another, less common phenomenon is the decision to peer or otherwise connect an internal virtual network with a 3rd-party’s virtual network. This would most typically be done between the same or similar network constructs such as those provided by the same cloud provider or the same co-location service. Without going into the DMZ/Trust-Untrust aspects (which absolutely must be considered for an implementation), there is a risk of the third-party’s network clashing with yours, especially in a heavily segmented hub-and-spoke configuration. Some providers will complain if you try to include a network that overlaps directly, but may not catch transitive peering problems. A common solution preferred by many organizations is to use a dedicated VPN (either directly with the third-party, or between one’s network and a network designated specifically to peer with the third-party; this can also help to address the security concerns glossed over previously).</p><p>Another example mentioned above is that of mergers and acquisitions. These are especially cumbersome, as it is very unlikely that both A) the private network allocations align, and B) there are no networks that overlap between the two firms. This topic is complex and takes many forms; it will be covered in more detail in a subsequent post.</p><h3>Reusing Segments — An Overview</h3><p>Clearly, there may come a time where it’s determined that the most appropriate course of action is to integrate private networks with clashing CIDRs. Regardless of the “why”, we need an answer to “how”.</p><p>There are a few options and a reasonable architecture for a massive organization may include more than one:</p><ul><li><strong>Network Address Translation (NAT)</strong> — “Translates” the outbound traffic from the originating IP to a designated IP of the NAT</li><li><strong>Virtual Private Network (VPN)</strong> — Similar in function to a NAT, but encrypts the traffic</li><li><strong>Internal Domain Name Server (DNS)</strong> — Routes traffic using an alphanumeric domain name to a “next hop” (either the destination itself, or the next DNS in the series)</li><li><strong>Second Network Interface Controller (NIC)</strong> — Essentially creates an alternative channel, (mostly) avoiding clashing CIDRs among the two channels</li><li><strong>Internet Protocol version 6 (IPv6)</strong> — A newer version of the Internet Protocol which has a 128-bit address space, versus IPv4’s 32-bit address space</li></ul><p>This is by no means intended to be exhaustive, but merely a list of common approaches. This area can be a bit dense, so another follow up post will go into further detail on these approaches.</p><h3>Example of Private Network Segmentation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5xK8ai4NYyqcZfFjZeWXiw.jpeg" /><figcaption>Image generated with Adobe Firefly</figcaption></figure><p>All things considered, given a hypothetical “large multinational corporation” under ideal circumstances, a successful network address schema might first group components into internal/hr-related vs. product-oriented components, with the latter further segmented into “primary” networks (those that carry traffic between the functional portions of products and services and to and from the public internet) and “secondary” networks (these can take several forms, including overlay networks for container orchestration or second network interfaces used for “back of the house” traffic such as for logging and monitoring<strong>[3]</strong>).</p><p>After the initial segmentation, it is likely most appropriate to split the application networks into regions and environments (e.g. by dev, test, and prod), depending on how even the markets between the various regions are, and then the other at the next level of segmentation. Next, segmenting by application layer (e.g. web server, database, etc.) may be most prudent<strong>[4]</strong>. Finally, networks can be split by product/application, which seems to be most common.</p><p>An additional recommendation would be to intentionally leverage “second NIC networks” (described above) wherever east-west traffic is required. Logs (including metrics) and automated remote operations are both common types of east-west traffic and are both appropriate use cases for this approach. The networks that support east-west traffic will need to be treated as a completely separate set, and can begin its segmentation from the top.</p><h3>Conclusion</h3><p>Hopefully this appropriately categorizes the challenges faced by network engineers as business needs introduce new complexities. I hope it also successfully illustrates meaningful approaches and technologies for mitigating issues.</p><p>We’ve reviewed typical approaches and patterns with which to split private network ranges to best accommodate growth and scale. We’ve also discussed various common ways in which networks can become overlapped. We then took a quick glance at various ways to integrate private networks in such cases. Finally, we considered a hypothetical network address schema that follows a normal approach.</p><p>Check out <a href="https://cidrclash.com/">cidrclash.com</a> today and don’t let clashing networks sneak up on you!</p><h3>Footnotes</h3><p><strong>[1]</strong> And then perhaps 10.172.0.0/10 for operations teams.</p><p><strong>[2]</strong> I won’t offer a definition of what “well utilized” might mean; however, the sentiment is that the segments aren’t arbitrarily chosen such that many segments are severely underutilized. That being said, forecasting network needs can seem about as nuanced and volatile as the weather or stock market, so it is necessary to employ high quality judgment.</p><p><strong>[3]</strong> We’ve mentioned this as a way to ignore overlaps, but under ideal conditions and perfect knowledge/forecasting (and for the sake of everyone’s sanity), it is best to avoid overlaps.</p><p><strong>[4]</strong> There may also be entire stacks, teams, and departments that get a block of IPs for their purposes, such as for an internally hosted platform.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70ef86c366c3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devoops-discourse/mitigating-network-overlap-70ef86c366c3">Mitigating Network Overlap</a> was originally published in <a href="https://medium.com/devoops-discourse">DevOops Discourse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Network Overlap Problem]]></title>
            <link>https://medium.com/devoops-discourse/the-network-overlap-problem-9363a0db3942?source=rss----d31c962fddc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/9363a0db3942</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[Robert Glenn]]></dc:creator>
            <pubDate>Mon, 11 Mar 2024 00:42:34 GMT</pubDate>
            <atom:updated>2025-01-13T22:25:16.167Z</atom:updated>
            <content:encoded><![CDATA[<h4>Clashing CIDR Ranges In Peering Groups</h4><p>A free version of this article may be found here:</p><p><a href="https://cidrclash.com/blog/the-network-overlap-problem">CIDR Clash - Free Online CIDR Range Overlap Calculator</a></p><h4>The Problem</h4><p>At a previous client, I was on a team tasked with moving all on premises assets (including network) to a popular cloud provider. The specific client team was part of a wholly owned subsidiary of the actual contracted client and the project involved moving the contents of their data center into the parent company cloud. The network ranges were extremely segmented and there was little obvious rhyme or reason for address packing or range banding. During the project, we found that we had to change some ranges to consolidate and we had to be careful to not overlap with the parent company’s ranges at common interfaces. Trying to change as few (in many cases, hardcoded) IPs as possible.</p><p>A problem that plagued us for some time was something like the following scenario:</p><ul><li>We would identify a new range we needed that was atypically large or small as compared to the network ranges they maintained in their data center (and thus not simply a contiguous selection from a well-known subnet size, raising the risk of the new range falling within or including an existing range),</li><li>A member of the client team would come up with a new range that, unbeknownst to them, collided with some other range,</li><li>We unwittingly encoded this new range expecting it to be unallocated across existing ranges,</li><li>We ran the plans (we were using terraform) and nothing was reported to be wrong,</li><li>Then we ran the apply jobs, and discovered, much to our chagrin, the overlaps in network ranges within a peering group.</li></ul><p>Without it being a feature of some tool we were using, how could we ensure there will be no overlapping ranges across peering groups (which were already pushing the default limits in terms of cardinality of included subnets)? In the absence of a simple solution handed to us, and due to a sequence of events and circumstances that could only be referred to as a ‘comedy of errors’ (albeit in no way humorous), it was a protracted and frustrating experience for all involved.</p><p>More generally, there are many ways an organization can find themselves tangled in so many network ranges that it becomes tedious to track and maintain. Decentralized teams managing product-specific network components, or disparate networks becoming joined together through mergers and acquisitions all are subject to this risk. How do we easily determine which ranges overlap?</p><p>While the story at the opening is a rather complicated example, and perhaps seems almost contrived, given the upswell of acquisition activity over the last couple of decades — and not always including a “world-class, ‘leading-edge’ digital infrastructure”, a simple proof of which might be established combining basic statistics (e.g. normal distributions) with basic game theory (e.g. the concept of “zero-sum”) — it is surely not an isolated incident.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Uo5NKdGpb4GZSjvb8wKmg.png" /><figcaption>Image generated with Midjourney</figcaption></figure><h4>Common Scenarios</h4><p>In any acquisition or merger there is a danger of overlap in internal/private subnetworks. Sometimes, with acquisitions, the proud parent company may decide to swaddle their newbought with promises of status quo: phrases like “no changes”, “operations as usual”, “same game, just a different coach” are thrown around with big, sincere grins. At some point, however, nearly all of them find themselves dealing with a fussy child when they break that promise and implement the dreaded change management (aka organizational shakeup), which nearly always includes changes to <em>someone’s</em> technology standards — naming conventions, development tools, and certain infrastructure — to ensure everyone fits into the new home.</p><p>The likelihood of a more massive (in terms of people, processes, policies, and technologies/licenses) parent company being the one to make widespread changes approaches zero, regardless of who’s strategy is “right”. In terms of infrastructure, the networks are among the most likely to be affected. What’s the likelihood that an environment at two different companies uses something utilizing the 10.0.0.0/24 range? I’d say likely enough not to want to bet against it.</p><p>In the case of a merger, especially among comparable sized companies, the impact of collisions may be even more acute. Large companies sometimes use the <a href="https://en.wikipedia.org/wiki/Private_network">RFC 1918 ranges</a> in strategic ways (e.g. to segment networks intended for internal vs. external applications), so there’s more chances of overlap (or even mismatching schemas). Certainly, there are some matches where one side brings the superior solution and infrastructure in some capability space, and the question of who aligns to whom is at least clear (sore feelings notwithstanding). In many other cases, the determination is not so straightforward, and both sides claim to strategic superiority may have some merit.</p><p>Finally, it is not uncommon for a centralized team to manage the network boundary and allow product teams to manage their own internal network structure. This is common with very large organizations or highly segmented services and product lines. The idea being an architecture in which the boundary networks and their interfaces are managed by a single team and are designed to interface with product-specific environment networks managed by the product teams themselves. Even in hub and spoke scenarios, care must be taken either in hub architecture or spoke configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BsDkSiv0TXan8l4eAjyOFg.jpeg" /><figcaption>Image generated with Midjourney (and edited)</figcaption></figure><h4>On Premises vs. The Cloud</h4><p>On premises, there is typically a single team that manages issuing the IPs and ranges. Even in the case of a self-service solution for requesting network ranges from blocks of larger pre-allocated ranges, someone at the organization had to pre-allocate those larger blocks. Once a range is carved out of the block, the expectation is that the system issuing the range would know what ranges are available and only allow them.</p><p>In the cloud, it’s the provider and — under the shared responsibility model, in this case the responsibility is to choose non overlapping ranges — consumer’s joint responsibility to maintain subnetworks. The cloud provider doesn’t have a way of knowing the expectations behind a particular subnet (Is this for an isolated sandbox? Part of a hub/spoke configuration? Can we leverage NATing to reuse ranges?), and for its own business to work, it must support any range duplicated by any customer. It is therefore up to the cloud service consumer to manage the non-overlapping (or otherwise) for their networks.</p><p>Moreover, however #theFutureIsNow we seem to be there’s still an awful lot of weight carried by spreadsheets when it comes to planning, tracking, and reporting network ranges. The more subnets, the more chance of collision, and more effort/frustration dedicated to determining where collisions may occur. This can be a time-consuming exercise if there is a battery of bureaucratic processes between determining a new network IP range and actually realizing an errant overlap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5dPppITU43YImgMZILe2UQ.png" /><figcaption>Image generated with Midjourney</figcaption></figure><h4>The “Solution”</h4><p>“Ok ok, Robbie I get it, this is a super-dee-duper important problem. So how do we solve it?” Well, just like any ‘senior engineer’ I skipped my way over to my preferred search engine and asked the internet, of course. I regularly use CIDR-to-IP conversion tools, so I was expecting there to be <em>something</em> readily available that fixed all of my problems. Instead, the most meaningful results were Stack Overflow answers with snippets of Python code and meandering Reddit threads that occasionally spiraled into a coherent resolution.</p><p>I cut my professional teeth on JavaScript so I’m more apt to use such code as a guide than to copy and paste directly, which is what I did: I created a module in TypeScript (don’t judge me if that’s uncool these days) and tested it in a node shell. Now I had a way to quickly print out colliding network ranges that were written in CIDR notation! But what about the ‘next guy’? What about the countless others, drowning in a sea of /26s and /18s, for god knows why? What about altruism?</p><h4>Check Out CIDR Clash Website</h4><p>In the case you don’t inhabit shells and text editors, but still want to benefit from a solution like this (or, I don’t know maybe you’re not one of those sickos who codes for pleasure despite a 40+ hour a week software job), head on over to <a href="https://cidrclash.com/">cidrclash.com</a> to check out a website hosted solution. The initial version takes as input a comma delimited list of IPv4 IP ranges in <a href="https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing">CIDR</a> notation and prints the collisions. It can obviously use some additional features and validation/flexibility, but it’s functional and (hopefully) relatively straightforward to use. I’ll follow up with a separate post that goes into more detail on the experience of building the website and the site itself.</p><p>In the meantime, good luck out there. Be wary out there in the digital wilderness!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9363a0db3942" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devoops-discourse/the-network-overlap-problem-9363a0db3942">The Network Overlap Problem</a> was originally published in <a href="https://medium.com/devoops-discourse">DevOops Discourse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform Patterns, Observed — Part 5: Logic in Terraform]]></title>
            <link>https://medium.com/devoops-discourse/terraform-patterns-observed-part-5-logic-in-terraform-a0b1207b3a56?source=rss----d31c962fddc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/a0b1207b3a56</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Robert Glenn]]></dc:creator>
            <pubDate>Wed, 18 Oct 2023 07:26:58 GMT</pubDate>
            <atom:updated>2023-10-18T07:26:58.729Z</atom:updated>
            <content:encoded><![CDATA[<h3>Terraform Patterns, Observed</h3><h4>Part 5: Logic in Terraform</h4><blockquote>Note — This is purely my perspective as a practitioner with firsthand visibility into several working solutions in my career as a software consultant. Much of the vocabulary used in this series is of my own imagination and will surely cede to better nomenclature from the community. Moreover, many implementations I have seen in practice include multiple types of patterns discussed in this series.</blockquote><blockquote>Note — This commentary is neither definitive nor endorsed and is in no way representative of the views of my employer, Accenture.</blockquote><p>While Terraform is primarily a configuration language that abstracts the APIs of another platform, there are logic expressions available in Terraform allowing developers to introduce inferences and abstractions, improving the flexibility (or general accessibility) of a solution, without introducing repetitive code or requiring deep technical knowledge on behalf of a module’s consumer. However, the tendency to see each of the world’s problems as a nail once one’s hammer is mastered can lead a team into a quagmire of complexity. On the other side, a draconian forbiddance of Terraform expressions can force teams into developing unreasonably large and repetitive codebase or splitting a team’s overall state into a vast wilderness of Terraform <em>state trees</em>.</p><p>In this post, we will discuss the most likely places for logic to appear, the major contributors to code complexity and developer confusion, and some gentle suggestions on how one might address or approach the spread of logic. We’ll wrap up with some general guidance and recommendations. It is worth reiterating that this is one person’s (severely non-exhaustive) perspective, and may be the most heretical or misguided post so far in the series. As such, I warmly welcome additional and dissenting viewpoints.</p><h3>Common Contexts of Logic</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z-MWSdKBXCchy8MQCIqlMw.png" /><figcaption>Image generated with Midjourney</figcaption></figure><h4><em>Field Generation</em></h4><p>This is some of the most common logic found in Terraform code. The most effective (and least problematic) field generation is string building via interpolation or map building over lists of input or output. The former is introduced to simplify the input fields of component blocks (i.e. named resources or modules), allowing for a greater number of fields to be constructed from fewer variables to be set by operators. This might be common for an organization that requires standardization around naming conventions and resource tagging[1] such as a repeated application or business codename that must appear as part of the standardized name of several managed resources, in their network and resource tags, or in the descriptions thereof.</p><p>Generating a map may be utilized to generate complex constructs from simpler module inputs and even combine them with the outputs of resources or <em>remote modules</em>, allowing an operator to keep input variables human-readable and the fields of modules or resources Terraform-readable.</p><p>Perhaps the most acceptable use of string interpolation is to clean up or directly manipulate the outputs of resources or <em>3rd-party remote modules</em> as it is not always immediately possible to address otherwise. It is often cleaner to keep this logic in a locals block (see next subsection).</p><h4><em>Local Variables</em></h4><p>It seems most common to have one locals block per module. If all component blocks are in main.tf (and the locals block isn’t more than 15–20 lines[2]), these can be too. Otherwise, put in its own locals.tf file. It is strongly recommended to name your locals well; consider “self documenting code” where the names tell what relationship or purpose the generated variable addresses. As discussed above, the locals block is the most appropriate place for field generation including restructuring the outputs of resources or <em>3rd-party remote modules</em>.</p><h4><em>Externalized Logic</em></h4><p>Terraform will most likely be embedded in your automation tool’s DSL, even if it amounts to yaml and freetext bash scripts. This unlocks the breadth of tools (or lack thereof) available to the execution context. In many popular platforms, these execution contexts are already containerized and even allow for custom container images to be leveraged for a job’s execution. This opens a world of possibilities to incorporate tools other than Terraform to perform operations on inputs and outputs. This may get messy if one leverages inline sed/awk commands or pipes, which brings me to the next point.</p><p>Consider using a friendlier scripting language like Python or Javascript if you have a lot of fancy logic to expand any inputs or analyze any outputs with code. These can be tested and stepped through with a debugging tool in ways Terraform cannot. This can even back a user interface or API, turning the system into fully self-service and event driven, perhaps negating the need for an automation tool at all.</p><h3>Expressions and Functions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O_akkAzVsi7Xofs4BA9J6A.png" /><figcaption>Image generated with Midjourney</figcaption></figure><h4><em>The for_each Expression</em></h4><p>Definitely always use for_each with a map and with thoughtful, fully deterministic keys to ensure the state stays well identified and does not rely on a strict order. Ideally, these expressions are used sparingly and don’t show up in too many places in a given hierarchy[3].</p><p>Using a for_each over a consistently variable set of the same component block is certainly superior to trying to duplicate full component blocks, especially as you manage dozens or hundreds of components in this way. However, if you know you will only ever have a small (2–3), set number of resources (e.g. an active/passive setup for a particular tool to stay available during outages or help recover from disaster), it may not be worth the loss of clarity and extra syntax. Also, it can be destructive or cumbersome to move from one paradigm to another, as it does affect the full underlying <em>state tree</em>[4]. This is explored more below in the <strong><em>Conjectures &amp; Conclusions</em></strong> section.</p><h4><em>The </em>count<em> Expression</em></h4><p>Pretty much never use count except as a way to conditionally include a block. The most common usage is to use a ternary statement inline within the component block.</p><p>The result will be either the presence of a list with a single element or nothing, so the output of a resource conditioned thusly requires the use of the list index notation. Referencing this output will also typically require the use of another ternary or the splat expression, leaking complexity into other parts of the codebase.</p><p>Moving from a count’ed component block to a for_each’ed component block (or one with neither) also suffers from the destructive/cumbersome attribute as before.</p><h4><em>Regarding dynamic Blocks</em></h4><p>These are also used along with a for_each or count statement. When resource fields are expressed as blocks, and when multiple blocks are allowed and required such that there is a high degree of variability in their use (examples may include disk attachments, network interfaces, or policy statements) and especially when the data to be passed to the block fields come from the outputs of resources or modules, it may be most prudent to leverage dynamic blocks. However, these should be used sparingly and never included in order to “future proof” (i.e. over-engineer) a module.</p><p>Furthermore, unless developing <em>3rd-party modules</em> (or intending to be used as such), it may be best to avoid nesting dynamic blocks. This introduces a level of complexity most likely better served by parallel dynamic blocks, even if this introduces some (or significant) code repetition or other logic types (e.g. using count as a conditional, as described below).</p><h4><em>The for/in(/if) Expression</em></h4><p>Although this may be common inline with e.g. for_each expressions, it is perhaps cleanest to put them all in a locals block. It is also recommended to break into individual lines and not to nest for/in expressions. Create a named local variable for each expression, rather than putting it all in one compound expression. The result of each expression could then be used in outputs for better traceability when desired.</p><p>For non-primitive lists, consider restricting the usage of for/in purely to casting the list to a map with fully deterministic keys. This allows for a later for/in or for_each expression to act over a map, resulting in a more predictable and deterministic (and therefore more robust) behavior.</p><p>The if statement can improve code readability when iterating over a collection for different reasons and when items are conditionally considered (e.g. in the case of a list of virtual machines that optionally have an attached disk, where the module creates both instances and disks): rather than looping over the same list, create multiple lists with suggestive names that include the condition as an if statement.</p><h4><em>Complex Variable Type Functions</em></h4><p>These are used in variable definitions to define the module’s input variables that are not of type string, number, or bool (boolean). The most common non-primitive type constructors are list(), set(), map(), and object()[5]. Defining these is strongly preferred to leveraging the any type (or not including a type at all) but object definitions in particular can get complex, especially when including a default value.</p><p>The presence of these functions follow the structure of the module code that operates on the input variable: a count or for_each statement requires a list , set , or map type. Field values of both map and object types can be accessed using the bracket notation on the field key a la many common programming languages.</p><h4><em>The length() Function</em></h4><p>This can be used on strings or collection types. For the former, length() is common in conjunction with the substr() function, to ensure strings adhere to maximum length guidelines imposed by a particular platform or policy.</p><p>For the latter, length() may be appropriate in the context of providing more or richer details in output definitions. Introducing more complex logic based on length()[6] is perhaps as ill-advised, avoidable, and awkward as it seems to be rare.</p><h4><em>String Matching Functions</em></h4><p>The strcontains() function produces a boolean result based on the presence of a substring. This limited behavior results in limited benefit, and may be better served with an additional (if repetitive) input variable.</p><p>The related regex()/regexall() functions each return a list of matching substrings. These allow for richer matching, but are unlikely to be directly actionable, requiring additional expressions to make use of the outputs. This too may be better served with abstracting the logic as additional input variables or with externalized tooling.</p><h4><em>The chomp() Function and the trim Functions</em></h4><p>Ideally, the strings the code is operating upon is clean enough to use directly. The behavior of the chomp() and the various functions related to trim() in Terraform is cumbersome to test as a unit and very likely does not perform any better than other common tools (see section above on externalized logic). Then again, there’s something to be said for “defense in depth”, so even the use of external tools to clean up text input does not necessarily preclude the use of the trim functions[7].</p><h4><em>Other String Manipulation Functions</em></h4><p>It is most likely best to avoid complex string manipulation like nesting split(), replace(), join(), etc., except perhaps in cases in which one must manipulate output values to make proper use of them. Because Terraform functions are purely functional (rather than e.g. object oriented), chaining commands can make for less readable code (see additional points on functional behavior below). There is support for multi-line strings using a “heredoc” syntax, but this should be a rare if not overly contrived occurrence.</p><h4><em>Directives</em></h4><p>It appears the natural inclination is to avoid directives, especially in root modules or any modules that change often. A more common and perhaps accessible approach is to use a conditional expression or a for/in expression (see above), even though this may make for more characters in the code file.</p><p>There is the ability to introduce whitespace stripping with directives, which might not be well handled by another method. There may also be sound rationale to introduce directives or the “heredoc” syntax in the case of “workflow driven IaC automation” (in which some ticketing system passes in the parameters that were input into a web form by a user or other automated system). However, I would strongly recommend cleansing the textual variable inputs <em>before</em> one feeds them to Terraform with a more commonly used tool (as described in the section above on externalized logic, in this case e.g. sed/awk, Jinja, etc.).</p><h3>Conjectures &amp; Conclusions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qCS-GLKHOXuv9gHk3V0S2g.png" /><figcaption>Image generated with Midjourney</figcaption></figure><h4><em>Functional Programming</em></h4><p>If you are familiar with functional languages like Lisp/Scheme/etc. or Haskell, it shouldn’t be too foreign of a concept to reason through function calls in Terraform. If you’re more familiar with the much more limited approach of many object oriented languages[8] to “lambda functions”, whereby pure functions are passed as arguments to specialized methods of collection types that iterate over the collection, sequentially calling said function on each iteration, the nesting behavior of inline function chaining in the functional programming paradigm may be both awkward and difficult to reason through. As such, it is strongly recommended that chained function calls be introduced as local variables (see the commentary above on <strong><em>Field Generation</em></strong> and <strong><em>Local Variables</em></strong>).</p><p>If your project is rife with function calls, I would recommend reading through a book on the subject, such as The Little Schemer (it’s worth the read even if your project has few functions). However, it may be most appropriate to reduce rather than endorse the use of functions.</p><h4><em>Looping &amp; Conditionals</em></h4><p>It is best to fully consider the use of for_each versus multiple named (or even conditioned using count) modules/resources, especially with resource instances that require extra care in the event of any interruption, as changing from one paradigm to another is often destructive (i.e. the resource will be deleted from one location of the <em>state tree</em> and recreated elsewhere) or requires manipulation of the Terraform <em>state object</em>, directly, to align the state with the resulting shape expressed by the new code.</p><p>Conditionals (e.g. using the ternary operator as part of an expression) can be leveraged as a way to provide runtime validation around fields, ensuring the values set to the fields are appropriate types. Excessive use of this is mostly a traditional pattern, as earlier versions of Terraform didn’t include more appropriate validation patterns or a good way to control emptiness, e.g. the validation and the nullable fields of variable definitions, the try() and can() functions, or the optional modifier. Unfortunately, a significant number of fairly substantial entities haven’t fully progressed beyond these earlier versions (either literally or developmentally/culturally), and while this pattern is rather syntactically expensive and unsightly, it still appears regularly.</p><h4><em>Complex Variable Types, A Slight Return</em></h4><p>Leveraging different complex types at multiple places in a <em>state hierarchy</em> (rather than passing along the same constructs along it) may produce a kind of telescoping effect: the parent module of a child with an object type variable may be inclined to embed this object type as a subfield of its own object type variable. However, the natural opposite of this–passing along less complex objects–may have an equally loathsome result of complex logic all along the hierarchy to generate complex local variables that skirt any semblance of a type check and don’t exactly simplify the codebase.</p><p>Perhaps the best way to broker a balance between the two unfortunate extremes is keeping whatever modules consuming–i.e. decomposing into it’s fields and using them entirely (and most importantly not repackaging them into some other object to be passed further downstream)–the complex variables close to the active module, at least ensuring low repetition.</p><h3>Next Time…</h3><p>We’ll discuss style and convention in Terraform. As this is perhaps most likely to ignite an emotionally spirited debate (see Tabs v. Spaces), I will attempt to generally revert back to observation and avoid over-editorializing.</p><h3>Footnotes</h3><p>[1] For cost attribution and/or as the subject of a network configuration.</p><p>[2] Your mileage may vary.</p><p>[3] Quantifying “too many” left as an exercise for the reader.</p><p>[4] Not the somewhat logical one we’ve considered up to now that hadn’t really considered the difference a for_each or count statement makes to a component.</p><p>[5] There is also tuple(), but the type tuple seems to be identical to the type list.</p><p>[6] — A poorly contrived example of which may include triggering a quota increase request when a certain threshold of inputs or outputs is reached.</p><p>[7] Perhaps the point is other tools may be more reliable than Terraform for cleaning up strings if only one tool is to be relied upon.</p><p>[8] At least c.a. the last time I professionally developed such software.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0b1207b3a56" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devoops-discourse/terraform-patterns-observed-part-5-logic-in-terraform-a0b1207b3a56">Terraform Patterns, Observed — Part 5: Logic in Terraform</a> was originally published in <a href="https://medium.com/devoops-discourse">DevOops Discourse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform Patterns, Observed — Part 4: State Management]]></title>
            <link>https://medium.com/devoops-discourse/terraform-patterns-observed-part-4-state-management-dccec970f554?source=rss----d31c962fddc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/dccec970f554</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Robert Glenn]]></dc:creator>
            <pubDate>Sun, 17 Sep 2023 05:17:08 GMT</pubDate>
            <atom:updated>2023-09-17T05:17:08.225Z</atom:updated>
            <content:encoded><![CDATA[<h3>Terraform Patterns, Observed</h3><h4>Part 4: State Management</h4><blockquote>Note — This is purely my perspective as a practitioner with firsthand visibility into several working solutions in my career as a software consultant. Much of the vocabulary used in this series is of my own imagination and will surely cede to better nomenclature from the community. Moreover, many implementations I have seen in practice include multiple types of patterns discussed in this series.</blockquote><blockquote>Note — This commentary is neither definitive nor endorsed and is in no way representative of the views of my employer, Accenture.</blockquote><p>As presented in the previous post, Infrastructure as Code has 3 major components (which I refer to as the <em>state triplet</em>) that comprise a known <em>state</em>: the codebase, the live, instantiated resources (or <em>state domain</em>), and the recorded <em>state object</em> (stored in a Terraform state file).</p><p>In this post, we will begin by establishing the common characteristics and constraints regarding state management, including how to approach the shape of the <em>state tree</em>. Then, we will consider how different approaches to module arrangement (see <a href="https://medium.com/devoops-discourse/terraform-observed-part-2-module-arrangement-109d2cf517e1">Part 2</a>) affect the complexity or cardinality of the state. Finally, I will present some additional thoughts for consideration.</p><h3>State Management</h3><p>Some of these insights may be quite intuitive, whereas others may seem subtle and perhaps uninteresting. Unfortunately, it tends to be the latter that lays dormant until it is far too late to address, economically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cdEBZpBPvUUSMg1U6tya9Q.png" /><figcaption>Image generated with Midjourney</figcaption></figure><h4><em>State and Input</em></h4><p>A Terraform <em>state tree</em> is constrained by the active module that it was built from: the active module, and the local and remote modules that it references, determine the shape of the <em>state tree</em> up to e.g. branch cardinality of blocks including the <em>for_each</em> or <em>count</em> meta-arguments. For better or worse, these latter cases are common, and when they are randomly dispersed throughout a <em>state tree</em>, they can make things unwieldy both conceptually and technically when issues arise.</p><p>In this way, the power of Terraform can be double-edged: abstraction allows for consistency and efficiency, while also inviting complexity, over-engineering, and spaghetti code. And this manifests in the <em>state tree</em>: maybe it’s okay to iterate over 100 configuration objects in a leaf module if one does it one time, each <em>terraform apply</em>, but what if that <em>leaf node</em> is being fed the exact same inputs in some other, non-<em>leaf node</em>? This creates costly redundancy and while it sounds contrived, it is based on a true story, so to speak[1].</p><p>Once an appropriate level of abstraction is reached and proven useful for a particular use case, it could be leveraged as an accelerator (sometimes directly) for other such instances of the use case. Although different inputs against the same backend will result in updates to that backend and <em>state domain</em>, a distinct set of inputs (e.g. in a separate .tfvars file) targeting a unique state file backend location will result in a new, full replica of the <em>state domain</em>[2], allowing for solutions to scale out to additional teams and products. Be sure to avoid the dangers of both overfitting and over-engineering. It may be better to fork and shape as necessary (again, targeting a unique state file backend location).</p><h4><em>State Size and Shape</em></h4><p>The blast radius of a broken or locked state is the entire <em>state tree</em>. All resources managed by a single state share a single lifecycle. Consider using a domain-driven approach to chunk states to a reasonable size and level of atomicity.</p><p>Big states certainly take longer to traverse than small states[3]. Logically, we also want a high ratio of <em>leaf nodes</em> to <em>state nodes</em> (or e.g. resource blocks to <em>module</em> blocks as they are proportional) so we’re spending memory and state data efficiently on running resources rather than on organizational metadata.</p><p>Perhaps unintuitively, this balance can also affect code volume as separate states may introduce data blocks in the case where one state’s resources reference a resource maintained by the other state. These references are very useful to ensure the dependencies exist, but they also duplicate (at least) every relevant code change. When the data references are hot, or change often, this can be cumbersome.</p><p>On the other hand, the over-presence of DAGs in multiple places along a <em>state hierarchy</em> implies either a bloated state or too deep of a nesting and would be better served by separating the different module references into distinct <em>state trees</em>. The balance between many hot data references and a bloated state is certainly a key (albeit more qualitative) indicator of IaC maintainability.</p><h4><em>State Relocation and Propagation</em></h4><p>When it is determined a state must be relocated or segmented into multiple states, there are a number of options, with some including direct edits to the state components like the state’s backend location, or even the <em>state tree</em> itself. These operations can be destructive (or frustratingly difficult to disentangle) if performed haphazardly, so they should be a late if not last resort. Ideally, one would just wholesale recreate the resource in the new state, altogether, and destroy the resource in the old state (possibly in reverse order to prevent specific collisions), rather than manipulating a resource’s remote state directly. However, as this isn’t always an acceptable approach, a serviceable approach would be to import the live resource into the new remote state and prune the <em>state tree</em> using <em>terraform state rm</em>, or in the case of wholesale relocation, one can use <em>terraform init — migrate-state</em>[4]. Neither method is atomic[5], which augments the risk of any mistake, so this must be done with extreme caution and care.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9dCuOgH5kzEKPbawwzSGVw.png" /><figcaption>Image generated with Midjourney</figcaption></figure><h3>Module Arrangement and State</h3><h4><em>Flat</em></h4><p>Each individual <em>state tree</em> is small, or at least very short (it’s essentially all <em>leaf nodes</em>). Likely dozens if not hundreds of <em>state trees</em>. Possibly many <em>state trees</em> served from the same codebase with unique variable assignments (e.g. by supplying mutually exclusive .tfvars configuration files to the execution of <em>terraform plan</em> as described above). The unique state objects will need to be carefully organized to avoid confusion or state collision.</p><h4><em>1-Level Remote Nesting (1LRN)</em></h4><p>This typically produces a relatively comfortable state management balance amongst the size and shape of the <em>state tree</em>(s), the time the state is under lock, and the blast radius of any broken <em>state triplet</em>. The biggest risk might be in over-segmenting states to the point of breaking apart logical groupings, unnecessarily injecting an order of operations that Terraform was designed to address in the first place[6]. More on this below.</p><h4><em>N-Level Remote Nesting (NLRN)</em></h4><p>This also has a high potential to produce a relatively comfortable state management balance just as in 1LRN. However, whereas 1LRN risks erring on the side of too many small state objects, NLRN might err on the side of too large and unwieldy state objects. It is the most challenging to optimize and is certainly the most overhead. However, as it is essentially the de facto pattern when leveraging many open-source community modules, the overhead may be offloaded. No matter how many remote modules (and development teams thereof) are involved in an active module’s execution, the state object itself should be well understood by the active module’s operators and should be shaped in such a way as to balance breadth, depth, and interdependencies.</p><h4><em>1-Level Local Nesting (1LLN)</em></h4><p>In theory, because the state tree is shallow[7], the memory Terraform will need to track the state should not become unmanageable, though the execution of an applied plan could still be unreasonably time-consuming if an entire <em>IaC Ecosystem</em> is served by one <em>state tree</em>. Also, because these modules are often tightly fit to the single-use case for which they were initially developed, a single state object is generally served from the module’s codebase (rather than servicing multiple state objects by e.g. supplying mutually exclusive .tfvars configuration files to the execution of <em>terraform plan</em>).</p><h4><em>N-Level Local Nesting (NLLN)</em></h4><p>This combines the dangers of both previous approaches. Very similar to <em>NLRN</em>, this might err on the side of large/ unwieldy <em>state trees</em>. This may also result in massive <em>state trees</em>, in general, if attempting to manage a single <em>IaC Ecosystem</em> from one <em>tree</em>. In any case, the <em>state tree</em> itself should be well understood by the module’s developers and should also be properly balanced/shaped.</p><h4><em>Mixed Approach — Local &amp; Remote Modules</em></h4><p>As before in <a href="https://medium.com/devoops-discourse/terraform-observed-part-2-module-arrangement-109d2cf517e1">Part 2</a> it is probably most useful to consider the costs and benefits of each individual approach, and to be deliberate (rather than aimless) in targeting a mixed approach. Because this is highly likely to include <em>3rd-party modules</em> at various depths, consider regularly considering/reviewing the <em>state trees</em> that are created by the <em>active module</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TvWYb3A6j0uPXC1ehW3uuw.png" /><figcaption>Image generated with Midjourney</figcaption></figure><h3>Additional Thoughts</h3><p>I think it’s fairly obvious that the state itself is the last of the <em>state triplet</em> to be seriously considered. The live, instantiated resources (those comprising the <em>state domain</em>) are certainly the first consideration given that we’re employed to deliver marketable utility. It is natural for software-oriented technologists to next consider the software interface with which we are oftentimes most familiar: code and scripts.</p><p>It’s unfortunately not uncommon to get all the way to productionland and find yourself in a mire of sub-optimal/terrible state management and, to quote David Byrne, “you may ask yourself ‘how did I get here?’”. The Terraform state is (thankfully) editable, for the very brave, so this won’t burn down your… never mind, what I mean is you can (probably) get out of whatever jam you get into with your Terraform state. However, it’s certainly <em>not</em> something one wants to have to learn (or shop for) when productionland is already in peril.</p><h4><em>State and Order of Operations</em></h4><p>It is most likely that the <em>IaC Ecosystem</em> is not a single <em>state tree</em>. It’s equally likely that at least two of the discrete states have some dependency or interdependency. The former is almost unavoidable in large enough <em>IaC Ecosystems</em> and happens e.g. when you have one <em>state tree</em> servicing compute resources and another <em>tree</em> servicing the networking components. So long as the references aren’t too hot, these should be manageable.</p><p>The latter is where problems more typically (and persistently/chronically) arise. When two states are interdependent — i.e. either the states’ dependencies are bidirectional[8] or the dependencies create a cycle[9] — a change can require an awkward level of coordination and overhead such as several interstitial, incremental updates to tear down, unclad/unlock[10], or otherwise prepare one or both states for certain, should-be-atomic changes to each respective state[11]. And just to be clear, this is the “happy path”: when such an <em>ecosystem</em> is operating by design.</p><p>A better way is to avoid any cyclical/bidirectional dependencies and consider redeveloping any portion of the <em>ecosystem</em> in which such dependencies arise. Remember, while it may feel icky to have to double your resource count (and thus operational expense) for a quarter or play some wacky sliding puzzle game with your networks to keep from running out of RFC 1918 space, this is the flexibility afforded by cloud and infrastructure as code. If the ickiness is truly overwhelming it may be exposing some missing tooling/convention/governance, architectural inefficiencies, or unleveraged mitigation techniques in your <em>ecosystem</em>[12].</p><h4><em>ClickOps Driven Development (CODD?)</em></h4><p>An emergent phenomenon is what I’ve heard most commonly referred to as “ClickOps Driven Development”[13]: resources are created manually to fully configure and ensure a working solution, and then tagged, named, or labeled to signify readiness for the resource to be incorporated into the <em>IaC Ecosystem</em> as code. While this is wildly inefficient, especially in contrast to well-architected solutions, it can bridge the gap in the case of inexperience, unfamiliarity, or under-coordination (e.g. or perhaps “severe asynchronicity”). Perhaps a more economical use case would be to reserve this approach for rapid prototyping or proof of architecture via the ClickOps activities, after which a full <em>IaC Ecosystem</em> architecture can be designed and scaled in Terraform.</p><h3>Next Time…</h3><p>We’ll explore logic and expressions in Terraform, both considering the common successful patterns in Terraform and the merits thereof. As always, we’ll also review anti-patterns and how to avoid them.</p><h3>Footnotes</h3><p>[1] It is actually based on countless true stories.</p><p>[2] Assuming of course e.g. name or IP collisions are avoided.</p><p>[3] Assuming similar resource types are targeted in proportion, equivalent compute and networking capacities are leveraged, and all other things “equal” from an Occam’s Razor perspective.</p><p>[4] See this post for more details: <a href="https://medium.com/codex/terraform-backend-migration-a-journey-worth-taking-84a937af0c90">Terraform Backend Migration: A Journey Worth Taking</a>.</p><p>[5] I.e. it is not possible to do in a single operation, so a failed insertion into the “to tree” won’t un-prune the “from tree”.</p><p>[6] A contrived example might be separating the state of a VM from the state of its attached disk, rather than allowing Terraform to manage their interdependence. While operations on the VM might not necessitate an update to the disk, the reverse might not be true; now each change elicits new consideration, possibly requires multiple triggers, and could go wrong if operations are performed out of order. There are probably good reasons to assume this risk, but I would err on the side of expecting them to also be so contrived.</p><p>[7] Each resource is 2–3 hops from the root, and any interdependencies (resulting in a graph, rather than a true tree) are in the leaf-most modules.</p><p>[8] I.e. where one state includes optional configuration, e.g. IAM/RBAC bindings, that require data generated from the second state, e.g. resource IDs, and that second state depends on data generated from the first state.</p><p>[9] Explanation and example left as an exercise to the reader.</p><p>[10] I.e. disabling a <em>prevent_destroy</em> or issuing a command to gracefully disengage one or more agents.</p><p>[11] I.e. the result of a single successful execution of <em>terraform apply</em> on a valid plan.</p><p>[12] In less roundaboutly cordial and more specific terms: if you can’t stand for any part of an internal network to be recreated as part of an <em>IaC Ecosystem</em> redesign, you probably have much bigger risks than even perfectly designed Terraform can mitigate.</p><p>[13] A somewhat curious nod to e.g. Test Driven Development.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dccec970f554" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devoops-discourse/terraform-patterns-observed-part-4-state-management-dccec970f554">Terraform Patterns, Observed — Part 4: State Management</a> was originally published in <a href="https://medium.com/devoops-discourse">DevOops Discourse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform Patterns, Observed — Part 3: State Misconceptions & Pitfalls]]></title>
            <link>https://medium.com/devoops-discourse/terraform-patterns-observed-part-3-state-misconceptions-pitfalls-e051ca1b7be9?source=rss----d31c962fddc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/e051ca1b7be9</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Robert Glenn]]></dc:creator>
            <pubDate>Sun, 20 Aug 2023 05:12:12 GMT</pubDate>
            <atom:updated>2023-09-17T04:54:56.691Z</atom:updated>
            <content:encoded><![CDATA[<h3>Terraform Patterns, Observed</h3><h4>Part 3: State Misconceptions &amp; Pitfalls</h4><blockquote>Note — This is purely my perspective as a practitioner with firsthand visibility into several working solutions in my career as a software consultant. Much of the vocabulary used in this series is of my own imagination and will surely cede to better nomenclature from the community. Moreover, many implementations I have seen in practice include multiple types of patterns discussed in this series.</blockquote><blockquote>Note — This commentary is neither definitive nor endorsed and is in no way representative of the views of my employer, Accenture.</blockquote><p>Infrastructure as Code as we know it today has 3 major components that comprise a known state: the codebase that is applied (or the IaC tool’s interpretation thereof), the resulting live, instantiated resources (which is accurate only to the point in time when it was last read by the IaC tool), and the recorded <em>state object</em> (definitions of terms used in this article are below, in an appendix section) which is stored in a Terraform state file. I refer to this as the “state triplet” and I believe it to be fundamental to how IaC works[1].</p><p>In this post, we will begin our discussion of Terraform state by presenting some common misconceptions and typical pitfalls that can arise. We will also establish some terminology used in this and subsequent posts. This post is intended to provide context for the following post which will discuss typical management of Terraform state.</p><h3>Common Terraform State Misconceptions</h3><p>While Terraform is wildly popular, I find that it is largely understood by even most proponents. I’ll discuss some common misconceptions I’ve encountered more than once.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lHyT1ddvJVD-XKWOoHfVwQ.png" /><figcaption>Image generated with Midjourney</figcaption></figure><h4><em>State Consistency &amp; Drift</em></h4><p>Terraform is generally idempotent[2], meaning the same code should produce the same outcome, no matter how many times it is run. Also, Terraform runs when you tell it to. Even if it’s running on a cron, it’ll only check on the schedule, and be accurate to that point in time. It won’t automatically detect changes you make in other contexts such as the platform’s CLI, UI, or API, or applied to a different Terraform state[3].</p><p>Thus, unless all updates to the platform are made through Terraform (or the Terraform/provider version is not kept well enough up-to-date), there’s a possibility of the dreaded configuration drift. When the drift is detected, Terraform will tell you how it will attempt to correct the drift, which oftentimes just means undoing the detected change. Sometimes, however, the configuration drift forces a resource to be recreated, which destroys the resource[4].</p><h4><em>State Atomicity</em></h4><p>Each terraform state is completely ignorant/agnostic of any other state. Each <em>terraform apply</em> results in exactly one state. Thus, if a <em>terraform apply</em> execution is 1–1 with a pipeline execution (a “job” or “run” or whathaveyou), then every pipeline execution is completely ignorant/agnostic of any other pipeline execution.</p><p>Moreover, one terraform state won’t detect changes you make in other contexts that don’t directly affect the encoded resources. Even changes to similar resources (e.g. a manually created VM that appears beside the encoded VMs in the platform’s user interface) via the platform’s CLI, UI, API, or another Terraform state will be undetected.</p><p>Thus, each <em>state tree</em> can be thought of as atomic[5]. Any connections between Terraform states would need to be implemented outside of the confines of the state triplet components (i.e. state file, Terraform code, and instantiated resources).</p><h4><em>Not for “1-Offs”</em></h4><p>Terraform does not lend itself well to operations that must be run once and only once, or those that are expected to be fire-and-forget. The very nature of Terraform is to support the consistency of the instantiated resources that make up your <em>IaC Ecosystem</em>. Many 1-off operations may be more appropriately performed directly with the underlying platform API request or CLI command.</p><p>Also, typically, the entire state under execution is checked any time <em>terraform plan</em> is run. Terraform does support a <em>-target</em> flag, which can be used to “apply changes incrementally”[6] and could conceivably be leveraged for “1-off operations” (maybe rolling some key or incrementing a stored object version). However, I would consider this an advanced feature and I would recommend building governance around its use.</p><p>Using Terraform for 1-off operations, especially within a larger, idempotently-consistent state, runs the risk of additional operational overhead either in encoded guardrails (e.g. building a wrapper around the <em>-target</em> flag for additional control over when it is leveraged[7]) and in clunky source control management (e.g. additional local git pull operations to keep the encoded stored object or encryption key versioning consistent, or when reading the terraform state information, locally).</p><h3>Common Pitfalls</h3><p>There are several things to watch out for when architecting for a healthy, well-shaped state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O5vYb1X_1McA5S7Qbo0vSQ.png" /></figure><h4><em>Locked State</em></h4><p>A state’s lock file is the mechanism that protects Terraform from attempting to make multiple concurrent attempts at applying plans. Especially in the case of automated builds, it can prevent unintended actions. Unfortunately, it also prevents intentional actions, potentially requiring otherwise automated jobs to be manually restarted. Ideally, the state is sized and organized in such a way that the action of applying a plan completes before a new plan is attempted. Aborting the execution of a <em>terraform apply</em> command may leave the state locked, which may require manual intervention.</p><h4><em>Broken or Inconsistent States</em></h4><p>Perhaps worse than a <em>locked state</em> is a <em>broken state</em>. A <em>broken state</em> occurs when Terraform discovers the <em>state object</em> does not reflect reality, and either reports a plan that makes a number of unexpected changes or fails altogether with an error message. It can also arise with mismanaged resource and module referencing or stale Terraform or provider versions. In any case, a <em>broken state</em> causes confusion and impedes further development until the state is reconciled (or the deltas are ignored and simply bashed away upon the next <em>terraform apply</em>). On the extreme (but by no means rare) end, a state requires field triage surgery, with an intrepid professional making momentous excisions to the <em>state tree</em>.</p><p>There are possibly uncountable paths to a <em>broken state</em>. Here is a short list of common missteps:</p><ul><li>Misusing <em>for_each</em> and <em>count</em> expressions [8] — Only use the <em>count</em> expression to provision a resource/module conditionally. Use <em>for_each</em> sparingly and never use a generated map whose keys are determined by an ordering/sequencing function. This will be discussed further in a subsequent post.</li><li>ClickOps development undoing changes applied with IaC (and vice versa).</li><li>A data resource (perhaps indirectly) references a managed resource elsewhere in the <em>state tree</em>. The non-deterministic data resource is then referenced in another managed resource for a field that forces the replacement of the resource each time <em>terraform apply</em> is executed[9].</li><li>Version constraints are introduced and forgotten; meanwhile, the Terraform CLI version is advanced per organizational policy, resulting in a version mismatch.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tFcxyfW3B_ZbFJROfm2tSw.png" /></figure><h4><em>Uneven State Trees</em></h4><p>While there is nothing inherently wrong with a <em>state tree</em> that is lopsided or jagged, I would suggest that uneven distributions of resource definitions increase the overall complexity of troubleshooting. The longer, the state hierarchy, the more places that may require attention when resolving a bug. Typically, a new variable definition at a <em>leaf node</em> requires analogous definitions at <em>ancestor nodes</em>. This creates a cascading effect up the <em>state hierarchy</em>, oftentimes all the way to the <em>root</em>. Sometimes it’s not possible to avoid some unevenness, but having a high ratio of <em>nesting cardinality</em> to <em>node cardinality</em> along a longest <em>node ancestry</em> (e.g. a giant module with a high degree of child submodules somewhere buried down a module chain) is more typical of haphazard detachment rather than mindful abstraction.</p><p>A more measurable outcome of an uneven <em>state tree</em> is a severely uneven distribution of activity throughout the tree. Even in well-balanced state trees, some nodes may see more activity than others; this is neither problematic nor especially interesting on its own. However, when a tree has a few resources that change often/regularly with many resources that change rarely/irregularly, it can suggest an unnatural coupling of the comprising resources. Certainly, including unrelated resources into the same <em>state tree</em> can be especially irksome with <em>locked</em> or <em>broken states</em>, when the offending <em>state node</em> is unrelated to the intended resource changes.</p><h4><em>Too Many Cooks</em></h4><p>For better or worse, multiple teams working in (sometimes loose) coordination on a single <em>IaC Ecosystem</em> is fairly standard. When a single repository is developed by multiple teams, it can very quickly cause friction with the presence of any <em>locked</em> or <em>broken states</em>. These are in turn at an increased potential, as multiple teams can compound the likelihood to include dissimilar resources. Extremely distanced teams or team members can magnify any such discomfort, especially with large disparities in solution exposure.</p><h4><em>State Bloat</em></h4><p>When the <em>state tree</em> has grown so large that it takes too much time and resources, it may begin to cause severe slowdowns to a <em>terraform plan</em> or <em>terraform apply</em>. Even with enough resources, the running time can be prohibitive in the case of a typical development team collaborating to rapidly construct a single state. Suddenly teams are met with artificial barriers to introducing their latest solutions, or far worse, a much-needed security patch. Avoiding state bloat is addressed further in the next post.</p><h3>Next Time…</h3><p>We’ll continue our discussion of state, describing how to manage and organize modules for well-balanced <em>state objects</em>.</p><h3>Appendix — Terms &amp; Definitions</h3><p><em>Remote Resource</em> — a resource maintained by a different Terraform state (or by a different mechanism altogether) can be called a <em>remote resource</em>. This must be referenced using a data block.</p><p><em>State Tree</em> — The full representation of all items and relationships that Terraform uses to maintain the desired behavior of instantiated platform resources as recorded by a single <em>terraform apply</em> execution. Note that this would be represented as a collection of <em>state objects</em> in the output of a <em>terraform plan</em> or <em>terraform apply</em>.</p><p><em>Backend State</em> — A representation of a state tree that is stored in a uniquely located file. The unique location is determined by a backend selector that can be configured at runtime (i.e. at the execution of a <em>terraform plan</em>). The backend state can be moved or manipulated using Terraform CLI.</p><p><em>State Node</em> — A named resource or module located somewhere in the state tree.</p><p><em>State Object</em> — generically, this can be any complete collection of recorded state elements. In Terraform, the most general <em>state object</em> is the <em>state tree</em> itself. For the purpose of this series, whenever we are discussing Terraform, we will use <em>state object</em> to denote any proper subtree of a <em>state tree</em>.</p><p><em>Root Node</em> — The <em>state node</em> at the root of a given <em>state object</em>. In the case of the <em>state tree</em>, the <em>root node</em> is always a module even in the case of a single resource.</p><p><em>Leaf Node</em> — A special <em>state node</em> at the leaf of the <em>state tree</em>. This is always a resource.</p><p><em>State Hierarchy</em> — The chain of <em>state nodes</em> along a single depth-first search from the root to a leaf.</p><p><em>Parent/Child/Sibling Nodes</em> — Describe relative <em>state nodes</em> in a typical tree configuration; however, in regular conversation, this may represent more of a logical relationship than a strict relationship, given how certain Terraform elements are stored in the state[10].</p><p><em>Locked State</em> — Typically, a state that is in the middle of a running <em>terraform apply</em> command. Sometimes, the lock is left behind when an execution is aborted without care.</p><p><em>Broken State</em> — A <em>state tree</em> that is inconsistent with the live, instantiated resources, thereby breaking the most important relationship of the state triplet. This must be reconciled[11] before safe usage can continue.</p><p><em>State Progeny</em> — The collection of nodes in a <em>state object</em>, with the exception of the <em>state object’s</em> <em>root node</em>.</p><p><em>State Ancestry</em> — The chain of <em>parent nodes</em> from the given <em>state node</em> to the <em>state tree’s</em> <em>root node</em>.</p><p><em>Node Cardinality</em>* — For any given <em>state node</em>, this is the number of <em>child nodes</em>.</p><p><em>Root Cardinality</em>* — For any given <em>state object</em>, this is the <em>node cardinality</em> of the <em>root node</em>.</p><p><em>Leaf Cardinality</em>* — For any given <em>leaf node</em> (i.e. resource), the number of <em>state nodes</em> in the <em>state ancestry</em>.</p><p><em>Nesting Cardinality</em>* — For any given <em>state object</em>, this is the largest value of <em>leaf cardinality</em>.</p><p>*Note that this is at least 1.</p><h3>Footnotes</h3><p>[1] I don’t believe it to be further decomposable, without compromising integrity.</p><p>[2] With I’m sure more than a few exceptions that I won’t enumerate. Let’s say that it can generally be considered idempotent unless a command/field/expression is documented otherwise.</p><p>[3] Not to suggest one couldn’t develop a solution that would run Terraform in this way, but it certainly doesn’t do so out of the box.</p><p>[4] This can even happen just out of poorly written code, or by using unsafe versions of certain API fields.</p><p>[5] It is certainly possible to encode a resource in two locations and import the instantiated resources into two distinct Terraform states; however, I seriously challenge the rationale for doing so beyond contrived oddity.</p><p>[6] <a href="https://developer.hashicorp.com/terraform/tutorials/state/resource-targeting">https://developer.hashicorp.com/terraform/tutorials/state/resource-targeting</a></p><p>[7] And what happens when you don’t want the key to be rolled or have no new object to store as a new version?</p><p>[8] These two are common enough to be discussed in the Hashicorp documentation (<a href="https://developer.hashicorp.com/terraform/language/meta-arguments/count#when-to-use-for_each-instead-of-count">here</a> and <a href="https://developer.hashicorp.com/terraform/language/meta-arguments/for_each#limitations-on-values-used-in-for_each">here</a>, respectively), but it bears including.</p><p>[9] For some resources, a temporary toggle of its existence may be invisible to the system and any users, but it should generally be avoided, nonetheless.</p><p>[10] The simplest example is how a <em>for_each</em>’ed block is represented; the state objects in such cases are technically children of a collection element, but we would logically consider them as children of what is actually their grandparent node.</p><p>[11] A healthy reconciliation process might be to ignore the inconsistency and blow away whatever changes were introduced else wise; be sure this inconsistency isn’t due to something else, e.g. referencing the wrong backend prefix.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e051ca1b7be9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devoops-discourse/terraform-patterns-observed-part-3-state-misconceptions-pitfalls-e051ca1b7be9">Terraform Patterns, Observed — Part 3: State Misconceptions &amp; Pitfalls</a> was originally published in <a href="https://medium.com/devoops-discourse">DevOops Discourse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>