<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by MrTrustworthy on Medium]]></title>
        <description><![CDATA[Stories by MrTrustworthy on Medium]]></description>
        <link>https://medium.com/@mrtrustworthy?source=rss-13fe4dbad10f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*oOhGB0Mu-1mGMfIy.</url>
            <title>Stories by MrTrustworthy on Medium</title>
            <link>https://medium.com/@mrtrustworthy?source=rss-13fe4dbad10f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 19:54:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mrtrustworthy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[When configuration becomes data]]></title>
            <link>https://medium.com/geekculture/when-configuration-becomes-data-9c8fd94ae4f2?source=rss-13fe4dbad10f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c8fd94ae4f2</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[configuration-management]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[MrTrustworthy]]></dc:creator>
            <pubDate>Tue, 03 Aug 2021 20:45:47 GMT</pubDate>
            <atom:updated>2021-08-05T06:18:52.264Z</atom:updated>
            <content:encoded><![CDATA[<p>What’s actually the difference between configuration and data? Admittedly, a very weird topic to think about, yet I still get pulled back to it regularly.</p><p>And, as someone in an ever-growing company, the question seems to become more and more important as we keep adding new developers, users and products in our company.</p><p>In DevOps (and Software Engineering, too), we work more and more with <strong>Infrastructure as Code</strong> (<strong>IaC</strong>) tools. Ansible, Puppet, Chef, Terraform, Kubernetes… the list of tools that allow you to <em>declaratively</em> define your servers and applications keeps growing.</p><p>The file formats are different from tool to tool. Some use JSON, some YAML, some make their own for whatever reasons (hello <a href="https://www.terraform.io/">Terraform</a>!). But overall, they are all <strong>configuration files</strong>. Most tools, like Helm and Terraform, have templating and some level of scripting built in to make working with larger and more complex configurations easier. And, as our systems and companies grow, those configs become more and more complex, in turn spawning ever more complex a̶b̶o̶m̶i̶n̶a̶t̶i̶o̶n̶s̶ tools to manage the configuration files themselves — like <a href="https://jsonnet.org/">JSONNET</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NYqQfmv0uweeauy4" /><figcaption>Photo by <a href="https://unsplash.com/@flowforfrank?utm_source=medium&amp;utm_medium=referral">Ferenc Almasi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Configuration management, the overarching name of this entire field, is a fun and exciting topic. As long as you only <em>think</em> about it, academically. Implementing (and more importantly: maintaining) an overlay-based meta-configuration across business domains <em>in practice</em> is, to put it mildly, somewhat less fun.</p><p>But this post isn’t about configuration management. Or rather, it’s about the question: <strong>At which point does configuration stop being configuration, and simply becomes boring data?</strong></p><h3>Configuration and data</h3><p>Well, to be precise about it, configuration is just one specific subset of data. So, for the purposes of this post, let’s make up some arbitrary definitions:</p><ul><li>Data, as we typically use the term, is stored in databases, accessed programmatically, and generally changed frequently during the runtime of a system.</li><li>Configuration, on the other hand, is stored in the above mentioned (human &amp; machine readable) file formats, checked into your <em>git</em> repositories, and a change in configuration typically requires code reviews, deploys, and/or restarts.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bPyuG70R5v9EFdi3" /><figcaption>Photo by <a href="https://unsplash.com/@benjaminlehman?utm_source=medium&amp;utm_medium=referral">benjamin lehman</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We (by “we” I mean everyone who works with DevOps-y things) generally have a good intuitive understanding of the differences between configuration and data.</p><p><em>New purchase on the shop</em>? Obviously goes into the database. <em>New AWS S3 bucket</em>? Put it into the terraform config file!</p><p>But, as the company I’m working in grows in size, some lines have started to get blurry.</p><h3>Growth of… complexity? No, just volume</h3><p>Each of our developers has their own test VM, which is used to… well, probably to test some stuff. Probably how to screw up our database again — who knows what else developers do. Those VMs are defined &amp; managed with an IaC system, so to add a new VM we simply have to add a new entry in our configuration file.</p><p>Sounds pretty convenient, right? Well, it was convenient when we had only about 30 developers. With ten times as much, and the industry typical fluctuation of engineers, this “easy” solution causes more and more work as more people get on- and offboarded each month. And, like all engineers ever, we love automation. So instead of this manual step there’s now an application that allows people to create new VMs in self-service whenever they want.</p><p><strong>And the way the application works is quite simple:</strong> Whenever someone new requests a VM in a ticket, the app automatically makes a change to our IaC configuration repo, pushes the change, and the CI deploys the new version — thereby creating the new VM.</p><p>That’s a totally sensible solution that does exactly what it should. It even re-uses existing structures like the CI pipeline to deploy new infrastructure. Yet, the longer I think about it, the more it sounds completely <strong>wrong</strong> to me.</p><p>With 30 developers, we considered each test VM to be an individual entity where changes <em>deserve</em> to go through the non-trivial process of JIRA Issues, merge requests, code review, and deploys. With 300 developers, we care so little about new test VMs that we didn’t even track how many are created. At least until the CFO showed up with a red face, high pitched voice, and our latest AWS bill — at that point we started tracking the number of VMs again at least.</p><h3>When configuration becomes data — it hurts</h3><p>So, the lesson we learned learned here was this: At some point, when there’s more VMs than story points in your next sprint, their management stops being configuration and becomes data. The Kubernetes enthusiasts may say it’s <a href="http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/">cattle, not pets</a>. Data, which may change quickly and at any time, without JIRA Issues, reviews or more effort than a button click. Data, that no longer belongs into a config file and should instead live in its ancestral home, a <em>database</em>.</p><p>By writing an application that automatically updated our Terraform setup via CI, we’ve essentially committed (no pun intended) the greatest sin of software engineering: <strong>We used files as a database</strong>. A few paragraphs above I made jokes about developers screwing up databases, but the joke’s on me now. I’m starting to question some long-held beliefs. Maybe those finance people were right, and Excel is actually an acceptable database? Maybe SAP really is the pinnacle of software development?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HD08okhX2huyd9M9" /><figcaption>Photo by <a href="https://unsplash.com/@anthonytran?utm_source=medium&amp;utm_medium=referral">Anthony Tran</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>But I must not stray so far from the light</em>. Let’s focus the entire team on how we can improve the VM-provisioning application. We can still save it. If a user requests a new VM, we write an entry into a database. We re-model our application to compare the state in the database with the existing VMs, and create or delete them as required. Sounds a bit similar to what <a href="https://kubernetes.io/docs/concepts/overview/components/">Kubernetes is already doing with etcd</a>, but it could be different enough to warrant its existence. We could even build a web UI for better self serving of your users, which allows them to provision a machine through a single click! Then… <em>oh damn it, we’re back at the place we were before IaC.</em></p><h3>Boundaries are very important, in both SE and SM</h3><p>Maybe, just maybe, there’s not a single approach to managing applications and systems that is actually “correct” at all scales.</p><p>Maybe, as much as I hate to admit it, both IaC and colourful<em> clicky-button-do-thing </em>tools have their place in modern tech companies.</p><p>Maybe there’s a breakpoint where configuration becomes data, after which point our lack of tight-gripped control over configuration matters less than easy <strong>self-service UX for users</strong>.</p><p>And maybe we simply have to figure out where the <strong>breakpoints</strong> are, so that we can decide how the best solution for today and tomorrow should be implemented.</p><p>As a developer at heart, I love<em> clear requirements</em> almost as much as automation. Consequently, I’ve spent far too much time, trying to define precise rules that would allow me to algorithmically decide whether an entity should be treated as configuration or data. All that effort, only to end up with a muddy collection of notes that can barely be called <em>guidelines</em>. Here they are:</p><ul><li>If there are more than <strong>14</strong> (now referred to as a “<a href="https://en.wikipedia.org/wiki/Baker%27s_dozen_(disambiguation)">DevOps’ dozen</a>”) practically equal entities in the same environment, it’s probably data</li><li>If you can tell your CTO “honestly, I have <strong>no idea</strong> how many entities there are, probably a lot” without pissing them off, it’s probably data</li><li>If entities are created/deleted once a month or more frequently on average, it’s probably data</li><li>If only a single person (or a single team, in large companies) would notice or care if an entity were to <strong>suddenly disappear</strong> for a bit, it’s probably data</li><li>If a change in an entity can, even in the worst case, only create as much <strong>monetary damage</strong> for your company than your conference budget, it’s probably data</li><li>If an entity isn’t just a sub-component of an overarching architecture, but is instead actually a defining a <strong>core part of your architecture</strong>,<em> it’s probably not data</em></li></ul><p>Writing out those rules made me realise how many things we currently store in configuration files that <em>don’t belong there</em>. We use <a href="https://www.vaultproject.io/">Vault</a> everywhere for secrets, and it’s well integrated in our IaC workflows. Maybe using Vault, or something similar that’s not focused on secrets, for storing things that have been promoted from configuration to data could be a good idea 🤔</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c8fd94ae4f2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/when-configuration-becomes-data-9c8fd94ae4f2">When configuration becomes data</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Data Driven to Driving Data— The dysfunctions of Data Engineering]]></title>
            <link>https://medium.com/@mrtrustworthy/from-data-driven-to-driving-data-the-dysfunctions-of-data-engineering-34c34496ed8e?source=rss-13fe4dbad10f------2</link>
            <guid isPermaLink="false">https://medium.com/p/34c34496ed8e</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[MrTrustworthy]]></dc:creator>
            <pubDate>Tue, 27 Jul 2021 19:22:43 GMT</pubDate>
            <atom:updated>2021-07-27T19:22:43.075Z</atom:updated>
            <content:encoded><![CDATA[<p>This is not the Data Engineering post you were looking for.</p><p>We’ll <strong>not </strong>talk about how to set up Spark, or what the difference between a HDFS Namenode and Datanode are. <strong>Those things don’t matter</strong>. I’ve never seen a Data Engineering project fail because someone picked ORC instead of Parquet. Quite the opposite: Many “data driven” initiatives are failing <em>even though</em> they had the best engineers on the task and picked the “best” stack of technologies.</p><p>The problems we’re facing in Data Engineering are not the technologies. Sure, the entire ecosystem could do with a few more years of maturing, consolidating and bugfixing. But that’s not the reason why so many Data Engineering initiatives end up as overpriced technology POCs. For too many organizations, their Big Data cluster with dozens of VMs can only be called <em>effective</em> when the goal is to parallelize the burn-down of their cloud budget.</p><p>We talk so much about <em>Data Driven</em> companies that base all their important decisions on methodical analysis of data. We admire (and curse) companies like Facebook for turning people’s data into a tradable asset. There are so many articles exclaiming that data is now the <a href="https://www.economist.com/leaders/2017/05/06/the-worlds-most-valuable-resource-is-no-longer-oil-but-data">most</a> <a href="https://www.paysafe.com/blog/the-worlds-most-valuable-resource-not-oil-but-data/">valuable</a> <a href="https://thevault.crownrms.com/data-valuable-resource/">resource</a> in the world. And all that sounds pretty reasonable in theory… but also vastly removed from reality. When asking companies whether they’d prefer a few more terabytes of data or a suitcase full of gold, my guess is that most would take the suitcase. At least with gold, they’d know how to turn it into money.</p><p>Yes, many companies are doing ok. They have a Analytics team and some dashboards with KPIs. The developers sometimes push events and logs into something called Kafka. There are Data Scientists doing some fancy Neural-Machine-Something. None of the C-level executives <em>really</em> understands it, but the teams seem happy with their work — and that’s a good sign, right? Yes, that is good. But we should be doing so much <strong>better </strong>by now.</p><p>To be successful in the future, we have to take it further. Being <em>Data Driven</em> isn’t enough. Data is the engine as well as the fuel on which our businesses run. Its flow shapes a company, from the value delivery processes to the inter-personal communication paths of its employees. And to realize the potential of our data to its fullest extent, we need to get in the driving seat. We need Data Engineering — not just in the shape of a team of specialized developers — but as a methodical approach at re-designing how a company treats data, processes and projects on a fundamental level.</p><p>Strap in, because this post will be quite a journey. It begins in the Shire, where Data Engineers live in small holes in the ground. It will lead through vast valleys where Developers and DevOps Engineers are fighting for supremacy over Kubernetes. Together, they will sail the shores of Lake Data, where Business Analysts and Data Scientists swim in unison. And the journey will end in Mordor, where we’ll throw the cursed ring of <strong>business value</strong> into the fires of Mount Doom (which is, coincidentally, where most Product Owners live).</p><h4><strong>What is a Data Engineer?</strong></h4><p><a href="https://medium.com/u/451599b1142a">Jeff Hale</a> did an analysis of the skills required in Data Engineering job offers and wrote a nice <a href="https://towardsdatascience.com/most-in-demand-tech-skills-for-data-engineers-58f4c1ca25ab">post</a> about his findings some time ago. By clicking through some job ads on <a href="http://indeed.com/">Indeed</a>, you can easily see the same pattern he found in his search. If you’re familiar with the industry, there’s actually not much of a surprise to be found by browsing those job postings. SQL, ETL, Spark… if you’re a Data Engineer, all those things should at least sound familiar to you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*tDylwBK8n-DzUaHNyVAnkQ.png" /></figure><p>What <em>is</em> surprising though, is the combination of breadth and depth in skills that seem to be required according to those job postings. Not only should you possess the skills of multiple different fields (like Cloud, Data Science, DevOps, Software Development and Business Intelligence), but you should also be <strong>good</strong> in all of them.</p><p>Sounds great, right? Someone who can do the work of at least four other jobs! Why even hire Developers or Cloud Engineers when a Data Engineer could also do their job? While we’re at it, why not replace Accounting, Marketing and HR with our golden-egg laying Data Engineers as well?</p><h4><strong>The different types of Data Engineers</strong></h4><p>Well, that’s obviously absurd. Looking back at individual job postings, we can quickly see the the source of this: No job requires you to have <em>all </em>the skills mentioned above, instead they all require <em>different</em> ones. The issue with all those job postings is that <strong>we don’t have a consistent role definition for a Data Engineer</strong>. There’s actually four different roles that companies tend to lump together under the broad umbrella of “Data Engineering”:</p><p><strong>Type 1: Specialized Software Developer: </strong>A developer who’s focusing on data-driven applications. Regularly searched for by companies that have moved to a microservice-oriented architecture that relies on streaming platforms like Kafka for inter-service communication. Common tasks are either writing services that produce streams of events, or services that listen for those events to perform some action. Sometimes, those developers also maintain the Kafka/RabbitMQ clusters, similar to how Search-teams often maintain their Solr/Elasticsearch setups themselves.</p><p><strong>Type 2: Business Intelligence (BI) Developer</strong>: Grossly simplified, there’s two different segments in the field of BI: One that’s business-focused and concerns itself with KPI definitions and Dashboards, and one that’s tech-focused and responsible for Data Warehouses and ETL processes. It’s harder to find people to do the second one, which is why (I assume) some companies try to make it more attractive by re-branding the job position as Data Engineering. After all, your ETL cronjobs run on half a terabyte of data — that counts as Big Data Engineering, right?</p><p><strong>Type 3: Data Scientist with a systems focus</strong>: Often also found under the name of <em>Machine Learning Engineer</em> (MLE), a role with a similarly inconsistent definition. During the height of the Data Science gold rush, companies quickly figured out that if you wanted to crunch through terabytes of data, you not only need algorithms but also infrastructure that can scale to those volumes. Setting up and maintaining Spark and/or Hadoop clusters are typical jobs that fall under this job profile. With surprising regularity, Data Science teams just pick the one among them who’s the <em>least</em> afraid of bash scripting and sacrifice that person to the MLE role. That person isn’t exactly <em>happy</em> about this, but hey, at least it looks good on the CV.</p><p><strong>Type 4: Big-Database Admin</strong>: When a company starts hiring Data Scientists, they first try to get the Ops/DBA/SysAdmin team to take over the work of managing Data Science infrastructure. The Ops team will then try and convince the CTO that they <em>really</em> don’t have time for that because developers keep breaking MySQL by writing logs into it. If the Ops team is successful, this responsibility might be handed to the Data Science team, who in turn starts recruiting MLEs to get rid of it, and you end up with <strong>type 3</strong>. As Ops teams typically have lots of practice in arguing <em>against</em> something, this is the most common scenario. If the Ops team<em> </em>somehow isn’t successful in convincing the CTO, you’ll end up with this kind of role instead — Database Admins who manage HDFS and Hive instead of MySQL and MongoDB.</p><h4><strong>Organic development of the Data Engineering role</strong></h4><p>There’s an underlying pattern to all of those types, a certain <strong>organic development</strong> (to anyone in IT known as “catastrophic mess”) through which the role is defined. It tends to start with some group in the company that’s unsatisfied with their (in)ability to work with data. Further analysis reveals a gap — some set of tasks that nobody wants to do because it’s not <em>their</em> job — and the company starts recruiting to fill it. The specific role of Data Engineer that’s recruited is effectively derived from the group that argues for their need first.</p><p>If it’s the lead developers/architects that want to move from <a href="https://martinfowler.com/articles/break-monolith-into-microservices.html">monolith to micro-service</a> first, you end up with a <strong>type 1 </strong>Data Engineer<em>. </em>If the analysts (or, more likely, the C-level executives) are not happy with their lack of insight in the companies operations, recruiting for <strong>type 2 </strong>starts. In the most common cases right now, it’s the newly-established Data Science team who blames the non-existent business value in their last project on the lack of usable infrastructure — and you end up with <strong>types 3 </strong>or <strong>4 </strong>instead.</p><p>Now, there’s a variety of issues with this approach. Let’s start with an obvious one: What happens once a second and third group starts to put in their requests for “more data liberation”? Assume that the Data Scientists got their voice in first, and the company got a <strong>type 3</strong> Data Engineer. Now the Lead Developer decides to split the existing PHP/Java monolith into neat little micro-services — and they apparently need Kafka for something called CQRS or DDD. Does the Machine Learning Engineer now have to build an Enterprise Service Bus for your Developers? Does the Lead Developer try to persuade the Ops team to set up and maintain a Kafka cluster for him? Do you hire another type of Data Engineer to fill this new gap? And what happens once the BI Analysts decide that they want to create nice dashboards based on the events that your micro-services send, or on the models that your Data Scientists built? After all, analyzing data is how data driven companies decide what to build next.</p><h4>The value of integration</h4><p>This is the point where many companies are getting stuck right now: The <strong>integration</strong> step. No matter from which perspective you approach Data Engineering first — Developer, Data Scientist, or BI/Analyst — you’ll have trouble integrating it with the other two angles later on.</p><p>This integration is definitely the hardest part about Data Engineering. Coincidentally, it’s also where the true <strong>value </strong>of Data Engineering comes from. It’s not only about the massive speed-up to all individual teams working with data. It’s also about the decoupling of organizational dependencies, the new opportunities that can be taken as a result, and the difference between <a href="https://svpg.com/product-vs-feature-teams/">Dev Teams and Empowered Product Teams</a>.</p><p>Making everyone happy at the same time sounds like a hard job. But why should we even bother? We have three groups with their individual needs — creating three specialized, bespoke platforms sounds like a much better idea. Developers get some Kafka Brokers, Data Scientists get a dedicated Spark Cluster somewhere in the cloud, and for Analysts we just buy <a href="https://www.tableau.com/">Tableau</a> or <a href="https://looker.com/">Looker</a> like everyone else does. Three rings for the Elven kings under the sky. <br>Go away Sauron, you can keep your Ring.</p><h4>True end-to-end cost</h4><p>Sadly, everyone who’s been in that situation knows that it’s not that easy. The one thing that’s often ignored is the true end-to-end time and effort to turn data into money. This doesn’t happen on purpose, but as a consequence of increasing specialisation and complexity. If you’re in a company with 5 employees (including 3 working students), you only need a daily excel export of your four MySQL tables for reporting. Your “recommender” is a bunch of if-else’s that runs on the operational database. Everyone knows all processes and systems, no need to over-engineer your data pipelines.</p><p>When the company grows, it will piece by piece add new processes, tools and teams to specialise in them. At some point, you’ll try to determine where a specific metric, KPI or recommendation is actually coming from, and you’ll find yourself on a surprisingly long journey. From the business department that’s accountable for the KPI, to the analysts, from the DBAs to the developers responsible for the component. You’ll find yourself calling the old working student (probably left the company 2 years ago already) who originally implemented the first version of that business process. At some point you give up asking “why?” and simply rename your KPI to reflect its actual meaning. <strong><em>Average cart value</em></strong> is now called <strong><em>Average cart value (excluding blue pencils)</em></strong> and you end up intently staring at the ceiling fan whenever someone asks you about it. There’s helicopter sounds in the distance.</p><p>As predictable as this is, those types of data pipelines structures keep appearing and re-appearing. It’s another type of organic growth. This time, not for job titles, but for data architectures, pipelines, batch processes, and workflows. And working with data keeps getting slower and slower as time goes on.</p><h4>Data Engineering as a Product Team</h4><p>My hypothesis on the main reason for this: Data Engineers are often treated as a service team. Some development team needs a Kafka integration? Send a mail to Data Engineering! An Analyst needs another pipeline? Put a ticket in JIRA, tag Data Engineering, mark it as high prio — hopefully they’ll do it fast. And yes, Data Engineers generally act as enablers for other departments, often simply reacting to requests from an outside user. But, Data Engineers can work much better if you give them more responsibility: as a Product Team.</p><p>If you think of a classic product team structure, something like the ubiquitous examples of the “Checkout Team” comes to mind. End-user facing teams with designers, frontend and backend devs, maybe even some SRE or DevOps sprinkled into it.</p><p>A Data Engineering product team has a slightly different anatomy, more akin to a B2B or internal-facing product team. Different people like different buzzwords, but I personally prefer the term <a href="https://softwareengineeringdaily.com/2020/02/13/setting-the-stage-for-platform-engineering/#:~:text=Platform%20engineers%20build%20systems%20that%20allow%20teams%20to%20build%20on.&amp;text=However%2C%20DevOps%20teams%20aren&#39;t,building%20these%20types%20of%20platforms."><strong>Platform Team</strong></a>. Think of a team at AWS which is building Redshift, or any other cloud provider building an analytics cloud product — that’s how internal Data Engineering teams can work, too. They don’t have to build a cloud data product that works well for every company. Instead they can use public cloud &amp; open source tools to build a data platform that works well for precisely your company, including all your weird communication structures, workflows, industry regulations, and the CEOs preferred color scheme.</p><p>Compared to a service team, which would own all 100 data pipelines for different departments, a Platform Data Engineering team would just own the core pipeline product. And this product must allow each department to create and manage their own pipelines in <strong>self service</strong>.</p><p>If a certain data pipeline or report crashes, the specific team that uses this pipeline should fix it themselves. They can’t? Well, then the Data Engineering team has to build better tools for their users, so they can manage (and recover) their pipelines on their own. After all, the platform is their product, and they’re responsible for its success.</p><h4>Inversion of responsibility</h4><p>This is an inversion of responsibility, and in its general thought very similar to the ideas behind DevOps. Data Engineers are no longer <em>directly</em> responsible for how quick they can build new pipelines. Now, they are responsible for how fast other teams can build new pipelines on top of their platform. It gives the developers, analysts, and data scientists more responsibility (and control!) over their own, and Data Engineering more freedom to build scalable self-service solutions for the entire company. Instead of being bogged down with dozens of service requests, they can now focus on building an integrated data platform,</p><p>If some other team approaches Data Engineering with a request, their first response should be “what is missing in the current platform for you to be able to do this yourself?”. Their key success metrics should reflect how fast new users can build pipelines and reports on their own. They should write release notes for big updates of their data platform products in the internal message boards, and do internal marketing and educational workshops to teach users how to best work with their platform.</p><p>It’s not an easy path to take, especially when there’s already an established “Data Engineering as a service team” structure. Explaining to other teams that, no, Data Engineering will not build your dashboard anymore, but they’ll teach you how to do it yourself… will create some organisational resistance. But, with some enthusiasm, a strong vision, and good communication, this can always be overcome. And, in the end, you’ll step by step work towards a more self-service oriented data usage for your company. And nobody will care whether you store it in ORC or Parquet.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34c34496ed8e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>