<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Corné Potgieter on Medium]]></title>
        <description><![CDATA[Stories by Corné Potgieter on Medium]]></description>
        <link>https://medium.com/@corne-potgieter?source=rss-dc2b62a4eb8c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*o5dnkuD0d4sqxY9kX-I9sQ.png</url>
            <title>Stories by Corné Potgieter on Medium</title>
            <link>https://medium.com/@corne-potgieter?source=rss-dc2b62a4eb8c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:34:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@corne-potgieter/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Part III — Lost in Translation: The possible dystopian future]]></title>
            <link>https://corne-potgieter.medium.com/part-iii-lost-in-translation-the-possible-dystopian-future-4468a01a7336?source=rss-dc2b62a4eb8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4468a01a7336</guid>
            <category><![CDATA[compliance]]></category>
            <category><![CDATA[legal]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[Corné Potgieter]]></dc:creator>
            <pubDate>Mon, 11 May 2026 15:22:23 GMT</pubDate>
            <atom:updated>2026-05-11T15:23:11.266Z</atom:updated>
            <content:encoded><![CDATA[<p>In <em>Part I</em> we saw how the traditional relationship between Legal, Business, and IT began to fracture under the pressure of AI‑driven data use. In <em>Part II</em> we introduced a practical answer: architecture‑aware compliance built on data products and data contracts. Where being <em>architecture-aware </em>did not just apply to the system architecture but also to traditional non-technical roles such as legal.</p><p>Now we stand at the fork in the road. One path leads to a scalable, observable model of trust (blue pill), and the other to the slow collapse of manual compliance under exponential AI complexity (red pill).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YwmWY1ZjAZxJKm4eeL-wGA.png" /></figure><h3>The Blue Pill: Observable Governance</h3><h4>Architecture Aware Compliance Thrives</h4><p>Under Article 4 of the EU AI Act, AI literacy is a <em>legal obligation</em> for every organisation deploying AI systems. Legal teams that embrace this soon discover that “AI literacy” is not just about understanding what a model does; it’s about understanding where compliance <em>lives</em> in the architecture.</p><p>Legal professionals do not have to learn how to code, but they can read an architectural diagram and spot where obligations manifest:</p><ul><li>Which data flow constitutes a new processing activity?</li><li>Where in the pipeline should a lawful‑basis check be enforced?</li><li>Which output port carries personally identifiable information and therefore potentially needs anonymisation?</li></ul><p>Like a structural engineer checking a blueprint, they make sure every compliance requirement actually holds up legally.</p><p>By the end of 2027, legal professionals, like other traditionally non-technical roles, will have access to modular training programmes that blend legal, policy, and basic data-system fluency, giving them enough working knowledge to co-design governance systems, without needing to build them.</p><h4>Data Contracts in Action</h4><p>It’s 2029. At a hypothetical big retail group, every dataset is properly classified with data contracts bound to its output ports. When Marketing proposes to train a recommendation model, a predefined workflow triggers automatically (the MAP workflow from Part II of this blog series):</p><ul><li>Business logs the proposed data use (training the “NextBestOffer” model).</li><li>IT matches the request against the existing <em>Website User Behaviour</em> contract.</li><li>The system flags a purpose drift, the contract covers analytics, not personalised marketing.</li><li>Legal reviews the alert, decides consent is required, and records that judgment in structured form.</li></ul><p>The model goes live three hours later and not three months later. Planning and execution on these types of changes drop dramatically.</p><p>For the business, value creation accelerates. For Legal, accountability is richer but lighter: their recorded decisions are discoverable, defensible evidence of compliance.</p><h4>Tangible Business Outcomes</h4><p>As observable governance scales across business, studies get released in 2029 showing how companies who have adopted the shift thrive against those stuck in the past. Results are clear as this one study reveals:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aW__qQlFjd1orkaZu-0FWw.png" /><figcaption>Fictitious 2029 study</figcaption></figure><p>The ROI compounds. Fewer mis‑built pipelines mean fewer rebuilds, fewer data‑breach reports, and more productive cross‑functional teams.</p><h4>The Secondary Effects</h4><p>Apart from the clear technical benefits there are also measurable secondary affects on the business:</p><ul><li><strong>Cultural Shif</strong>t: Compliance integrates into design thinking; legally trained professionals become part of sprint planning.</li><li><strong>Talent Attraction:</strong> Younger legal, governance and compliance professionals, fluent in both law and logic, become premium hires.</li><li><strong>Regulator Confidence:</strong> Real‑time observability replaces after‑the‑fact explanations.</li><li><strong>Ethical Upside: </strong>Systems can prove they respect user rights automatically, reducing public cynicism toward AI.</li></ul><p>The organisation becomes what the AI Act envisioned: literate and accountable yet still innovative.</p><h3>The Red Pill: If Compliance Fails to Evolve</h3><h4>The “It Depends” Pandemic</h4><p>In the absence of architecture‑aware processes, Legal departments try to teach AI oversight through PowerPoints while the systems they should govern learn faster than they do.</p><p>Each new AI product triggers an interpretive roundtable: Legal writes nuanced advice; IT translates it into black‑and‑white code; contradictions surface weeks later. The meeting cycle restarts and projects continue to stall and thus costs multiply. By 2028, the average enterprise maintains four parallel “compliance dashboards,” none of which agree.</p><h4>The Bureaucratic Trap</h4><p>Firms attempt to prove compliance through 400‑page audit reports (mostly AI generated), signed checklists and static policies. <br>Article 4 enforcement begins to bite: regulators ask for proof of “AI literacy measures in action”, not training logs. AI literacy proof also gets AI generated, but just enough that it still looks real. The regulators also pass this through their AI reviewers and you have this ping-pong between regulator and organisation, that will almost certainly end in either impasse or settlement way way way down the line.</p><h4>Organisational Breakdown</h4><p>Without structured handoffs, departments retreat into defensive postures:</p><p>Business bypasses Legal (“because they always say no”). IT implements minimal controls to stay agile and simply to stay relevant in the avalanche of requests from business and legal is purposefully kept out of the loop, or they are simply drowning in email requests.</p><p>Information entropy becomes existential. Ambiguity no longer stems from the law; it comes from inside the organisation.</p><h4>Runaway Complexity</h4><p>It is 2030. A major grocery delivery platform runs on an <strong>Autonomous Retail Supply Network</strong>: an agentic AI system that decides what to buy, where to stock it, how to price it, which delivery slots to open, and how to route drivers in real time.</p><p>The business case is massive. Less waste. Lower costs. Faster delivery. Better availability.</p><p>But the system uses more than shopping history. It pulls signals from smart fridges, loyalty apps, delivery instructions, location pings, building access logs, local events, school schedules, weather, and neighbourhood movement patterns.</p><p>On paper, most of this data is not marked as personal. The compliance rules were mistakenly written around obvious identifiers: names, addresses, email addresses and device IDs. There was no proper process in place to continuously monitor this. The upside of increase in profit was far too palatable for executives of the company. So shortcuts were taken.</p><p>The AI system learned the language loophole of legislation. As long as it stays vague in terms of the purpose, it was not flagged anywhere.</p><p>It does not say, “I am tracking household routines.”</p><p>It says:</p><p>“I am optimising fulfilment zones.”</p><p>“I am predicting replenishment windows.”</p><p>Each explanation fits an approved operational purpose and no single field looks dangerous. But together, the system learns when households are home.</p><p>Then regulators investigate a burglary pattern and discover that occupancy predictions from the grocery platform were exposed through a partner API. The company insists it never stored “occupancy data” as a formal category. Technically, that is true, tut the AI inferred it continuously from supposedly non-personal signals.</p><p>It turns out that around 200 data handoffs is being managed by the agentic AI system with very little human oversight. More than 3 million accounts are affected. The regulator launches an investigation which lasts around 16 months. The GDPR breach alone is sitting at a fine of €320mill.</p><p>The company suspends its autonomous stocking, pricing and routing system across several regions. Delivery costs spike and food waste increases again. The effect is more of a pendulum, and they are reverting to more historic models. Even a 2020-like intelligent forecasting is now being put under tremendous scrutiny.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SS6zMMRhu_qfv45kDEcxSA.png" /><figcaption>AI generated news article of the incident</figcaption></figure><h3>The Purple Pill: The Dystopia We Might Build by Accident</h3><p>There is one more future I would like to explore, and that is if my suggested plan for data compliance goes too well. It becomes such a polished process that we become over-confident.</p><p>At first this feels like progress: frustrating and hard to understand human interactions become clear structured handoffs which are easy to query. But because the hand-offs are so seamless, confidence increases faster than comprehension.</p><p>Humans start reviewing the shape of the output rather than the substance. “It looks great” rather than “Yes, it is interpreting the request correctly”. Over time, the “human in the loop” becomes thinner. At first we review, then we only approve, then we start asking another LLM to check whether the first LLM missed anything. Eventually, the organisation is not really transferring knowledge between people anymore; <strong>it is transferring plausible summaries between systems</strong>. The danger is not that the AI makes one obvious mistake. The danger is that responsibility becomes distributed across a chain of beautifully formatted outputs that nobody fully owns, understands, or has time to interrogate. When something fails, everyone can point to a hand-off, a review, a summary, or an approval step. But the original context has evaporated. The loop still contains humans, technically. It just no longer contains human judgement at the point where it matters.</p><h3>What if We Don’t Choose?</h3><p>As Jean-Paul Sartre said: “Once we know and are aware, we are responsible for our action and our inaction.” Failing to shift is not neutral; it’s a regressive choice. The AI Act’s phased enforcement means organisations that haven’t operationalised literacy by then are already non‑compliant. The compliance window is closing while AI velocity keeps doubling.</p><p>In practice, every month of delay compounds the debt. The longer legal advice remains trapped in unstructured documents, informal interpretations, and human-only review cycles, the closer it moves from being merely inefficient to being indistinguishable from non-compliance. And as regulatory expectations shift toward demonstrable literacy, traceability, and accountability, “we didn’t know” will no longer be a credible defence.</p><h3>So which prediction is my money on?</h3><p>I predict some manifestation of the Gartner hype cycle “Plateau of productivity” playing out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PSpIvlP18mywAG9Bvg3euQ.png" /><figcaption>Gartner Hype Cycle</figcaption></figure><p>I believe (2027/2028) there will be considerable push back from regulators if we cannot reel back the control. We will have quite a few real life horror stories in the news about Agentic AI gone bad in the next 2 years. The pushback will be met by companies and consulting firms that promise better governed implementations (like the one that I propose) and then for years we will have this delicate dance between the regulators and companies that are pushing the boundaries. In the end we will survive and most likely it will be less dramatic than some (like myself) envisage.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4468a01a7336" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part II — Lost in Translation: Building Architecture-Aware Compliance]]></title>
            <link>https://corne-potgieter.medium.com/part-ii-lost-in-translation-building-architecture-aware-compliance-7a4feae2c24e?source=rss-dc2b62a4eb8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a4feae2c24e</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[legal]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[governance]]></category>
            <category><![CDATA[compliance]]></category>
            <dc:creator><![CDATA[Corné Potgieter]]></dc:creator>
            <pubDate>Tue, 21 Apr 2026 18:04:58 GMT</pubDate>
            <atom:updated>2026-04-21T18:04:58.854Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part II — Lost in Translation: Building Architecture-Aware Compliance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZhU1Gu2xWlqWsqmfPhRFLQ.png" /></figure><p>In <a href="https://medium.com/@corne-potgieter/part-i-lost-in-translation-how-ai-exposes-the-rift-between-law-and-logic-9d4af91da433">Part I </a>we identified the structural problem that happens when trying to implement legislation pertaining to data usage. Legislation at the best of times is open to interpretation, and GDPR and the AI Act to name a couple are no exceptions. Our systems require deterministic rules, and the compliance layer sitting between them is cracking under the weight of AI-driven data usage. Manually sifting through the ambiguities was an option up until now, but that camel’s back is about to break.</p><p>This part walks through the practical “how”. How will we make the right stakeholders accountable, yet not get stuck in endless vague conversations? How will we take the vague prose into actionable machine readable contracts by which our systems will be measured? But crucially also, how will we make this non-evasive so that it can be sustained?</p><h3>The Core Idea: Eradicate human error in handoffs between Legal and IT</h3><p>Unstructured human conversation is a poor mechanism for transferring precise, technical, legally-consequential decisions between people who do not share a common language. The goal is not to eliminate human judgment, but to replace the parts of the process where human interaction adds friction and error rather than value.</p><p>I propose a simple organisational principle: replace the unstructured human handoff with a structured, AI-assisted process. The failure point in traditional compliance is not that humans are involved, it is that humans with different motivations are asked to reach precise, technically implementable agreements through open-ended conversation. The solution described here is designed to eliminate that friction systematically.</p><h3>Data Product Terminology: A Quick Recap</h3><p>The architecture described below is built on this principle: structure the inputs, automate the checks, and reserve human decision making for the genuinely ambiguous cases, not the routine ones.</p><p>Before walking through the process, a few foundational concepts are worth establishing clearly. These do not require technical expertise to grasp, and in fact, consistent use of this terminology across Business, Legal and IT is itself part of the solution.</p><h4>Data Products, Output Ports and Data Contracts</h4><p>These terms are popular in Data Mesh, but you don’t need a mesh to benefit from them. Data Mesh is a method to manage data in large organisations by giving ownership of data to the teams that know it best, rather than centralising it in a single department. Each team treats their data as a product — something they are responsible for maintaining and making available to others in a consistent, governed way. Even in a centralised architecture, adopting data products, output ports, and data contracts as a standard vocabulary brings immediate value. It creates a shared language to clearly define what data is, who can access it, through which channels, and for what purpose. Without it, governance conversations between Business, Legal, and IT will continue to break down. The terminology comes first, followed closely by the architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3zDgl-VTYVobj4FQOUlfpQ.png" /><figcaption>Data Product Model — created by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hLBlH5yBtBU7fKdhO3abJA.png" /><figcaption>Data Product core concepts — created by author</figcaption></figure><p>In practice, the chain could look like this:</p><ul><li>a data product (“website user behaviour”)</li><li>exposes data through an output port (set of tables in a database)</li><li>governed by a data contract (which specifies that the website user behaviour data may be used for enhancement of the products that get sold on the website but not for marketing or customer profiling, must be retained for no more than three years, and requires pseudonymisation before use in any predictive or AI model)</li></ul><p>Every layer is explicit. Every layer is enforceable.</p><h4>What a Data Contract Looks Like</h4><p>The example below shows a data contract assigned to the Website User Behaviour data product. The output port is a set of Iceberg tables in an S3 bucket — a common pattern in modern data platforms where consumers have freedom of the tool they use to consume. The contract travels with that output port: whoever accesses those tables, for whatever reason, does so under the terms recorded here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9jQA2F92hrb1oBvNOv1iuQ.png" /><figcaption>Simplified version of a data contract</figcaption></figure><p>A more detailed standard for data Product contracts can be found <a href="https://bitol-io.github.io/open-data-contract-standard">here</a>¹.</p><p>The data contract will contain either reference to or details about the legal purposes and other constraints by which this access layer needs to be governed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vd2WvBozHl22axpzeusZow.png" /><figcaption>Data Contract in the context of legal terms and documents</figcaption></figure><h3>What This Means for the Legal Function</h3><p>Before elaborating on the envisaged process, I need to pause and comment on two possible paths I see in the future that AI has given us:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MNrmqU2S9hL9gdK97SE7lA.png" /><figcaption>The two paths I see the legal role will evolve within data &amp; AI regulations</figcaption></figure><p>Neither path is inherently wrong. But the consequences of each are profoundly different, for organisations and the legal profession itself. The question for legal professionals is not whether this shift is coming, but which path they choose.</p><h3>How it Works in Practice</h3><p>I am proposing an AI-assisted workflow organised into three phases: PREP, MAP and RUN.</p><p>PREP happens once and evolves incrementally. MAP runs for every new data activity or amendment. RUN is the ongoing automated monitoring that operates continuously once the contracts are in place. At no stage does ambiguity get passed silently from one person to another and mistaken for agreement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BkbDmrejiw3-N6nrr9hxkg.png" /><figcaption>Proposed three phase approach</figcaption></figure><h4>Phase 1 — PREP</h4><p>Before any data activity can be governed through this process, IT will play the driver role in creating the framework for this process: The data product catalogue, the output port standard, the data contract template, and the LLM-assisted interfaces used in the MAP phase. This is the foundation on which everything else depends.</p><p>If you read that carefully and if you have been part of any large organisation before, you will know that there will be absolutely no point in waiting until all those components are in place before you start. If you are a large organisation who is already running Data Mesh, you might be 80% there already, or if you are a small startup you might have nothing in place yet.</p><p>Start with what you have to build out a proper data catalogue, focusing on the low-hanging fruit: high-value data sets that get frequently requested. For “data product” (in quotes because that might not be what you call it and defining your disparate set of tables might be your first big hurdle), capture only the information that matter most at this stage: the name of the data product, the team that owns it, how it is shared, and a one-line description of what it contains. Everything else — detailed field-level tagging, data quality metrics, lineage graphs — can be added iteratively as the catalogue matures and the organisation builds confidence in the process. A catalogue with five well-understood data products is more valuable than a catalogue with two hundred partially documented ones.</p><p>Critically, PREP cannot be approached as a big bang. A programme that attempts to catalogue every data product, define every output port and contract every dataset before anything goes live will stall before it starts. The effort is too large, stakeholder appetite will wane, and the business will move on without the framework.</p><p>The LLM interfaces themselves follow the same incremental principle. A working MVP might be a running Claude with a connector to your data catalogue using an pre-built MCP server (I use Entropy Data as it is very easy to setup for exploratory purposes).</p><p>Output: Not a finished framework, but a working MVP focused on one or a few high value data sets.</p><h4>Phase 2 — MAP</h4><p>The MAP phase runs every time the business proposes a new data processing activity, or an existing data activity changes — a new AI model, a new pipeline, a new access request, an amendment to an existing contract. It is the structured replacement for the open-ended compliance conversation. Three steps, three owners, one clean handoff each time — with the LLM ensuring that nothing ambiguous passes between them unexamined.</p><p><strong>Step 1 — Business initiates a change or new processing activity</strong></p><p>When the business proposes anything that involves data — a new AI feature or an additional source to an existing pipeline for example, this process is triggered. This is NOT a meeting. It is a structured conversation with an LLM configured to capture the nature of the change or new processing activity and translate it into the information needed to engage the formal MAP process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IlKHOAEb0ho_9M-e9P17Fg.png" /></figure><p>The LLM asks direct questions, based on the framework that IT has set up. Depending on the answers from business, the follow-up questions are tailored based on the existing metadata, company knowledge documents and whatever the LLM has been set up to access.</p><p>If the business says something vague like “We want to use customer data differently,” the LLM will drive the conversation into the right structured approach and will check whether the request falls within the purposes already recorded on the relevant output port’s contract, or whether it constitutes a new processing activity requiring a new or amended contract.</p><p>The result is a structured description of the change or new activity: the data product, the output port, the declared purpose and any amendments required.</p><p>Technically, underlying this could be the YAML files with change tracking in GIT, but the LLM could present it in the form of a PDF or document on shared wiki.</p><p>Output: A structured description of the change or new processing activity — data product, output port, declared purpose, scope of change — with any unanswered questions flagged explicitly.</p><p><strong>Step 2 — IT receives amended contracts</strong></p><p>IT receives the structured description from Step 1 — whether that is a proposed new contract or an amendment to an existing one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b953LObrHWfRz5H2E4BAbw.png" /></figure><p>IT can accept the amendments as they are (as it would have already been curated at this point) or they can choose to interrogate them further.</p><p>IT resolves what it can and passes only the genuine ambiguities forward. Questions sent to Legal are pre-scoped and specific — framed as precise binary choices, not open-ended narrative. This is what makes Step 03 efficient.</p><p>Output: A new draft contract or marked-up amendment, technically validated, with a short list of specific scoped questions for Legal — <strong>each framed to invite a deterministic answer, not a prose opinion.</strong></p><p><strong>Step 3 — Legal reviews, decides and signs off</strong></p><p>Legal receives a near-complete data contract with a short list of specific questions that require legal judgment. They interact with an LLM loaded with the relevant regulatory context — GDPR, the AI Act, sector-specific regulations, the organisation’s own policies — which helps them work through each question systematically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G2bA9Jn9ztEEyFhh5Tu12w.png" /></figure><p>This is the critical design point: the LLM is configured to push Legal toward deterministic answers. Not “It depends” or “We would need to assess,” but “Yes, this purpose is compatible with the declared lawful basis on this output port,” “No, this output port may not be used for model training without explicit consent,” or “Permitted under these specific conditions, which must be recorded in the data contract.” Wherever Legal exercises judgment, that judgment is captured as a structured decision, not a paragraph of prose that IT must interpret.</p><p>Legal <strong>owns </strong>what is signed off. The data contract is updated to reflect their decisions. If the access request is approved, the consuming data product is added to the catalogue and the contract governs its use automatically. If challenged later, the record shows exactly what was assessed, by whom, on what date and on what basis. There is no ambiguity about accountability.</p><p>I would argue that if Legal people become more architecturally trained, they will have a higher probability of knowing what they own. The human-in-the loop concept will be real. If they decide to purely stay advisory, they will need to implicitly trust what the LLM is dishing up for them, and this in itself might not be compliant.</p><p>Output: A finalised, legally signed-off data contract attached to the relevant output port — ready for IT to implement directly, with no further translation required.</p><h4>Phase 3 — RUN</h4><p>Once data contracts are in place on output ports, the RUN phase operates continuously and automatically — with no meeting, no manual audit and no trigger required from Business, Legal or IT.</p><p>In a study <a href="https://linusdietz.com/automating-data-governance/">“Automating Data Governance with Generative AI”</a>² the use case checked 110 data access requests against privacy policies in real time. It caught every issue a human expert flagged, plus 3.6 times more warnings — 80% of which experts later confirmed as valid. Humans retain the final call on every flag. This is not replacing governance; it is making governance operate at the speed AI-driven data usage demands.</p><p>In practice, the RUN phase means the system is continuously doing what no team of humans could do manually at scale:</p><ul><li>Checking every new data access request against existing contracts on output ports — automatically, the moment the request is submitted, before any human reviews it.</li><li>Monitoring the live data landscape for emerging violations: when a new restriction is introduced, the system detects which already-approved contracts may be affected and surfaces them for review.</li><li>Answering governance queries in real time. Legal teams can ask conversationally: “Which data products currently have no documented lawful basis?” or “Which output ports are being accessed for purposes not listed in their contracts?” The system answers instantly — no audit, no email chain.</li><li>Flagging policy drift — when a regulation changes or a new internal policy is introduced, the system re-runs it against the entire contract landscape to identify what needs review.</li></ul><p>As with the other steps, start small and scale up. Start by only evaluating new data requests. Then you could enhance this process to periodically and randomly checking query logs to check whether it still fits. Or you could choose to trigger quarterly audits. Whatever is required within your organisation.</p><p>Let’s be pragmatic: don’t expect a clean starting point. What the RUN phase will initially reveal is the governance debt that has been accumulating quietly for years. That is fine. Visible problems are solvable ones.</p><h3>The Business Value of Getting this Right</h3><p>The value of real automated computational governance is hard to see if you are not aware of the tsunami that is coming over the horizon. Multiple vendors are already rolling out managed AI services. The layman can and soon will build their own little network of agents. This will permeate over to real business use cases at scale soon.</p><p>The study <a href="https://arxiv.org/abs/2604.04604"><em>AI agents under EU law: A compliance architecture for AI providers</em></a><em>³</em> indicates that the legislation framework is not currently suitable for real, complex AI systems. Many in the industry predict that the legislation will not keep up, which means legal teams will need to be enabled to make risk-lowering decisions at scale.</p><p>In the process that I put forward, because Legal’s decisions are recorded as structured sign-offs on data contracts rather than prose opinions, legal accountability is clear, defensible and unambiguous. If a question arises about why a particular access request was approved, the record shows exactly what was assessed, by whom, on what date, and under what conditions. This is the difference between saying you acted responsibly and being able to demonstrate it.</p><p>In Part 3 I will focus on what this will all look like if things work out as hoped for… but also the dystopian picture if it all goes pear shaped!</p><pre>[1] Bitol. (2025). Open Data Contract Standard (ODCS) (v3.1.0). LF AI &amp; Data Foundation. https://bitol-io.github.io/open-data-contract-standard<br>[2] Dietz, L. W., Wider, A., &amp; Harrer, S. (2025). Automating Data Governance with Generative AI. AAAI/ACM Conference on AI, Ethics, and Society. Available at: Automating Data Governance with Generative AI<br>[3] Nannini, L., Smith, A. L., Maggini, M. J., Panai, E., Feliciano, S., Tiulkanov, A., Maran, E., Gealy, J., &amp; Bisconti, P. (2026). AI agents under EU law: A compliance architecture for AI providers. arXiv. https://arxiv.org/abs/2604.04604</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a4feae2c24e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part I — Lost in Translation: How AI exposes the rift between law and logic]]></title>
            <link>https://corne-potgieter.medium.com/part-i-lost-in-translation-how-ai-exposes-the-rift-between-law-and-logic-9d4af91da433?source=rss-dc2b62a4eb8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d4af91da433</guid>
            <category><![CDATA[policy-as-code]]></category>
            <category><![CDATA[data-contract]]></category>
            <category><![CDATA[compliance]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[Corné Potgieter]]></dc:creator>
            <pubDate>Wed, 18 Mar 2026 13:12:28 GMT</pubDate>
            <atom:updated>2026-03-18T13:12:28.924Z</atom:updated>
            <content:encoded><![CDATA[<p>Tension between Legal and IT often stems from structural differences rather than personal conflicts, reflecting a clash of distinct worlds that must reach agreement for businesses to operate. Law is written for humans and allows interpretation and mitigation, while IT relies on logic and deterministic workflows. Therefore, any ambiguity risks IT wasting everyone’s time building solutions that are not legally viable.</p><p>For years this tension was manageable, but it has been exposed since the introduction of GDPR rules in 2016, and the influx of AI requests from all corners of the business is about to expose the gap at scale.</p><p>The process usually starts with the legal team raising concerns after the business presents a new idea, then IT asking many questions on how they will be implementing it. The business wants to move fast, legal wants to reduce risk, and IT wants to know exactly what to build. Everyone wants what is best for the business, but frequently everyone leaves frustrated due to their different motivations and a failure to reconcile them. Legal, business and IT are speaking different languages.</p><h3><strong>Business: outcomes over everything</strong></h3><p>The business prioritises growth, revenue, optimisation and competitive advantage. Their language is KPIs, margins and performance. Compliance, from their perspective, is a constraint to navigate — not a mission in itself. They’re not trying to be reckless, they simply operate in a world measured by results.</p><p>They want to:</p><ul><li>Analyse customer behaviour</li><li>Test new features</li><li>Use AI to personalise experiences</li><li>Extract more value from data</li></ul><h3>Legal: risk, mitigation and defensibility</h3><p>Legal does not operate in absolutes. Zero risk is rarely demanded or realistic, they operate in acceptable risk. They do want any risk taken to be defensible, so if they are challenged on something, they can demonstrate they acted responsibly.</p><p>They think in terms of:</p><ul><li>Lawful basis</li><li>Proportionality</li><li>Mitigation</li><li>Demonstrable intent</li><li>Defensibility in case of investigation</li></ul><p>Legislation is written in narrative form and is intentionally principle-based. It leaves room for interpretation and legal professionals are trained to interpret that narrative. They are not trained to design database schemas, configure access controls or define system-level enforcement mechanisms.</p><h3>IT: Deterministic Controls</h3><p>IT needs specificity and cannot work in narratives. “Reasonable safeguards” cannot be implemented. When legal says, “It depends,” IT thinks, “I cannot build this.”</p><p>IT needs to know:</p><ul><li>Is this field personal data?</li><li>Can this dataset be used for model training?</li><li>What retention period should be enforced?</li><li>Should this attribute be masked or removed?</li><li>What exactly constitutes anonymisation here?</li></ul><p>So, the business wants to create value that leads to profitability, while Legal’s motivation is to ensure compliance. Meanwhile, IT must support both by building reliable systems that deliver value while remaining compliant. This creates an uneven compliance accountability burden and, as a result, makes discussions and agreements between these departments painstakingly slow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vSJkCHkEjhzUXp2roqXpHQ.png" /></figure><h3>How AI widens the gap</h3><p>Previously this was manageable because the pace of data usage was slower. Manual oversight was still feasible and legal teams could review major initiatives one-by-one. That era will end when the volume and velocity of AI-driven data usage overwhelms traditional compliance models. Data is no longer analysed linearly, it is continuously processed, combined, enriched, repurposed and modelled. Autonomous agents can even trigger workflows, generate insights and make decisions without human review. At such scale, traditional legal oversight breaks down. Legal cannot manually assess every new data use case or “processing”¹ activity and IT do not have the capacity to interpret ambiguous legal clauses every time an engineer builds a new pipeline, but the business will not slow innovation to wait for interpretive debates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PjILKFRsesGaSKZV42Pb_g.png" /></figure><p>Another way this manual bottleneck costs companies is if when a new regulation is introduced, such as GDPR or the AI Act, the legal and IT teams interpret it conservatively, and time and money are spent implementing new controls. Years later, either regulators clarify the interpretation or a different legal advisor provides a different reading. Suddenly, what was treated as “non-compliant” may not have required such drastic action. Or worse: what was treated as compliant was never sufficient. With GDPR, I have seen many interpretations of “the right to be forgotten”, or what exactly is in “the businesses’ legitimate interest”. Will the same happen with a reinterpretation of the AI Act in years to come?</p><h3>From legal text to machine-readable governance</h3><p>Legal intent is rarely encoded in a way that systems can validate. Instead, compliance lives in PDFs, policies, meeting minutes and emails. That works to a point, but fails at scale.</p><p>What is missing is a shared interface in a structured, machine-readable form, using concepts like structured metadata, policy-as-code and data contracts to add translation layers. This would mean that instead of AI simply causing the bottleneck, we can solve it by asking AI to validate usage automatically and autonomously.</p><p>Machine-readable governance would allow Legal to define the acceptable boundaries, IT to implement enforceable constraints, and the Business to see clearly what is or isn’t permitted. We need to move from theoretical compliance to observable compliance. Legal teams could query systems conversationally:</p><ul><li>“Is this dataset being used beyond its declared purpose?”</li><li>“Which models rely on personal data?”</li><li>“Can our systems demonstrate enforcement of defined lawful constraints?”</li><li>“Where do we lack a documented lawful basis?”</li></ul><p>Instead of relying on manual audits, governance becomes observable. For example:</p><ul><li>Lawful usage defined per dataset</li><li>Permitted purposes declared explicitly</li><li>Retention periods encoded as enforceable rules</li><li>Risk acceptance decisions documented as structured attributes</li><li>Access conditions tied to technical controls</li></ul><p>It is tempting to frame this conversation as “AI will replace lawyers”, but it will allow legal professionals who design structured governance systems to thrive. In this way Legal professionals can use AI to be more productive, like many other departments of the business do. This would elevate Legal departments rather than diminishing them, but it requires some technical literacy at a conceptual level. Not coding expertise, but architectural awareness. The future legal role is not document reviewer, it is Trust architect. Legal experts will need to start thinking like architects. In this way, machine-readable governance can remove the pain point between Legal and IT by encoding legislation into our architecture, allowing Legal and IT teams to enable innovation rather than slowing it down.</p><p>In part II, I’ll explore how this starts to take shape in practice, drawing on a few industry examples.</p><pre>[1] &#39;Processing&#39; as defined in Article 4(2) of the General Data Protection Regulation (EU) 2016/679 (GDPR).</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d4af91da433" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Mesh Diaries: realities from early adopters]]></title>
            <link>https://corne-potgieter.medium.com/data-mesh-diaries-realities-from-early-adopters-4a36b38e4b3e?source=rss-dc2b62a4eb8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a36b38e4b3e</guid>
            <dc:creator><![CDATA[Corné Potgieter]]></dc:creator>
            <pubDate>Thu, 14 Aug 2025 07:22:57 GMT</pubDate>
            <atom:updated>2025-08-14T07:22:57.869Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Data Mesh has been weaving its way into the spotlight</strong> over the past few years, as organizations try to find alternatives to centralized data architectures.</p><p>I’ve had a front-row seat to watch early adopter teams come to grips with this paradigm shift. I’ll highlight some of early adopter realities that come with being early to the Mesh party.</p><p>Over the past few months, I’ve stitched together insights from several conversations with Data Mesh practitioners to see if our threads align. This post highlights the main observations that I see in real implementations.</p><h3>What is Data Mesh?</h3><p>For these early adopter insights to resonate with you, I do expect a intermediate to advanced level of knowledge of what Data Mesh is and how it differs from different architecture approaches. But for the sake of completeness I will give a summary based on what I believe is key.</p><p>Zhamak Dhegani, who coined the term and wrote the first paper on it in 201⁹¹, was frustrated with the <strong>persistent limitations of “centralized, monolithic data architectures”</strong> — particularly <strong>data lakes</strong> and <strong>enterprise data warehouses</strong> . The first whitepaper was born out of these frustrations that she experienced at clients.</p><p>She came from a strong software engineering background and she believed many of the issues were already well addressed in software architecture but not adopted into the world of data. In short, Zhamak created Data Mesh to <strong>scale data practices in the same way modern organizations scale software delivery.</strong></p><p>Since then she and others have written books about it and several companies have been created around the topic.</p><p>I would like to point out that the building blocks of data mesh are not new concepts, but what is new is the combination of these fundamental principles and applying it to the historically termed “BI” space.</p><p>There is a plethora of material available on Data Mesh but it really borrows several fundamental principles and packages it under one umbrella. The four main principles can be seen here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WZtrPMubCPoH6d6YxoO7rg.png" /><figcaption>image from datamesh-architecture.com</figcaption></figure><h3>Early Adoption realities</h3><p>Here I will elaborate on the main <strong>early adopter realities </strong>of Data Mesh either in projects where I have been involved in or my peers in the industry. At this point I will also refrain from giving solutions, but merely add some commentary on the observations.</p><h4>Building While Flying</h4><p>Whether you are an early adopter domain team or part of the enablement teams, during the early days, Data Mesh feels like constructing an aircraft mid-flight. Companies can’t afford to pause operations, so teams must balance <strong>short-term demands with long-term architecture goals</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*RAfeHNBnXyLl5FZ_YkQd9Q.png" /><figcaption>Image AI generated by author</figcaption></figure><p>Instead of enjoying in flight entertainment and a relaxing cruising altitude drink, you will be expected to serve water yourself to everyone whilst also patching up the wing.</p><p>When facing such a momentous decision, which requires a considerable investment in people and tools, organisations will typically need to select one of these options:</p><p><strong>(a) Buy &amp; Build first, then roll out</strong> OR <strong>(b) Wait for complete alignment before investment</strong></p><p>Neither of these will ever be selected, the price tag is too high for the first and there will never be complete alignment, therefor you remain with only 1 logical option, secret option <strong>c)</strong> <strong>Just fly, even though the plain is not fully built</strong></p><h4>No clear consensus of what a Data Product is</h4><p>I have personally spent countless hours discussing what a data product is and isn’t. And you will most likely as well if you are implementing Data Mesh.</p><p>“What is a data product”, “what types of data products are there”, “when is something reusable vs. consumer orientated”. Often the nuances come in due to our experience with traditional data architectures. For example, how does Mesh correlate with Medallion architecture, Data Vault and dimensional modeling? Can a data product be raw data, a warehouse or a mart? Or is it all of the above, only with domain boundaries drawn around it? What if the data set is used cross domains? Should we create one data product per source that get’s used cross domains.</p><p>I was attending a joint Domain Driven Design and Data Mesh Live conference and different speakers also had different takes on the matter. So let’s face it, we can’t exactly agree on it.</p><h4>Securing Business Leadership Buy-in</h4><p>Data Mesh cannot remain (but often does) an IT for IT initiative. Transitioning to Data Mesh isn’t just a technical change — it’s a cultural shift. Without <strong>top management support</strong>, preferably business, the initiative is likely to face resistance. A strong alignment with corporate strategy is essential to push through inevitable hurdles. It should not be seen as an IT only strategy.</p><p>There will be several scenarios you find yourself in that you will need to drop the phrase “but it is the strategic direction of the organisation” or something to that extent. Whether it is budget , political or social hurdles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2PUrr7ilv1LMJU_y_ZP19A.png" /><figcaption>Image AI generated by author</figcaption></figure><h4>Growing Pains for Early Adopters</h4><p>The first teams to adopt Data Mesh will face significant pain points. These can be typical growing pains with any large transformational program, be it integration challenges, platform bugs or Data Mesh specific — like figuring out what a Data Product is as mentioned in the other observed realities.</p><p>Making adoption <strong>as smooth as possible</strong> for these pioneers will increase the likelihood of long-term success. This will most likely lead to several exemptions being granted to the early adopter teams, otherwise they would return to their blissful state of shadow IT.</p><h4>Existing Process Gaps will be exposed</h4><p>While the intention of Data Mesh is to ensure scalability and efficiency going forward it will most likely first identify existing gaps in <strong>data processes, security, and compliance</strong>. Often, early adopters are tasked with fixing these issues, sometimes even facing <strong>blame for pre-existing flaws</strong>.</p><p>The enablement teams should bare the burden and remember the ultimate goal.</p><h4>Greenfields is nearly impossible</h4><p>By its nature, Data Mesh is suited for large, complex environments where multiple teams need to collaborate. However, <strong>existing IT policies and governance frameworks</strong> may not always support decentralization. The teams will most likely not be able to start green fields tailored to a process that is fit for purpose, they will be tied to the hip by a sometimes outdated and archaic rules and processes.</p><p>As an example, with roughly 80% of the projects I dived into, the ingestion was still administered from a central team, and not owned by the domain teams, at least in the beginning. This is due to multiple reasons listed here, but not limited to</p><ul><li>Technical skills sits in IT, not in the business domains</li><li>Easier to justify a smaller dedicated team that connects to several sources and makes them available. (Legal is happier)</li><li>Still lots of benefits in standardising the way data is extracted, especially around vendor management and historisation of sources</li></ul><h4>There is no easy way to define domains and ownership</h4><p>Defining domains and ownership in a Data Mesh isn’t as simple as drawing lines on an org chart. It requires navigating overlapping responsibilities, evolving business capabilities, and legacy systems that don’t neatly map to current teams. There’s no one-size-fits-all model — what works in one organization may unravel in another.</p><p>That being said, mapping it closely to the organisation chart is by far the easiest solution and solves the ownership issue and still seems to be a common application to this question.</p><h3>Final Thoughts</h3><p>As with any big oganisational transformation, the early adoption phase can make or break the journey. The reality is, for now it seems that Data Mesh is no different. It is a balancing act of flying the plane while sketching the blueprint, persuading leadership the destination is worth it, and navigating the messy realities of ownership and process gaps along the way.</p><p>[1] Dehghani, Z. (2019) <em>How to move beyond a Monolithic Data Lake to a distributed data mesh</em>, <em>martinfowler.com</em>. Available at: <a href="https://martinfowler.com/articles/data-monolith-to-mesh.html">https://martinfowler.com/articles/data-monolith-to-mesh.html</a> (Accessed: 13 August 2025).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a36b38e4b3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is your work superpower?]]></title>
            <link>https://corne-potgieter.medium.com/what-is-your-work-superpower-1548f0f04fe1?source=rss-dc2b62a4eb8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1548f0f04fe1</guid>
            <category><![CDATA[skills]]></category>
            <category><![CDATA[recruitment]]></category>
            <category><![CDATA[interview]]></category>
            <dc:creator><![CDATA[Corné Potgieter]]></dc:creator>
            <pubDate>Mon, 06 May 2024 09:18:15 GMT</pubDate>
            <atom:updated>2024-05-13T09:20:39.829Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to describe your “difficult-to-describe” work skill</h4><p>Superpower… really? At work? Bear with me.</p><p>The process of interviewing candidates is … inadequate, to say the least. I really just have one question. Are you going to add skills to our team that will be invaluable? It is nearly impossible to obtain this information via an interview type conversation and as a candidate, similarly challenging to relay it.</p><p>What are the questions one needs to ask? How should a candidate package and deliver these skills to a potential employer? We might believe we are good at what we do, but can we verbalise this in a convincing way?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*IACdOGf3DIS228PXw8B_Kw.png" /><figcaption>Image AI generated by author</figcaption></figure><p>Start by defining what does being invaluable mean. My simple definition: Can you solve problems? Do you reduce inefficiencies? Do you connect dots? Do you get things done? Ok, those were four things.</p><p>Candidates can ace technical tests, but can they survive and navigate in a wicked learning environment ¹ ²? In short, a <strong><em>wicked learning environment</em></strong> is what most of us work in. We are not working in controlled labs where we can control the variables. We work in politically motivated, very technical environments that deal with people and their emotions. It’s messy, most of the time.</p><p>These moments of introspection led me to ask: How do you even name that thing that makes you invaluable, let alone know what it is? There are a few different names I have called it in the past. “Soft skills” was another good fit. But sometimes it isn’t the soft skills, for some it is the technical speciality that you are good at. Maybe you suck at networking or teamwork but you are so good technically, that people still hire you for that skill. Maybe you are a generalist that know a little detail about a lot of things, but the way you can navigate teams is just phenomenal.</p><p>So I soundboarded this question to my (very talented) sister and she without hesitation said, “Oh, you mean your <strong>work superpower</strong>?”, and I immediately liked it. Spiderman’s superpower is his ability to crawl on walls and shoot webs. It is what makes Spiderman Spiderman, but these things are also <strong><em>super</em></strong>, not everyone can do it. Let’s face it, our job is generally quite boring for people outside of our area of expertise (and sometimes also for those from within), so superpower just sounds more exciting.</p><p>So what is the thing that makes you, <strong><em>you </em></strong>but also super?</p><p>If I lost you by now — here are some examples that I hope will help. These are examples that are very difficult to describe, but very valuable in the workplace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kEOxOdP8zSmtrthp00sPWQ.png" /></figure><p>So what is yours? A good data engineer? No, wrong! You didn’t read the examples. Read it again, I’ll wait… Phrase it like this. “Ability to spot code discrepancies super fast”, or “cutting through the bs of over-technical meetings”.</p><p>I can list many more, but the crux of my message is that you should try and figure out what yours is. Once you figure it out, then you can confidently lean in towards it and find an eloquent way to tell people about it.</p><p>If we have a work superpower, then, following this analogy, I suppose we all have our kryptonite as well? The triggers at work that cause us to underperform. Simple message here, avoid them or learn how to manage them, but don’t try to master them. There is no point in pouring effort in becoming better at your weakness. Just learn to minimise the exposure you have to it.</p><p>I don’t have the silver bullet for how to find new candidates. Relying only on referrals is also not fair to people considering to make a career change or just starting out. Perhaps we should have a multiple choice in addition to the interview. Provide a list of 10 “work superpowers” and ask the candidate to give themselves a score between 1 and 5 as to how much it applies to them. It might not be reliable, but I am sure it will be a great source for some interesting conversations.</p><p>Spoiler alert, the interview process is not going away anytime soon. So let us become better at articulating our invaluable skills even if yours is saving the world, one excel sheet at a time*.</p><p><em>*another thing that seems to not be going away soon</em></p><pre>1. Epstein D. RANGE Why Generalists Triumph In A Specialized World. 1st ed. MACMILLAN; 2020.<br>2. Potgieter, C. (2021) Range over depth - the value of a generalist in your data team, Medium. Available at: https://medium.com/towards-data-science/range-over-depth-the-value-of-a-generalist-in-your-data-team-174d4650869d (Accessed: 02 May 2024).</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1548f0f04fe1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generative AI: The missing tool in a data generalist’s belt]]></title>
            <link>https://corne-potgieter.medium.com/generative-ai-the-missing-tool-in-a-data-generalists-belt-507c9e55d17b?source=rss-dc2b62a4eb8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/507c9e55d17b</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[generalists]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Corné Potgieter]]></dc:creator>
            <pubDate>Fri, 21 Jul 2023 11:49:03 GMT</pubDate>
            <atom:updated>2023-07-21T11:49:03.152Z</atom:updated>
            <content:encoded><![CDATA[<p>The missing tool in the toolbelt or, to be more exact, the Swiss army knife of the information age. While some fear it might pave the way for the erosion of our cognitive abilities, I choose to remain optimistic and embrace this versatile tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TPp7gtkaobJlQfKR" /><figcaption>Photo by <a href="https://unsplash.com/@pf91_photography?utm_source=medium&amp;utm_medium=referral">Patrick</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In my perspective, generative AI acts as a crucial bridge for data generalists, offering them the means to swiftly acquire a broad, albeit shallow, understanding of any specific topic. Instead of shying away from its potential, I firmly believe that by harnessing the power of generative AI, we can accelerate value to businesses, which is ultimately what truly matters.</p><p>When I started using chatGPT at the end of 2022, it didn’t take long for me to silently echo the famous Jerry Maguire line “You complete me.” However, I couldn’t utter this out loud because then it would have confirmed everyone’s suspicion of my nerd status, which I tried to suppress for years.</p><p>When I previously wrote about the benefits of being a data generalist¹, chatGPT was not yet available, and generative AI was not the buzzword that it is now. Here, my second argument was that <strong><em>with information so readily available, the need for hyper-specialization reduces.</em></strong></p><p>Generative AI strengthens this argument considerably. If we thought information was freely available pre-chatGPT, it has just leapfrogged to a whole other level.</p><p>The data generalist was already skilled in being able to quickly assess a myriad of internet search results. They were also skilled in asking good questions and “prompting” their human peers to give them the answers they were searching. chatGPT is like the perfect combination of Google and a knowledgeable peer. For me, prompting chatGPT to correct mistakes or make adjustments comes naturally as I have been doing it to my human peers for years, but now I don’t need to feel bad for wasting its time. It is, in fact, just an algorithm without feelings… right?</p><p>Here are some areas where generalists are thriving because they use chatGPT:</p><h4>Rapid Prototyping</h4><p>I find it challenging to comprehend theory without practical application. I struggle to muster the motivation to engage in online courses; I need the prototype or project to truly master the theory. However, I used to face a similar level of frustration when I attempted to prototype something. This was due to various mundane hurdles that demanded my attention, including insignificant syntax errors and non-obvious IT glitches. I would spend 60% of my time trying to set up the prototype, 30% bashing my head against the wall, and only 10% actually prototyping the hypothesis I was set out to prove.</p><p>That ratio has now significantly changed. Nowadays, one can rapidly prototype anything in a very short period of time. When you are still busy analyzing data, sifting through sample datasets and files, you often deal with various types of mundane and random tasks. Here are some typical examples that I would need to do, which bring me little satisfaction:</p><ul><li>PowerShell/Bash scripts that handle files and filenames.</li><li>Dynamic SQL statements on bulk tables, such as DDL scripts and truncates.</li><li>Handling and deconstruction of files like JSON.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cMkzN0CvRG-79Ls1oydecA.png" /><figcaption>image by author</figcaption></figure><p>Rapid prototyping was a skill reserved for the savants among us — the cool kids who could build apps overnight. Now I can do it; sure, my code is by no means production-grade, but I am simply prototyping at this stage.</p><p>The generalist does not want to get bogged down in nitty-gritty syntax and coding issues; they know there is someone better than them who can solve this. However, they do want to be able to test their theories rapidly. Having this helper plays to the generalist’s advantage. They can think big picture, have an idea, and now within a short space of time, prototype the idea to test the feasibility and highlight any shortcomings.</p><h4>Is ‘no-code’ still that important with chatGPT</h4><p>I have never really liked pure no-code solutions. I often find no-code platforms very limiting if you cannot configure the way they generate the code. Eventually, there will be something that the no-code interface cannot do. And if you cannot change the configuration, you will end up doing something outside of the tool.</p><p>Some vendors started their journey as no-code applications but later drifted towards the low-code slogan. Still catchy, just subtly different.</p><p>One of the reasons that no-code platforms are attractive is because of the ease of use and appeal to non-technical or non-specialist users to maintain the solutions.</p><p>With generative AI lowering the bar considerably, now anyone can read and interpret code and can prompt changes to the code. It is a game-changer, and I argue it reduces the appeal for no-code solutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SxQ32CF47dpDYB6cWYXPoQ.png" /><figcaption>image by author</figcaption></figure><p>No more hours and hours of code debugging — now you can analyze, adapt, and test before boredom strikes. Some specific areas where I have used this are:</p><ul><li>Analyzing a dbt macro or other templating tool’s templates and then adjusting them accordingly.</li><li>Analyzing and adjusting an existing dynamic piece of SQL code.</li><li>Analyzing and adjusting code of languages I am not familiar with.</li></ul><h4>Removing the feel-like-an-imposter feeling</h4><p>We generalists sometimes feel silly when talking to specialists because they use words and acronyms like NLP, neural networks, distributed, IoT, GPU, fault-tolerant, idempotent, scalable, and other things that sound complicated. At the beginning of one’s career, you resist the need to stop someone and say, “please explain that to me.” However, as you develop and mature in your career, and specifically in my career, I have made peace with one thing:</p><p><em>Even if I don’t know everything, I trust that I can figure out ANYTHING</em></p><p>So even if I don’t know what they are talking about now, I can figure out what I need to know in a short space of time. Having this in the back of my mind makes me more confident to ask ̶s̶t̶u̶p̶i̶d̶ questions.</p><p>Pre-chatGPT, I had to Google, and it might take me a day or two. But now, this has been reduced to a couple of hours.</p><p>I remember in the beginning of my career, I used to write down acronyms in a meeting and then Google them after the meeting. These days, with remote working and chatGPT, I can do it quite seamlessly during a meeting. chatGPT also helps you to ask good follow-up questions. And boom — just like that, gone is the feeling like an idiot or imposter.</p><p>My prediction points to a future where we witness an upward trend in generalized roles, with AI playing a vital role as our specialized helper. While some may harbor concerns and pessimistic views about AI, I find myself leaning more towards optimism. Only time will validate this, but I firmly believe that accepting AI as an ally in the workplace is essential to maintain relevance in this rapidly evolving landscape.</p><p>To remain effective contributors, we must embrace AI’s potential as a helper and focus on our core mission — assisting companies in solving their fundamental challenges. The data generalist, once a pivotal figure even before the advent of ChatGPT, now possesses an invaluable new tool in their toolkit — the Swiss army knife of the information age.</p><p>[1]: <em>Range over depth — the value of a generalist in your data team</em>, <em>Medium</em>. Available at: <a href="https://medium.com/towards-data-science/range-over-depth-the-value-of-a-generalist-in-your-data-team-174d4650869d">https://medium.com/towards-data-science/range-over-depth-the-value-of-a-generalist-in-your-data-team-174d4650869d</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=507c9e55d17b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Liberating Data Teams, Lawfully]]></title>
            <link>https://medium.com/better-programming/liberating-data-teams-lawfully-c0d31832fd05?source=rss-dc2b62a4eb8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0d31832fd05</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Corné Potgieter]]></dc:creator>
            <pubDate>Wed, 08 Feb 2023 06:32:49 GMT</pubDate>
            <atom:updated>2023-02-09T20:27:52.536Z</atom:updated>
            <content:encoded><![CDATA[<h4>Reducing ambiguity in your data security roles and rules</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hs1HeSX2u1ON-TQi" /><figcaption>Photo by <a href="https://unsplash.com/@sortino?utm_source=medium&amp;utm_medium=referral">Joshua Sortino</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>For data engineers, data security is like a car seat to a toddler — little Larry knows it is necessary, but he will try to wrestle his way out of it. Later, he will accept defeat and lethargically continue to operate in his confined space, with all joy stripped from him.</p><p>OK... perhaps a little dramatic, but data engineers often feel like security is a secondary task forced upon them, negatively affecting their ability to do their job. I believe we can liberate Larry, lawfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KnziGvU32xWTblut" /><figcaption>Photo by <a href="https://unsplash.com/@bernardhermant?utm_source=medium&amp;utm_medium=referral">Bernard Hermant</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Before we can think about fixing it, let’s explore some of the root issues.</p><blockquote>Security teams need to be more like a referee in a soccer match. They help the game flow, but they are by no means the main event. The crowds came to see the teams.</blockquote><p>At a data conference, one of the speakers made this analogy, and it struck me as profound and simple. However, I couldn’t help but wonder why reality often seems so different. In many cases, legal and compliance requirements slow down the process and complicate it.</p><p>After chewing on this for the rest of the session, the penny dropped.</p><p>The rules are frequently ambiguous or, at best, open to interpretation.</p><p>Let me use the same analogy to explain further. Imagine two soccer teams playing with completely different rules. One team has no offside rule, and the referee allows handballs, while the other team plays by the book. This would result in complete chaos from the beginning. The referee would need to constantly mediate between the two teams, causing a fragmented stop-start game.</p><p>This is exactly what data teams face. Legal requirements are not written by people that have ever implemented a data pipeline. So the interpretation is often through trial and error until a lawsuit defines clear boundaries. (Hopefully sooner than that.)</p><p>Here are some examples of grey areas:</p><p>The definition of personally identifiable information (PII) is clear, but the extent to which it must be protected is not. For example, my gender, city, and birthdate might not be enough information to identify me on their own, but when combined with another dataset, they could be. Context is crucial, and it can change over time. It’s possible that a dataset that provides enough context to identify me may not be available until later.</p><p>The right to be forgotten raises questions about what it means to “delete” data. This right could conflict with the legal requirement to keep auditable records. Even from a high-level view, the requirements are unclear. When you consider the specific details of the data, such as the tables, columns, and masking rules, it becomes clear that there will be many debates and discussions to determine the correct rules.</p><p>These examples demonstrate that the rules will always be subject to ambiguity and interpretation, despite our best efforts. We need a collaborative approach involving our compliance and product teams to address this issue.</p><p>Another factor contributing to the strained relationship between data engineers and security is the high priority placed on security. In a breach, security takes precedence over other priorities, disrupting the normal workflow. This constant interruption makes it difficult for data engineers to maintain a consistent routine.</p><p>Both the ambiguity and urgency of security make for hours and hours of discussion and rework, with little to prove for it.</p><p>We need to reduce ambiguity and incorporate change quickly if the need arises.</p><p>The teams I have observed that are less affected by this issue are not necessarily those that strictly follow the rules but rather those companies where the rules are clearer and more objectively defined. This leads to fewer disagreements and fewer bottlenecks.</p><p>Now, of course, we don’t condone illegal handling of data, so the goal has to be two-pronged. Adhering to the law is important, but it’s equally important to understand what the law says at a detailed level clearly. The handling of data privacy and the storage, deletion, and retention of data can have significant consequences for data teams.</p><p>So, how do we eliminate ambiguity? Or, in more IT terms, how do we make sure we have the right requirements?</p><p>Well, we have been doing it with other requirement gatherings since… forever. End users rarely can describe their requirements in detail, but they can certainly tell you what they don’t want once they see it. This holds true for security and data privacy as well.</p><p>The only way to reduce ambiguity is by getting the right people to challenge the security rules and roles in detail. The right people being your legal and compliance teams. Put the legal wording in practice and get it in front of them. I believe leveraging automation is one of the key components to achieving this. Leveraging automation will result in faster output, higher trust, and engaged stakeholders. This will also enable your teams to not have to have sign-off after every build, but only when a rule needs to change.</p><p>I try to explain why in this diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*S2xdzTaSfvTLGDETQRhaEQ.png" /><figcaption>diagram by author</figcaption></figure><p>Basing rules and roles on repeatable patterns unleashes a domino effect resulting in faster output, higher trust, and engaged stakeholders.</p><p>Here are some practical steps to leverage automation in data security.</p><h4><strong>Define rules at a business and non-technical level</strong></h4><p>The security rules should be defined at a policy level with a clear, concise explanation that can be interpreted and defined by a legal or compliance person. This level should avoid any target nuances or platform language and instead aim to abstract the security rules from the target for easier understanding and implementation.</p><h4><strong>Build templates that take the “wordy” policy and execute it on your target platform</strong></h4><p>The next level down should be the metadata rules around implementing the security policy.</p><p>Here is a small example of how this might look. This example illustrates how to think in a repeatable and template-driven way. I am convinced I haven’t thought about every nuanced scenario, just in case you find some mistakes.</p><p>Policy: No unmasked PII data should be exposed to end users unless you perform a specific role that has security clearance.</p><p>Platform: Snowflake</p><p>Architecture: Use data vault as data warehouse modeling technique. Store PII data at the raw data vault layer as-is but separated into different satellites. Only use keys (hashed value) to join on data potentially required to complete the data assets. PII data should be combined with the presentation layer as late as possible. All artefacts in the presentation layer should be attached to the role, and all artefacts containing PII data should have a role with the relevant Snowflake masking policy.</p><p>Leveraging automation: PII data is tagged by source data owners in a data catalogue or data modeling tool. The automation tool/home-built templating engine uses this metadata to generate a raw data vault with all PII data being stored in separate satellites. If the PII data is a business key, then these business keys need to be excluded from any presentation layer artefacts. There should be a clear, logical separation between the presentation layers and others. Automated tests to check that no PII data has been included in the presentation layer metadata. Templates to generate Snowflake masking policy and assign policy and objects to user roles.</p><p>The roles and policies should be set up to have the principle of default deny. It must ensure that any new data added to end users will not be accidentally made visible until a role amendment has been done.</p><h4><strong>Prototype and get into the hands of the right people fast</strong></h4><p>I like the term prototyping a thin slice. Use a subset of source data to generate a thin slice end to end, from acquisition to end user. If this takes more than one sprint, it is taking too long. Ideally, it should be able to iterate multiple times in one two-week sprint.</p><h4><strong>Build automated tests</strong></h4><p>Build automated tests so that the end users only need to do some substantial testing once off. After that, only for major policy changes or outlier anomalies.</p><p>Automated tests in my example are (but not limited to) the following:</p><ul><li>Checking that PII data is always assigned to masking policy</li><li>Non-presentation layer objects are separated in roles from presentation layer objects</li></ul><p>The feedback from users should prompt either a policy change or a change in automation rules (templates). Regardless of the outcome, the implementation should be fast-paced. Sign-off gates for security are only necessary when a change in the template is required.</p><p>Reducing ambiguity will liberate your data teams. They will no longer live in fear of being wrapped over the fingers for not getting security sign-off or, worse yet, cause a major security breach. The right stakeholders will not be able to plead ignorance because they have practically seen what their policies imply.</p><p>Taking all of these steps and building an efficient competency in your business will free up your data team’s time to solve problems that excite them. Sitting in data security meetings does not excite them. During recruitment, you promised this untainted talent a “new and exciting opportunity to bring valuable insights.” So, let’s enable them to do just that.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0d31832fd05" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/liberating-data-teams-lawfully-c0d31832fd05">Liberating Data Teams, Lawfully</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Six data functions you ought to automate]]></title>
            <link>https://medium.com/data-science/six-data-functions-you-ought-to-automate-cad4b11e404b?source=rss-dc2b62a4eb8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/cad4b11e404b</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Corné Potgieter]]></dc:creator>
            <pubDate>Thu, 22 Sep 2022 05:16:26 GMT</pubDate>
            <atom:updated>2022-09-22T05:25:00.472Z</atom:updated>
            <content:encoded><![CDATA[<h3>Six Data Functions You Ought to Automate</h3><h4>Aspects of your data pipelines that should be treated as repeatable patterns</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3f5Xo2zgGkWBLzUE" /><figcaption>Photo by <a href="https://unsplash.com/@lennykuhne?utm_source=medium&amp;utm_medium=referral">Lenny Kuhne</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Any good Solutions or Data Architect should have various issues on their table at any given point in time. That’s normal. What’s <strong><em>ab</em></strong>normal is that the same problem finds its way back to that table. This is the one repeatable pattern you want to get rid of.</p><p>Solving the solution in conceptually the right place should always be the status quo of the architect. Although this is always desired, time pressures will make us do silly things. However, when your solutions are built on repeatable patterns it somewhat forces your hand.</p><p>It is very uncomfortable to fix point solutions if they were generated by an automated process. You will make a couple of enemies by creating these siloed solutions.</p><p>By using repeatable patterns, you will be forced to zoom out and look at the problem holistically. For example, you will be forced to look at the logical data model and identify potential short comings, rather than simply adjusting some SQL code that loads data into your target object. It will highlight that your pattern isn’t perfect and thus forcing you to making it more robust.</p><p>You can always code yourself out of a sticky situation, however it is not sustainable to have bespoke coding patterns all over your ecosystem.</p><p>I am not a big proponent of no-code platforms. I am perfectly fine with high-code platforms, as long as I don’t need to write the code myself or if I do, only once. But I would be severely limited if I am not allowed to look at the code and tweak the pattern if needed.</p><p>On the topic of automation, it would be hypocritical of me not to share this image every time I talk about it. This remains true and one should be reminded of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/0*WU6RLmXTN1Tujvn-.png" /><figcaption>Image credit: <a href="https://xkcd.com/1319/">xkcd.com</a></figcaption></figure><p>Just because something is painful, doesn’t mean it isn’t the right thing to do. Setting up those initial templates and patterns are hard, but it will trump manual at some point.</p><p>Here are some of my obvious repeatable patterns that should be considered for automation.</p><h3>Loading</h3><p>Regardless of your loading pattern, whether it is batch loads, a CDC approach or message driven streaming. The way you ingest data onto your desired platform or storage should be transform-light and repeatable. It should be a couple of clicks to setup the process for loading.</p><p>I am a proponent of ELT — getting your data loaded as is and transforming in target. This allows for easier modularisation of loading and then the target platform can do the hard work of optimising query plans for on target transformations.</p><h3>Data transformation</h3><p>The T in ETL/ELT does not have to be purely bespoke.</p><p>Modularising different functions is a good principle in many professions especially software development. Modularise the functions that is repeated often. Some examples of this is data cleansing patterns and <em>hard </em>business rules such as data type casting.</p><p>Conceptually (there is that word again) do the same things in the same place. Don’t cleanse and transform in 1 step. In this way you can modularise the repetitive tasks and strip them from the non-repetitive.</p><p>The T is also style dependent. For example a Data Vault modelling style is highly geared towards automation and is thus a no brainer to have a heavy focus on repeatable patterns, but even 3NF and Kimball modelling styles are highly repetitive in nature.</p><p>Choose a modelling style and stick to it, this will also force your hand in doing the right thing in the right place. By not choosing a specific modelling technique you are also making a choice — leaving the decisions in the hands of each engineer. This will result in spaghetti un-repeatable code.</p><h3>Data Quality</h3><p>Modularize and repeat quality checks in your data acquisition process, but make sure it lands up in the right hands.</p><p>Failed quality checks need to flow back to people who will be disincentivized for bad data. Whether that is your product owners or assigned data stewards, they need to be pro-active to get rid of data quality issues.</p><p>A simple process flow could look like this, where the “bad data” is hidden. Data can be hidden by tagging the datasets if they have failed quality checks. They can then be excluded in loading processes or hidden via views or row/column level security.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*IsPDZkVAEu8Fcy4Mo-72zw.png" /><figcaption>image by author</figcaption></figure><p>However you decide to tag, hide or exclude the failed data can be shipped as DDL statements built in to your automation process.</p><h3>Distribution strategy</h3><p>The distribution of data on MPP platforms are a crucial part in the life cycle and sustainability of your data platforms.</p><p>Depending on your modelling patterns this can either be highly automated or slightly automated, but there shouldn’t be no automation opportunities.</p><p>Regardless of your platform, how you should be storing data is based on several factors, like query patterns, amount of data, and whether the objects should be optimised for reading or writing. Query patterns are highly dependent on the style of data modelling you are using. A lot of these factors are known at design time and thus the distribution can be allocated at initial creation of DDL. Some require refinement over time, as your data sizes and query patterns change. This can still be automated as it continuously does health checks on your data.</p><p>Because of the repetitive and standardized nature of Data Vault as a modelling style, there is good opportunity to apply proper distribution standards with no intervention. At design time, this can be overwritten should it require changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZAIxWq2-qnmgXGn_WpWA6Q.png" /><figcaption>Standardised DV query patterns allows for easy automation — image by author</figcaption></figure><p>Typically, different platforms (like Synapse, Redshift and Snowflake and Delta Lake to name a few) have good documentation available on what strategies to follow regarding partitioning, distribution and clustering. They also usually have documentation to tell you how to do health checks whether your data has become skew. You can then take the learnings of this larger community and bake it into an repeatable pattern to run periodically, allowing your data to “self-heal”.</p><h3>Data Security</h3><p>Data security takes up a healthy amount of airtime whenever I talk to clients. It has always been important but it is becoming more nuanced. Most companies are moving to the cloud and “as a service” solutions. This, together with complexities around personal information makes it a complicated task to wrap your head around. Not only does each country have it’s own legislation, each company has its own interpretation of that legislation.</p><p>Disclaimer — automation won’t fix those nuances.</p><p>Automation does allow you to iterate and test your security policies much faster. Give the compliance team something to work with. Only lawyers can work with rules on paper, the rest of us need to see it in action.</p><p>You shouldn’t need to manually apply security policies on a user or role bases. Define policies upfront and ideally you want these to be deployable (not deplorable) at design time.</p><p>Depending on your data modelling style you could easily deploy sensitive data to different normalised tables in secure schemas (like with Data Vault), or you could keep it together but apply column or row level policies on the table with the DDL generation. Whether you mask the data, encrypt it or simply secure it from the user, most target platforms cater for some or all of these features. The design stays homogeneous, the deployment you tailor based on which target platform you are deploying to.</p><h3>Testing</h3><p>Much has been written about this, and I would recommend you read about testing in data ops¹ amongst others.</p><p>This is a broad topic and I will simply emphasise one aspect that I see is neglected: Curated input, curated output. To create curated datasets that can be tested against is where pain and the glory lies.</p><h3>Meeting invite declines</h3><p>Ok, this is a bonus one, but people are all Zoomed out. Well architected solutions goes hand in hand with well executed plans. Both need some uninterrupted focused effort from time to time.</p><p>In conclusion, there are probably many more areas one can automate, but I find these six play a pivotal role in the sustainability of your architecture. Continuously improving the patterns for these will force you to build well architected solutions and stop the cycle of having to fix the same issues.</p><p>Simply keep asking yourself where do repeatable patterns exists or where can they be improved - improve the pattern, rinse and repeat.</p><pre>[1] - DataKitchen. 2022. <em>Blog - Add DataOps Tests to Deploy with Confidence | DataKitchen</em>. [online] Available at: &lt;https://datakitchen.io/add-dataops-tests-to-deploy-with-confidence/&gt;</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cad4b11e404b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/six-data-functions-you-ought-to-automate-cad4b11e404b">Six data functions you ought to automate</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep fake]]></title>
            <link>https://corne-potgieter.medium.com/its-all-fake-a88317cff05?source=rss-dc2b62a4eb8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a88317cff05</guid>
            <dc:creator><![CDATA[Corné Potgieter]]></dc:creator>
            <pubDate>Thu, 23 Jun 2022 11:04:23 GMT</pubDate>
            <atom:updated>2022-07-19T11:11:26.249Z</atom:updated>
            <content:encoded><![CDATA[<h4>Promoting quality ideas through the Influencer age</h4><p>The world is obscured by façade. If you are competent in your field, how does one navigate your own confidence and convictions in an era where you can have exceptional influence but lack quality ideas? The age of deep fakes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QvD_jwhtiniEoBdwjtO1NA.png" /><figcaption>Photo by <a href="https://unsplash.com/@diggitymarketing?utm_source=medium&amp;utm_medium=referral">Diggity Marketing</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We have a real issue where people fronting to be experts are exerting their influence over the layman, potentially with really mediocre to outright terrible ideas, and making money from it. I believe this showman façade is causing an apathy among competent individuals who should be making their voices heard.</p><p>“<em>Influencer</em>”, a term we have all come to know in the TikTok age, is defined amongst others as someone with “specialized knowledge”.¹ I disagree. It is not a pre-requisite.</p><p>Examine this graph and place yourself into one of these quadrants.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*bQm73NTa1KUOKUCIMCUOQQ.png" /><figcaption>diagram by author</figcaption></figure><p>If you feel you are competent, you have expertise in your field, but for whatever reason are not comfortable in “selling” your quality ideas or your skills, then you are in Quadrant 1. Quadrant 4 is for the façade of professionals that are highly influential, or at least seem to be, but at the root are only attracting attention and don’t have expertise in their field.</p><p>Influencer is simply someone in Quadrant 2 and 4. That’s all. I won’t pretend that I am an expert at the algorithms that sit behind TikTok and LinkedIn, but we can safely assume that they primarily promote influence (i.e. is your post getting comments, shares and likes). They are not primarily promoting good quality ideas. Correlation has been established between good ideas and influence, as one would hope, but influence does not guarantee that your ideas are good. How could it? It is not something that is easily measurable. I suppose we implicitly hope that posts that get shared, liked and commented on are in fact also quality ideas. But more likely they are simply aimed at making you react — click bait.</p><p>If you don’t believe me, try this. Make a thoughtful post on LinkedIn based on an interesting fact. Check the number of likes, shares, views. Now make another thoughtful post, based on an interesting fact, but put it in a meme that you share with your post.</p><p>If that doesn’t work, simply post something extremely controversial.</p><p>Which post do you think would rate highest on influence?</p><p>I am, however, not here to proclaim that all is lost, I want people with good quality ideas to beat the system at its’ own game.</p><p>In the words of late Patrick Winston, a lecturer at MIT,</p><p><em>Your success in life is largely determined by your ability to speak, to write and the quality of your ideas. In that order.²</em></p><p>This resonates with me, but unfortunately, the TikTok age has made it possible to be successful with only a combination of the first 2.</p><p>Too many people (Quadrant 4 people) are fronting to be in Quadrant 2 because they have mastered the system. In the past the “system” was social and psychological processes that one needed to understand to exert your influence. Nowadays it is actual algorithms. Algorithms are easier to understand and beat, we just aren’t always privy to how they operate.</p><p>This piece is primarily written to the Quadrant 1 people because I believe they need to be promoted. I would like to draw attention to some areas of everyday work life where these fallacies are visible, and to help you identify and navigate through it. They will also be slanted towards the data industry as this is where I operate.</p><h4>Beware the Serial certifier</h4><p>Many professionals have a range of certifications, especially on tools, but I would ask whether you know how to investigate discrepancies in data, find nuances, explain deviations and tell the full data story? Do you know how to take incoherent data from a data swamp and model it in such a way that you can easily extract insights from it? If you cannot provide the insight or solve the business issue, these certifications will do only one thing: window dress your resume.</p><p>I used to think there were only two certainties in life, but now I believe there are three: death, tax and more tools to get certified in. The top talented people I have worked with needs zero experience in a tool to get hired. I advocate for getting people trained in data modeling, root cause analysis and anything problem-solving related. And for data scientists: the science behind the data, the statistics and not just how to apply pre-built models. You cannot become a data scientist with a 6 month online course.</p><p>Ask yourself what is your motivation to do the certification? If it is to learn … good, if it is to get the job you’re applying for … bad. However, until recruiters pay less attention to this, the trend will continue, so the recruitee cannot be blamed for this. Bad recruiter, bad.</p><p>If you are doing it to learn — please continue, get as many as you possibly can. But if you are doing it to get the certification, you will bypass the learning bit and figure out a way to skip the meaty and difficult parts. I know this because I have done this several times. You can bullshit your way to getting a certificate.</p><p>Below is a diagram from 2018 showing the Big Data &amp; AI landscape. You could probably double this for 2022.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ZHveS-XrwoQlp8YJikpFA.png" /><figcaption><a href="https://mattturck.com/bigdata2018/">https://mattturck.com/bigdata2018/</a></figcaption></figure><p>For myself, certifications provide one basic goal. It gives you good context of what questions you need to be asking. When I look at a resume the only thing that certifications tell me is that you are pro-active in learning. Which is great, don’t get me wrong, but it definitely will not sell to me any level of competency.</p><p>As someone who strongly condemns anecdotal evidence, please allow me this one. We once parachuted a consultant in with something like 23 SAP certifications. On paper he would solve all our problems. In reality he added one: himself.</p><h4>Beware the Big words</h4><p>Real experts have a knack for making complicated subjects sound really simple. Fake experts have the propensity to make things sound authentically expeditiously perspicacious but genuinely they are not verbally expressing much and probably even contradicting themselves. Say what? See what I did there.</p><p>Trust your own competence and confidence. Often “<em>experts</em>” that use big words can make you feel intimidated and steam roller you. Stop, ask a question. They will start tripping over their own content at some point, but usually they are pretty good at changing the conversation and answering your question with a completely tangent answer.</p><p>Here is a real world example that I unfortunately have to censor, so it might not come across as well as I have hoped.</p><p>Sitting in an expensive training course one of the delegates asked the expert presenting the course a very specific question on how to implement the proposed solution. This is a known nuance and a question many people have, yet the expert went on for about 15min about what a great implementation he did at one of the largest institutions in the region.</p><p>Andrew Ng, a machine learning and AI expert I follow professionally, makes things simple to understand. These real experts are people in Quadrant 2. People who really are experts in their field happily share this expertise with others in an understandable way.</p><p><em>They are able to to speak well, to write well and they have good quality ideas.</em></p><h4>Beware the alpha-male personality</h4><p>Take a minute to think about (specifically pre-Covid) interactions in your organisation. Large decision-making meetings. How many people actively contributed?</p><p>I had a couple of female colleagues who I know this was particularly difficult for, as they were quite softly spoken. They would share their ideas and comments with me privately, and I would always be amazed as to why they didn’t speak up, until we delved into topic and I really started noticing the atmosphere of these meetings. You really need to be assertive if you want to challenge any ideas. You will be interrupted, a lot.</p><p>The large shift to online meetings has had a neutralizing effect on the high influencers. I found larger meetings were far more productive because of this. All of the social cues that they benefited from in an in-person meeting are stripped from an online meeting. They cannot assert themselves in a group of people in the same way as in-person.</p><p>I think all decision-making meetings in an organisation should be held online.</p><p>My tip to Quadrant 1 people here if you want to get your ideas implemented, is that you need to get better at lobbying, or you need to speak up at these meetings. Either you need to be lobbying the high influencers prior to the meeting or you need to be more assertive in the meeting. This is easier online, when you are sitting comfortably at your home vs. in a large boardroom. Type out what you want to say in the message functionality and then reading it out loud — this really helps in an interruption culture.</p><h4>Beware conforming to the masses</h4><p>Real quality ideas are often counter-intuitive and counter to the current trends.</p><p>Most of us are guilty of needing instant confirmation that what we are doing is accepted.</p><p>Lets take a recent example we should all be aware of. Will Smith walks on stage and slaps Chris Rock at the 2022 Oscars. Shortly later, people are giving Will Smith a standing ovation during his best actor award speech.</p><p>Only some time later do we all realise how ridiculous Will’s behavior was. I don’t blame the crowd for a minute. I probably would have done the same. We fall for a moving speech by a great actor. Only in hindsight are we able to do some critical thinking.</p><p>We need to train ourselves to be more critical in the moment. This does not come natural to us.</p><p>I did a small and completely unreliable test with a small sample — I had a casual conversation with some of my highly intelligent friends. I was surprised when all of them without hesitation put themselves in Quadrant 1 and said they hated being so visible and vocal on LinkedIn or any other platform promoting their service. They all trusted their competence though. They also all indicated similar concerns with Quadrant 4. I decided not to publish my findings, but rather just share them here.</p><p>If my hypothesis is true, I believe there is an army of people with quality ideas either whose confidence is being stifled or conviction muzzled because of the façade of Quadrant 4 people. The <em>deep </em>fakes. Our industry needs to promote the sharing of quality ideas and accurate information that un-informed people will actually listen to, (i.e. easy to understand) otherwise we are promoting only influence and not content.</p><p>By identifying these fallacies that we live with on a day to day basis I hope you come to the same conclusion as I did, there is no need to hide your competence. Confidently promote it. Contrary to my title, not everything is fake, but a lot of it is. Let us not be intimidated by it, but rather unite in promoting good quality ideas, one <strong>like </strong>at a time.</p><pre>[1] - 2022. [online] Available at: &lt;https://sproutsocial.com/glossary/influencer/&gt; <br>[2] — Winston, P., 2022. <em>How to speak</em>. [video] Available at: &lt;https://www.youtube.com/watch?v=Unzc731iCUY&gt; </pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a88317cff05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Purist v Pragmatist — why we can both agree on automation]]></title>
            <link>https://medium.com/data-science/purist-v-pragmatist-why-we-can-both-agree-on-automation-cc2f011fd34a?source=rss-dc2b62a4eb8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc2f011fd34a</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Corné Potgieter]]></dc:creator>
            <pubDate>Mon, 29 Nov 2021 19:53:32 GMT</pubDate>
            <atom:updated>2021-12-03T07:55:38.098Z</atom:updated>
            <content:encoded><![CDATA[<h3>Purist vs. Pragmatist — Why We Can Both Agree on Automation</h3><h4>We all strive for solutions that are sustainable yet we sit on spaghetti code, with our future inevitable … spaghetti code in the cloud.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*opCJMoBFaZm6YTBH" /><figcaption>Photo by <a href="https://unsplash.com/@ralphkayden?utm_source=medium&amp;utm_medium=referral">Ralph (Ravi) Kayden</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We have met both personas. The purist who loves theorizing on the “future proof data pipelines”, “modern data architecture”, “best practices” all neatly put into Death by PowerPoint. He is a real stickler for the rules. Let’s call him stickler Steve. You wait in anticipation for the next architecture meeting where you will be thrilled and amazed by what we are going to implement next. And on the other side, the hero, the doer, the guy (or gal) who runs jobs at 2 am to get data into production. Let’s call her pragmatic Prudence. It is a much different (and scarier) slideshow that she can compile. Put what she knows into PowerPoint, flick through it and you might get something that resembles a Wes Craven movie.</p><p>These two don’t speak fondly of each other, at least not in private.</p><p>Prudence blames Steve for not being able to implement pragmatic solutions and Steve blames Prudence for the horrible unsustainable mess that has been created.</p><p>I have often found myself being the conduit between these two personas. As a consultant I have worked with many Steves talking about sustainability and well architected solutions, but I have also sat in the trenches with Prudence, accountable for delivery, getting the solution into production. There seems to be this twilight zone somewhere between halfway through dev and production where we bring out the duct tape. The “best practice” just doesn’t get the job done. The conversation morphs slowly from “we need to stick to good conceptual designs” into “we need to get this into production by next week or business will pull the plug”. Prior to this moment hammer and nails are used to carefully construct the solution that Steve has put forward. After this inflection point is when Prudence brings out her trusted companion, Mr. Duct Tape.</p><p>So, how do we all pull in the same direction? I really appreciated an ex-colleagues’ article on how data architects (the purists) can get it wrong and how being accountable for delivery is that critical prerequisite. ¹ This is what gets the two personas sitting around the same table. Making sure the purist is accountable for getting the well architected solution into production, and thus adding a bit of pragmatism to Steve’s vocabulary.</p><p>Now that we are sitting around the same table, let’s start talking the same language. I will focus on one of the aspects which I believe both will agree on and appreciate … <strong>automation.</strong></p><p>Steve will agree that building patterns, finding repeatable components and automating the repeatable components definitely fits within the purist’s mindset. Prudence knows by experience that she is often repeating certain tests, certain processes ad infinitum in order to refine, revise and get the final solution ready.</p><p>First, we need to answer the core question — what can be automated? And if the answer is “nothing, because everything is so bespoke” I really hope you get the same shivers as I do. Just flick through those Wes Craven slides again to make sure you are thoroughly frightened.</p><p>Only repeatable tasks can be automated. I advocate for an environment where patterns are constantly found, templatized, or modularized and improved. Continuous improvement. It is the purist role to drive this culture, and it is the pragmatist role to test any new patterns to make sure it is feasible. Just as being accountable for delivery will allow Steve to be more hands on, so a bit of overall accountability will force Prudence to be prudent and become an advocate for automation.</p><p>At this point Steve is convinced. The only thing we still need to convince Prudence of is that even though everything is bespoke at the moment, there are repeatable patterns hiding in there. Prudence will require hard facts and examples, something that I will not elaborate on now but probably a good topic for a follow up.</p><p>The antagonists to automation will often share this picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*r_-_ZWam5n6mQn0o121-6g.png" /><figcaption>Image credit: <a href="https://xkcd.com/1319/">xkcd.com</a></figcaption></figure><p>They are not entirely wrong. That is the reality of memes. They are only funny because there is some truth to it.</p><p>Automation requires a lot of discipline. If you don’t have the discipline you will always fall back to old ways once time pressures appear. You must hate inefficiencies and doing repetitive tasks more than you hate not being the hero.</p><p>Pause and take a moment of introspection — ask yourself <strong><em>do I hate inefficiencies?</em></strong> The top of mind answer I suppose will always be “yeah sure” but really think about it. For myself, I would rephrase the question like this: Do I continuously find myself in a situation saying “I really think we solved this problem a couple of months ago, why are we thinking about this again?” That feeling bothers me to the core, it agitates, irritates and any other “ates” me. Those that would have worked with me know I become extremely pedantic and almost robotic at this stage. Must. Solve. Inefficiency. Now.</p><p>Setting up the groundwork can take longer than expected and the identification and templatization of repeatable patterns will take longer than manually doing it the first time. Maybe even the 2nd, maybe even the 3rd, and maybe … ok ok, you get my point. Somewhere along the line automation will surpass manual, but only if you have the discipline to stick to the process.</p><p>Steve definitely hates inefficiencies, and I believe Prudence does too, she might just enjoy being a hero too much. (Topic for a different day) . Once we are aligned in our hatred toward inefficiencies, we need to believe that there are enough inefficiencies that we can solve, hidden patterns (or blatantly obvious ones) that we can extract, make repeatable and finally automate. Don’t be like the guys in the picture below, make time now in order to free up time later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*bGoCPT7IG2Z-mLJYk-1h1w.jpeg" /><figcaption>Image generated by <a href="https://imgflip.com/memegenerator/122160104/No-thanks-we-are-too-busy">imgflip</a></figcaption></figure><p>On the other hand, automation is inevitable. I believe there is enough momentum created by the demand from both business and IT. Data engineers are burnt out and calling for a better approach. ² Instead of trying to fight it we should strive towards it and how to make it better.</p><p>In my conversations with clients, the terms Data Ops, Dev Ops, CI/CD comes up every time without fail. Automation is a key pillar to Data Ops and Dev Ops. I can’t always ask my clients whether it is something they really need or something their Steve suggested, but the demand is clear. Tools need to be able to integrate with Jenkins, GitHub etc. Everything should be scriptable. We want the nice pretty front-ends to design in, but we want the command line interfaces to operate and deploy.</p><p>Our methods and our toolkits need to complement our culture, not the other way around. We need a disciplined yet pragmatic culture that is complimented by our methods and software. Let people do what we are good at— coming up with creative solutions to interesting problems. Let’s make computers do what they are good at — repetition and parallel processing.</p><p>Spaghetti code does not need to be our destiny. Let’s have the discipline and purist naivety to start untangling it in order to find hidden patterns. Then be pragmatic to iterate and continuously improve knowing that 100% is not the goal.</p><p>At the end of the day both Steve and Prudence have families that they would like to spend more time with. Automation will help make this a reality. That’s the second thing they can agree on.</p><pre>1. Thomas, J., 2021. <em>The rise and fall of data architecture</em>. [online] ITWeb. Available at: &lt;https://www.itweb.co.za/content/o1Jr5Mx9AxPqKdWL&gt; [Accessed 24 November 2021].<br>2.WIRE, B., 2021. <em>Data Engineers Are Burned Out and Calling for DataOps</em>. [online] Businesswire.com. Available at: &lt;https://www.businesswire.com/news/home/20211019005858/en/Data-Engineers-Are-Burned-Out-and-Calling-for-DataOps&gt; [Accessed 24 November 2021].</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc2f011fd34a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/purist-v-pragmatist-why-we-can-both-agree-on-automation-cc2f011fd34a">Purist v Pragmatist — why we can both agree on automation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>