<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Digital Power on Medium]]></title>
        <description><![CDATA[Stories by Digital Power on Medium]]></description>
        <link>https://medium.com/@digitalpower?source=rss-e272dacd376------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vpVqnwKF2DkRoyCDGio2mA.png</url>
            <title>Stories by Digital Power on Medium</title>
            <link>https://medium.com/@digitalpower?source=rss-e272dacd376------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:34:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@digitalpower/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data governance in Azure Databricks]]></title>
            <link>https://medium.com/@digitalpower/data-governance-in-azure-databricks-97e75a73e567?source=rss-e272dacd376------2</link>
            <guid isPermaLink="false">https://medium.com/p/97e75a73e567</guid>
            <category><![CDATA[databricks-unity-catalog]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[access-control]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <dc:creator><![CDATA[Digital Power]]></dc:creator>
            <pubDate>Tue, 27 Oct 2026 15:25:36 GMT</pubDate>
            <atom:updated>2026-10-27T15:26:16.877Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to structure access control with Unity Catalo</h3><h4>When organisations experience data breaches, the discussion often focuses on external threats: attackers, vulnerabilities and inadequate security measures.</h4><p>But another risk receives less attention: users who retain access to data they no longer need.</p><p>An employee leaves the organisation, but direct permissions on a Databricks schema remain. Or a data scientist receives temporary access to sensitive data, only for those permissions to remain active eighteen months later.</p><p>No malicious intent. No cyberattack. Just governance debt accumulating over time.</p><p>As data access becomes increasingly democratised through modern data platforms and AI-powered tools, managing permissions becomes more important.</p><p>In Azure Databricks, this requires a structured approach to identity management, access control and auditing.</p><p>This article explains how to implement those principles using Unity Catalog, Automatic Identity Management (AIM), RBAC, ABAC, system tables, Terraform and Databricks Asset Bundles.</p><h3>Part four of our data governance series</h3><p>This article is part of a four-part series covering data governance from organisational foundations to technical implementation.</p><ol><li><a href="https://digital-power.com/en/inspiration/why-you-can-t-afford-to-wait-on-data-governance-any-longer/">Why you can no longer afford to wait with data governance</a></li><li><a href="https://digital-power.com/en/inspiration/in-3-steps-towards-effective-data-governance/">Effective data governance in 3 steps</a></li><li><a href="https://digital-power.com/en/inspiration/gain-control-of-data-governance-for-your-data-and-ai-platform-in-5-steps/">Gain control of data governance for your data and AI platform in 5 steps</a></li><li><strong>Data governance in Azure Databricks</strong> (this article)</li></ol><p><em>The principles apply broadly, but the examples focus specifically on Azure Databricks and Unity Catalog.</em></p><h3>Who actually has access to your data?</h3><p>Managing access involves more than determining who can read a particular table. Organisations also need to understand why permissions were granted, who approved them and whether they are still necessary.</p><p>Without a clear process, permissions accumulate. Users receive access for individual projects, responsibilities change and permissions are not consistently reviewed.</p><p>This governance debt becomes more difficult to manage as the number of users, datasets and applications grows.</p><p><strong>Governance should therefore be part of the platform’s foundation rather than something introduced once the environment has reached a certain level of maturity.</strong></p><h3>Ownership: the organisational side of governance</h3><p>One of the most important governance decisions concerns ownership.</p><p>Who manages the data model? Who defines what constitutes a customer? Who decides which users should be allowed to access sensitive information?</p><p>Technology alone cannot answer these questions.</p><p>A platform team understands infrastructure and technical configuration, but the business context of individual datasets usually sits with data owners.</p><p>A useful separation of responsibilities is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*p2ZSXOizPzUud7bPCUKrew.png" /></figure><p>This approach enables data owners to manage access based on business requirements without relying on the platform team for every decision.</p><h3>SCIM, RBAC and ABAC: managing identities separately from permissions</h3><p>Identity management and access management serve different purposes.</p><p>Identity management determines which users, groups and service principals exist in the platform. Access management determines what those identities are allowed to do.</p><h3>Identity management with AIM</h3><p>Azure Databricks supports <a href="https://www.databricks.com/blog/automatic-identity-management-azure-databricks">Automatic Identity Management (AIM)</a>, which synchronises users, service principals and groups directly from Microsoft Entra ID.</p><p>This removes the need for a separate SCIM provisioning application in supported configurations.</p><p>With AIM:</p><ul><li>Entra ID remains the source of truth for identities.</li><li>Group membership changes are synchronised automatically.</li><li>New users can be created when they first sign in through just-in-time provisioning.</li></ul><p>AIM is enabled at account level. Adding synchronised groups to individual workspaces remains a separate administrative action.</p><h3>Role-based access control (RBAC)</h3><p><a href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-privileges/">RBAC</a> assigns permissions to roles or groups.</p><p>For example, a group named data-analysts might receive SELECT privileges on tables within a particular catalog.</p><p>Managing permissions through groups simplifies administration and reduces the number of individual assignments.</p><h3>Attribute-based access control (ABAC)</h3><p><a href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/">ABAC</a> applies policies based on attributes and tags.</p><p>For example, columns containing personally identifiable information (PII) can be tagged and automatically masked for users who do not meet the relevant access conditions.</p><p>This becomes especially useful when maintaining individual table-level permissions is no longer practical.</p><p>Regardless of which model is used, permissions should generally be assigned to synchronised groups rather than individual users.</p><h3>Using Databricks system tables to audit access</h3><p>A structured access model requires visibility.</p><p>Security and compliance teams must be able to investigate which permissions have been granted and who can access sensitive data.</p><p>Databricks provides <a href="https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/">system tables</a> and information schema views that expose governance-related information.</p><p>For example, this SQL query can retrieve table privileges for a particular dataset:</p><pre>SELECT<br>    grantee,<br>    privilege_type,<br>    table_catalog,<br>    table_schema,<br>    table_name<br>FROM system.information_schema.table_privileges<br>WHERE table_name = &#39;finance_transactions&#39;;</pre><p>The result shows granted privileges on matching tables within the querying user’s visibility. Filtering by catalog and schema may be necessary when table names are reused.</p><p>System tables also support auditing, while Unity Catalog provides lineage metadata to trace relationships between data assets.</p><p>An important distinction remains: <strong>Auditing provides visibility, not control.</strong></p><p>System tables help teams inspect permissions and activity. They do not replace the processes needed to approve, grant and review access.</p><h3>Purview, Collibra and Alation: cataloguing versus access control</h3><p>Microsoft Purview, Collibra and Alation can support data discovery, documentation and governance across multiple platforms.</p><p>They help organisations understand which datasets exist, what they contain and where they can be found.</p><p>However, a data catalogue does not replace the underlying permission-management process.</p><p>An organisation may have a comprehensive catalogue while still maintaining excessive or outdated permissions in Databricks.</p><p><strong>Data cataloguing improves discoverability. Access management determines who can actually use the data.</strong></p><p>These capabilities are complementary, not interchangeable.</p><h3>Three ways to provision access in Azure Databricks with Unity Catalog</h3><p>Once identities, roles and responsibilities are clear, the next question is how permissions should be provisioned.</p><p>Three common approaches are available: Terraform, Databricks Asset Bundles and the graphical user interface.</p><figure><img alt="Three ways to provision access in Azure Databricks with Unity Catalog" src="https://cdn-images-1.medium.com/max/1024/0*jQX3LZwtJe7_GH-B" /><figcaption>Three ways to provision access in Azure Databricks with Unity Catalog</figcaption></figure><h4>1. Terraform: governing the platform</h4><p>Terraform is suited to infrastructure-level configuration, including workspace settings and foundational Unity Catalog permissions.</p><p>Configuration can be managed as code, reviewed in version control and deployed through CI/CD pipelines.</p><p>For example:</p><pre>resource &quot;databricks_grants&quot; &quot;analyst_read&quot; {<br>  catalog = &quot;production&quot;<br><br>  grant {<br>    principal  = &quot;data-analysts&quot;<br>    privileges = [&quot;USE_CATALOG&quot;]<br>  }<br>}</pre><p>This example grants the data-analysts group permission to use the specified catalog. Additional schema and table permissions are needed to read data.</p><p>The main advantage is reproducibility. Changes can be tracked, reviewed and deployed consistently.</p><p>The trade-off is complexity. Terraform requires infrastructure expertise, which can introduce unnecessary overhead when data owners need to adjust permissions for individual data products.</p><p>For platform-wide configuration, however, this approach offers strong control.</p><p>See the <a href="https://registry.terraform.io/providers/databricks/databricks/latest/docs">Databricks Terraform provider documentation</a> for supported resources.</p><h4>2. Databricks Asset Bundles: governing data products</h4><p><a href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/">Databricks Asset Bundles</a> allow resources to be defined and deployed together using configuration files.</p><p>When a data product is managed as a bundle, permissions can be maintained alongside its pipelines, notebooks and schemas.</p><p>For example:</p><pre>resources:<br>  schemas:<br>    bronze_layer:<br>      name: bronze<br>      catalog_name: production<br>      grants:<br>        - principal: data-engineers<br>          privileges:<br>            - USE_SCHEMA<br>            - SELECT</pre><p>This enables domain teams to manage permissions as part of the same deployment process used for their data products.</p><p>It also supports a clearer separation of responsibilities. The platform team maintains the infrastructure, while domain teams manage access to their own datasets.</p><p>For implementation details, see the <a href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/permissions">Databricks Asset Bundles permissions documentation</a>.</p><h4>3. The graphical interface: convenient but less reproducible</h4><p>The Databricks graphical interface allows authorised users to assign permissions directly.</p><p>This is useful for development, experimentation and temporary changes.</p><p>However, manual changes are not automatically represented in version-controlled configuration. Although audit information is available, understanding why a particular permission was assigned may require additional investigation.</p><p>For production environments, relying extensively on manual changes can therefore make governance more difficult.</p><p>The interface is convenient, but it should be used within a defined permission-management process.</p><h3>Avoid unnecessary complexity</h3><p>Organisations may use Terraform, Asset Bundles and the graphical interface together.</p><p>However, doing so without clear responsibilities makes configuration harder to maintain.</p><p>A better approach is to define which tool manages each type of permission and establish which configuration is authoritative.</p><h3>Conclusion</h3><p>Effective data governance in Azure Databricks depends on a clear separation of responsibilities.</p><ul><li><strong>The platform team</strong> manages identity synchronisation and foundational infrastructure.</li><li><strong>Data owners</strong> determine access to their data products.</li><li><strong>Unity Catalog</strong> provides the technical mechanisms to enforce permissions.</li><li><strong>System tables and audit information</strong> make access and activity verifiable.</li><li><strong>Terraform and Databricks Asset Bundles</strong> support controlled, version-managed permission provisioning.</li></ul><p>The technology is available. The more difficult part is establishing ownership and maintaining the processes that keep access appropriate over time.</p><p>Good governance is not a one-time configuration exercise. It requires continuous attention to who has access, why that access is necessary and who is responsible for maintaining it.</p><p><em>Written by Cas Hortensius, Data Engineer at Digital Power.</em></p><p><em>Originally published on </em><a href="https://digital-power.com/en/inspiration/data-governance-in-azure-databricks/"><em>Digital Power</em></a><em>, where we share practical knowledge and experiences in data engineering, analytics and AI.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97e75a73e567" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Databricks Genie Agents make data accessible through natural language]]></title>
            <link>https://medium.com/@digitalpower/how-databricks-genie-agents-make-data-accessible-through-natural-language-c5f18f42cab0?source=rss-e272dacd376------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5f18f42cab0</guid>
            <category><![CDATA[databricks-genie]]></category>
            <category><![CDATA[conversational-ai]]></category>
            <category><![CDATA[conversational-analytics]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Digital Power]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 14:46:08 GMT</pubDate>
            <atom:updated>2026-09-29T14:46:08.058Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rCdWKRryqyojLBJAPun6Nw.png" /></figure><h3>How to design and manage a Genie Agent for reliable answers</h3><p>Many organisations face the same challenge. Someone has a question about revenue, customer behaviour or operational performance. The data exists, but finding the answer takes time. A dashboard needs to be opened, filters have to be configured correctly, or a Data Analyst needs to perform an additional analysis.</p><p>Despite years of investment in dashboards and self-service analytics, there is often still a gap between a business question and the answer. Users need to know where information is stored, how dashboards are structured and which definitions are being used. This not only slows decision-making but also creates dependency on data specialists and leads to different interpretations of the same KPIs.</p><p>At the same time, organisations are looking for faster and more accessible ways to gain insights from their data. Increasingly, they are exploring how generative AI can make those insights instantly available to business users.</p><h3><strong>TL;DR</strong></h3><p><strong>Databricks Genie Agents</strong> allow users to ask questions about data in natural language and receive immediate answers, without having to perform analyses themselves.</p><p>However, AI alone is not enough. The quality of a Genie Agent depends on well-structured datasets, rich metadata and business context, including KPI definitions and business terminology. Strong governance ensures that users only receive answers based on the data they are authorised to access.</p><h3>From dashboards to conversations with data</h3><p>This is where Databricks Genie comes in. With a <a href="https://learn.microsoft.com/en-gb/azure/databricks/genie/"><strong>Genie Agent</strong></a>, users can ask questions in natural language and receive immediate answers based on the available data. Instead of navigating dashboards and tables, they can simply have a conversation with their data.</p><p>For example, a user could ask: <em>“Which productcategory achieved the highest revenue growth lastquarter?”</em> Rather than searching through dashboards or asking a Data Analyst to investigate, they can ask Genie directly and receive an immediate answer, potentially accompanied by a visualisation.</p><p>That does not mean a Genie Agent automatically provides reliable answers. The quality of the responses depends heavily on how data, business definitions and business logic have been organised. Without sufficient context, Genie can run into the same challenges as other AI solutions, such as misinterpretations, inconsistent definitions or answers that do not fully reflect the business domain.</p><p>That is why we see implementing a Genie Agent not as a one-off project, but as an ongoing process of continuous improvement. This is where the real value lies: users gain faster access to trustworthy insights, while organisations achieve greater returns on their data investments.</p><h3>A Genie Agent is a collaboration between business and data</h3><p>A successful Genie Agent is about more than technology. It requires close collaboration between business users and data specialists.</p><p>On one side is the <strong>curator</strong>. This is typically a Data or Analytics Engineer, or a domain expert responsible for the quality of the Genie Agent. The curator selects relevant datasets, captures business logic, manages definitions and monitors the quality of the answers.</p><p>On the other side are the business users. They ask questions as part of their daily work and expect accurate answers and visualisations without needing to understand the underlying data model or SQL. Their feedback is essential for identifying unclear definitions, missing context and incorrect interpretations.</p><p>The result is more than just a new way of accessing data. Employees can independently find answers to their questions much faster, while data specialists spend less time responding to recurring requests. This creates more capacity to focus on complex analyses, new data products and challenges that require specialist expertise.</p><h3>How do you build an effective Genie Agent?</h3><p>For the curator, configuring a Genie Agent is one of the most important success factors. The order in which context is added has a major impact on the reliability of the answers.</p><p>Each step expands the knowledge available to Genie, gradually improving both the quality and consistency of its responses.</p><h4>1. Define the objective and scope</h4><p>Start by defining a clear objective.</p><p>Who is the Genie Agent intended for? Which business domain does it support? And what questions should users be able to answer independently?</p><p>Keep the scope deliberately focused. A Genie Agent that supports a single process or business domain typically delivers more reliable answers than one that attempts to cover every possible question across the organisation.</p><h4>2. Add only the relevant tables</h4><p>Next, select only the data required to achieve that objective.</p><p>Ideally, work with a limited set of well-documented tables or views. Ensure that relationships between datasets are clearly defined and simplify complex data models wherever possible.</p><p>The clearer the data model, the more likely Genie is to use the correct data when answering questions.</p><h4>3. Enrich the Genie Agent with metadata</h4><p>Provide context by enriching your data with metadata. Adding descriptions and synonyms to tables and columns helps Genie match user questions to the correct data.</p><p>It is also useful to include common terminology, abbreviations and frequently used value patterns. This allows Genie to recognise different ways users may ask the same question and map them to the appropriate data.</p><p>In many cases, this step alone delivers a significant improvement in answer quality, even before any additional business logic has been introduced.</p><h4>4. Add SQL expressions for business logic</h4><p>SQL expressions provide the foundation for consistent business definitions.</p><p>Use them to define KPIs, metrics, filters and dimensions. This ensures that concepts such as revenue, active customers and conversion rate are calculated consistently across the organisation.</p><p>This eliminates discussions about definitions and enables different teams to work from the same version of the truth. As a result, users have greater confidence in Genie’s answers and decision-making becomes more reliable.</p><h4>5. Add example queries</h4><p>Some questions involve complex business logic that cannot easily be derived from metadata or SQL expressions alone.</p><p>For these situations, you can add example queries by combining a natural language question with validated SQL.</p><p>These examples help Genie interpret similar questions more accurately. This can significantly improve accuracy, particularly for complex analyses and frequently asked business questions.</p><h4>6. Use prompts for exceptions and behaviour</h4><p>General instructions can help define answer formatting, language or specific behavioural guidelines.</p><p>However, prompts should mainly be used for exceptions. Business logic, KPI definitions and relationships between datasets are better captured in metadata and SQL, as structured context is generally more reliable than text-based instructions.</p><h4>7. Governance within the Genie Agent</h4><p>Genie automatically respects the access permissions and security settings configured in <a href="https://learn.microsoft.com/en-gb/azure/databricks/data-governance/unity-catalog/"><strong>Unity Catalog</strong></a>. This means users only see data they are authorised to access.</p><p>As a result, governance, security and compliance remain centrally managed while users can independently obtain the answers they need. This makes it possible to increase data accessibility without compromising control or security.</p><p>Finally, Genie supports language models from Anthropic and OpenAI with EU data residency, provided the environment has been configured accordingly. This allows data to be processed within the European Union, helping organisations meet internal governance and compliance requirements.</p><p>Want to learn more about governance in Databricks? Read our article on <a href="https://digital-power.com/en/inspiration/data-governance-in-azure-databricks/"><strong>Data Governance in Azure Databricks</strong></a><strong>.</strong></p><h3>Start small and continuously improve</h3><p>It can be tempting to build a broad Genie Agent that supports every possible question from day one. In practice, a phased approach usually delivers better results.</p><p>Start with a limited number of datasets, preferably well-managed and cleansed tables, together with a small set of frequently asked questions. Then monitor how users interact with Genie, which answers meet their information needs and where additional context is required.</p><p>This allows the Genie Agent to gradually evolve into a trusted layer between users and data.</p><p>Not by creating ever more dashboards and visualisations, but by reducing the distance between a question and an answer. That is where the greatest value lies: faster insights, less dependency on data specialists and better decisions based on trusted data.</p><h3>Unlock more value from your data with Databricks Genie Agents</h3><p>Would you like your employees to find reliable answers in your data more quickly, without relying entirely on dashboards or data specialists? We help organisations design and optimise Databricks Genie Agents so AI can generate reliable answers based on well-structured data, business context and robust governance. This makes data accessible to a wider audience, accelerates decision-making and helps you maximise the value of your existing data investments.</p><p><a href="https://digital-power.com/en/contact/"><strong>Get in touch</strong></a> or <a href="https://meetings-eu1.hubspot.com/rene-westerdijk/salesteam-meetinglink-website-eng-2025?__hstc=10321962.ac85e5aae5300acde127083780e55568.1789133426934.1790687680920.1790692351529.47&amp;__hssc=10321962.2.1790692351529&amp;__hsfp=855383a8f8da27697784c5dccc61044c"><strong>schedule an introductory meeting</strong></a> to discuss how Databricks Genie Agents can support your organisation.</p><h3>This is an article by Bas Verburg</h3><p>Bas is a Data Engineer at Digital Power and is specialised in Databricks. He works with organisations on complex data engineering projects, helping them build scalable data platforms, improve data quality and create a solid foundation for analytics and AI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5f18f42cab0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From smart meter data to real-time decisions]]></title>
            <link>https://medium.com/@digitalpower/from-smart-meter-data-to-real-time-decisions-601be842dc5f?source=rss-e272dacd376------2</link>
            <guid isPermaLink="false">https://medium.com/p/601be842dc5f</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[internet-of-things]]></category>
            <category><![CDATA[real-time-streaming-data]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[Digital Power]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 14:30:46 GMT</pubDate>
            <atom:updated>2026-09-29T14:30:46.567Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why real-time data needs more than a data warehouse</h3><p>By 2028, the world will generate 394 zettabytes of data a year, more than double the roughly 181 zettabytes produced in 2025 (IDC, cited by TechTarget).</p><p>For the energy sector, this growth is particularly relevant. More and more data no longer comes from spreadsheets or traditional databases, but from smart meters, grid sensors and AI systems that continuously generate information.</p><p>That changes how data infrastructure needs to work.</p><p>Batch processing was designed for a world in which data could be collected, stored and analysed later. But if you need to balance supply from thousands of solar panels in real time, waiting for the next batch is no longer sufficient.</p><p>Smart meters illustrate this shift clearly. Monthly readings used to be enough. Today, meters can transmit electricity, gas, heat or water measurements every fifteen minutes or even more frequently.</p><p>A relatively simple reporting workflow therefore becomes a continuous stream of telemetry that needs to be ingested, processed and made available for analytics with minimal delay.</p><p>For many organisations, this exposes a gap in their existing architecture. The problem is usually not the data warehouse itself. It is the missing streaming ingestion layer in front of it.</p><h3>Why a data warehouse alone is not enough</h3><p>Most organisations already have a modern data platform. Snowflake, Databricks, Microsoft Fabric and BigQuery are designed to handle large amounts of data, scale storage and compute independently, support ELT patterns and integrate with BI tooling.</p><p>But high-frequency sensor data introduces a different engineering problem.</p><p>Before data reaches the warehouse, your architecture needs to continuously absorb readings from thousands or potentially millions of distributed devices. It also needs to handle ordering, reliable delivery and backpressure.</p><p>In practical terms, the data needs to arrive in the correct order, without getting lost and without overwhelming the system when volumes increase.</p><p>That is fundamentally different from loading a nightly CSV export from an ERP system or periodically retrieving JSON through a REST API.</p><p>With continuously streaming smart meter and sensor data, you have to deal with high volumes, protocol differences, out-of-order events and strict latency requirements.</p><figure><img alt="Four streaming challenges" src="https://cdn-images-1.medium.com/max/1024/0*BAofAUZSbpB_hwrH" /><figcaption>4 streaming challenges</figcaption></figure><p>Without a dedicated streaming layer, you are left with two imperfect options. You can collect the data into larger batches, which removes much of the value of having high-frequency sensors. Or you can continuously send small jobs to your warehouse, increasing processing overhead and cost.</p><h3>What you need between the stream and your warehouse</h3><p>Before data can enter the warehouse, it first needs to exist as a continuous stream of events.</p><p>In many smart metering environments, this producer side is already available. Devices communicate with a backend system or IoT platform, which validates the readings and publishes them to a streaming platform such as Kafka, Azure Event Hubs or AWS Kinesis.</p><p>If that event pipeline does not exist yet, it is the first component you need to establish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*48DpzQd1RfC16Z9x" /></figure><p>Once the stream is available, the next requirement is an ingestion, or consumption, layer.</p><p>A streaming ingestion layer is a continuously running process that reads events from the stream and turns them into data that can reliably be loaded into the warehouse.</p><p>Modern warehouses are very good at storing and querying large volumes of data. They are not necessarily designed to continuously consume external event streams.</p><p>That responsibility sits with the ingestion process. It needs to manage concerns such as consumer coordination, offsets, schema evolution, retries and reliable delivery before the data reaches the warehouse.</p><p>There are three main ways to build this layer.</p><h4>Option A: Build a self-managed consumer</h4><p>The first approach is to develop and operate the consumer yourself.</p><p>A Kafka Consumer application, Kafka Streams application or Apache Flink job can subscribe directly to a topic, deserialize events using the schema registry and perform enrichment or aggregation before writing the results downstream.</p><p>The application can use idempotent, offset-aware logic to prevent restarts or consumer rebalances from producing duplicate rows.</p><p>The main advantage is control. You decide exactly how events are processed, transformed and delivered.</p><p>That control also creates operational responsibility. Your team needs to monitor consumer lag, deal with partition rebalancing and manage offsets correctly.</p><p>A self-managed consumer therefore makes most sense when the processing requirements justify that additional complexity.</p><h4>Option B: Use a managed connector</h4><p>Instead of building the consumer yourself, you can use an existing sink connector to perform the read-and-land step.</p><p>Examples include Kafka Connect sinks for Snowflake, Databricks or Delta Lake, as well as integrations provided by managed streaming platforms such as Azure Event Hubs and AWS Kinesis.</p><p>This reduces the amount of custom code your team needs to build and maintain.</p><p>The trade-off is flexibility. Managed connectors become less suitable when significant transformation, enrichment or custom processing needs to happen before the data reaches the warehouse.</p><h4>Option C: Use native warehouse streaming ingestion</h4><p>A third option is to use streaming ingestion capabilities provided directly by the warehouse.</p><p>Examples include Snowflake Snowpipe Streaming and Databricks Zerobus Ingest.</p><p>Operationally, this can produce the simplest architecture because there is one less intermediate system to manage.</p><p>The trade-off is again control. Native ingestion provides less room for enrichment, aggregation or complex event processing before the data lands in the warehouse.</p><p>That becomes particularly important when the use case goes beyond analytics and starts supporting real-time alerts or operational decisions.</p><h3>Choosing the right approach</h3><p>There is no single correct architecture.</p><p>The right approach depends on factors such as stream volume, processing complexity, the amount of transformation required before ingestion and your team’s capacity to operate additional infrastructure.</p><p>A team building sophisticated real-time alerting may need the control of a self-managed consumer. For straightforward analytics ingestion, that same architecture could introduce unnecessary complexity.</p><p>The decision is therefore not simply about which technology can ingest the data. It is about matching the ingestion architecture to what the organisation needs to do with the data.</p><h3>Why the streaming ingestion layer changes what’s possible</h3><p>Once you’ve closed the consumption gap, the operational and strategic benefits you were promised at a business level start to become technically concrete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gk2SrqiNHPoeck53" /></figure><h3>From data delivery to actionable decisions</h3><p>The streaming ingestion layer is where your real-time sensor data starts delivering business value.</p><p><a href="https://digital-power.com/en/inspiration/from-sensor-data-to-decisions-that-genuinely-improve-your-operations/">As outlined in our analysis on sensor data and operational decisions</a>, you get the most value when you combine sensor data with business data to create actionable management information.</p><p>Take a practical example from one of our customers. When an energy utility modernised its metering infrastructure to fifteen-minute intervals and added heat supply and return-flow measurements, having more data wasn’t enough.</p><p>You only start seeing meaningful patterns, anomalies and efficiency losses when that data is reliably consumed, processed in near real time and combined with your other operational data in a modern data warehouse.</p><p>If you’ve already invested in Snowflake or Databricks, you might expect your smart meter data to be available in near real time. But if you’re still relying on daily or hourly batch jobs, the challenge probably isn’t the warehouse. It’s the missing ingestion layer in between.</p><p>By closing that gap, you can move from looking back at what happened last month to understanding what’s happening now, what’s likely to happen next and where you should intervene.</p><p>You don’t necessarily need more infrastructure on the producer side. Instead, you need a dedicated, correctly configured ingestion process that reads the stream you already have and makes that data available where your business can act on it immediately.</p><h3>Ready to make your sensor data work for you in near real time?</h3><p>Need help designing or improving the ingestion layer between your event streams and data platform? Our Data Engineering specialists can help you build a scalable infrastructure that fits your existing architecture and real-time requirements. <a href="https://digital-power.com/en/expertises/data-engineering/">Learn more about Data Engineering</a></p><p><a href="https://digital-power.com/en/contact/">Get in touch</a> or <a href="https://meetings-eu1.hubspot.com/rene-westerdijk/salesteam-meetinglink-website-eng-2025?__hstc=10321962.ac85e5aae5300acde127083780e55568.1789133426934.1790682801309.1790687680920.46&amp;__hssc=10321962.5.1790687680920&amp;__hsfp=855383a8f8da27697784c5dccc61044c">schedule a meeting</a> to discuss the possibilities.</p><h3>This is an article by Timo Gruwèl</h3><p>With almost 10 years of experience in data engineering and analytics, Timo helps organisations turn business ambitions into robust technical solutions. Starting his career as a data analyst before growing into a data engineer, he has built broad, all-round expertise across data warehousing, orchestration and the wider data ecosystem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=601be842dc5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configuring Claude Code with CLAUDE.md]]></title>
            <link>https://medium.com/@digitalpower/configuring-claude-code-with-claude-md-826f3de3ff8a?source=rss-e272dacd376------2</link>
            <guid isPermaLink="false">https://medium.com/p/826f3de3ff8a</guid>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Digital Power]]></dc:creator>
            <pubDate>Fri, 05 Jun 2026 14:36:57 GMT</pubDate>
            <atom:updated>2026-06-05T14:39:36.115Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nT62Q2-JTyN9pCmRuwp5iw.png" /></figure><h3>How to Get AI Agents to Write Better Code Consistently</h3><h4>The File That Determines How Your AI Agent Writes Code</h4><p>There is a noticeable difference between teams that occasionally get good results from Claude Code and teams that consistently produce high-quality code. That difference rarely comes from writing better prompts.</p><p>More often, it comes from something fundamental: the standards and constraints defined before the agent starts generating code.</p><p>With modern coding models, the nature of software development is changing. Engineers are no longer only writing code. They are also shaping the behaviour of the agents that write it.</p><p>Recently, Andrej Karpathy described this shift as <a href="https://x.com/karpathy/status/2019137879310836075"><em>Agentic Engineering</em></a>, the logical next step beyond what is commonly called “vibe coding”.</p><p>Without explicit guidance, an agent falls back on patterns learned during training. The result is often generic implementations, unnecessary complexity, and technical decisions that may be valid in isolation but do not align with the conventions of a specific codebase.</p><p>The solution is not more prompting. It is one file.</p><h4>Why Claude Code Goes Off Track Without Context</h4><p>Coding agents naturally try to align with what they believe the user wants. Even when the most appropriate response would be that information is missing or a request does not make sense.</p><p>This behaviour is visible in benchmarks such as the <a href="https://github.com/petergpt/bullshit-benchmark">Bullshit Benchmark</a>, where models frequently validate incoherent input instead of challenging it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/0*6HdfBzjTgIJMmpgq.png" /><figcaption>Benchmark ranking AI models by their ability to recognise and correct incorrect assumptions.</figcaption></figure><p>In software development, this tendency often appears as overengineering: additional abstractions, excessive error handling, or architectural layers that were never requested.</p><p>The problem is not a lack of intelligence. The problem is a lack of context.</p><p>Without a clear definition of what “good” looks like inside a repository, Claude Code fills in the gaps itself. Those assumptions come from patterns seen during training, not from your technology stack, engineering standards, or team agreements.</p><h4>CLAUDE.md as a Behavioural Baseline for AI Agents</h4><p>CLAUDE.md addresses this problem.</p><p>The file lives in the root of a repository and defines the standards, boundaries, and behavioural rules that should always apply. Claude Code automatically reads it at the start of every session.</p><p>A useful comparison is .editorconfig. The difference is that .editorconfig standardises code formatting, while CLAUDE.md standardises agent behaviour.</p><p>Although this article focuses on CLAUDE.md, the same concept exists elsewhere. Other coding agents support similar approaches through files such as AGENTS.md.</p><p>Defining expectations at project level removes the need to repeat them in every prompt.</p><p>In practice, this leads to:</p><ul><li>More consistent architectural decisions</li><li>Fewer surprises during code review</li><li>Less time spent correcting solutions that are almost, but not quite, appropriate</li></ul><h3>Why Fewer Rules Produce Better Results</h3><p>One important detail is often overlooked.</p><p>CLAUDE.md is included with every API call as part of the system prompt. A file containing 2,000 tokens that is sent 40 times consumes 80,000 input tokens purely for instructions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NsN1FAFsSHeGbquj.png" /><figcaption><em>Diagram showing a 2,000-token CLAUDE.md file results in 80,000 input tokens across 40 API calls</em></figcaption></figure><p>Those instructions also compete with the code context the model needs to perform its task.</p><p>Even the latest models show reduced performance when context windows become overloaded.</p><p>Recent research supports this observation. One study found that large amounts of generic AI-generated instructions can lead to worse outcomes than having no rules at all.</p><p>The goal is not to create a comprehensive handbook. The goal is to provide the minimum amount of guidance required to improve decisions.</p><h3>What to Include in CLAUDE.md</h3><p>Effective CLAUDE.md files typically contain three categories of rules.</p><h3>1. Hard Technical Constraints</h3><p>These are decisions the agent should never deviate from:</p><ul><li>Programming languages</li><li>Frameworks</li><li>Architectural principles</li><li>Established coding conventions</li></ul><pre>## Hard Limits<br><br>- Functions: &lt;= 100 lines<br><br>- Cyclomatic complexity: &lt;= 8<br><br>- Positional parameters: &lt;= 5<br><br>- Files: &lt;= 500 lines</pre><p>These constraints prevent the agent from introducing alternatives that do not belong in the codebase.</p><h3>2. Stack and Tooling</h3><p>Without explicit instructions, agents default to the tools and workflows most frequently represented in their training data.</p><p>That is how teams end up receiving pip install recommendations when they use uv, or suggestions for black when the standard is ruff.</p><pre>## Python Tooling<br><br>- Package manager: uv<br><br>- Linting + formatting: ruff<br><br>- Type checking: ty<br><br>- Testing: pytest</pre><p>These rules are simple to document and often provide immediate improvements in output quality.</p><h3>3. Behavioural Guidelines</h3><p>This is often the most undervalued category.</p><p>Behavioural guidelines define how the agent should reason and make decisions:</p><ul><li>When refactoring is appropriate</li><li>How uncertainty should be handled</li><li>When clarification should be requested</li><li>When assumptions should be avoided</li></ul><p>These rules often make the difference between code that is technically correct and code that is genuinely useful.</p><h3>CLAUDE.md Is a Living Document</h3><p>A CLAUDE.md file should not be treated as a checklist that is written once and forgotten.</p><p>In practice, it works best as a feedback mechanism.</p><p>If the agent repeatedly makes the same mistake, add a rule. If a rule provides little measurable benefit, remove it.</p><p>Over time, this process produces a compact set of instructions that provides enough guidance without overwhelming the model with unnecessary context.</p><p>For teams using AI systematically in software development and data engineering, this is a natural step towards more mature adoption.</p><p>Not by writing more prompts, but by defining what “good” looks like before the agent starts coding.</p><p>If you are exploring how AI agents fit into modern engineering workflows, we regularly share practical perspectives on data, AI, and software systems at Digital Power. <a href="https://eu1.hubs.ly/H0vZf7-0">You can find more articles and insights on our website</a>.</p><h3>Author</h3><p><a href="https://www.linkedin.com/in/paul-hoekstra/">Paul Hoekstra</a> is a Data Engineer who builds AI-ready data platforms end-to-end, covering cloud infrastructure, DevOps, data warehousing, and machine learning pipelines.</p><p>Representing Digital Power, he is currently working at Vodafone, where he develops Apache Flink streaming applications and contributes to MLOps initiatives for a composable Customer Data Platform (CDP).</p><p>His interests span data engineering, machine learning, and agentic workflows, with a particular focus on how AI can improve day-to-day engineering practices.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=826f3de3ff8a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing value-adding ML systems]]></title>
            <link>https://medium.com/@digitalpower/designing-value-adding-ml-systems-5e2b21eb07b1?source=rss-e272dacd376------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e2b21eb07b1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[shap]]></category>
            <category><![CDATA[machine-learning-tools]]></category>
            <category><![CDATA[ml-model-deployment]]></category>
            <category><![CDATA[ml-system-design]]></category>
            <dc:creator><![CDATA[Digital Power]]></dc:creator>
            <pubDate>Fri, 01 May 2026 08:58:44 GMT</pubDate>
            <atom:updated>2026-05-01T09:51:55.780Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="machine learning model" src="https://cdn-images-1.medium.com/max/1024/1*GCprFgPKVNEUwTt_E-oAtg.png" /></figure><p><strong>Why Machine Learning Models Fail, and How System Design Fixes It</strong></p><p>Machine learning is often approached as a modelling problem. Choose an algorithm, train it on historical data, validate performance, deploy. If the metrics look good, the job feels done.</p><p>In practice, that is almost never where the real work ends, and it’s rarely where value begins.</p><p>This article is for data scientists, ML engineers, and analytics leads who have built models that looked strong in testing, yet struggled to make a real impact once deployed. If you’ve ever shipped a model with great offline metrics only to watch it quietly fade into irrelevance, the issue is unlikely to be your algorithm.</p><p>What separates ML projects that deliver business value from those that stall in notebooks is <strong>system design</strong>: the structured process of translating a business question into a production‑ready decision system.</p><p>To make this concrete, we’ll walk through six stages of ML system design, using a SaaS churn example throughout:</p><ol><li>Translating the business question</li><li>Framing the ML problem</li><li>Making data decisions</li><li>Selecting and training a model</li><li>Evaluating offline and online</li><li>Integrating, monitoring, and maintaining</li></ol><p>The difference between “ML modelling” and “ML system design” becomes obvious when you work through them side by side.</p><h3><strong>Models don’t create value. Decisions do.</strong></h3><p>Businesses don’t ultimately care about probabilities, scores, or ROC curves. They care about changed behaviour: different actions taken at the right time, under real operational constraints.</p><ul><li>An ML model outputs predictions.</li><li>An ML system supports decisions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hk5MiSVN_MGauTCA" /></figure><p>That distinction explains why so many technically sound ML projects fail. They optimise for statistical performance without ever locking down how predictions will be turned into action.</p><p>System design forces that commitment.</p><h3><strong>Step 1: Translate the business question</strong></h3><p><strong>Goal:</strong> define what decision will change, under which constraints, and how success is measured in business terms.</p><p>This is the most underestimated step, and also the most important.</p><p>Business stakeholders rarely speak in ML language. They speak in outcomes:</p><ul><li>“Reduce churn.”</li><li>“Increase retention.”</li><li>“Improve conversion.”</li></ul><p>Your task is to translate those aspirations into something operational. That starts by answering questions most projects skip:</p><ul><li>What <em>decision</em> will be made differently because this system exists?</li><li>Who will act on the output?</li><li>How often?</li><li>Under what constraints?</li><li>How will success be measured?</li></ul><p>Model accuracy is almost never a useful success metric. Business KPIs are.</p><p><strong>Churn example</strong></p><p>The client asks: “Can we reduce churn?”</p><p>Translated requirement:</p><ul><li>Identify customers likely to cancel within 30 days</li><li>The retention team can contact <strong>500 customers per week</strong></li><li>Cost per intervention is <strong>€10</strong></li><li>Outreach must happen <strong>at least 7 days before renewal</strong></li><li>Success means reducing monthly churn from <strong>5% to 3%</strong></li></ul><p>At this point, we’re not talking about algorithms yet, we’re designing a decision.</p><h3><strong>Step 2: Frame the ML problem</strong></h3><p><strong>Goal:</strong> turn the business need into a precise prediction problem with a defined horizon, scope, and integration path.</p><p>Once the decision is clear, you can frame a formal ML problem. This is where vagueness causes long‑term damage if left unresolved. Key choices must be made explicit:</p><p><strong>Prediction target:</strong> What exactly counts as “churn”? Is it only full cancellation, or also downgrades? Do trial users count?</p><p>Ambiguous labels lead to fragile models.</p><p><strong>Prediction horizon: </strong>How far ahead must the model predict for the business to act?</p><ul><li>Too short, and there’s no time to intervene.</li><li>Too long, and predictions become unreliable.</li></ul><p><strong>Scope</strong>: Which customers are included? High‑value enterprise accounts often behave very differently from self‑serve users, and may need separate treatment.</p><p><strong>Integration alignment:</strong> How will predictions be consumed? CRM workflows, sales dashboards, automated campaigns — each implies different latency, format, and reliability requirements.</p><p><strong>Churn example</strong></p><ul><li>Churn is defined as <strong>account cancellation within 30 days</strong></li><li>Downgrades are excluded</li><li>Trial users are modelled separately</li><li>Scope: paying customers with ≥3 months tenure</li><li>Output feeds directly into the CRM to trigger retention workflows</li></ul><p>With this, the ML problem stops being abstract.</p><h3><strong>Step 3: Make your data decisions</strong></h3><p><strong>Goal:</strong> identify the right sources, engineer meaningful features, and guard against quality pitfalls.</p><p>Data decisions form the backbone of any ML system. Start by mapping internal systems that might carry signal:</p><ul><li>Product analytics</li><li>CRM data</li><li>Billing and finance</li><li>Support interactions</li></ul><p>For each source, assess availability, freshness, and reliability.</p><p><strong>Feature engineering matters more than raw data</strong></p><p>Raw data rarely predicts business behaviour on its own. Features become powerful when combined with domain knowledge. Three categories are especially useful:</p><ul><li><strong>Rolling aggregates</strong> (e.g. usage over last 30/60/90 days)</li><li><strong>Trends and slopes</strong> (is engagement rising or declining?)</li><li><strong>Recency indicators</strong> (days since last meaningful interaction)</li></ul><p><strong>Data quality is where projects silently fail</strong></p><p>Common failure modes include:</p><ul><li><strong>Data leakage</strong> (using future information)</li><li><strong>Survivorship bias</strong> (only learning from customers who stayed)</li><li><strong>Class imbalance</strong></li><li><strong>Poor label quality</strong></li></ul><p><strong>Churn example</strong></p><p>Data sources:</p><ul><li>Behavioural: login frequency, feature usage, support tickets</li><li>Relational: tenure, NPS</li><li>Financial: balances, billing complaints</li><li>Demographic: segment, acquisition channel</li></ul><p>Key engineered features:</p><ul><li>30‑day balance trend</li><li>Login frequency slope</li><li>Days since last meaningful interaction</li></ul><p>A seemingly useful feature like “account closed date” is excluded, it leaks the label.</p><p><strong>Handling imbalance</strong></p><p>Only <strong>5%</strong> of customers churn. A naive model predicting “no churn” for everyone would achieve 95% accuracy while being useless.</p><p>To address this:</p><ul><li>Oversample churners (e.g. SMOTE)</li><li>Undersample non‑churners</li><li>Weight churned customers more heavily during training</li></ul><p>After training, the classification threshold is lowered from <strong>0.5 to 0.35</strong>. This increases recall at the cost of more false positives — a trade‑off the business explicitly accepts.</p><h3><strong>Step 4: Select and train your model</strong></h3><p><strong>Goal:</strong> balance performance, interpretability, and maintenance cost.</p><p>Model selection is not a leaderboard exercise.</p><p>A pragmatic approach follows three phases:</p><ol><li><strong>Establish a baseline</strong> <br>Logistic regression provides a performance floor and interpretability. If it’s good enough, stop there.</li><li><strong>Explore complexity</strong> <br>Gradient boosting models like LightGBM handle tabular data, missing values, and imbalance well.</li><li><strong>Tune deliberately</strong> <br>Use stratified cross‑validation and Bayesian hyperparameter search, optimised for metrics that reflect business goals (e.g. F1, recall).</li></ol><p>The key question isn’t “Which model scores highest?”. It’s “Is the performance gain worth the added complexity and maintenance burden?”</p><p><strong>Churn example</strong></p><ul><li>Logistic regression reveals useful signals.</li><li>LightGBM improves performance meaningfully.</li><li>Bayesian tuning optimised for F1 yields final gains.</li></ul><p>LightGBM is chosen as the best trade‑off.</p><h3><strong>Step 5: Evaluate offline and online</strong></h3><p><strong>Goal:</strong> prove statistical soundness <em>and</em> real‑world impact.</p><p>Offline metrics can mislead, especially on rare events. Accuracy is useless here. Instead:</p><ul><li>Precision</li><li>Recall</li><li>F1‑score</li><li>AUC‑ROC</li></ul><p>Explainability matters too. SHAP values help:</p><ul><li>Build stakeholder trust</li><li>Detect unexpected drivers</li><li>Explain individual predictions</li></ul><p>But offline metrics only show potential. Online evaluation demonstrates reality.</p><p><strong>Churn example</strong></p><ul><li>False negatives are expensive, so recall is prioritised.</li><li><a href="https://shap.readthedocs.io/en/latest/">SHAP</a> shows balance decline and inactivity as dominant drivers.</li><li>An 8‑week A/B test shows <strong>35% lower churn</strong> in the intervention group. That’s the result that matters.</li></ul><h3><strong>Step 6: Integrate, monitor, and maintain</strong></h3><p><strong>Goal:</strong> embed predictions into workflows and detect decay before value erodes.</p><p>A model that isn’t embedded into daily work delivers nothing.</p><p>Predictions must:</p><ul><li>Arrive where decisions are made</li><li>Trigger concrete actions</li><li>Include context and explanation</li></ul><p><strong>Feedback loops are essential</strong></p><p>Production performance must be tracked by comparing predictions with outcomes. ML systems learn — and decay — in the real world, not in notebooks.</p><p><strong>Watch for drift</strong></p><p>Customer behaviour changes. Products evolve. Markets shift.</p><p>Monitor:</p><ul><li>Feature distributions</li><li>Prediction scores</li><li>Business KPIs</li></ul><p><strong>Churn example</strong></p><p>Predictions flow directly into the CRM. Each week, high‑risk customers automatically enter the retention workflow with:</p><ul><li>Risk score</li><li>Contributing factors</li><li>Pre‑approved offer</li></ul><p>Three months in, a product redesign changes login patterns. Drift monitoring catches performance decay early, and the model is retrained.</p><h3><strong>The bigger picture</strong></h3><p>ML system design is a decision‑making discipline.</p><ul><li>An ML model produces predictions.</li><li>An ML system turns predictions into better decisions.</li></ul><p>Most ML projects fail not because the algorithm was wrong, but because predictions were never aligned with real actions, constraints, and incentives.</p><p>Once you design for decisions — not just metrics — machine learning starts delivering what it promised.</p><h3>This article was written by George Pavlidis</h3><p>George Pavlidis is a Data Scientist at <a href="https://digital-power.com/en/">Digital Power</a> with 6 years of experience spanning data science, machine learning engineering, and data engineering. He has worked across industries including finance, renewable energy, and manufacturing, always focused on translating business problems into production-ready ML solutions. George holds a B.Sc. in Applied Informatics from the University of Macedonia and has published research in areas ranging from speech recognition to energy demand forecasting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e2b21eb07b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Should you run LLMs locally?]]></title>
            <link>https://medium.com/@digitalpower/should-you-run-llms-locally-d4f9dfc09481?source=rss-e272dacd376------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4f9dfc09481</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[lm-studio]]></category>
            <dc:creator><![CDATA[Digital Power]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 13:00:19 GMT</pubDate>
            <atom:updated>2026-03-30T13:02:00.412Z</atom:updated>
            <content:encoded><![CDATA[<h3>When local AI models outperform API-based services</h3><p>Large Language Models (LLMs) have quickly become a standard component in modern applications. Most developers start by integrating models such as OpenAI, Claude or similar providers through APIs. It is fast, convenient and requires very little infrastructure.</p><p>But what happens when:</p><ul><li>API costs suddenly increase?</li><li>A provider updates a model and your prompts stop working?</li><li>You need to process sensitive data that cannot leave your environment?</li></ul><p>For organisations working with proprietary data, high request volumes or strict compliance requirements, running open-source LLMs locally is becoming an increasingly attractive alternative.</p><p>In this article, we explain when local LLM deployment makes sense, how to get started, and what infrastructure you need to move from experimentation to production.</p><h3>This article in short</h3><ul><li>Use API-based LLMs for fast prototyping and low-volume use cases</li><li>Run LLMs locally when data privacy, cost control or scale are critical</li><li>Local LLMs remove rate limits and give full control over model behaviour</li><li>Mid-size models (~70B) offer the best balance between performance and cost</li><li>A hybrid setup (API + local models) is often the most practical strategy</li></ul><h3>Why not just use API-based LLMs?</h3><p>API services such as OpenAI or Anthropic offer powerful models that are easy to integrate. For many use cases, they remain the simplest option.</p><p>However, there are situations where local models offer clear advantages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R2s09Yaw5sAVhTjx" /><figcaption>API-based LLMs</figcaption></figure><h3>When running LLMs locally becomes attractive</h3><p>Local deployment becomes particularly valuable when:</p><ul><li>You process sensitive or regulated data</li><li>Usage volumes become large and predictable</li><li>API costs grow significantly</li><li>You need stable and reproducible model behaviour</li></ul><h3>Understanding local LLM deployment</h3><p>Running LLMs locally involves several components. To simplify the landscape, we divide the process into five key areas:</p><ol><li>Local development tools</li><li>Interfaces and user experience</li><li>Model selection and performance</li><li>Deployment infrastructure</li><li>Cost considerations</li></ol><h4>1. Local development tools</h4><p>Before deploying models in production, you will typically start by testing them locally. Two tools make this particularly straightforward:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LKO-7dzmvHELgWRa" /><figcaption>Local development tools</figcaption></figure><p>Both tools allow you to prototype locally before committing to production infrastructure.</p><p>In practice, LM Studio is often preferred for quick testing, while Ollama offers more flexibility for automation and integration.</p><h4>2. Interface and user experience</h4><p>Once models run locally, you need a way to interact with them.</p><p>While LM Studio and Ollama include basic chat interfaces, many teams prefer a dedicated interface layer.</p><p><strong>Open WebUI</strong> provides a modern web interface similar to ChatGPT, but connected to your own models.</p><p>Key capabilities:</p><ul><li>Connect to multiple model backends</li><li>Compare model responses side-by-side</li><li>Share access with team members</li><li>Manage conversation history</li><li>Upload documents for context</li></ul><p>This makes it especially useful when you want to give non-technical colleagues access to local AI tools.</p><h4>3. Model selection and performance</h4><p>Choosing the right model is essential when running LLMs locally. Model size is typically measured in parameters (billions).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p3HXXON2yFYsFWOI" /><figcaption>LLM model selection</figcaption></figure><p>For many organisations, mid-size models around 70B parameters provide the best balance between performance and infrastructure cost. Smaller models can handle high-volume, simpler tasks, while larger models are typically reserved for specialised workloads where maximum capability is required.</p><h4>4. Deployment and infrastructure</h4><p>Once you move beyond experimentation, you need infrastructure capable of serving models reliably.</p><p><strong>Running models locally</strong></p><p>Your own laptop or workstation is often sufficient for development.</p><p>This works well when:</p><ul><li>testing integrations</li><li>experimenting with prompts</li><li>running smaller models</li></ul><p><strong>Deploying to GPU virtual machines</strong></p><p>For production workloads, a GPU-enabled VM is typically required.</p><p>This is necessary when:</p><ul><li>multiple users access the model</li><li>uptime and reliability are important</li><li>larger models require more VRAM</li><li>higher throughput is required</li></ul><p><strong>Typical deployment setup</strong></p><p>A common architecture includes:</p><ul><li>GPU VM</li><li>Ollama running the model</li><li>Nginx reverse proxy</li><li>client applications connecting via API</li></ul><p>This setup provides an OpenAI-compatible endpoint that you control entirely.</p><h4>5. Cost considerations</h4><p>There are generally three ways to serve LLMs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z_hL2A2W3LL1xeL-" /><figcaption>Cost considerations</figcaption></figure><h3>When does local deployment make sense?</h3><p>Local models are particularly valuable when:</p><ul><li>Workloads are high volume and predictable</li><li>Batch processing tasks analyse large datasets</li><li>Multiple use cases share the same infrastructure</li><li>Data privacy requirements prevent external sharing</li></ul><p>Many organisations adopt a hybrid strategy, combining local models with external APIs depending on the task.</p><h3>Conclusion</h3><p>Running open-source LLMs locally gives organisations significantly more control over their AI infrastructure.</p><p>You gain:</p><ul><li>full data privacy</li><li>predictable infrastructure costs</li><li>independence from API rate limits</li><li>stable model behaviour in production</li></ul><p>For teams exploring this approach, a practical path forward is:</p><ol><li>Start small</li><li>Experiment locally using tools such as LM Studio or Ollama.</li><li>Choose the right model</li><li>Match model size to the complexity of your task.</li><li>Scale thoughtfully</li></ol><p>When moving to production, evaluate whether a budget GPU provider or a major cloud platform fits your architecture best.</p><p>Local LLMs are not always the right solution, but for many organisations they give greater flexibility, control and cost efficiency in modern AI systems.</p><h3>This is an article by Bob Strube</h3><p>Bob is a Data Engineer at Digital Power, specialising in AI and scalable data platforms. He has built and deployed machine learning and GenAI solutions on Azure and Databricks. He helps organisations move from experimentation to production with reliable, cost-efficient and privacy-conscious infrastructure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4f9dfc09481" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How AI is transforming programming: From autocomplete to agentic coding]]></title>
            <link>https://medium.com/@digitalpower/how-ai-is-transforming-programming-from-autocomplete-to-agentic-coding-4a57095dc94b?source=rss-e272dacd376------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a57095dc94b</guid>
            <category><![CDATA[ai-tools]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[agentic-coding]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Digital Power]]></dc:creator>
            <pubDate>Tue, 13 Jan 2026 14:34:10 GMT</pubDate>
            <atom:updated>2026-01-13T14:34:10.176Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3OPzVJFfKrjQhS4DiOBtzw.avif" /></figure><h3>An overview of the AI tools we use when programming</h3><p>Artificial Intelligence is transforming how you design, build, and maintain digital solutions. From code generation to data pipeline automation, AI has become a trusted companion in technical workflows.</p><p>At Digital Power, we see this evolution every day. What began as simple code suggestions has grown into intelligent agents that understand entire projects, refactor files, and even review pull requests. These systems free us to focus on architecture, analytics, and creativity: the parts of our work that make the biggest impact. At the same time, they introduce new responsibilities: we now spend more of our effort guiding these tools, validating their output, and ensuring they strengthen the long-term quality of our solutions.</p><p>In this article, you’ll get a clear view of how AI-assisted coding has evolved, how different types of tools fit into modern development, and which platforms currently define the landscape. You’ll also gain insight into the trade-offs and emerging responsibilities that come with this new way of working.</p><h3>This article in short</h3><ul><li>AI coding tools have evolved from autocomplete to agentic systems that plan, edit, and verify code.</li><li>There are now several types of tools: from code editor assistants to reviewer bots, each fitting a different workflow.</li><li>We provide an overview of the most relevant tools in 2025, organised by category.</li><li>The key takeaway: AI is becoming an active collaborator that helps teams move faster and maintain quality, while also introducing new responsibilities around validation, reliability, and long-term maintainability.</li></ul><h3>From autocomplete to agentic coding</h3><p>Early AI tools worked like predictive text: completing lines of code based on what you were typing. That alone saved time. But the newest generation goes far beyond.</p><p>Modern assistants now reason across whole repositories. They can plan a change, edit multiple files, run tests, and verify the results. Some even suggest improvements to infrastructure or data models. We call this shift <strong>agentic coding</strong>: AI that collaborates with us instead of simply following prompts.</p><p>In data and software projects alike, these tools have changed how we think about development. Instead of asking “<em>what should I type next?</em>”, we ask “<em>how should this system behave?</em>” The AI helps bridge that gap between intention and implementation, allowing faster iteration.</p><h3>What the data shows: Global trends in AI programming</h3><figure><img alt="Graph showing Copilot release in June 2020, ChatGPT release in November 2022, Cursor Release in 2023 and Copilot free in December 2024. This all in a rising graph of AI-tagged repositories" src="https://cdn-images-1.medium.com/max/1024/0*XKbq_hqyz6-wbIHr" /></figure><p>The graph shown here uses data from the GitHub Innovation Graph and focuses specifically on public repositories tagged with AI technologies. This subset is useful for showing the trend over time. It highlights clear spikes in developer activity following major AI milestones, such as the launch of ChatGPT, the rise of AI coding environments like Cursor, and the introduction of the free mode of GitHub Copilot.</p><p>As for the overall numbers, <a href="https://github.blog/news-insights/octoverse/octoverse-a-new-developer-joins-github-every-second-as-ai-leads-typescript-to-1">GitHub Octoverse 2025</a> reports that more than 180 million developers now build on GitHub, with over 36 million new developers joining in the past year, who collectively pushed nearly 1 billion commits in 2025, a 25.1% increase YoY.</p><h3>The types of AI coding tools</h3><p>AI-assisted coding now comes in several distinct types, each fitting a different stage of a technical workflow. These can be grouped by how they integrate into your workflow and how they collaborate with you.</p><figure><img alt="Table of all the tools. Inline code assistants like Github Copilot and Cursor, or CLI agents like Cline and Claude Code, Container-native agents like OpenAI Codex and Devin, Reviewers like Coderabbit and Github Copilot Reviewer" src="https://cdn-images-1.medium.com/max/1024/0*7SXo7Fzk35Pp_GTd" /></figure><h3>Choosing the Right Fit</h3><p>Selecting the right AI tool depends less on programming language and more on workflow.</p><ul><li><strong>IDE based agents</strong> such as <strong>Cursor</strong> and <strong>GitHub Copilot </strong>support everyday development inside your editor. They help with writing, debugging, and exploring code, and many now integrate with the terminal to run commands and act across multiple files.</li><li><strong>CLI tools</strong> like <strong>Claude Code </strong>or <strong>Cline </strong>operate through the command line and work efficiently across whole repositories. They are strong options for large refactoring tasks, automation, and generating tests or documentation.</li><li>In secure or regulated environments, <strong>container native agents</strong> such as <strong>Devin </strong>offer isolated workspace where tasks run safely without touching your main system.</li><li><strong>Reviewer tools </strong>like <strong>CodeRabbit </strong>or <strong>GitHub Copilot</strong> can analyse pull requests and provide automated feedback for teams that want an extra quality check.</li></ul><p>Ultimately, the right tool depends on your goals and workflow. Each category serves a different need, and the best fit is the one that supports how you work.</p><h3>Looking ahead</h3><p>As AI coding tools continue to evolve, the role of the programmer is changing too.</p><p>Writing code is no longer the bottleneck. Understanding business context, designing systems, and ensuring reliability are becoming the real differentiators.</p><p>By taking over repetitive and mechanical work, AI gives you more time and focus to work on what truly matters: <strong>delivering business value</strong>. At the same time, it introduces new responsibilities. You need to validate AI output, safeguard code quality, monitor long-term maintainability, and remain in control of architectural decisions. The craft of programming is shifting from producing code to shaping solutions; from execution to direction.</p><p>At Digital Power, we see this as an opportunity to elevate our impact. We explore and adopt these tools thoughtfully, balancing speed with responsibility, always with one goal in mind: empowering people to create data-driven solutions that make organisations smarter, faster, and more future-proof.</p><h3>This is an article by Paul Hoekstra</h3><p>Paul is a Data Engineering Consultant at Digital Power. He focuses on building automated data pipelines, scalable data models and reliable ETL transformations. With a hands-on and analytical approach, he helps organisations turn complex data landscapes into clear insights that truly make a difference.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a57095dc94b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating to European Tag Management and Analytics platforms]]></title>
            <link>https://medium.com/@digitalpower/migrating-to-european-tag-management-and-analytics-platforms-4f479a613146?source=rss-e272dacd376------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f479a613146</guid>
            <category><![CDATA[eurostack]]></category>
            <category><![CDATA[web-analytics]]></category>
            <category><![CDATA[piwik]]></category>
            <category><![CDATA[tracking]]></category>
            <category><![CDATA[tag-management]]></category>
            <dc:creator><![CDATA[Digital Power]]></dc:creator>
            <pubDate>Fri, 10 Oct 2025 08:59:39 GMT</pubDate>
            <atom:updated>2025-10-10T08:59:39.795Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lh-8wT3Vj0TyQpSynv2Vsw.png" /></figure><h3>How to make the right choice</h3><p>Due to recent geopolitical developments, such as heightened concerns about privacy and legislation, many organisations are reassessing their dependence on US-based software solutions. At the same time, the call for greater control over data and reduced reliance on external parties is growing louder.</p><p>Migrating your tag management system (TMS) or web analytics platform (WAP) to European alternatives can be an important step in reducing this dependency. But what are the real implications of such a migration?</p><p><em>This article is part of the “EuroStack” series, in which we help organisations make the transition to European alternatives for their data and analytics environment.</em></p><p><strong>Series overview:</strong></p><ul><li><a href="https://digital-power.com/en/inspiration/is-your-organisation-ready-for-independence-from-the-us-and-migrate-to-eurostacks/"><strong>Blog 1</strong> — Is your organisation ready for independence from the US and migrate to EuroStack?</a></li><li><a href="https://digital-power.com/en/inspiration/what-european-data-warehouse-solutions-are-available/"><strong>Blog 2</strong> — What European data warehouse solutions are available?</a></li><li><a href="https://digital-power.com/en/inspiration/how-to-migrate-your-data-warehouse/"><strong>Blog 3</strong> — Migrating to a European data warehouse: where to start?</a></li><li><strong>Blog 4</strong> <em>(current article)</em> — Migrating to European Tag Management and Analytics platforms: where to start?</li></ul><h3>This article in a nutshell</h3><ul><li><em>Many organisations are rethinking their reliance on US-based tag management and analytics platforms due to privacy, regulation, and data sovereignty concerns.</em></li><li><em>European alternatives are becoming mature and competitive — often excelling in compliance and transparency.</em></li><li><em>Migrating doesn’t need to be all-or-nothing: hybrid setups, self-hosting, or phased transitions can reduce risk and keep you flexible.</em></li><li><em>Choose platforms that fit your strategy, stay vendor-agnostic, and keep your data future-proof.</em></li></ul><h3>Why these platforms?</h3><p>Tag management systems (TMS) and web analytics platforms (WAP) form the foundation of many data-driven ambitions. They translate behaviour on websites and apps into insights, activation, and personalisation. Although they are closely related, they fulfil fundamentally different roles and therefore deserve separate consideration.</p><h3>Tag Management Systems (TMS)</h3><p>A TMS allows you to manage tracking codes (tags) easily, without having to modify your website or app code directly. This gives you greater control over which data you collect and how it is processed or forwarded to other systems.</p><p>TMS platforms remain essential because they:</p><ul><li>pre-process data before it is sent to analytics or advertising platforms</li><li>read and structure information via cookies or device IDs</li><li>translate consent status into actions and data flows</li></ul><h3>Web Analytics Platforms (WAP)</h3><p>Web analytics platforms provide insights into how users interact with your digital environment. They help you understand where visitors come from, what they do, and where friction or opportunities arise. These insights are crucial for optimisation, personalisation, and strategic decision-making.</p><p>However, WAPs are under pressure. Their impact has declined, mainly due to:</p><ul><li>stricter regulation, browser tracking prevention, and rising privacy awareness</li><li>a growing need to connect online data with offline sources</li><li>more complex analysis and activation requirements, such as audience segmentation or predictive models</li></ul><h3>The role of TMS and WAP</h3><p>We increasingly see that TMS and WAP data do not stand alone but form part of broader data strategies. Interactions on digital platforms are combined with other data sources to create a more complete and accurate customer view. This serves not only analysis but also activation in marketing, service, and product development.</p><p>But which solutions are available? And how do European alternatives compare to the established US-based platforms? In the next section, we compare several leading platforms based on the most relevant criteria.</p><h3>What are your options?</h3><p>Just like their US counterparts, many European providers now offer integrated solutions: a Tag Management System (TMS) combined with a Web Analytics Platform (WAP). This can offer integration benefits, but most web analytics platforms also work perfectly well with external TMS solutions, so vendor lock-in can often be avoided.</p><p>To provide a fair comparison, we examine both types of platforms separately, using relevant and commonly applied criteria.</p><h3>Tag management systems</h3><p>For TMS solutions, we focus on five core categories:</p><ol><li><strong>Functionality:</strong> Think of data processing, decision logic, vendor support, and server-side tracking.</li><li><strong>Performance:</strong> What is the impact on your site? How scalable is the solution? How does the platform handle peak traffic?</li><li><strong>Ease of use:</strong> How intuitive is the interface, and how easy is it to manage</li><li><strong>Compliance:</strong> Support for consent management, self-hosting, and customisable processing logic.</li><li><strong>Support:</strong> The level of assistance from the provider, documentation, community, and training resources.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YYxAQE0B0pPBunbbn4NAsw.avif" /></figure><p>Although all platforms offer value, their focus and functionality differ significantly.<br><strong>Commanders Act</strong> and <strong>JENTIS</strong> offer a full-fledged server-side tracking platform for organisations with strict compliance and enterprise requirements — though this comes with limited client-side capabilities.</p><p><strong>Piwik PRO</strong> is accessible, well supported, and strikes a good balance between features and flexibility.<br><strong>Matomo</strong> is particularly attractive for organisations seeking maximum control, with features such as self-hosting and server tracking.<br>Piwik PRO and Matomo offer more flexible tag management options, with hybrid client- and server-side setups.</p><p>Finally, there is also <strong>EU-hosted sGTM</strong>. Unlike the other tools mentioned, this is not a European product — but you can configure it entirely within Europe. You can manage this yourself or outsource it to parties like Stape. However, you will still be subject to the known limitations and risks of sGTM.</p><h3>Web Analytics Platforms</h3><p>Web analytics requires a deeper set of criteria because of its greater functional complexity. We use six main categories:</p><ol><li><strong>Reporting: </strong>From standard reports to real-time insights, dashboards, attribution, segmentation, and predictive analytics</li><li><strong>Ease of use: </strong>Collaboration options, UI design, data processing speed, and management tools</li><li><strong>Data integrations: </strong>Import/export, APIs, native connections, and ecosystem extensions</li><li><strong>Data &amp; implementation: </strong>Flexibility of the data model, availability of prebuilt options, SDKs for apps</li><li><strong>Compliance: </strong>Consent management, self-hosting, and control over data processing</li><li><strong>Other factors: </strong>Cost structure, vendor support, and quality of the user community</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tJkBlf3TBybBvP-Exc8jYg.avif" /></figure><p><strong>Piano Analytics</strong> is the most powerful option in terms of reporting and data processing, ideal for enterprise use cases, but less accessible and affordable for smaller teams.<br><strong>Piwik PRO</strong> provides a strong all-round experience, with excellent compliance, support, and solid dashboards.</p><p><strong>Plausible</strong> and <strong>Fathom</strong> are extremely lightweight, privacy-friendly, and easy to use, but lack advanced features like segmentation or custom attribution.<br><strong>Matomo</strong> sits somewhere in between: open source, flexible in terms of hosting, but limited in reporting depth and integrations.</p><h3>What about hybrid approaches?</h3><p>Migration does not need to be an all-or-nothing decision. There are several hybrid approaches that help you mitigate risk and maintain flexibility. Examples include self-hosting Google Tag Manager scripts to bring more client-side tracking under your control or deploying a server-side tag management configuration via a European cloud provider.</p><p>Some organisations also choose to run two web analytics platforms in parallel: one for day-to-day operations and another as a privacy-friendly alternative. Whatever you choose, make sure your implementation remains agnostic, so you can quickly adapt when legislation, technology, or company policy changes. This helps you avoid being locked in and prevents costly and lengthy migrations later.</p><h3>Conclusion</h3><p>The digital data infrastructure landscape is changing rapidly due to geopolitics, regulatory shifts, and growing concerns about data sovereignty. It makes sense to prepare your organisation now, rather than wait until you are forced to act.</p><p>This does not mean you must immediately abandon US solutions — they often offer powerful features and mature support. At the same time, European alternatives are rapidly maturing and already excel in areas such as compliance and transparency.</p><p>By exploring which platforms best fit your strategy and technical maturity today, you give yourself the freedom to move gradually towards a future-proof setup. Choose deliberately, implement agnostically, and stay agile.</p><p>Are you considering a migration or looking for a sparring partner to explore your options? We’d be happy to help as your data partner — <a href="https://digital-power.com/en/contact/">get in touch or schedule a call.</a></p><h3>This is an article by Bram Ooms</h3><p>Bram started as a Technical Web Analyst in 2019, where he focused on data implementations at clients such as Univé, DPG Media, Boels and Vodafone. Through his experiences with the impact of legislation on enabling data flow he developed an interest in data privacy, which he is now actively pursuing within Digital Power.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f479a613146" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to migrate your data warehouse]]></title>
            <link>https://medium.com/@digitalpower/how-to-migrate-your-data-warehouse-bcdafe0e2185?source=rss-e272dacd376------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcdafe0e2185</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-warehousing]]></category>
            <category><![CDATA[data-warehouse-migration]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Digital Power]]></dc:creator>
            <pubDate>Fri, 10 Oct 2025 08:54:46 GMT</pubDate>
            <atom:updated>2025-10-10T08:54:46.769Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GEBYMpKhyZ0WA14sWm3CdQ.avif" /></figure><h3>A strategic move from big tech to European cloud solutions</h3><p>If you’ve decided to migrate your data warehouse to a European environment, taking a structured approach is key. This blog focuses on the steps needed for a smooth and successful transition.</p><p><em>This article is part of the “EuroStack” series, in which we help organisations make the transition to European alternatives for their data and analytics environment.</em></p><p><strong>Series overview:</strong></p><ul><li><a href="https://digital-power.com/en/inspiration/is-your-organisation-ready-for-independence-from-the-us-and-migrate-to-eurostacks/"><strong>Blog 1</strong> — Is your organisation ready for independence from the US and migrate to EuroStack?</a></li><li><a href="https://digital-power.com/en/inspiration/what-european-data-warehouse-solutions-are-available/"><strong>Blog 2</strong> — What European data warehouse solutions are available?</a></li><li><strong>Blog 3</strong> <em>(current article)</em> — Migrating to a European data warehouse: where to start?</li><li><a href="https://digital-power.com/en/inspiration/migrating-to-european-tag-management-and-analytics-platforms/"><strong>Blog 4</strong> — Migrating to European tag management and analytics solutions: where to start?</a></li></ul><h3>This article in short</h3><p><em>This article summarises the eight essential steps for migrating your data warehouse to a European cloud solution. It covers defining objectives, assessing your current setup, designing architecture, selecting a migration strategy, and moving over in stages. For full details, checklists, and best-practice examples, you can download our migration guide.</em></p><p><strong>We identify the following eight steps:</strong></p><ol><li>Define migration objectives and ensure internal alignment</li><li>Assess your current data warehouse and gather business requirements</li><li>Design target architecture and governance</li><li>Choose a migration strategy</li><li>Implement the initial use case</li><li>Validate the initial implementation</li><li>Carry out the full migration</li><li>Decide on the final cut-over</li></ol><p>You can download our migration guide for a detailed step-by-step plan. This article is a concise summary, ideal if you want to understand the process before you start the full migration.</p><p>Download our mirgration guide <a href="https://digital-power.com/en/inspiration/download-migration-guide-for-modern-data-warehousing/">here</a>.</p><h3>1. Define migration objectives and ensure internal alignment</h3><p>Before you begin, make sure everyone is aligned on the main reasons for moving away from your current setup. When moving to a European cloud, these reasons may include data sovereignty, better regulatory compliance, or reducing dependency on the big tech ecosystem. Quantify these drivers where possible, and secure buy-in from all relevant stakeholders.</p><h3>2. Assess your current data warehouse and gather business requirements</h3><p>A clear understanding of your existing environment and how it is used across the business forms the foundation for your new solution. Alongside reviewing current use cases, gather requirements for potential future use cases to ensure the new warehouse supports them.</p><p>This involves discussions with stakeholders ranging from business analysts and engineers to marketers and managers. Map all data sources, data models, and integrations to understand the current structure. Gather relevant data governance policies for the new system, and analyse major cost drivers to identify opportunities for savings.</p><h3>3. Design target architecture and governance</h3><p>Using the business requirements, create a visual of the technical architecture for the new warehouse, including all relevant compute, storage, and orchestration components. Define governance rules and processes — such as data ownership, access management, and handling of personal data (PII) — to ensure safe and compliant use of data. Consider incorporating a data catalogue to help govern your data assets.</p><h3>4. Choose a migration strategy</h3><p>Set your migration strategy before creating a detailed plan. Will you migrate all data sources up-front, or start with those required for the first high-impact use case? We recommend starting small with the highest-value case and building out the model from there.</p><h3>5. Implement the initial use case</h3><p>There are two main streams of work here:</p><ul><li>Setting up the platform infrastructure and ingesting the first data sources</li><li>Modelling the ingested raw data according to the business requirements</li></ul><p>Once this is complete, you can proceed to testing the initial implementation.</p><h3>6. Validate the initial implementation</h3><p>Run thorough checks on the first use case. Compare outputs from the old and new systems, check query performance, and run integration tests. Have business users perform user acceptance testing via the BI tool or application that will consume the modelled data.</p><h3>7. Carry out the full migration</h3><p>Once the initial use case is successfully validated, proceed with the migration of remaining data sources. Ingest these one by one, modelling them to fit the new structure. It’s common to redesign and improve models at this stage based on fresh business insights.</p><p>Update BI tools and other connected applications to ensure a smooth transition. Expect some manual adjustments in BI tools depending on changes to models and schemas.</p><h3>8. Decide on the final cut-over</h3><p>This is the definitive switch from the old big tech warehouse to the new European solution. Only cut over once all use cases are migrated, tested, and users are fully onboarded. Remember to remove any unused resources to save costs.</p><h3>Conclusion</h3><p>As these migration steps show, moving your data warehouse to a European alternative requires a structured approach. You can find more detail on each step, including deliverables, in our migration guide.</p><p>At Digital Power, we help clients successfully navigate this journey. <a href="https://digital-power.com/en/solutions/data-warehousing/">Find out more about our data warehousing solution</a> and read practical use cases.</p><p><strong>This is an article by Casper Damen.</strong></p><p>Casper Damen is a senior Data Engineer. He has a passion for automation, cloud- and data pipelines. By applying software development best practices to analytics, he ensures more reliable, maintainable and scalable data solutions. Over the years Casper has built several data platforms in multiple cloud providers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bcdafe0e2185" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What European data warehouse solutions are available?]]></title>
            <link>https://medium.com/@digitalpower/what-european-data-warehouse-solutions-are-available-78e1169ff11c?source=rss-e272dacd376------2</link>
            <guid isPermaLink="false">https://medium.com/p/78e1169ff11c</guid>
            <category><![CDATA[european-tech]]></category>
            <category><![CDATA[data-warehousing]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Digital Power]]></dc:creator>
            <pubDate>Fri, 10 Oct 2025 08:50:37 GMT</pubDate>
            <atom:updated>2025-10-10T08:50:37.528Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B2pkE3nqeQBWIklSCS_7zw.avif" /></figure><h3>An evaluation of data warehouse alternatives</h3><p>Due to geopolitical developments and growing concerns about data sovereignty, more and more businesses are looking to reduce their dependence on US cloud providers.</p><p>A logical next step is to explore European alternatives <strong>(EuroStack)</strong>. After all, access to business-critical data should never be at risk. But where do you store your data securely, and how do you set up a data warehouse environment that is just as powerful and user-friendly as solutions like Databricks, BigQuery or Snowflake?</p><p><em>This article is part of the “EuroStack” series, in which we help organisations make the transition to European alternatives for their data and analytics environment.</em></p><p><strong>Series overview:</strong></p><ul><li><a href="https://digital-power.com/en/inspiration/is-your-organisation-ready-for-independence-from-the-us-and-migrate-to-eurostacks/"><strong>Blog 1</strong> — Is your organisation ready for independence from the US and migrate to EuroStack?</a></li><li><strong>Blog 2</strong> <em>(current article)</em> — What European data warehouse solutions are available?</li><li><a href="https://digital-power.com/en/inspiration/how-to-migrate-your-data-warehouse/"><strong>Blog 3</strong> — Migrating to a European data warehouse: where to start?</a></li><li><a href="https://digital-power.com/en/inspiration/migrating-to-european-tag-management-and-analytics-platforms/"><strong>Blog 4</strong> — Migrating to European tag management and analytics solutions: where to start?</a></li></ul><h3>This article in short</h3><ul><li><em>You want to protect your data and reduce dependence on US cloud providers, especially with fast-changing geopolitics and regulations.</em></li><li><em>European alternatives (EuroStack) are emerging, and you face a choice: the convenience of a managed service or the full control of self-hosting.</em></li><li><em>Managed EU data warehouses exist but still don’t match BigQuery, Snowflake, or Databricks in ease of use and completeness.</em></li><li><em>Choosing a self-hosted setup with open-source tools like Kubernetes, ClickHouse, and Airflow means more investment in operations and DevOps skills.</em></li><li><em>A self-hosted setup is more effort, but you gain sovereignty, flexibility, and less dependency on major vendors in return.</em></li><li><em>Start small with a proof of concept on European infrastructure to gain experience, reduce risks, and prepare for the future.</em></li></ul><h3>Three main approaches</h3><p>There are three main approaches to building a future-proof European data infrastructure:</p><ol><li><strong>Managed</strong> data warehouse solutions from European cloud providers</li><li><strong>Self-hosted</strong> open-source data warehouse stacks on European infrastructure</li><li><strong>On-premise hosting</strong> (not covered here)</li></ol><p>On-premise gives maximum control but demands heavy investment in hardware, security, and specialist skills. That’s why we focus on cloud-based options in this blog.</p><p>Our scope is <strong>Online Analytical Processing (OLAP)</strong> — complex analysis and reporting on large volumes of historical data — rather than <strong>Online Transaction Processing (OLTP)</strong>, which handles high-speed daily transactions. As transactional data is becoming more relevant for AI and ML, we also assess the data streaming capabilities of each solution.</p><h3>Solution 1: Managed data warehouse solutions</h3><p>Managed data warehouses have gained a dominant position in many organisations’ data infrastructure in recent years. These “plug-and-play” solutions, such as Databricks, BigQuery, and Snowflake, enable businesses to set up robust data warehouse environments without requiring in-depth technical expertise on the underlying infrastructure.</p><p>These platforms, typically hosted on hyperscalers like AWS, Azure, and Google Cloud, stand out because of:</p><ul><li>Automatic scalability</li><li>Minimal infrastructure maintenance</li><li>Rapid implementation</li><li>Seamless integration with other cloud services and BI tools</li></ul><p>For many organisations, these are decisive advantages. The question remains: are European cloud providers ready to deliver a comparable experience?</p><h3>Research into 5 European cloud providers</h3><p>We compared 5 European cloud providers on the key components for a complete data warehouse stack:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pH65DflxTXxHAn_dam1dmw.avif" /></figure><p><strong>Key patterns observed:</strong></p><ol><li><strong>Data storage and management:</strong> Object storage (often S3-compatible) and standard backup and disaster recovery features are widely available.</li><li><strong>Compute and processing:</strong> Virtual machines and containers are standard; serverless compute is rare. Kubernetes is reasonably well supported, but managed big data engines such as Apache Spark or Flink are generally absent.</li><li><strong>Databases and analytics:</strong> Relational and NoSQL databases are widely available, while column-oriented databases and analytics engines are rare.</li><li><strong>Data orchestration and integration:</strong> Streaming tools are scarce; managed ETL/ELT services are entirely absent.</li><li><strong>Developer experience:</strong> Terraform is often supported; secrets and user management are limited. Python SDKs are frequently still in an early development stage.</li><li><strong>Cost management and monitoring:</strong> Many providers lack a clear cost dashboard; budget alerts are uncommon.</li></ol><h3>Conclusion managed solutions</h3><p>Currently, no European cloud provider offers a fully managed data warehouse that rivals Databricks, BigQuery, or Snowflake. While many have the building blocks, none deliver a fully integrated, user-friendly platform. OVHCloud and Scaleway are developing their own, but still lack essentials like a governance-ready data catalogue, orchestration, and built-in AI/ML and streaming support.</p><p>For alternatives, Exasol offers a high-performance warehouse engine deployable on European clouds. For real-time use cases such as marketing personalisation, Ververica provides an end-to-end streaming platform based on Apache Flink, Paimon, and Fluss, hostable in your own environment.</p><h3>Solution 2: Self-hosting a data warehouse on European infrastructure</h3><p>Because managed European solutions are currently limited, self-hosting an open-source data warehouse provides a realistic alternative. However, this does require technical expertise and sufficient DevOps capacity — meaning your organisation must have the people and resources to not only run the software but also maintain the underlying infrastructure securely, stably, and up to date. This includes cloud hosting expertise, deployment automation, monitoring, backups, and troubleshooting.</p><p>Based on our research, we focused on a <a href="https://github.com/digital-power/open-source-data-warehouse-poc">proof of concept on Scaleway</a>, which stood out as a complete and developer-friendly European provider in our testing.</p><p>The setup consists of:</p><ul><li>Kubernetes cluster as the foundation</li><li>Apache Airflow for data orchestration</li><li>ClickHouse as the analytical database</li><li>dbt for SQL-based data transformations</li><li>Traefik for routing web traffic</li><li>Python for data ingestion</li><li>Supporting managed services such as Object Store, Key Vault, and Container Registry</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e5ntTcoSXCYUlskLQlL7UA.avif" /></figure><h3>Conclusion self-hosted solutions</h3><p>Most European cloud providers now offer managed Kubernetes clusters. This makes it possible to build your own data warehouse using open-source technologies. While it requires significant DevOps investment, it allows you to create a powerful European data warehouse solution that is fully under your own control. The advantages and disadvantages of hosting your own environment are summarised below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rQTGdpJmJ9Rn8w3pK3dg4w.avif" /></figure><h3>Final thoughts</h3><p>Right now, no European cloud provider offers a fully managed data warehouse on par with Databricks, BigQuery, or Snowflake. For full data sovereignty and independence from US Big Tech, self-hosting open-source technologies is the most viable route.</p><p>While this requires more effort in infrastructure and management, tools like Helm charts and Operator templates make it easier to spin up a Kubernetes-based setup. With ClickHouse and Airflow, you can have a functional data warehouse running in weeks.</p><p>We recommend starting with a proof of concept on European infrastructure to build expertise, reduce reliance on US providers, and prepare for future geopolitical or legal changes.</p><p>At Digital Power, we are happy to <a href="https://digital-power.com/en/contact/">support you</a> in this process: from proof of concept to implementation. As an independent data consultancy with extensive experience in setting up data platforms, we can determine the best technical solution based on your specific business needs.</p><p><strong>This is an article by Casper Damen.</strong></p><p>Casper Damen is a senior Data Engineer. He has a passion for automation, cloud- and data pipelines. By applying software development best practices to analytics, he ensures more reliable, maintainable and scalable data solutions. Over the years Casper has built several data platforms in multiple cloud providers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78e1169ff11c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>