<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Samuel Van Ackere on Medium]]></title>
        <description><![CDATA[Stories by Samuel Van Ackere on Medium]]></description>
        <link>https://medium.com/@samuel.vanackere?source=rss-86fd8b1c761e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*LVveLN7gwmgl0SHBO-t_sw.jpeg</url>
            <title>Stories by Samuel Van Ackere on Medium</title>
            <link>https://medium.com/@samuel.vanackere?source=rss-86fd8b1c761e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:18:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@samuel.vanackere/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How We Built a Smart Matching Algorithm to Connect Two of Belgiums’ Most Important Registers]]></title>
            <link>https://medium.com/ai-analytics-digital-flanders/how-we-built-a-smart-matching-algorithm-to-connect-two-of-belgiums-most-important-registers-2a9a82406865?source=rss-86fd8b1c761e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a9a82406865</guid>
            <dc:creator><![CDATA[Samuel Van Ackere]]></dc:creator>
            <pubDate>Tue, 12 May 2026 13:36:00 GMT</pubDate>
            <atom:updated>2026-05-12T13:37:07.973Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Bridging the gap between building units and cadastral parcels, using graph analysis, probabilistic matching, and AI.</em></p><p>In Flanders, two government registers hold critical information about every building: the <strong>GRAR</strong> (Gebouwen- en Adressenregister, the Buildings and Address Register) and the <strong>AAPD</strong> (Algemene Administratie van de Patrimoniumdocumentatie, the Federal Cadastral Register). One knows where people live. The other knows who owns what. But until now, these two registers were not linked at the level where it matters most: the individual apartment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EqeymAx5iFPksQUE" /><figcaption>Photo by <a href="https://unsplash.com/@jasonsung?utm_source=medium&amp;utm_medium=referral">Jason Sung</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The Data Insights team at Digitaal Vlaanderen set out to change that. What started as a data quality initiative grew into a full-blown AI matching pipeline that now connects hundreds of thousands of building units to their correct cadastral parcels. This article tells the story of why that link matters, why it is surprisingly hard to make, and how we solved it.</p><h3>Why does this link matter?</h3><p>The GRAR is maintained by Digitaal Vlaanderen and contains information about buildings, building units, and their official addresses. A building unit (gebouweenheid) is the smallest functional unit within a building: think of an individual apartment, a studio, or a commercial space with its own entrance.</p><p>The AAPD is maintained by the Federal Public Service Finance and contains cadastral parcels and partitions, including who owns what and the cadastral address of each partition. However, the cadastral address usually lacks the precision of a full apartment-level bus number.</p><p>Connecting these two registers at the building unit level unlocks several high-impact use cases. Energy performance certificates (EPC) for apartments can be correctly assigned. Enforcement of energy regulations becomes possible at the unit level rather than the building level. And perhaps most importantly, property tax (onroerende voorheffing) can be assessed more accurately when each building unit is linked to its corresponding cadastral partition.</p><p>Without this link, government agencies are working with an incomplete picture. The data exists; it is just not connected.</p><p>Why is this so hard?</p><p>For a detached family home, it is simple. The GRAR says “Ter Elstlaan 39, Leuven” and the cadastral register says the same thing. One building unit, one parcel, done.</p><p>But Flanders has a lot of apartment buildings. And that is where things get complicated.</p><p>Consider a building at “Onze-Lieve-Vrouwstraat 34, Leuven.” The GRAR lists six apartments with bus numbers 101, 102, 201, 202, 301, and 302. The cadastral register lists partitions for the same address, but instead of bus numbers it uses descriptions like “A1/APP1A/”, “A2/APP2B/”, or just “HA.GV.” There is no shared identifier. The naming conventions are completely different. And sometimes the street names do not even match exactly between the two systems.</p><p>Multiply this by hundreds of thousands of building units across Flanders, and you have a serious data matching challenge.</p><h3>Our approach: identify, then solve</h3><p>We quickly realized that no single algorithm could handle the full variety of cases. Instead, we adopted a two-phase approach: first identify the different problem types, then develop a targeted solution for each.</p><h3>Phase 1: graph analysis and exact matching</h3><p>The first step was understanding how entities in the data model relate to each other. Building units, addresses, plan parcels, and cadastral parcels are all connected through various linking tables, and these connections have complex many-to-many relationships.</p><p>We performed a <strong>graph analysis</strong> using PySpark and GraphFrames. Every building unit, address, and parcel became a node in a graph, and every link between them became an edge. By finding connected components in this graph, we identified <strong>clusters</strong>: sets of entities that are directly or indirectly linked through the data model. These clusters became our fundamental unit of analysis.</p><p>Next, we ran three rounds of exact matching: one using the data model’s own structure, one using AAPD address data, and one using National Register (Rijksregister) address data. After combining the results, every building unit fell into one of five categories:</p><p><strong>Suspicious cases</strong> (data quality issues detected), flagged for manual review. <strong>Unique match via the data model</strong> (about 54% of all building units), already solved, no further action needed. <strong>Common areas without a unique match</strong> (about 2.3%), handled through geometric point-in-polygon analysis. <strong>Multiple candidates with one confirmed mapping</strong> (about 3%), solvable with deterministic address comparison. <strong>Multiple candidates without a confirmed mapping</strong> (about 21%), the hard cases requiring AI.</p><p>That last category, roughly one in five building units, is where the real challenge lives.Phase 2: AI-powered probabilistic matching</p><p>For building units with multiple candidate parcels and no clear mapping, we needed a fundamentally different approach. The goal: for each building unit, produce a <strong>ranked list of candidate parcels</strong> with a confidence score, so that the best match can be identified and, where the confidence is high enough, automatically accepted.</p><h3>Thinking like a detective</h3><p>A useful analogy is a detective investigation. With no evidence, any suspect is equally likely. A single clue, say a matching street name, narrows the field but does not close the case. Add a second clue, the municipality also matches, and confidence rises. Add a third, a floor number extracted from a free-text description matches the cadastral partition, and you are approaching certainty.</p><p>Our matching algorithm works the same way. It gathers multiple “clues” from the data, weighs their reliability, and combines them into a probability score. The more signals that align, the more confident the match.</p><h3>Data enrichment: getting more clues</h3><p>A big part of the challenge was extracting structured information from unstructured fields. Cadastral descriptions like “A2/APP2A/” or “HA.GV” encode floor numbers and positions, but in inconsistent, abbreviated formats. On the GRAR side, EPC certificates contain free-text descriptions like “Studio on the 4th floor, left side” or “Located on the ground floor.”</p><p>We built extensive preprocessing pipelines to clean, normalize, and extract features from these messy fields. For floor number inference, we combined deterministic rules (regex patterns on bus numbers and cadastral descriptions) with an LLM (Llama 3.3 70B hosted on Databricks) to parse free-text location descriptions. This hybrid approach gave us floor information for a much larger share of building units than either method alone.</p><h3>Model selection: simple versus complex</h3><p>A fundamental trade-off in any matching system is between model simplicity and the ability to handle complex patterns. We experimented with several approaches.</p><p><strong>Simpler models</strong> like Splink, a probabilistic record linkage framework, work well for structured fields (municipality, street, house number) but struggle with the unstructured cadastral descriptions that often hold the key distinguishing information.</p><p><strong>Deep learning models</strong> like DeepMatcher and Ditto can handle unstructured text by learning complex patterns, but they are heavier to run and harder to interpret.</p><p>Our best results came from <strong>Ditto</strong>, a transformer-based entity matching model built on RoBERTa. Ditto serializes both entities (building unit and cadastral partition) into a single text representation and applies text classification to predict whether they match. This approach is particularly well-suited for our problem because the features on both sides are asymmetric: an apartment’s bus number “201” needs to be matched against a description like “A2/APP2A/”, and a transformer model can learn these cross-field correspondences.The final pipeline</p><p>Our production pipeline uses Ditto with 3-fold cross-validation. The model is trained on building units where we have a confirmed match through the National Register, and then applied to the remaining unmatched units. The ensemble of three models produces an average match score with a standard deviation that indicates model agreement.</p><p>After scoring, a postprocessing step determines whether a match is confident enough to accept automatically. A prediction is considered reliable when it is the top-ranked candidate, exceeds the optimized threshold, and has a sufficient margin over the second-best candidate.</p><h3>Results</h3><p>The iterative approach paid off. Over the course of multiple experiments, adding features, refining preprocessing, and testing different model architectures, we steadily improved performance. Starting from a baseline of address comparison only (36% precision, 53% MAP), adding DeepMatcher brought us to 59% precision. The hybrid strategy combining fuzzy address comparison with Ditto and logistic regression reached 66% precision. The final Ditto end-to-end model achieved 69% precision and 78% MAP.</p><figure><img alt="The matching algorithm achieved a matching rate of 80.72%, with an average confidence score of 84.1%." src="https://cdn-images-1.medium.com/max/752/1*NcpTkkp7gnZ9qPpt_CRZHw.png" /></figure><p>When applying a confidence threshold, 80.72% of building units can be matched with an average confidence score of 84.1%. This means that hundreds of thousands of building units can be matched without human intervention, while uncertain cases are flagged for review with a helpful ranking of candidate matches.</p><p>Combined with the deterministic methods that handle the simpler cases, the full pipeline resolves a large majority of all building units across Flanders.</p><h3>Lessons learned</h3><p><strong>Problem decomposition beats one-size-fits-all.</strong> By categorizing building units into problem types first, we could apply the right tool for each situation, from simple lookups to AI models, rather than forcing everything through one pipeline.</p><p><strong>Feature engineering matters as much as model choice.</strong> Extracting floor numbers from free-text descriptions, cleaning inconsistent bus numbers, and inferring structure from cadastral codes contributed as much to accuracy gains as switching between model architectures.</p><p><strong>Graph analysis is a powerful starting point.</strong> Before any matching even begins, the cluster analysis already resolves more than half of all building units and provides a clean, workable structure for everything that follows.</p><p><strong>Evaluation design is critical.</strong> Using the National Register mapping as ground truth required careful thought. We had to ensure that the evaluation set was representative of the actual prediction target: building units without a confirmed mapping. This meant deliberately excluding certain features from the model to avoid overly optimistic results.</p><h3>What comes next?</h3><p>The matching pipeline continues to evolve. The team is working on incorporating new problem types, further improving accuracy through additional data sources, and building an integration layer that will make the match results available through coupling tables and APIs. A dispute resolution mechanism will allow domain experts to review and override uncertain matches, feeding corrections back into future model iterations.</p><p>The bigger picture is about interoperability between Flemish and Federal registers. When building units are reliably linked to cadastral parcels, it opens the door for more accurate policy enforcement, better citizen services, and a more connected government data ecosystem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a9a82406865" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-analytics-digital-flanders/how-we-built-a-smart-matching-algorithm-to-connect-two-of-belgiums-most-important-registers-2a9a82406865">How We Built a Smart Matching Algorithm to Connect Two of Belgiums’ Most Important Registers</a> was originally published in <a href="https://medium.com/ai-analytics-digital-flanders">AI &amp; Analytics Digital Flanders</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decision-making with Linked Data Event Streams and PowerBI]]></title>
            <link>https://medium.com/geekculture/decision-making-with-linked-data-event-streams-and-powerbi-5cd8379d32?source=rss-86fd8b1c761e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5cd8379d32</guid>
            <category><![CDATA[data-driven-decisions]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[linked-data-event-stream]]></category>
            <category><![CDATA[ldes]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Samuel Van Ackere]]></dc:creator>
            <pubDate>Wed, 15 Mar 2023 15:01:30 GMT</pubDate>
            <atom:updated>2023-03-15T15:01:30.945Z</atom:updated>
            <content:encoded><![CDATA[<p>How to monitor, analyse and make informed decisions with Linked Data Event Streams</p><figure><img alt="Linked data event stream, LDES, Linked Data Event Stream" src="https://cdn-images-1.medium.com/max/1024/0*NsSH_9b6f5XzBHBk" /><figcaption>Photo by <a href="https://unsplash.com/@firmbee?utm_source=medium&amp;utm_medium=referral">Firmbee.com</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In recent years, the use of <strong>Linked Data Event Stream (LDES) </strong>has grown rapidly to represent and share large amounts of real-time data. However, visualizing and monitoring fast-moving real-time data can be a challenge. This is where PowerBI comes in. It can be used to create powerful visualizations and dashboards of LDES data, making it easy to monitor the content and performance and use it as a <strong>decision support system</strong>.</p><p><strong>We explore how PowerBI can visualize LDES data and how it can be used as a decision support system. </strong>We will also discuss the benefits of using PowerBI for LDES data visualization and the key features that make it an ideal tool for this purpose.</p><h4>Synced dashboard in PowerBI</h4><p>A <strong>Linked Data Event Stream</strong> is a constant flow of immutable objects (such as version objects, sensor observations or archived representation) containing information about an event or state change originating from a system that continuously creates data.</p><p><a href="https://medium.com/@samuel.vanackere/linked-data-event-streams-explained-in-8-minutes-e1c76d077bb9">Linked Data Event Streams explained in 8 minutes</a></p><blockquote>Since Linked Data Event Streams are always in sync with the source (e.g. sensors, financial databases, etc.), <strong>every dashboard that consumes an LDES can be kept in sync and display real-time data.</strong></blockquote><p>PowerBI is an intelligence tool that enables users to connect to multiple data sources and create and share interactive and intuitive dashboards. One of its key features is the ability to visualize geospatial data in real time through a link with <strong>PostgreSQL</strong>. In this way, it allows users to create <strong>real-time</strong> custom visualizations from LDES streams that are consumed in sync in a PostgreSQL database.</p><figure><img alt="Linked data event stream, LDES, Linked Data Event Stream" src="https://cdn-images-1.medium.com/max/494/0*0ZRChBl_V7UYh0X6.png" /><figcaption>Image by author</figcaption></figure><p>The Linked Data Event Streams specification provides a standardized way of representing real-time geospatial data, which can be easily integrated into PowerBI.</p><p>This allows organizations to create dynamic and up-to-date dashboards that display real-time information about the events occurring in a specific geographic location. The characteristics of the Linked Data Event Streams specification ensure that the data is always in sync, providing organizations with an accurate and current representation of their data. PowerBI leverages this data to create interactive maps, heat maps, and other visualizations that help organizations better understand and analyse their geospatial data.</p><p>The Generic Information Public Domain (GIPOD) data stream is used in the example underneath. This data stream brings together all information about (earth) works, events and nuisances in the public domain. <a href="https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances?generatedAtTime=2021-01-06T18:04:12.173Z">This data stream is published as an LDES</a>, and is consumed into PostgreSQL. Afterwards, this LDES is visualized in PowerBI.</p><p>Underneath you can see a part of a fragment of the published <a href="https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances?generatedAtTime=2021-01-06T18:04:12.173Z">LDES</a>:</p><pre>{<br>&quot;@context&quot;: [<br>&quot;<a href="https://private-api.gipod.vlaanderen.be/api/v1/context/gipod.jsonld">https://private-api.gipod.vlaanderen.be/api/v1/context/gipod.jsonld&quot;</a><br>],<br>&quot;@id&quot;: &quot;<a href="https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances?generatedAtTime=2021-01-06T18:04:12.173Z">https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances?generatedAtTime=2021-01-06T18:04:12.173Z&quot;</a>,<br>&quot;@type&quot;: &quot;Node&quot;,<br>&quot;viewOf&quot;: &quot;<a href="https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances">https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances&quot;</a>,<br>&quot;collectionInfo&quot;: {<br>&quot;@id&quot;: &quot;<a href="https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances">https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances&quot;</a>,<br>&quot;@type&quot;: &quot;EventStream&quot;,<br>&quot;shape&quot;: &quot;<a href="https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances/shape">https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances/shape&quot;</a>,<br>&quot;timestampPath&quot;: &quot;prov:generatedAtTime&quot;,<br>&quot;versionOfPath&quot;: &quot;dct:isVersionOf&quot;<br>},<br>&quot;tree:relation&quot;: [<br>{<br>&quot;tree:node&quot;: &quot;<a href="https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances?generatedAtTime=2021-01-06T18:05:43.477Z">https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances?generatedAtTime=2021-01-06T18:05:43.477Z&quot;</a>,<br>&quot;@type&quot;: &quot;tree:GreaterThanRelation&quot;,<br>&quot;tree:path&quot;: &quot;prov:generatedAtTime&quot;,<br>&quot;tree:value&quot;: &quot;2021-01-06T18:05:43.477Z&quot;<br>}<br>],<br>&quot;items&quot;: [<br>{<br>&quot;@id&quot;: &quot;<a href="https://private-api.gipod.vlaanderen.be/api/v1/mobility-hindrances/10590330/15">https://private-api.gipod.vlaanderen.be/api/v1/mobility-hindrances/10590330/15&quot;</a>,<br>&quot;@type&quot;: &quot;MobilityHindrance&quot;,<br>&quot;gipodId&quot;: 10590330,<br>&quot;identifier&quot;: [<br>{<br>&quot;@type&quot;: &quot;Identifier&quot;,<br>&quot;Identifier.identifier&quot;: {<br>&quot;value&quot;: &quot;10590330&quot;,<br>&quot;type&quot;: &quot;gipodId&quot;<br>},<br>&quot;assignedByName&quot;: &quot;<a href="https://gipod.vlaanderen.be/">https://gipod.vlaanderen.be&quot;</a><br>}<br>],<br>&quot;isConsequenceOf&quot;: [<br>{<br>&quot;@id&quot;: &quot;/api/v1/events/9197361&quot;,<br>&quot;@type&quot;: &quot;Event&quot;,<br>&quot;gipodId&quot;: 9197361<br>}<br>],<br>&quot;description&quot;: &quot;Verhuiswagen&quot;,<br>&quot;owner&quot;: {<br>&quot;isVersionOf&quot;: &quot;/api/v1/organisations/14768eaa-6c54-0075-837a-90178a74ce8c&quot;,<br>&quot;@type&quot;: &quot;Organisation&quot;,<br>&quot;preferredName&quot;: &quot;Gemeente Kapellen&quot;<br>},<br>&quot;contactOrganisation&quot;: [],<br>&quot;zone&quot;: [<br>{<br>&quot;@id&quot;: &quot;/api/v1/mobility-hindrances/10590330/zones/019e77f3-e2fd-4624-b6d7-8cfdbe57683f&quot;,<br>&quot;@type&quot;: &quot;Zone&quot;,<br>&quot;consequence&quot;: [],<br>&quot;geometry&quot;: {<br>&quot;@type&quot;: &quot;Geometry&quot;,<br>&quot;wkt&quot;: &quot;&lt;<a href="http://www.opengis.net/def/crs/EPSG/9.9.1/31370&gt;">http://www.opengis.net/def/crs/EPSG/9.9.1/31370&gt;</a> POLYGON ((154536.4910038893 221859.32163325883, 154538.4706818894 221854.19150346238, 154540.80870575312 221855.08900541347, 154538.82902577805 221860.21913343482, 154536.4910038893 221859.32163325883))&quot;<br>},<br>&quot;zoneType&quot;: {<br>&quot;@id&quot;: &quot;/api/v1/taxonomies/zonetypes/0fb72ef7-6ac9-4a70-b295-a30ea215d250&quot;,<br>&quot;prefLabel&quot;: &quot;HinderZone&quot;<br>}<br>}<br>],<br>&quot;period&quot;: [<br>{<br>&quot;@type&quot;: &quot;Period&quot;,<br>&quot;start&quot;: &quot;2021-02-26T23:00:00Z&quot;,<br>&quot;end&quot;: &quot;2021-02-27T22:30:00Z&quot;<br>}<br>],<br>&quot;timeSchedule&quot;: null,<br>&quot;permittedBy&quot;: [],<br>&quot;status&quot;: {<br>&quot;@id&quot;: &quot;/api/v1/taxonomies/statuses/a411c53e-db33-436a-9bb9-d62d535b661d&quot;,<br>&quot;prefLabel&quot;: &quot;Onbekend&quot;<br>},<br>&quot;generatedAtTime&quot;: &quot;2021-01-06T18:04:12.173Z&quot;,<br>&quot;eventName&quot;: &quot;MobilityHindranceWasImportedFromLegacy&quot;,<br>&quot;isVersionOf&quot;: &quot;/api/v1/mobility-hindrances/10590330&quot;,<br>&quot;memberOf&quot;: &quot;<a href="https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances">https://private-api.gipod.vlaanderen.be/api/v1/ldes/mobility-hindrances&quot;</a>,<br>&quot;lastModifiedOn&quot;: &quot;2021-01-06T17:11:43.167Z&quot;,<br>&quot;lastModifiedBy&quot;: {<br>&quot;isVersionOf&quot;: &quot;/api/v1/organisations/14768eaa-6c54-0075-837a-90178a74ce8c&quot;,<br>&quot;@type&quot;: &quot;Organisation&quot;,<br>&quot;preferredName&quot;: &quot;Gemeente Kapellen&quot;<br>},<br>&quot;createdOn&quot;: &quot;2021-01-06T17:11:43.167Z&quot;,<br>&quot;createdBy&quot;: {<br>&quot;isVersionOf&quot;: &quot;/api/v1/organisations/14768eaa-6c54-0075-837a-90178a74ce8c&quot;,<br>&quot;@type&quot;: &quot;Organisation&quot;,<br>&quot;preferredName&quot;: &quot;Gemeente Kapellen&quot;<br>}<br>},...</pre><h4>LDES data flow</h4><p>To store Linked Data Event Streams in PostgreSQL/PostGIS, Apache NiFi is utilized. Apache NiFi is a highly flexible and scalable open-source platform that streamlines data processing and system integration. The platform boasts a user-friendly web interface allowing users to easily create, manage, and monitor data flows. It offers a variety of pre-configured connectors and processors for executing data processing operations.</p><p>The structure of this data transformation flow has already been described in the next article (data flow is the same for importing data to PostgreSQL or to TimescaleDB):</p><p><a href="https://pub.towardsai.net/linked-data-event-streams-and-timescaledb-for-real-time-timeseries-data-management-9e82ba336f82">Linked Data Event Streams and TimescaleDB for Real-time Timeseries Data Management</a></p><p>Power BI is a widely used business intelligence tool encompassing services, applications, and connectors. It enables you to extract raw data from multiple sources and generate informative reports. To link Power BI with a data source like PostgreSQL, an appropriate ODBC driver can be utilized.</p><figure><img alt="Linked Data Event Stream, LDES, Linked data event stream" src="https://cdn-images-1.medium.com/max/1024/1*IVS2LBwoGj73WaEyPNQVHg.png" /><figcaption>Source: Image by author</figcaption></figure><h4>Full code</h4><p>The full code is available on Github if you want to dive into this use case. Follow the instructions to ingest Linked Data Event Stream into Power BI.</p><p><a href="https://github.com/Informatievlaanderen/VSDS-LDESDemo/tree/master/geoserver">VSDS-LDESDemo/geoserver at master · Informatievlaanderen/VSDS-LDESDemo</a></p><h4>Conclusion</h4><p>In conclusion, Power BI provides a powerful tool for <strong>visualizing Linked Data Event Streams in real-time</strong>, enabling effective analysis and interpretation of large and complex data sets. Its interactive dashboards, customizable visualizations, and real-time streaming capabilities make it a crucial tool for data-driven decisions.</p><p>If you like what you read, be sure to ❤️ it — as a writer, it means the world. Stay in touch by following me as an author.</p><p>Contributors to this article are <a href="https://github.com/ddvlanck">ddvlanck (Dwight Van Lancker) (github.com)</a>, <a href="https://github.com/sandervd">sandervd (Sander Van Dooren) (github.com)</a> at Smart Data Space (Digital Flanders). In a rapidly changing society, governments need to be more agile and resilient than ever. As a strategic partner, we realize and supervise digital transformation projects for Flemish and local governments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cd8379d32" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/decision-making-with-linked-data-event-streams-and-powerbi-5cd8379d32">Decision-making with Linked Data Event Streams and PowerBI</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-Time Data Linkage via Linked Data Event Streams]]></title>
            <link>https://pub.towardsai.net/real-time-data-linkage-via-linked-data-event-streams-e1aab3090b40?source=rss-86fd8b1c761e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1aab3090b40</guid>
            <category><![CDATA[linked-data-event-stream]]></category>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[linked-data]]></category>
            <category><![CDATA[ldes]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Samuel Van Ackere]]></dc:creator>
            <pubDate>Wed, 15 Mar 2023 00:02:08 GMT</pubDate>
            <atom:updated>2023-03-15T00:02:08.618Z</atom:updated>
            <content:encoded><![CDATA[<h4>Seamless and efficient interlinking of a data stream across various domains and applications</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tzg6ulQShlg3OURQ" /><figcaption>Photo by <a href="https://unsplash.com/@heftiba?utm_source=medium&amp;utm_medium=referral">Toa Heftiba</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Real-time interchanging data across domains and applications is challenging; data format incompatibility, latency and outdated data sets, quality issues, and lack of metadata and context.</p><p>Using Linked Data Event Stream (LDES), data can be fluently shared between different systems and organizations. <strong>In this way, companies and organizations can ensure that their data is well-structured, interoperable, and easily consumable by other systems and services. </strong>LDES has emerged as a standard for representing and sharing up-to-date data streams.</p><p>This proof of concept explores how LDES can interchange and combine three different data sets: address registers, parcels, and building units assuming different organizations publish them.</p><blockquote><strong>In this way, companies and organizations can ensure that their data is well-structured, interoperable, and easily consumable by other systems and services.</strong></blockquote><p>If you are not familiar with Linked Data Event Stream, you can find more information about it in this article:</p><p><a href="https://medium.com/@samuel.vanackere/linked-data-event-streams-explained-in-8-minutes-e1c76d077bb9">Linked Data Event Streams explained in 8 minutes</a></p><h4>Onboarding of a Linked Data Event Stream</h4><p>Every data stream can finally be transformed into a Linked Data Event Stream with some conversions. In this example, we start from data streams outputted via Apache Kafka.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/1*R-HcgXMfZ2L4ijwxmzBqEQ.png" /><figcaption>Image by author</figcaption></figure><p>A data flow is configured in Apache Nifi that converts these Kafka topics to an LDES server. The LDES server is a configurable component used to ingest, store, transform, and (re-)publish a Linked Data Event Stream.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6l_0uUzphq8XgUi5qKlCMQ.png" /><figcaption>Publishing data of addresses, building units and parcels as LDES (Image by author)</figcaption></figure><p>In this case, data streams from <a href="https://grar.smartdataspace.dev-vlaanderen.be/parcels/by-page">parcels</a>, <a href="https://grar.smartdataspace.dev-vlaanderen.be/building-units/by-page">building</a> units, and <a href="https://grar.smartdataspace.dev-vlaanderen.be/addresses/by-time">data registry</a> from Flanders are converted into an LDES to interchange and combine afterward. These LDES all have the same structure, are always in sync with the source data, and are presented in a machine-readable format, including metadata and relationships, enabling automated discovery and integration of information across various domains and applications. The figure underneath shows one part of the address registry LDES fragment. It embeds the context of the data and information about the city or municipality, street name, full address, and location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OcQlJA1xuHOW2Bdbvp9pmw.png" /><figcaption>Image of author</figcaption></figure><h4>Interlink multiple Linked Data Event Streams</h4><p>The value of publishing data sets through an LDES specification lies in the easy interlinking of data between companies and organizations over the Web. With this capability, data consumers can consume various data sets using an LDES Client, enabling a streamlined approach to accessing diverse data sources.</p><p>For this purpose, the three Linked Data Event streams are stored in a <a href="https://graphdb.ontotext.com/">GraphDB</a> to facilitate efficient and effective data consumption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*8YN0OTX88C8-borkD28I4w.png" /><figcaption>Image for author</figcaption></figure><p>GraphDB supports complex semantic queries and inference, making discovering meaningful relationships between different data sources possible.</p><p><a href="https://github.com/samuvack/ldes-grar">In this Github repo</a>, you will find a docker file with the configuration of GraphDB and Apache NiFi. A data flow is configured in Apache NiFi to convert these data streams into GraphDB. You will find an <a href="https://github.com/samuvack/ldes-grar/blob/main/NiFi_Flow.json">Apache NiFi configuration file</a> containing the necessary data flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lVyW_IUdVpghNM1HcyJqdQ.png" /><figcaption>Image by author</figcaption></figure><p>Once the Apache NiFi data flow is executed, the LDES members will flow into GraphDB. Subsequently, the data can be queried via SPARQL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4eq6rPaHWy0YbqoN9meCvA.png" /><figcaption>Image by author</figcaption></figure><p>GraphDB is a powerful tool for storing and querying graph data. Still, it also offers a range of visualization possibilities that allow users to explore and understand their data in new and intuitive ways. One of the most popular visualization features in GraphDB is Graph Explorer, which provides an interactive view of a graph and allows users to navigate and search through it in real-time. This can be particularly useful for understanding complex relationships between entities, such as in social networks or biological systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_wU8xUPh55l5LIZUwfAnhg.png" /><figcaption>Image by author</figcaption></figure><h4>SPARQL query</h4><p>SPARQL is the standard query language and protocol for Linked Open Data on the web or for <a href="https://www.ontotext.com/knowledgehub/fundamentals/what-is-rdf-triplestore/">RDF triplestores</a>. SPARQL enables the execution of semantic queries through data, offering a flexible and powerful method for searching, filtering, and retrieving structured information.</p><p>Below is an example of a SPARQL query is shown, which returns triples with information on parcels.</p><pre>PREFIX gebouwregister: &lt;https://<a href="http://basisregisters.vlaanderen.be/implementatiemodel/gebouwenregister">basisregisters.vlaanderen.be/implementatiemodel/gebouwenregister</a>#&gt;<br>PREFIX adres: &lt;https://<a href="http://data.vlaanderen.be/id/adres&gt;&lt;/a&gt;">data.vlaanderen.be/id/adres/&gt;</a><br>PREFIX perceel: &lt;https://<a href="http://data.vlaanderen.be/id/perceel&gt;&lt;/a&gt;">data.vlaanderen.be/id/perceel/&gt;</a><br>PREFIX rdf: &lt;http://<a href="http://www.w3.org/">www.w3.org/</a>1999/02/22-rdf-syntax-ns#&gt;<br>PREFIX prov: &lt;http://<a href="http://www.w3.org/ns/prov">www.w3.org/ns/prov</a>#&gt;<br>PREFIX ns: &lt;https://<a href="http://data.vlaanderen.be/ns/generiek">data.vlaanderen.be/ns/generiek</a>#&gt;<br><br>CONSTRUCT{<br>    <br>?perceel rdf:type ?type .<br>?perceel prov:generatedAtTime ?generatedAtTime .<br>?perceel ns:lokaleIdentificator ?lokaleIdentificator .<br>?perceel ns:naamruimte ?naamruimte .<br>?perceel ns:versieIdentificator ?versieIdentificator .<br><br>} where {<br> ?perceel gebouwregister:Adresseerbaar%20Object adres:2327687 .<br>    OPTIONAL { ?perceel rdf:type ?type .<br>         ?perceel prov:generatedAtTime ?generatedAtTime .<br>       ?perceel ns:lokaleIdentificator ?lokaleIdentificator .<br>             ?perceel ns:naamruimte ?naamruimte .<br>          ?perceel ns:versieIdentificator ?versieIdentificator .}<br>    #      ?perceel gebouwenregister:Perceel%3Astatus ?status .}<br>}</pre><h4>Combining multiple data streams via their semantics</h4><p>The three data streams contain unique address IDs linked respectively to the parcel, building unit, and address registry data streams. Through the use of SPARQL queries, we are able to link these data streams semantically.</p><p>In this way, developing a REST API that retrieves an RDF containing details on addresses, parcels, and building units becomes possible. Using LDES as an intermediate data format, this API ensures that the output always remains synchronized with the source database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uqwpxMBK68ZNTyaUqLti6Q.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t2n7UWgxaK_MjwSoB0Ivuw.png" /><figcaption>Image by author</figcaption></figure><h4>Full code</h4><p>To replicate the whole proof of concept (LDES 2 GraphDB), please go to this <a href="https://github.com/samuvack/ldes-grar">Github repository</a>. It describes how to set up the GraphDB and Apache NiFi via docker, after which the data flow can be started using the supplied <a href="https://github.com/samuvack/ldes-grar/blob/main/NiFi_Flow.json">Apache NiFi setup file</a>.</p><p><a href="https://github.com/samuvack/ldes-grar">GitHub - samuvack/ldes-grar</a></p><p>If you like what you read, be sure to ❤️ it — as a writer, it means the world. Stay in touch by following me as an author.</p><p>Contributors to this article are <a href="https://github.com/ddvlanck">ddvlanck (Dwight Van Lancker) (github.com)</a>, <a href="https://github.com/sandervd">sandervd (Sander Van Dooren) (github.com)</a> at Flanders Smart Data Space (Digital Flanders, Belgium). In a rapidly changing society, governments need to be more agile and resilient than ever. Digital Flanders realizes and supervises digital transformation projects for Flemish and local governments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1aab3090b40" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/real-time-data-linkage-via-linked-data-event-streams-e1aab3090b40">Real-Time Data Linkage via Linked Data Event Streams</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualizing Linked Data Event Streams with QGIS]]></title>
            <link>https://medium.com/geekculture/visualizing-linked-data-event-streams-with-qgis-da25b6ccfc1b?source=rss-86fd8b1c761e------2</link>
            <guid isPermaLink="false">https://medium.com/p/da25b6ccfc1b</guid>
            <category><![CDATA[linked-data-event-stream]]></category>
            <category><![CDATA[qgis]]></category>
            <category><![CDATA[ldes]]></category>
            <category><![CDATA[linked-data]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Samuel Van Ackere]]></dc:creator>
            <pubDate>Thu, 09 Mar 2023 06:02:44 GMT</pubDate>
            <atom:updated>2023-03-09T06:02:44.418Z</atom:updated>
            <content:encoded><![CDATA[<p>How to monitor and analyse Linked Data Event Streams and make informed decisions with QGIS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sdd-KHvmVZE7AEDd" /><figcaption>Photo by <a href="https://unsplash.com/@sadswim?utm_source=medium&amp;utm_medium=referral">ian dooley</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Linked Data Event Stream (LDES) is a powerful way to represent and share real-time data, but visualizing and monitoring this data can be challenging. QGIS, a free and open-source geographic information system (GIS) software, can create effective maps of LDES data, making it easy to monitor the content in real-time and use it as a decision support system.</p><p>One way to use QGIS is to visualize sensor data in the context of smart city applications. For example, you can use QGIS to display sensor data such as air quality, noise levels, or traffic flow on a map. By layering this data on top of a base map, you can gain insights into how different factors are related and make more informed decisions about disasters and other rapidly changing events.</p><h4>Linked Data Event Stream</h4><p>A Linked Data Event Stream is a constant flow of immutable objects (such as version objects, sensor observations or archived representation) containing information about an event or state change originating from a system that continuously creates data.</p><p>For more information about Linked Data Event Stream (LDES), you can read this article:</p><p><a href="https://medium.com/@samuel.vanackere/linked-data-event-streams-explained-in-8-minutes-e1c76d077bb9">Linked Data Event Streams explained in 8 minutes</a></p><h4>Visualizing geospatial data</h4><p>QGIS can be used with PostgreSQL/PostGIS to visualize geo data in real-time. In this way, the powerful, open-source relational database management system PostgreSQL stores and manages large amounts of spatial data, and QGIS visualizes all the data. PostGIS is an extension of PostgreSQL that provides spatial data types and functions for working with spatial data.</p><p>For more information about how to ingest Linked Data Event Streams into PostgreSQL/PostGIS, read this article (data flow is the same for TimescaleDB and PostgreSQL):</p><p><a href="https://pub.towardsai.net/linked-data-event-streams-and-timescaledb-for-real-time-timeseries-data-management-9e82ba336f82">Linked Data Event Streams and TimescaleDB for Real-time Timeseries Data Management</a></p><p>Connecting QGIS to a PostgreSQL/PostGIS database lets you access and visualize your geo data in real-time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*9S3sfhxeozb7nDE8l7GgVw.png" /><figcaption>Image by author</figcaption></figure><p>For example, you can set up a live feed of sensor data to a PostgreSQL /PostGIS database and then use QGIS to display that data on a map in real time. Using the Time Manager plugin makes adding a time bar to the map possible. This allows you to see spatiotemporal data in a user-friendly manner. Because an LDES makes it possible to stay in sync with the source database, it is possible to make quick decisions based on the current state of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*wDGRab5Nlb73wKj84elLDw.png" /><figcaption>Image by author</figcaption></figure><h4>Conclusion</h4><p>In conclusion, QGIS is a tool that can be used to visualize and monitor Linked Data Event Streams (LDES) data with a spatiotemporal component. It offers a wide range of spatial analysis and visualization tools that can be used to create maps and visualizations of real-time data from LDES streams. Additionally, QGIS can be connected to a PostgreSQL/PostGIS database to visualize the data in real time.</p><p>Now you should be in good shape to use QGIS in combination with Linked Data Event Streams and their advantages. You can find more information on the websites underneath.</p><ul><li><a href="https://academy.europa.eu/courses/publishing-data-with-linked-data-event-streams-why-and-how">Publishing data with Linked Data Event Streams: why and how</a></li><li><a href="https://semiceu.github.io/LinkedDataEventStreams/">Linked Data Event Streams</a></li></ul><p>If you like what you read, be sure to ❤️ it — as a writer, it means the world. Stay in touch by following me as an author.</p><p>Contributors to this article are <a href="https://github.com/ddvlanck">ddvlanck (Dwight Van Lancker) (github.com)</a>, <a href="https://github.com/sandervd">sandervd (Sander Van Dooren) (github.com)</a> at Smart Data Space (Digital Flanders, Belgium). In a rapidly changing society, governments need to be more agile and resilient than ever. Digital Flanders realizes and supervises digital transformation projects for Flemish and local governments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da25b6ccfc1b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/visualizing-linked-data-event-streams-with-qgis-da25b6ccfc1b">Visualizing Linked Data Event Streams with QGIS</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enriching Digital Twins with Linked Data Event Streams]]></title>
            <link>https://medium.com/geekculture/enriching-digital-twins-with-linked-data-event-streams-285630a02b82?source=rss-86fd8b1c761e------2</link>
            <guid isPermaLink="false">https://medium.com/p/285630a02b82</guid>
            <category><![CDATA[linked-data-event-stream]]></category>
            <category><![CDATA[digital-twin]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ldes]]></category>
            <category><![CDATA[semanticweb]]></category>
            <dc:creator><![CDATA[Samuel Van Ackere]]></dc:creator>
            <pubDate>Fri, 03 Mar 2023 13:02:43 GMT</pubDate>
            <atom:updated>2023-03-03T13:02:43.640Z</atom:updated>
            <content:encoded><![CDATA[<p>A Guide to Embed and Visualize Real-Time Linked Data Events in a Digital Twin</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jbKv6yBldTCRwECe.jpg" /><figcaption>Source: Digital Urban European Twins: The DUET Project (<a href="https://www.digitalurbantwins.com/">https://www.digitalurbantwins.com/</a>)</figcaption></figure><p>Digital Twins are becoming increasingly important for decision-makers to help understand and manage real-world assets and processes. Creating a digital replica of the physical world enables real-time monitoring and analysis of data, providing a complete picture of the system.</p><p>Linked Data Event Streams can be used to enrich these digital twins by adding real-time data to provide even more insights into the system. <strong>Here we explore the role of digital twins and how Linked Data Event Streams can be used to enhance the digital replica.</strong></p><p><a href="https://medium.com/@samuel.vanackere/linked-data-event-streams-explained-in-8-minutes-e1c76d077bb9">Linked Data Event Streams explained in 8 minutes</a></p><h4>Create a Digital Twin</h4><p>Cesium-3D is an open-source JavaScript library for creating 3D globes and maps, which makes it an ideal tool for developing digital twins. Developing a digital twin using Cesium-3D is a powerful way to represent a physical object or environment visually. By integrating various data sources, such as real-time sensor data, CAD models, and geographic information, a digital twin can be developed that provides a highly detailed and accurate representation of the physical object. For example, <a href="https://demos.cesium.com/NewYork/?view=-74.01881302800248%2C40.69114333714821%2C753.2406554180401%2C21.27879878293835%2C-21.343905508724625%2C0.0716951918898415">New York City has developed a digital twin of the city using Cesium-3D</a>. The digital twin allows city planners and decision-makers to test different scenarios and make informed urban planning and development decisions.</p><h4>Embed real-time linked data</h4><blockquote><strong>Linked Data Event Streams can play a crucial role in enriching digital twins in real-time</strong>.</blockquote><p>GeoServer is an open-source server for sharing geospatial data, which supports a wide range of data sources, including databases like PostgreSQL. Publishing Linked Data Event Streams (LDES) into GeoServer after saving the data into PostgreSQL is a powerful way to make data available through a Web Map Service (WMS). The WMS service GeoServer allows developers to visualize this data in Cesium-3D.</p><p>To insert LDES into PostgreSQL, you can follow the same data flow as is described in this article:</p><p><a href="https://pub.towardsai.net/linked-data-event-streams-and-timescaledb-for-real-time-timeseries-data-management-9e82ba336f82">Linked Data Event Streams and TimescaleDB for Real-time Timeseries Data Management</a></p><p>The code snippet underneath shows how the WMS layer is added in the Cesium 3D code.</p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html lang=&quot;en&quot;&gt;<br>&lt;head&gt;<br>  &lt;title&gt;Hello World!&lt;/title&gt;<br>  &lt;script src=&quot;Cesium/Cesium.js&quot;&gt;&lt;/script&gt;<br>  &lt;style&gt;<br>  @import url(https://medium.com/Cesium/Widgets/CesiumWidget/CesiumWidget.css);<br><br>  #cesiumContainer {<br>      position: absolute;<br>      top: 0;<br>      left: 0;<br>      height: 100%;<br>      width: 100%;<br>      margin: 0;<br>      overflow: hidden;<br>      padding: 0;<br>      font-family: sans-serif;<br>  }<br><br>  body {<br>      padding: 0;<br>      margin: 0;<br>      overflow: hidden;<br>  }<br><br><br>&lt;/style&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>  &lt;div id=&quot;cesiumContainer&quot;&gt;&lt;/div&gt;<br>  &lt;script&gt;<br><br>var widget = new Cesium.CesiumWidget(&#39;cesiumContainer&#39;);<br>var url=&#39;<a href="http://localhost:8080/geoserver/wms&#39;">http://localhost:8080/geoserver/wms&#39;</a>; //Geoserver URL<br>var layers = widget.scene.globe.imageryLayers;<br>    layers.removeAll();<br>    layers.addImageryProvider(new Cesium.WebMapServiceImageryProvider({<br>        url : url,        <br>        layers: &#39;LDES Demo&#39;<br>    }));<br><br>Sandcastle.finishedLoading();<br><br>  &lt;/script&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>This code creates the virtual globe; whereafter it becomes possible to add air quality data, noise levels, etc., via WMS service, together with 3D models of the city.</p><p>The output of the DUET project is visualized underneath.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*iJKZgfIQTRsmXCnxtxGRLg.gif" /><figcaption>Source: Digital Urban European Twins: The DUET Project (<a href="https://www.digitalurbantwins.com/">https://www.digitalurbantwins.com/</a>)</figcaption></figure><h4>Full code</h4><p>The full code is available on GitHub if you want to dive into this use case. Follow the instructions to visualize geospatial Linked Data Event Streams via GeoServer in WMS or WFS. Adding the WMS layers to the Cesium 3D globe becomes possible from there.</p><p><a href="https://github.com/Informatievlaanderen/VSDS-LDESDemo/tree/master/geoserver">VSDS-LDESDemo/geoserver at master · Informatievlaanderen/VSDS-LDESDemo</a></p><h4>Conclusion</h4><p>Digital twins have become vital for decision-makers looking to optimize operations, make informed decisions, and enhance performance. By creating a virtual replica of real-world assets, systems and processes, digital twins provide valuable insights and improve decision-making processes.</p><p><strong>Linked Data Event Streams can enrich digital twins in real-time</strong>. This leads to better decision-making and increased efficiency.</p><p>If you like what you read, be sure to ❤️ it — as a writer, it means the world. Stay in touch by following me as an author.</p><p>Contributors to this article are <a href="https://github.com/ddvlanck">ddvlanck (Dwight Van Lancker) (github.com)</a>, <a href="https://github.com/sandervd">sandervd (Sander Van Dooren) (github.com)</a> at Smart Data Space (Digital Flanders). In a rapidly changing society, governments need to be more agile and resilient than ever. As a strategic partner, we realize and supervise digital transformation projects for Flemish and local governments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=285630a02b82" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/enriching-digital-twins-with-linked-data-event-streams-285630a02b82">Enriching Digital Twins with Linked Data Event Streams</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasting Linked Data Event Streams on the Internet of Water]]></title>
            <link>https://pub.towardsai.net/forecasting-linked-data-event-streams-on-the-internet-of-water-826171085bae?source=rss-86fd8b1c761e------2</link>
            <guid isPermaLink="false">https://medium.com/p/826171085bae</guid>
            <category><![CDATA[ldes]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[linked-data-event-stream]]></category>
            <dc:creator><![CDATA[Samuel Van Ackere]]></dc:creator>
            <pubDate>Wed, 01 Mar 2023 12:02:40 GMT</pubDate>
            <atom:updated>2023-03-01T12:02:40.672Z</atom:updated>
            <content:encoded><![CDATA[<p>Applying Facebook’s powerful forecasting model to Linked Data Event Stream IoT sensor data for improved water management</p><figure><img alt="Linked data event stream, LDES, Linked Data Event Stream" src="https://cdn-images-1.medium.com/max/1024/1*Gyjwl7sigVG7AJnfwF63HQ.jpeg" /><figcaption>Digital image created with Dall-E 2. Image by author</figcaption></figure><p>As our world becomes increasingly data-driven, the ability to accurately forecast future events has become crucial for businesses and organizations. Linked Data Event Streams are known as a powerful standard to represent and share big data in real time.</p><p>In this article, <strong>I explore the potential of using Prophet for forecasting </strong><a href="https://medium.com/p/e1c76d077bb9"><strong>Linked Data Event Streams (LDES)</strong></a><strong>, specifically focusing on an Internet of Water use case.</strong></p><h4>Linked Data Event Stream</h4><p>A Linked Data Event Stream (LDES) is a continuous flow of immutable objects, such as version objects, sensor observations, or archived representations. Each object contains information about an event or change of state that originates from a system that continuously generates data. The use of LDES for Internet of Water IoT sensor time-series data allows for integrating multiple data streams from different sources, making it possible to connect and analyse data from a wide range of IoT devices.</p><p>For more information on what an LDES is, please read this article:</p><p><a href="https://medium.com/@samuel.vanackere/linked-data-event-streams-explained-in-8-minutes-e1c76d077bb9">Linked Data Event Streams explained in 8 minutes</a></p><p>Before forecasting the LDES stream, I need to ingest this into a database. After this materialization, it becomes possible to forecast the stream’s data in one batch.</p><p>This is how to ingest an LDES in a TimescaleDB or PostgreSQL:</p><p><a href="https://pub.towardsai.net/linked-data-event-streams-and-timescaledb-for-real-time-timeseries-data-management-9e82ba336f82">Linked Data Event Streams and TimescaleDB for Real-time Timeseries Data Management</a></p><p>To show the potential of Machine Learning on Linked Data Event Streams, I showcase a Prophet forecasting model on an <a href="https://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations">Internet of Water (IoW) LDES</a>. Immediately one of the benefits of LDES becomes apparent; in addition to the fact that you are always in sync with the most up-to-date data via an LDES, <strong>the LDES also contains historical data.</strong> As a result, it becomes possible to immediately harvest all the necessary historical data to train the offline forecasting model.</p><p>Underneath, a part of the historical data of the IoW sensor data is visualized (temperature and conductivity).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Vr4_brrURk1hydJjyhhhMQ.png" /><figcaption>fragment of IoW Linked Data Event Stream (Image by author)</figcaption></figure><h4>Forecasting IoW data</h4><p>I will use the <a href="https://facebook.github.io/prophet/"><strong>Prophet forecasting model</strong></a> to forecast the Internet of Waters (IoW) LDES. Prophet is a powerful forecasting model developed by Facebook that is particularly well-suited for time-series data. It uses a combination of linear and non-linear methods to model trends and seasonality in data, enabling it to handle both short-term and long-term forecasts. The model can also handle missing data points and outliers, making it more robust to real-world data.</p><p>When applied to time-series sensor data, Prophet can predict future events such as water usage, temperature, and equipment performance. These predictions can improve operations&#39; efficiency, safety, and sustainability by enabling organizations to anticipate and prevent potential issues. Additionally, by incorporating data from multiple sensors, Prophet can be used to predict the interactions and relationships between different data streams.</p><p>Underneath is a code snippet of the prophet forecasting model trained on the IoW LDES and generating a prediction.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6429f936124d6e440b6c8269b05bb1e/href">https://medium.com/media/d6429f936124d6e440b6c8269b05bb1e/href</a></iframe><p>This code generates a forecasting of the time series for three consecutive days. The benefit of using Prophet model is that it is easy to add prediction intervals together with the forecasted data. This is a good way to provide uncertainty estimation in your forecasting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*iPwWdm8zmWCOEVsx.png" /><figcaption>Forecasting of time-series data with Prophet model. Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*LRWR3AHgkojCh8s3.png" /><figcaption>Forecasting of time-series data with Prophet model. Image by author</figcaption></figure><h4>Full code</h4><p>To replicate the data flow in this article, please go to the <a href="https://github.com/samuvack/ML-LDES-server">ML-LDES server</a>. It describes how to set up the dockerized PostgreSQL/PostGIS, PgAdmin, and Apache NiFi, after which the data flow can be started using the supplied Apache NiFi setup file.</p><p><a href="https://github.com/samuvack/ML-LDES-server">GitHub - samuvack/ML-LDES-server: The Machine Learning (ML) server for Linked Data Event Streams (LDES)</a></p><h4>Wrapping Up</h4><p>A Linked Data Event Stream is considered the core API of fast and slow-moving data. It is a data event stream of a group of immutable objects described as machine-readable RDF (such as sensor observations, address registers or financial data).</p><p><strong>In this article, I tried to use an offline forecasting model Prophet on such an LDES, explaining from start to finish how to set up this data flow.</strong></p><p>If you like what you read, be sure to ❤️ it — as a writer, it means the world. Stay in touch by following me as an author.</p><p>Contributors to this article are <a href="https://github.com/ddvlanck">ddvlanck (Dwight Van Lancker) (github.com)</a>, <a href="https://github.com/sandervd">sandervd (Sander Van Dooren) (github.com)</a> at Flanders Smart Data Space (Digital Flanders, Belgium). In a rapidly changing society, governments need to be more agile and resilient than ever. Digital Flanders realizes and supervises digital transformation projects for Flemish and local governments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=826171085bae" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/forecasting-linked-data-event-streams-on-the-internet-of-water-826171085bae">Forecasting Linked Data Event Streams on the Internet of Water</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linked Data Event Streams and TimescaleDB for Real-time Timeseries Data Management]]></title>
            <link>https://pub.towardsai.net/linked-data-event-streams-and-timescaledb-for-real-time-timeseries-data-management-9e82ba336f82?source=rss-86fd8b1c761e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e82ba336f82</guid>
            <category><![CDATA[timeseries-forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[linked-data]]></category>
            <category><![CDATA[timescaledb]]></category>
            <category><![CDATA[linked-data-event-stream]]></category>
            <dc:creator><![CDATA[Samuel Van Ackere]]></dc:creator>
            <pubDate>Sat, 25 Feb 2023 00:02:39 GMT</pubDate>
            <atom:updated>2023-10-11T12:25:06.258Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to consume a Linked Data Event Stream and store it in a TimescaleDB database</h4><figure><img alt="Linked Data Event Streams, TimescaleDB, Timeseries, Linked Data Event Streams, Linked Data Event Streams, Linked Data Event Streams, LDES" src="https://cdn-images-1.medium.com/max/1024/0*mCAe0fxGJ9Vn2lrr" /><figcaption>Photo by <a href="https://unsplash.com/@homajob?utm_source=medium&amp;utm_medium=referral">Scott Graham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Linked data event stream</h3><p>Linked Data Event Streams represent and share fast and slow-moving data on the Web using the Resource Description Framework (RDF). This allows data to be linked and connected to other data sources using unique identifiers (URIs).</p><p>A Linked Data Event Stream is a data event stream of a group of immutable objects described as machine-readable RDF (such as sensor observations, address registers, or financial data).</p><p><a href="https://medium.com/@samuel.vanackere/linked-data-event-streams-explained-in-8-minutes-e1c76d077bb9">Linked Data Event Streams explained in 8 minutes</a></p><p>LDES streams provide a flexible and interoperable way of describing and exchanging events as Linked Data, enabling different systems and applications to easily consume and act on data streams in a consistent and standardized way.</p><p><strong>This article shows how to effortlessly insert sensor data in the form of an LDES into a TimescaleDB database.</strong></p><h3>Managing large amounts of time-series data</h3><p><a href="https://www.timescale.com/">TimescaleDB </a>is an open-source database for storing and querying large amounts of time-series data. It extends PostgreSQL with time-series support, offering features like fast ingestion and querying of large data, flexible data granularity, and long-term data storage.</p><figure><img alt="Linked Data Event Streams, Linked Data Event Streams, Linked Data Event Streams, Linked Data Event Streams, TimescaleDB, Linked data" src="https://cdn-images-1.medium.com/max/1024/0*jGLm8GGKMY2og0_2" /><figcaption><a href="https://www.timescale.com/blog/postgresql-timescaledb-1000x-faster-queries-90-data-compression-and-much-more/">Insert performance comparison</a> between TimescaleDB 2.7.2 and PostgreSQL 14.4</figcaption></figure><p>TimescaleDB is a powerful and efficient database system well-suited for storing and querying time-series data at scale. It is widely used in various applications, including IoT, finance, and telemetry.</p><p>First, a data flow must be configured to ingest a Linked Data Event Stream into PostgreSQL. This can be done in the Apache NiFi environment.</p><h3>LDES to TimeScaleDB</h3><p>To persist LDES in a database, we use the Apache NiFi platform. Apache NiFi is an open-source data processing, and integration platform designed to automate data flow between systems. It provides a Web-based user interface for creating, managing, and monitoring data flow and a range of pre-built connectors and processors for performing data processing tasks.</p><figure><img alt="Linked Data Event Streams and TimescaleDB for Real-time Timeseries Data Management" src="https://cdn-images-1.medium.com/max/1024/1*DfPnxFtw3i2iUsqxhuD_Rw.png" /><figcaption>Data pipeline in Apachine NiFi (image by author)</figcaption></figure><p>To consume an LDES stream, an LDES client processor is needed in the Apache NiFi flow. An LDES client is a component capable of consuming, processing, and analyzing the events in the stream.</p><p>The output of the LDES Client (working with the following <a href="https://apim-iow-apimanagement.azure-api.net/ldes/water-quality-observations-by-time">LDES</a>):</p><pre>_:B5edf92f59913b8d14f45818d5bde1d51 &lt;http://<a href="http://www.w3.org/">www.w3.org/</a>1999/02/22-rdf-syntax-ns#type&gt; &lt;<a href="http://def.isotc211.org/iso19156/2011/Measurement#OM_Measurement&gt;">http://def.isotc211.org/iso19156/2011/Measurement#OM_Measurement&gt;</a> .<br>_:B5edf92f59913b8d14f45818d5bde1d51 &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Observation#OM_Observation.observedProperty&gt; &lt;<a href="https://data.vmm.be/concept/waterkwaliteitparameter/conductiviteit&gt;">https://data.vmm.be/concept/waterkwaliteitparameter/conductiviteit&gt;</a> .<br>_:B5edf92f59913b8d14f45818d5bde1d51 &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Observation#OM_Observation.phenomenonTime&gt; &quot;2022-11-09T20:30:00.000Z&quot;^^&lt;<a href="http://www.w3.org/2001/XMLSchema#datetime&gt;">http://www.w3.org/2001/XMLSchema#datetime&gt;</a> .<br>_:B5edf92f59913b8d14f45818d5bde1d51 &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Observation#OM_Observation.result&gt; _:B727a9b2b6a6311cc83b677d439f4cf68 .<br>_:B5edf92f59913b8d14f45818d5bde1d51 &lt;http://<a href="http://www.w3.org/ns/sosa/madeBySensor&gt;">www.w3.org/ns/sosa/madeBySensor&gt;</a> &lt;urn:ngsi-v2:cot-imec-be:Device:dwg-iow-csHTUVdGuPYK89L34yi88j&gt; .<br>_:B6003104aae33209ee0e6a26e14ba38cb &lt;https://schema.org/value&gt; &quot;1.152E1&quot;^^&lt;http://<a href="http://www.w3.org/">www.w3.org/</a>2001/XMLSchema#double&gt; .<br>_:Bec20a7e293639ee5a985d621d7b2d6d5 &lt;http://<a href="http://www.w3.org/">www.w3.org/</a>1999/02/22-rdf-syntax-ns#type&gt; &lt;<a href="http://def.isotc211.org/iso19156/2011/Measurement#OM_Measurement&gt;">http://def.isotc211.org/iso19156/2011/Measurement#OM_Measurement&gt;</a> .<br>_:Bec20a7e293639ee5a985d621d7b2d6d5 &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Observation#OM_Observation.observedProperty&gt; &lt;<a href="https://data.vmm.be/concept/waterkwaliteitparameter/temperatuur&gt;">https://data.vmm.be/concept/waterkwaliteitparameter/temperatuur&gt;</a> .<br>_:Bec20a7e293639ee5a985d621d7b2d6d5 &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Observation#OM_Observation.phenomenonTime&gt; &quot;2022-11-09T20:30:00.000Z&quot;^^&lt;<a href="http://www.w3.org/2001/XMLSchema#datetime&gt;">http://www.w3.org/2001/XMLSchema#datetime&gt;</a> .<br>_:Bec20a7e293639ee5a985d621d7b2d6d5 &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Observation#OM_Observation.result&gt; _:Bdd20913615f9b4c0169c7770bdc770d4 .<br>_:Bec20a7e293639ee5a985d621d7b2d6d5 &lt;http://<a href="http://www.w3.org/ns/sosa/madeBySensor&gt;">www.w3.org/ns/sosa/madeBySensor&gt;</a> &lt;urn:ngsi-v2:cot-imec-be:Device:dwg-iow-csHTUVdGuPYK89L34yi88j&gt; .<br>_:Bdd20913615f9b4c0169c7770bdc770d4 &lt;http://<a href="http://def.isotc211.org/iso19103/">def.isotc211.org/iso19103/</a>2005/UnitsOfMeasure#Measure.value&gt; _:B6003104aae33209ee0e6a26e14ba38cb .<br>&lt;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z&gt; &lt;http://<a href="http://www.w3.org/">www.w3.org/</a>1999/02/22-rdf-syntax-ns#type&gt; &lt;<a href="https://www.w3.org/TR/vocab-ssn-ext/#sosa:ObservationCollection&gt;">https://www.w3.org/TR/vocab-ssn-ext/#sosa:ObservationCollection&gt;</a> .<br>&lt;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z&gt; &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/SamplingFeature#SF_SamplingFeatureCollection.member&gt; _:B5edf92f59913b8d14f45818d5bde1d51 .<br>&lt;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z&gt; &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/SamplingFeature#SF_SamplingFeatureCollection.member&gt; _:B3105a27f0059867877fb28653f5a6abc .<br>&lt;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z&gt; &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/SamplingFeature#SF_SamplingFeatureCollection.member&gt; _:Bec20a7e293639ee5a985d621d7b2d6d5 .<br>&lt;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z&gt; &lt;http://<a href="http://purl.org/dc/terms/isVersionOf&gt;">purl.org/dc/terms/isVersionOf&gt;</a> &lt;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn&gt; .<br>&lt;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z&gt; &lt;http://<a href="http://www.w3.org/ns/prov">www.w3.org/ns/prov</a>#generatedAtTime&gt; &quot;2022-11-09T20:30:00.000Z&quot;^^&lt;<a href="http://www.w3.org/2001/XMLSchema#dateTime&gt;">http://www.w3.org/2001/XMLSchema#dateTime&gt;</a> .<br>&lt;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z&gt; &lt;http://<a href="http://www.w3.org/ns/sosa/hasFeatureOfInterest&gt;">www.w3.org/ns/sosa/hasFeatureOfInterest&gt;</a> &quot;spt-00029-97&quot; .<br>_:B3105a27f0059867877fb28653f5a6abc &lt;http://<a href="http://www.w3.org/">www.w3.org/</a>1999/02/22-rdf-syntax-ns#type&gt; &lt;<a href="http://def.isotc211.org/iso19156/2011/Measurement#OM_Measurement&gt;">http://def.isotc211.org/iso19156/2011/Measurement#OM_Measurement&gt;</a> .<br>_:B3105a27f0059867877fb28653f5a6abc &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Observation#OM_Observation.observedProperty&gt; &lt;<a href="https://data.vmm.be/concept/observatieparameter/hydrostatische-druk&gt;">https://data.vmm.be/concept/observatieparameter/hydrostatische-druk&gt;</a> .<br>_:B3105a27f0059867877fb28653f5a6abc &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Observation#OM_Observation.phenomenonTime&gt; &quot;2022-11-09T20:30:00.000Z&quot;^^&lt;<a href="http://www.w3.org/2001/XMLSchema#datetime&gt;">http://www.w3.org/2001/XMLSchema#datetime&gt;</a> .<br>_:B3105a27f0059867877fb28653f5a6abc &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Observation#OM_Observation.result&gt; _:B2598b988faa2635d5dd520f59f376b8e .<br>_:B3105a27f0059867877fb28653f5a6abc &lt;http://<a href="http://www.w3.org/ns/sosa/madeBySensor&gt;">www.w3.org/ns/sosa/madeBySensor&gt;</a> &lt;urn:ngsi-v2:cot-imec-be:Device:dwg-iow-csHTUVdGuPYK89L34yi88j&gt; .<br>&lt;https://<a href="http://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations&gt;">iow.smartdataspace.beta-vlaanderen.be/water-quality-observations&gt;</a> &lt;https://<a href="http://w3id.org/tree">w3id.org/tree</a>#member&gt; &lt;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z&gt; .<br>_:B727a9b2b6a6311cc83b677d439f4cf68 &lt;http://<a href="http://def.isotc211.org/iso19103/">def.isotc211.org/iso19103/</a>2005/UnitsOfMeasure#Measure.value&gt; _:B9776c69148d2ef600aed62aa2c85bd40 .<br>_:B2598b988faa2635d5dd520f59f376b8e &lt;http://<a href="http://def.isotc211.org/iso19103/">def.isotc211.org/iso19103/</a>2005/UnitsOfMeasure#Measure.value&gt; _:B436b5950c7725d6d9a38aaf6ca88b00f .<br>_:B9776c69148d2ef600aed62aa2c85bd40 &lt;https://schema.org/value&gt; &quot;1120&quot;^^&lt;http://<a href="http://www.w3.org/">www.w3.org/</a>2001/XMLSchema#integer&gt; .<br>_:B436b5950c7725d6d9a38aaf6ca88b00f &lt;https://schema.org/value&gt; &quot;673&quot;^^&lt;http://<a href="http://www.w3.org/">www.w3.org/</a>2001/XMLSchema#integer&gt; .</pre><p>Next in the Apache NiFi flow is a version materialization component. “Version materialization” refers to the process of removing version information from an LDES member and reverting it to reconstruct a “state” object, which only reflects the current state of the LDES member without historical information about changes.</p><p>Version materialization is needed since a consumer doesn’t want to store these versions in a database.</p><p>The data is first converted to JSON-LD to convert to a tabular structure easily. This JSON-LD file looks like this:</p><pre>{<br>  &quot;@id&quot; : &quot;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-g9kEXxUeP28TNc6wJvFsau&quot;,<br>  &quot;@type&quot; : &quot;Observatieverzameling&quot;,<br>  &quot;Bemonsteringsobjectverzameling.lid&quot; : [ {<br>    &quot;@id&quot; : &quot;_:b8&quot;,<br>    &quot;@type&quot; : &quot;Meting&quot;,<br>    &quot;Observatie.geobserveerdKenmerk&quot; : &quot;<a href="https://data.vmm.be/concept/waterkwaliteitparameter/temperatuur">https://data.vmm.be/concept/waterkwaliteitparameter/temperatuur&quot;</a>,<br>    &quot;Observatie:.phenomenonTime&quot; : {<br>      &quot;@type&quot; : &quot;<a href="http://www.w3.org/2001/XMLSchema#datetime">http://www.w3.org/2001/XMLSchema#datetime&quot;</a>,<br>      &quot;@value&quot; : &quot;2022-11-09T19:30:00.000Z&quot;<br>    },<br>    &quot;Observatie.resultaat&quot; : {<br>      &quot;@id&quot; : &quot;_:b6&quot;,<br>      &quot;Maat.maat&quot; : {<br>        &quot;@id&quot; : &quot;_:b1&quot;,<br>        &quot;<a href="https://schema.org/value">https://schema.org/value&quot;</a> : {<br>          &quot;@type&quot; : &quot;<a href="http://www.w3.org/2001/XMLSchema#double">http://www.w3.org/2001/XMLSchema#double&quot;</a>,<br>          &quot;@value&quot; : &quot;1.153E1&quot;<br>        }<br>      }<br>    },<br>    &quot;Observatie.uitgevoerdMetSensor&quot; : &quot;urn:ngsi-v2:cot-imec-be:Device:dwg-iow-8JgF3vxoYKoXzwWGUiX3Yc&quot;<br>  }, {<br>    &quot;@id&quot; : &quot;_:b3&quot;,<br>    &quot;@type&quot; : &quot;Meting&quot;,<br>    &quot;Observatie.geobserveerdKenmerk&quot; : &quot;<a href="https://data.vmm.be/concept/waterkwaliteitparameter/conductiviteit">https://data.vmm.be/concept/waterkwaliteitparameter/conductiviteit&quot;</a>,<br>    &quot;Observatie:.phenomenonTime&quot; : {<br>      &quot;@type&quot; : &quot;<a href="http://www.w3.org/2001/XMLSchema#datetime">http://www.w3.org/2001/XMLSchema#datetime&quot;</a>,<br>      &quot;@value&quot; : &quot;2022-11-09T19:30:00.000Z&quot;<br>    },<br>    &quot;Observatie.resultaat&quot; : {<br>      &quot;@id&quot; : &quot;_:b4&quot;,<br>      &quot;Maat.maat&quot; : {<br>        &quot;@id&quot; : &quot;_:b5&quot;,<br>        &quot;<a href="https://schema.org/value">https://schema.org/value&quot;</a> : {<br>          &quot;@type&quot; : &quot;<a href="http://www.w3.org/2001/XMLSchema#integer">http://www.w3.org/2001/XMLSchema#integer&quot;</a>,<br>          &quot;@value&quot; : &quot;920&quot;<br>        }<br>      }<br>    },<br>    &quot;Observatie.uitgevoerdMetSensor&quot; : &quot;urn:ngsi-v2:cot-imec-be:Device:dwg-iow-8JgF3vxoYKoXzwWGUiX3Yc&quot;<br>  }, {<br>    &quot;@id&quot; : &quot;_:b7&quot;,<br>    &quot;@type&quot; : &quot;Meting&quot;,<br>    &quot;Observatie.geobserveerdKenmerk&quot; : &quot;<a href="https://data.vmm.be/concept/observatieparameter/hydrostatische-druk">https://data.vmm.be/concept/observatieparameter/hydrostatische-druk&quot;</a>,<br>    &quot;Observatie:.phenomenonTime&quot; : {<br>      &quot;@type&quot; : &quot;<a href="http://www.w3.org/2001/XMLSchema#datetime">http://www.w3.org/2001/XMLSchema#datetime&quot;</a>,<br>      &quot;@value&quot; : &quot;2022-11-09T19:30:00.000Z&quot;<br>    },<br>    &quot;Observatie.resultaat&quot; : {<br>      &quot;@id&quot; : &quot;_:b2&quot;,<br>      &quot;Maat.maat&quot; : {<br>        &quot;@id&quot; : &quot;_:b0&quot;,<br>        &quot;<a href="https://schema.org/value">https://schema.org/value&quot;</a> : {<br>          &quot;@type&quot; : &quot;<a href="http://www.w3.org/2001/XMLSchema#integer">http://www.w3.org/2001/XMLSchema#integer&quot;</a>,<br>          &quot;@value&quot; : &quot;11133&quot;<br>        }<br>      }<br>    },<br>    &quot;Observatie.uitgevoerdMetSensor&quot; : &quot;urn:ngsi-v2:cot-imec-be:Device:dwg-iow-8JgF3vxoYKoXzwWGUiX3Yc&quot;<br>  } ],<br>  &quot;<a href="http://www.w3.org/ns/prov#generatedAtTime">http://www.w3.org/ns/prov#generatedAtTime&quot;</a> : {<br>    &quot;@type&quot; : &quot;<a href="http://www.w3.org/2001/XMLSchema#dateTime">http://www.w3.org/2001/XMLSchema#dateTime&quot;</a>,<br>    &quot;@value&quot; : &quot;2022-11-09T19:30:00.000Z&quot;<br>  },<br>  &quot;<a href="http://www.w3.org/ns/sosa/hasFeatureOfInterest">http://www.w3.org/ns/sosa/hasFeatureOfInterest&quot;</a> : &quot;spt-00027-06&quot;<br>}</pre><p>Hereafter, a transformation into a tabular structure is needed and can be done via a SPARQL query or a JOLT transformation. For this case, I use a JOLT transformation. A JOLT transformation filters out the relevant parameters and puts them in a structured JSON file (see output underneath).</p><pre>{<br>    &quot;id&quot;: &quot;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn&quot;,<br>    &quot;temperature_value&quot;: &quot;1.152E1&quot;,<br>    &quot;temperature_date&quot;: &quot;2022-11-09T20:30:00.000Z&quot;,<br>    &quot;temperature_sensor&quot;: &quot;urn:ngsi-v2:cot-imec-be:Device:dwg-iow-csHTUVdGuPYK89L34yi88j&quot;,<br>    &quot;conductivity_value&quot;: &quot;1120&quot;,<br>    &quot;conductivity_date&quot;: &quot;2022-11-09T20:30:00.000Z&quot;,<br>    &quot;conductivity_sensor&quot;: &quot;urn:ngsi-v2:cot-imec-be:Device:dwg-iow-csHTUVdGuPYK89L34yi88j&quot;,<br>    &quot;hydro_pressure_value&quot;: &quot;673&quot;,<br>    &quot;hydro_pressure_date&quot;: &quot;2022-11-09T20:30:00.000Z&quot;,<br>    &quot;hydro_pressure_sensor&quot;: &quot;urn:ngsi-v2:cot-imec-be:Device:dwg-iow-csHTUVdGuPYK89L34yi88j&quot;<br>}</pre><p>After this transformation, the data can be written to a PostgreSQL or TimescaleDB database. Analysis that can be done with data in the TimescaleDB database includes but is not limited to time-series visualization, anomaly detection, forecasting, and aggregation of time-series data over various time intervals.</p><figure><img alt="Linked Data Event Streams and TimescaleDB for Real-time Timeseries Data Management" src="https://cdn-images-1.medium.com/max/1024/0*MEpbintjlADMaiM_.png" /><figcaption>(image by author)</figcaption></figure><h3><strong>Full code</strong></h3><p>To replicate the data flow in this article, please go to the <a href="https://github.com/samuvack/LDES2TimescaleDB">LDES2TimescaleDB GitHub repo</a>. It describes how to set up the dockerized TimescaleDB and Apache NiFi, after which the data flow can be started using the supplied Apache NiFi setup file.</p><p><a href="https://github.com/samuvack/LDES2TimescaleDB">GitHub - samuvack/LDES2TimescaleDB: manage timeseries LDES members in TimescaleDB</a></p><h3>Wrapping Up</h3><p>A Linked Data Event Stream is the core API of fast and slow-moving data. It is a data event stream of a group of immutable objects described as machine-readable RDF (such as sensor observations, address registers or financial data).</p><p>To write a LDES to a PostgreSQL or TimescaleDB database, a data conversion flow is configured in Apache NiFi.</p><p>With this Medium article as a guideline, you should be able to write Linked Data Event Streams to a TimescaleDB database.</p><p>If you like what you read, be sure to ❤️ it — as a writer, it means the world. Stay in touch by following me as an author.</p><p>Contributors to this article are <a href="https://github.com/ddvlanck">ddvlanck (Dwight Van Lancker) (github.com)</a>, <a href="https://github.com/sandervd">sandervd (Sander Van Dooren) (github.com)</a> at Smart Data Space (Digital Flanders, Belgium). In a rapidly changing society, governments need to be more agile and resilient than ever. Digital Flanders realizes and supervises digital transformation projects for Flemish and local governments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e82ba336f82" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/linked-data-event-streams-and-timescaledb-for-real-time-timeseries-data-management-9e82ba336f82">Linked Data Event Streams and TimescaleDB for Real-time Timeseries Data Management</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Incremental Machine Learning for Linked Data Event Streams]]></title>
            <link>https://pub.towardsai.net/incremental-machine-learning-for-linked-data-event-streams-e5441cb4c65a?source=rss-86fd8b1c761e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5441cb4c65a</guid>
            <category><![CDATA[linked-data]]></category>
            <category><![CDATA[semanticweb]]></category>
            <category><![CDATA[linked-data-event-stream]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Samuel Van Ackere]]></dc:creator>
            <pubDate>Thu, 09 Feb 2023 20:02:31 GMT</pubDate>
            <atom:updated>2023-10-11T12:24:27.003Z</atom:updated>
            <content:encoded><![CDATA[<h4>Unlocking the Power of Real-time Predictions: An Introduction to Incremental Machine Learning for Linked Data Event Streams</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wiPJZiTZH5FTIGj9" /><figcaption>Photo by <a href="https://unsplash.com/@isaacmsmith?utm_source=medium&amp;utm_medium=referral">Isaac Smith</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article discusses online machine learning, one of the most exciting subdomains of machine learning theory. The potential of using incremental machine learning becomes more and more apparent when working on f<strong>ast-moving Linked Data Event Streams (LDES)</strong>.</p><p>With the conventional machine learning method, a lot of time is lost when training models from scratch repeatedly. It is better to use all parameters of previously trained models to arrive at faster predictions and analyses of fast-moving Linked Data Event Streams. A practical example applied to forecasting a Linked data event stream is used to show its potential.</p><h3>Linked Data Event Stream</h3><p>A <strong>data stream</strong> is typically a <strong>constant flow of </strong>distinct data points, <strong>each containing information about an event or change of state </strong>that originates from a system that continuously creates data. More comprehensive, a Linked Data Event Stream is a constant flow of immutable objects (such as version objects, sensor observations, or archived representation), each containing information about an event or change of state that originates from a system that continuously creates data.</p><p>It is the linked data version of data event streams, which is considered <strong>the core API of fast- and slow-moving data.</strong></p><p>For more info about this, please read this article on medium:</p><p><a href="https://medium.com/@samuel.vanackere/linked-data-event-streams-explained-in-8-minutes-e1c76d077bb9">Linked Data Event Streams explained in 8 minutes</a></p><h3>Incremental or Online Machine Learning</h3><p>Of all services that can be built on top of one or more Linked Data Event Streams, a machine learning service is one of them. The concept of an ML-LDES server is that you can send an LDES via HTTP POST request, after which the ML-LDES server can harvest the data in real-time and run a machine-learning model on some relevant parameters in the LDES. To show the potential of such an ML-LDES server, we showcase an incremental forecasting model on an <a href="https://apim-iow-apimanagement.azure-api.net/ldes/water-quality-observations-by-time">Internet of Water (IoW) LDES</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/0*bed65enK8hcCSg3a" /><figcaption>Linked data event streams are continuously sending objects via HTML POST to the ML server (<a href="https://github.com/samuvack/ML-LDES-server">The Machine Learning (ML) server for Linked Data Event Streams</a>) (Image by the author.)</figcaption></figure><p>But first things first, what is incremental or online machine learning? Conventional machine learning algorithms train a model on the full training dataset at once. A potential disadvantage is that they frequently create new models from scratch rather than continuously integrating new data into already-built models. This could result in outdated models and take a lot of time to retrain the model every time from scratch.</p><p>Unlike these batch learning techniques, <strong>incremental learning</strong> or <strong>online machine learning</strong> updates the best predictor for future data at each step as new data becomes available.</p><p>Online learning and incremental learning have recently drawn attention, particularly in the context of big data and learning from data streams. This contradicts the conventional premise that all data is available at all times.</p><p>Continuous adapting of a machine learning model based on a stream of data that keeps coming in is known as incremental learning. With incremental learning, the machine learning model should adapt to new data while maintaining its prior understanding.</p><blockquote>An online learner needs to make predictions about a sequence of instances, one after the other and receives feedback after each prediction. <a href="https://medium.com/u/27553a827c69">Danny Butvinik</a></blockquote><p>Machine learning entails instructing a model one sample at a time during training. Therefore, an online model is a stateful, dynamic object. It never needs to review old data because it is constantly learning.</p><p>Data event streams are often incrementally analyzed, and real-time aggregation, enrichment, transformation, correlation, filtering, or sampling are conducted on the fly. As a result, it enables the possibility to detect emerging trends, strange events, and substantial departures from the norm, approaching alarming limits. Afterward, real-time answers and data-driven decisions can be made by it.</p><h3><strong>Linked data event stream (LDES) of Internet of Water case (IoW)</strong></h3><p>First, the Linked Data Event Stream is fetched by the LDES client, after which all LDES members are sent via HTTP request to the <a href="https://github.com/samuvack/ML-LDES-server">ML-LDES server</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7YFTbMcXVCy8u_YDZBwqvw.png" /><figcaption>LDES workbench in Apache NIFI (Image by the author.)</figcaption></figure><p>An example of one of those LDES members is added underneath:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bf2I4nydUBw1tzwVDBFYUw.png" /><figcaption>N-triple flow file fragment (Image by the author.)</figcaption></figure><p>If we convert this N-triple to a Terse RDF Triple language (for easier interpretation), we get this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*5trJAvgSIYrvG2_TU63L1w.png" /><figcaption>Turtle output of one LDES member (Image by the author.)</figcaption></figure><h3>Incremental forecasting with River</h3><blockquote>River is the sklearn library for machine learning on streaming data, <a href="https://medium.com/u/4ae76d513ea6">Alexandra Amidon</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f2BPgnxfTcsJb0U1" /><figcaption>Photo by <a href="https://unsplash.com/@jonflobrant?utm_source=medium&amp;utm_medium=referral">Jon Flobrant</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://riverml.xyz/dev/">River</a> is a Python library composed of numerous classes that carry out different online processing methods. Most of these classes are machine learning models that can analyze a single sample for learning or inference purposes.</p><p>For the IoW case, we use the <a href="https://riverml.xyz/dev/api/time-series/SNARIMAX/">SNARIMAX forecasting module</a>. SNARIMAX stands for Seasonal Non-linear AutoRegressive Integrated Moving-Average with eXogenous inputs model.</p><p>It is a time series forecasting model that considers the data’s trend and seasonality, as well as any additional predictor variables (also known as exogenous variables) that may be relevant for forecasting.</p><p>In the SNARIMAX model, the “seasonal” component accounts for periodic fluctuations in the data (such as monthly or quarterly cycles). The “natural additive” component accounts for long-term trends and patterns, and the “regressive” feature allows the model to incorporate the influence of one or more predictor variables on the forecast. The “integrated” and “moving average” parameters of the model help to smooth out short-term fluctuations and noise in the data.</p><p>Now to perform an incremental learning model on the time-series values of the Linked Data Event Stream, all RDF members are pulled in one by one (and remain in sync with the CoW (City of Water) sensor). The code snippet below illustrates what such an RDF member roughly looks like (simplified). As soon as a new RDF member is available, the LDES Client reads this value and sends it to the incremental learning model. This model will run a new forecast starting from the parameters it already had from its previous forecasting.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c53c354af2ed8aebb655aa6f93b3b637/href">https://medium.com/media/c53c354af2ed8aebb655aa6f93b3b637/href</a></iframe><p>We can visualize this continuous prediction per point in time and plot the whole data stream at once underneath it. In that case, we see how the incremental learning process becomes better and better in predicting future values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*gZKK6-EONfkbyJKBXympHg.gif" /><figcaption>Image by the author.</figcaption></figure><p>Note that in the graph above, the plotted time series data is for reference and is not used in one batch to learn the model. Instead, at each iteration, the data sample is sent to the model for learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*gSH2Q70c-rA1kGQw_d1m5A.gif" /><figcaption>Online machine learning (forecasting) using SNARIMAX method (Image by the author.)</figcaption></figure><p>When we use this Snarimax forecasting model for the IoW case, it is important to use the correct Snarimax parameters (<strong>p: </strong>Order of the autoregressive part, <strong>d: </strong>Differencing order, <strong>q: </strong>Order of the moving average part, <strong>m: </strong>Season length used for extracting seasonal features, <strong>sp: </strong>Seasonal order of the autoregressive part, <strong>sd: </strong>Seasonal differencing order, <strong>sq: </strong>Seasonal order of the moving average part). This is demonstrated in the figure below. See <a href="https://riverml.xyz/dev/api/time-series/SNARIMAX/">River specs</a> for more info.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*GcyEMLrXYrG7FDw5C0Jg_A.gif" /><figcaption>It is important to use the correctly chosen, applicable Snarimax parameters when running a forecasting model (Image by the author.)</figcaption></figure><p>To demonstrate how well the Snarimax model scores, we evaluated the model each time on the last twelve forecast points against the reference value and calculated a Mean Absolute Error of this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xXmDKj0Hmm6a3vlqWfrRrA.gif" /><figcaption>Snarimax forecasting of a Linked Data Event Stream with accompanying Mean Absolute Error</figcaption></figure><p>This article demonstrates how incremental learning can be applied to Linked Data Event Streams. At the time of writing, there was only data available over a time period of two weeks, with only a slight variation in the reference value.</p><h3>Conclusions</h3><p>The use of incremental learning offers numerous benefits over conventional machine learning methods, allowing for faster predictions and analyses of linked data event streams. The SNARIMAX forecasting module within River, which considers seasonal fluctuations, long-term trends, and exogenous variables, provides a practical example of the potential of incremental learning in real-world applications.</p><p>To replicate the data flow in this article, please go to the <a href="https://github.com/samuvack/ML-LDES-server">ML-LDES server</a>. It describes how to set up the dockerized PostgreSQL/PostGIS, PgAdmin, and Apache NiFi, after which the data flow can be started using the supplied Apache NiFi setup file.</p><p><a href="https://github.com/samuvack/ML-LDES-server/blob/master/server_forecasting_snarimax.py">ML-LDES-server/server_forecasting_snarimax.py at master · samuvack/ML-LDES-server</a></p><h3>References</h3><p>[1] Van Lancker, D., Colpaert, P., Delva, H., Van de Vyvere, B., Rojas Melendez, J. A., Dedecker, R., … Verborgh, R. (2021). Publishing base registries as linked data event streams. In M. Brambilla, R. Chbeir, F. Frasincar, &amp; I. Manolescu (Eds.), <em>WEB ENGINEERING, ICWE 2021</em> (Vol. 12706, pp. 28–36). <a href="https://doi.org/10.1007/978-3-030-74296-6_3">https://doi.org/10.1007/978-3-030-74296-6_3</a></p><p>[2] <em>river — River</em>. (n.d.). Retrieved February 7, 2023, from <a href="https://riverml.xyz/0.15.0/">https://riverml.xyz/0.15.0/</a></p><p>[3] <em>Linked Data Event Streams</em>. (n.d.). Retrieved February 7, 2023, from <a href="https://semiceu.github.io/LinkedDataEventStreams/">https://semiceu.github.io/LinkedDataEventStreams/</a></p><p>[4]European commission. (n.d.). <em>Publishing data with Linked Data Event Streams: why and how</em>. Retrieved February 7, 2023, from <a href="https://academy.europa.eu/courses/publishing-data-with-linked-data-event-streams-why-and-how">https://academy.europa.eu/courses/publishing-data-with-linked-data-event-streams-why-and-how</a></p><p>If you like what you read, be sure to ❤️ it — as a writer, it means the world. Stay in touch by following me as an author.</p><p>Contributors to this article are <a href="https://github.com/ddvlanck">ddvlanck (Dwight Van Lancker) (github.com)</a>, <a href="https://github.com/sandervd">sandervd (Sander Van Dooren) (github.com)</a> at Smart Data Space (Digital Flanders). In a rapidly changing society, governments need to be more agile and resilient than ever. As a strategic partner, we realize and supervise digital transformation projects for Flemish and local governments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5441cb4c65a" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/incremental-machine-learning-for-linked-data-event-streams-e5441cb4c65a">Incremental Machine Learning for Linked Data Event Streams</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linked Data Event Streams explained in 8 minutes]]></title>
            <link>https://medium.com/@samuel.vanackere/linked-data-event-streams-explained-in-8-minutes-e1c76d077bb9?source=rss-86fd8b1c761e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1c76d077bb9</guid>
            <category><![CDATA[ldes]]></category>
            <category><![CDATA[linked-data-event-stream]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[linked-data]]></category>
            <dc:creator><![CDATA[Samuel Van Ackere]]></dc:creator>
            <pubDate>Wed, 01 Feb 2023 14:40:51 GMT</pubDate>
            <atom:updated>2023-02-01T16:34:50.578Z</atom:updated>
            <content:encoded><![CDATA[<p>What is a Linked data event stream, and how can it help you make your dynamic data accessible?</p><figure><img alt="Linked data event stream, Linked data event stream, Linked data event stream, Linked data event stream, Linked data event stream, Linked data event stream, Linked data event stream" src="https://cdn-images-1.medium.com/max/1024/1*-FoDF-T-J0uHvxF4xpyDaQ.jpeg" /></figure><h4>1. Linked (open) data</h4><p>In 2009, the pioneer of the worldwide web, <a href="https://en.wikipedia.org/wiki/Tim_Berners-Lee">Tim Berners-Lee</a>, presented a <a href="https://www.ted.com/talks/tim_berners_lee_the_next_web?language=en">TED talk</a> about linked data as the next phase of the internet, sometimes referred to as the Semantic Web. It is a vision of the evolution of the web of documents into the semantic web, where meaning is added through linked data so that, in addition to the current -human- users of the web, machines and computers can understand and interpret the meaning of data.</p><p>The Semantic Web allows more advanced and sophisticated search capabilities and machines to understand and process data like humans do.</p><p>Linked <em>open </em>data is a community project overseen by the <a href="https://www.w3.org/">W3C organisation</a> and aims to enrich the Web by making open datasets even more accessible through the linked data method.</p><blockquote>“Linked Open Data is Linked Data which is released under an open license, which does not impede its reuse for free.” Tim Berners-Lee</blockquote><p><a href="https://en.wikipedia.org/wiki/Tim_Berners-Lee">Tim Berners-Lee</a> has proposed a 5-star system for rating the quality of open data on the Web, with <strong>Linked Open Data receiving the highest ranking</strong>:</p><p>⭐: data is openly available in some format (e.g. pdf);<br>⭐⭐: data is available in a structured format(e.g. .xls);<br>⭐⭐⭐: data is available in a non-proprietary structured format (.csv);<br>⭐⭐⭐⭐: data follows <a href="https://en.wikipedia.org/wiki/World_Wide_Web_Consortium">W3C</a> standards, like using <a href="https://en.wikipedia.org/wiki/Resource_Description_Framework">RDF</a> and <a href="https://en.wikipedia.org/wiki/Uniform_Resource_Identifier">URIs</a>;<br>⭐⭐⭐⭐⭐: all of the other, plus links to other Linked Open Data sources to provide a context.</p><p>Linked data is structured data linked to other data by <strong>relationships or connections </strong>to make it more valuable through semantic searches. It expands on established Web technologies like HTTP and URIs. Still, it uses them to communicate information in a way that computers can read automatically rather than only serving web pages for human readers. The idea behind linked data is to turn the Internet into a global decentralized machine-readable database.</p><figure><img alt="Linked data event stream, Linked data event stream, Linked data event stream, Linked data event stream, Linked data event stream, Linked data event stream, Linked data event stream" src="https://cdn-images-1.medium.com/max/850/0*1KmhNWz1rer3SE-A.jpg" /><figcaption>Visualization of an RDF (<a href="https://www.w3.org/TR/rdf11-primer/">RDF 1.1 Primer (w3.org)</a>)</figcaption></figure><p>The figure above visualizes Resource Description Framework (RDF). RDF is a standard data model for representing and sharing information on the Web, based on the idea of using triples to represent data. A triple consists of three parts: a subject, a predicate, and an object.</p><p>In the context of linked data, subjects and objects refer to individual pieces of data, such as people, places, or things. Predicates refer to the connections or associations between different nodes, such as the fact that a person is the painter of a painting.</p><p>In the figure above, the subject <a href="http://example.com/person/alice">http://example.com/person/alice</a> represents Alice, the predicate <a href="http://example.com/vocab/isAFriendOf">http://example.com/vocab/isAFriendOf</a> represents the relationship “is a friend of,” and the object <a href="http://example.com/person/bob">http://example.com/person/bob</a> represents Bob.</p><h4>2. Linked Data Event Stream</h4><p>Linked Data Event Streams (LDES) apply- as the term implies — the linked data concept to data event streams. A <strong>data stream</strong> is typically a <strong>constant flow of </strong>distinct data points, <strong>each containing information about an event or change of state </strong>that originates from a system that continuously creates data. Some examples of data streams include <strong>sensor and other IoT data, financial data, and so forth.</strong></p><p><a href="https://semiceu.github.io/LinkedDataEventStreams/">Linked Data Event Streams</a></p><p>LDES has several technical advantages:</p><ul><li>LDES is a technical standard that allows data to be exchanged sustainably and cost-effectively across silos using domain-specific ontology;</li><li>LDES is one standard for both fast and slow-changing data;</li><li>LDES offers a solution for managing historical records, versions, and retention policies efficiently.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AMZX65ogF0Sc8CCE" /><figcaption>Photo by <a href="https://unsplash.com/@johnschno?utm_source=medium&amp;utm_medium=referral">John Schnobrich</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>One can easily link datasets in a standardized way through linked data principles, giving enormous potential.</p><blockquote>A Linked Data Event Stream is a <strong>constant flow </strong>of immutable objects (such as version objects, sensor observations or archived representation) <strong>each containing information about an event or change of state </strong>that originates from a system that continuously creates data.</blockquote><h4>3. Example of a published Linked data event stream</h4><p>What does a Linked data event stream look like? We take an example of IoW (Internet of Water) sensor data published as a Linked data event stream to show an LDES in action.</p><p>This LDES stream describes the observations of water temperature and conductivity by multiple sensors spread across Flanders (Belgium). Like all LDES, it uses the <a href="https://treecg.github.io/specification">TREE specification</a> for its collection and fragmentation features. For the specific compatibility rules, read the <a href="https://treecg.github.io/specification">TREE specification</a>.</p><p>When reading the LDES endpoint, we start at the ‘main page’ or ‘starting point’ of the Linked Data Event Stream (ldes:EventStream). This so-called tree:view indicates a root node from where all members (fragments of data) can be reached. Here we can already deduce that we are dealing with a time-based fragmentation.</p><pre>@prefix ldes:  &lt;https://<a href="http://w3id.org/ldes#&gt;">w3id.org/ldes#&gt;</a> .<br>@prefix prov:  &lt;http://<a href="http://www.w3.org/ns/prov#&gt;">www.w3.org/ns/prov#&gt;</a> .<br>@prefix terms: &lt;http://<a href="http://purl.org/dc/terms&gt;&lt;/a&gt;">purl.org/dc/terms/&gt;</a> .<br>@prefix tree:  &lt;https://<a href="http://w3id.org/tree#&gt;">w3id.org/tree#&gt;</a> .<br><br>&lt;https://<a href="http://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations&gt;">iow.smartdataspace.beta-vlaanderen.be/water-quality-observations&gt;</a><br>        a                   ldes:EventStream ;<br>        ldes:timestampPath  prov:generatedAtTime ;<br>        ldes:versionOfPath  terms:isVersionOf ;<br>        tree:view           &lt;<a href="https://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations-timebased&gt;">https://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations-timebased&gt;</a> .</pre><p>Climbing deeper down the tree, we find tree:relation that shows a node with the first time-based fragment.</p><pre>@prefix tree: &lt;https://<a href="http://w3id.org/tree#&gt;">w3id.org/tree#&gt;</a> .<br><br>&lt;https://<a href="http://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations-timebased&gt;">iow.smartdataspace.beta-vlaanderen.be/water-quality-observations-timebased&gt;</a><br>        a              tree:Node ;<br>        tree:relation  [ a          tree:Relation ;<br>                         tree:node  &lt;<a href="https://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations-timebased?generatedAtTime=2022-11-08T17:28:27.680Z&gt;">https://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations-timebased?generatedAtTime=2022-11-08T17:28:27.680Z&gt;</a><br>                       ] .</pre><p>If we climb further down, we can find the first time-based fragment (generatedAtTime=2022–11–08 17:28). Important to note: although this fragment was generated on 2022–11–08, the observation data of the sensors were measured on a different time (e.g., 2021–05–14”). The time-based fragmentation is used to place the measured values of the observations in chronological order.</p><p>Here we can find all the relevant information of the measured data of the dataset:</p><pre>@prefix ns0: &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/SamplingFeature#SF_SamplingFeatureCollection.&gt; .<br>@prefix ns1: &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Observation#OM_Observation.&gt; .<br>@prefix xsd: &lt;http://<a href="http://www.w3.org/">www.w3.org/</a>2001/XMLSchema#&gt; .<br>@prefix ns2: &lt;http://<a href="http://def.isotc211.org/iso19103/">def.isotc211.org/iso19103/</a>2005/UnitsOfMeasure#Measure.&gt; .<br>@prefix ns3: &lt;https://schema.org/&gt; .<br>@prefix sosa: &lt;http://<a href="http://www.w3.org/ns/sosa&gt;&lt;/a&gt;">www.w3.org/ns/sosa/&gt;</a> .<br>@prefix dc: &lt;http://<a href="http://purl.org/dc/terms&gt;&lt;/a&gt;">purl.org/dc/terms/&gt;</a> .<br>@prefix prov: &lt;http://<a href="http://www.w3.org/ns/prov">www.w3.org/ns/prov</a>#&gt; .<br><br>&lt;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:imec-iow-ZibrvBsM4DQpYyDNSTXg3V/2021-05-14T11:46:08.000Z&gt;<br>  a &lt;https://<a href="http://www.w3.org/TR/vocab-ssn-ext/">www.w3.org/TR/vocab-ssn-ext/</a>#sosa:ObservationCollection&gt; ;<br>  ns0:member [<br>    a &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Measurement#OM_Measurement&gt; ;<br>    ns1:observedProperty &lt;https://<a href="http://data.vmm.be/concept/sensor/batterijniveau&gt;">data.vmm.be/concept/sensor/batterijniveau&gt;</a> ;<br>    ns1:phenomenonTime &quot;2021-05-14T11:46:08.000Z&quot;^^xsd:datetime ;<br>    ns1:result [ ns2:value [ ns3:value 3.250000e-1 ] ] ;<br>    sosa:madeBySensor &lt;urn:ngsi-v2:cot-imec-be:Device:imec-iow-kfoRZfEsBmK9Kuo4EpUBJm&gt;<br>  ], [<br>    a &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Measurement#OM_Measurement&gt; ;<br>    ns1:observedProperty &lt;https://<a href="http://data.vmm.be/concept/waterkwaliteitparameter/conductiviteit&gt;">data.vmm.be/concept/waterkwaliteitparameter/conductiviteit&gt;</a> ;<br>    ns1:phenomenonTime &quot;2021-05-14T11:46:08.000Z&quot;^^xsd:datetime ;<br>    ns1:result [ ns2:value [ ns3:value 9.470978e+2 ] ] ;<br>    sosa:madeBySensor &lt;urn:ngsi-v2:cot-imec-be:Device:imec-iow-kfoRZfEsBmK9Kuo4EpUBJm&gt;<br>  ], [<br>    a &lt;http://<a href="http://def.isotc211.org/iso19156/">def.isotc211.org/iso19156/</a>2011/Measurement#OM_Measurement&gt; ;<br>    ns1:observedProperty &lt;https://<a href="http://data.vmm.be/concept/waterkwaliteitparameter/temperatuur&gt;">data.vmm.be/concept/waterkwaliteitparameter/temperatuur&gt;</a> ;<br>    ns1:phenomenonTime &quot;2021-05-14T11:46:08.000Z&quot;^^xsd:datetime ;<br>    ns1:result [ ns2:value [ ns3:value 1.870000e+1 ] ] ;<br>    sosa:madeBySensor &lt;urn:ngsi-v2:cot-imec-be:Device:imec-iow-kfoRZfEsBmK9Kuo4EpUBJm&gt;<br>  ] ;<br>  dc:isVersionOf &lt;urn:ngsi-v2:cot-imec-be:WaterQualityObserved:imec-iow-ZibrvBsM4DQpYyDNSTXg3V&gt; ;<br>  prov:generatedAtTime &quot;2021-05-14T11:46:08.000Z&quot;^^xsd:dateTime .<br><br>...</pre><p>The above <a href="https://www.w3.org/TR/turtle/">Turtle fragment</a> describes parameters (e.g., battery level, conductivity, and temperature), their values and the definition of the used parameters (which can be found via the <a href="https://www.w3.org/wiki/URI">URI</a>).</p><p>The tree:GreaterThanOrEqualToRelationpoints to the following time-based fragment in the tree. It becomes possible to read all the stream&#39;s data by following this link over and over.</p><pre>&lt;<a href="https://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations-timebased?generatedAtTime=2022-11-08T17:28:27.680Z&gt;">https://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations-timebased?generatedAtTime=2022-11-08T17:28:27.680Z&gt;</a><br>        a              tree:Node ;<br>        tree:relation  [ a           tree:GreaterThanOrEqualToRelation ;<br>                         tree:node   &lt;<a href="https://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations-timebased?generatedAtTime=2022-11-08T17:28:37.086Z&gt;">https://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations-timebased?generatedAtTime=2022-11-08T17:28:37.086Z&gt;</a> ;<br>                         tree:path   prov:generatedAtTime ;<br>                         tree:value  &quot;2022-11-08T17:28:37.086Z&quot;^^&lt;<a href="http://www.w3.org/2001/XMLSchema">http://www.w3.org/2001/XMLSchema</a>#dateTime&gt;</pre><h4>4. LDES Client</h4><p>To consume this LDES, a software application or component called an LDES client is used. LDES clients can process and analyze events published in an LDES stream, allowing you to extract valuable insights and perform actions based on those events.</p><p>A subsequent article on this topic will be published soon.</p><h4>5. The advantages of linked data event streams</h4><p>Working with data, one of the main challenges we face is the burden of harvesting data without or with insufficient metadata. Acquiring data from a wide range of sources and formats (from data dumps in zip files to various databases, services, APIs, etc.) makes it a terrible task to transform and convert the data in a structured, matching structure. Gaining no accompanying information about the data’s origin, context, or significance leads to misinterpreting data. In addition, working with outdated or incomplete information can lead to incorrect or misleading conclusions. As a result, there is a need for a specification where data is always up-to-date, metadata is included and can be easily linked with other data sets across the Web.</p><blockquote>Linked data event stream is considered the core API that brings a solution to all these needs.</blockquote><p>LDES increases the usability and findability of data, as it comes in a uniform linked data standard published on an online URI endpoint. LDES holds historical data and ensures that the dataset is always up to date.</p><p>As a result, the user is always in sync with the publisher’s dataset and can fetch all the historical data with its metadata.</p><p>In a nutshell, there are several reasons why there was a need to add <strong>Linked Data Event Streams</strong> as a specification:</p><ol><li>Linked Data is a powerful paradigm for representing and sharing data on the Web. Still, it has traditionally focused on representing static data rather than events or changes to that data.</li><li>The use of event streams is becoming increasingly prevalent on the Web, as it enables applications to exchange information about changes to data in real-time efficiently.</li><li>There was a need for a standard format for representing Linked Data events and changes so that different systems could easily interoperate and exchange this information.</li><li>Linked Data Event Streams allow applications to subscribe to event streams and receive updates in real-time, which can be useful for various applications, such as real-time data analysis, event-driven architecture, and more.</li></ol><h4>6. Wrapping Up</h4><p>Using linked data event streams has many advantages for data scientists and other data users. Some of the main benefits include:</p><ol><li>Real-time analysis: With linked data event streams, data can be analyzed in real-time, allowing for relevant real-time analysis and decision-making.</li><li>Interoperability: Linked data event streams can be easily integrated with other data sources and systems, allowing for a more comprehensive and holistic view of the data.</li><li>Flexibility: Linked data event streams can be easily customized and adapted to different contexts and applications.</li><li>Improved clarification: Linked data event streams can help reduce errors and inconsistencies by misinterpreting the data, resulting in more reliable analysis.</li><li>Increased data accessibility: Linked data event streams provide access to data in a standardized and machine-readable format, making it easier to access and analyze data from diverse sources.</li><li>Improved data governance: Linked data event streams enable the tracking and auditing of data flows, facilitating better data management and governance.</li><li>Scalability: Linked Data Event Streams can handle large volumes of data, making them suitable for big data applications.</li></ol><p>Now you should be in good shape to know the basics of Linked Data Event Streams and their advantages. You can find more information on the websites underneath.</p><ul><li><a href="https://academy.europa.eu/courses/publishing-data-with-linked-data-event-streams-why-and-how">Publishing data with Linked Data Event Streams: why and how</a></li><li><a href="https://semiceu.github.io/LinkedDataEventStreams/">Linked Data Event Streams</a></li></ul><p>If you like what you read, be sure to ❤️ it — as a writer, it means the world. Stay in touch by following me as an author.</p><p>Contributors to this article are <a href="https://github.com/ddvlanck">ddvlanck (Dwight Van Lancker) (github.com)</a>, <a href="https://github.com/sandervd">sandervd (Sander Van Dooren) (github.com)</a> at Smart Data Space (Digital Flanders, Belgium). In a rapidly changing society, governments need to be more agile and resilient than ever. Digital Flanders realizes and supervises digital transformation projects for Flemish and local governments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1c76d077bb9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>