<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maria Claudia Bodino on Medium]]></title>
        <description><![CDATA[Stories by Maria Claudia Bodino on Medium]]></description>
        <link>https://medium.com/@mariaclaudiabodino?source=rss-24c78bc18411------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kxW7lcf5ANecFjI5f4C75Q.png</url>
            <title>Stories by Maria Claudia Bodino on Medium</title>
            <link>https://medium.com/@mariaclaudiabodino?source=rss-24c78bc18411------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:54:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mariaclaudiabodino/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How the EU supports the public sector in overcoming data challenges]]></title>
            <link>https://medium.com/@mariaclaudiabodino/how-the-eu-supports-the-public-sector-in-overcoming-data-challenges-684115d1e0f5?source=rss-24c78bc18411------2</link>
            <guid isPermaLink="false">https://medium.com/p/684115d1e0f5</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[public-sector]]></category>
            <dc:creator><![CDATA[Maria Claudia Bodino]]></dc:creator>
            <pubDate>Fri, 29 Sep 2023 10:49:15 GMT</pubDate>
            <atom:updated>2023-09-29T11:27:54.610Z</atom:updated>
            <content:encoded><![CDATA[<h3>Exploring the Big Data Test Infrastructure (BDTI) and its capabilities using an example use case</h3><blockquote>Public Administrations face increasingly complex and urgent socio-economic problems in their line of work. Using big data and data-informed decision making can help to come up with new and better solutions and provide transparency and trust to your citizens on decision making.</blockquote><blockquote>Big data and open data are everyday terms. However, knowing how to implement these technologies in our daily work requires skills, experience and resources, but most of all the use of the right digital tools which are often missing.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u3wjKzJH0cxwBD_J" /><figcaption>Photo by <a href="https://unsplash.com/@ryoji__iwata?utm_source=medium&amp;utm_medium=referral">Ryoji Iwata</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The European Commission is helping to address this challenge with a service called the <a href="https://big-data-test-infrastructure.ec.europa.eu/index_en"><strong>Big Data Test Infrastructure</strong></a><strong> (BDTI)</strong>. <strong>The BDTI offers public administrations a set of mainstream open-source tools for data storage, processing and analytics, hosted in the cloud and free of charge. </strong>It is offered to civil servants at all levels of government in Europe so that they can autonomously run pilot projects that demonstrate the value that data can bring to policymaking or administration. Civil society, academia and even the private sector can join the pilot projects, so long as a public administration body is at the centre of the use case. Within BDTI’s 6 months pilot, you can learn all about the value of using data in your sector and what it takes to land this in your organization.</p><p>This article will walk you through the BDTI service and its capabilities. After a general introduction to the service and its characteristics, an example use case will be presented to demonstrate the practical implementation and capabilities of the platform, making use of a variety of open-source tools and re-uses multiple open datasets.</p><p>If you wish to learn more about the Big Data Test Infrastructure (BDTI) with a hands on approach,<strong> </strong><a href="https://europa.eu/regions-and-cities/programme/2023/sessions/3006#close-modal"><strong><em>join</em></strong></a><strong><em> </em>the dedicated session</strong> <strong>at the EU Week of Regions and Cities in Brussels on the 11th of October for an engaging workshop to learn more about the platform</strong>. Cannot make it? Then subscribe to the <a href="https://joinup.ec.europa.eu/collection/egovernment/solution/big-data-test-infrastructure-bdti/event/bdti-goes-european-week-regions-and-cities"><em>Joinup page</em></a> to stay up to date with future events and initiatives.</p><h4><strong><em>The EU’s Big Data Test Infrastructure: what it is and how it works</em></strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bRh9HrnCPYxsam84.png" /></figure><p>The EU’s <a href="https://big-data-test-infrastructure.ec.europa.eu/index_en"><strong>Big Data Test Infrastructure</strong></a>, <strong>BDTI</strong>, was created in 2019, as part of the <a href="https://digital-strategy.ec.europa.eu/en/activities/digital-programme"><em>Digital Europe Programme</em></a>, which aims to accelerate Europe’s economic recovery and shape the digital transformation of Europe’s society and economy, increasing the easy availability, quality and usability of public sector information in compliance with the requirements of the European <a href="https://digital-strategy.ec.europa.eu/en/policies/legislation-open-data"><em>open data directive</em>.</a></p><p><strong>The purpose of the BDTI is to foster the reuse of public sector data and enable a data-informed public sector in the EU Member States. </strong>By providing a free-of-charge analytics test environment with open-source tools, the BDTI allows public administrations to prototype solutions before deploying them in the production environment on their own premises.</p><p>The <strong>BDTI </strong>applications are offered as a cloud service, enabling users to experiment with data in a <strong>pilot project</strong>. Once the project is over, users can take the source code and data with them to continue the work using the cloud of their choice or other resources. The platform consists of open-source tools and the required cloud infrastructure, which includes virtual machines, analytics clusters, storage facilities and networking facilities. To learn more about the tools available, you can have a look at <a href="https://big-data-test-infrastructure.ec.europa.eu/service-offering_en">the service offering page</a>.</p><h4><strong><em>Use cases and success stories</em></strong></h4><p>There are several real-life <a href="https://big-data-test-infrastructure.ec.europa.eu/success-stories_en">success stories</a> of public administrations using <strong>BDTI </strong>to improve their decision making. For example, <a href="https://digital-strategy.ec.europa.eu/en/library/experimenting-bdti-led-new-dimensions-labour-market-intelligence">Eurostat and its partners</a> used the BDTI to experiment with data in the development of official statistics. In this pilot project, open online job advertisement data was used to provide timely information about European labour markets.</p><p>Other real-life success stories are text mining by the Health services of the city of Valencia, the <a href="https://digital-strategy.ec.europa.eu/en/library/norwegian-digitalisation-agency-uses-commission-big-data-tool-optimise-public-procurement">optimisation of public procurement by the Norwegian Digitalisation Agency</a>, data sharing efforts by the European Blood Alliance, and work to facilitate the <a href="https://digital-strategy.ec.europa.eu/en/library/city-florence-uses-big-data-tool-understand-covid-19-impact">understanding of COVID-19’s impact on the city of Florence</a>.</p><p>Additionally, the <strong>BDTI </strong>website provides multiple <a href="https://big-data-test-infrastructure.ec.europa.eu/resources/use-cases_en"><em>use cases</em></a> that showcase the platform’s capabilities, based on open data. For example, the <a href="https://big-data-test-infrastructure.ec.europa.eu/resources/use-cases/search-analytics_en"><em>search analytics use case</em></a> is based on the <a href="https://data.europa.eu/data/datasets/cordish2020projects?locale=en"><em>CORDIS</em></a> open dataset, and the <a href="https://big-data-test-infrastructure.ec.europa.eu/resources/use-cases/low-code-data-analytics_en"><em>low code analytics use case</em></a> works with the <a href="https://setis.ec.europa.eu/emhires-dataset-part-ii-solar-power-generation_en"><em>EMHIRES</em></a> — European Meteorological derived High Resolutions RES generation time series for present and future scenarios — open dataset on solar power generation.</p><p>In the following sections, let’s walk through the fictional example of the Municipality of Dublin called “Government Spending”, to present how the service’s capabilities and tools can be applied to generate valuable insights from data. The project is available open-source on the <a href="https://code.europa.eu/bdti/bdti-demonstrator"><em>European Commission GitLab</em></a><em>.</em></p><h4><strong><em>The “Government Spending” use case</em></strong></h4><p>The ‘government spending’ use case as developed by the <strong>BDTI</strong> team consists of three stages typical of a data science project:</p><p>(1) data ingestion;</p><p>(2) visualisation and analysis;</p><p>(3) decision-making (using machine learning).</p><p>The subject of the fictional use case is the municipality of Dublin, based on open data (from <a href="https://data.smartdublin.ie/"><em>data.smartdublin.ie</em></a>; <a href="https://data.gov.ie/"><em>data.gov.ie</em></a>; <a href="https://open-meteo.com/"><em>open-meteo.com</em></a><em>) </em>and complemented where needed by fictional data. The <a href="https://code.europa.eu/bdti/bdti-demonstrator"><em>GitLab page</em></a> for this use case is open and visible to anyone who wish to see the full documentation, including user guides and scripts needed to replicate it, as well as the details regarding the (open) datasets used.</p><p><strong><em>Data ingestion</em></strong></p><p>The first step of the demonstration use case is the ingestion of government spending data. The specific challenge that needs to be addressed relates to energy invoices, which are only available in PDF format in our scenario. These non-machine-readable PDF invoices must be transformed into data that can be easily processed later.</p><p>To solve this challenge, a solution is built using a tool available on the BDTI, the KNIME Analytics platform. This tool is an open-source software that has an intuitive, visual interface and does not require coding, including optical character recognition features. Optical character recognition is a technology that interprets human-readable documents and transforms them into machine-readable data.</p><p>The output table can be reused to further process and analyse the data. BDTI offers a solution to store the output data in preparation for the next step in a relational database system.</p><p><strong><em>Visualisation and analysis</em></strong></p><p>After completing the data ingestion, the next step is to visualise and analyse the government spending data. To do so, Apache Superset, another open-source tool on data exploration and visualisation available on the BDTI, has been used. The result is a dashboard that visualises the actual government spending data of Dublin, by presenting the proportion of spending for each category relative to the total expenditure. Although this information is interesting in itself, it does not help to understand whether spending is high or low.</p><p>To provide more context to Dublin’s government spending data, a comparison of Dublin’s spending against two similar, but fictional, cities (reference city A and reference city B) has been created. This comparison reveals that Dublin allocates a relatively large part of its budget to public lighting. This kind of benchmarking will not directly tell where Dublin spends too much or too little but can provide hints as to what needs to be further investigated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_uIMK8Ca-RbYe-ZLdId6Mg.png" /></figure><p><strong><em>Decision-making</em></strong></p><p>The third and last step of the use case aims to build a solution for data-informed decision-making on public lighting related to expected traffic levels. Three open-source tools available on the <strong>BDTI </strong>to build this solution have been used. Thanks to this combination of tools, you can build a solution that supports civil servants in achieving savings on public lighting.</p><p>To propose a solution, a simple machine-learning model that predicts traffic for the upcoming week has been trained. The tool used for it can be applied to data science, statistical modelling and more. The machine-learning model uses weather data and traffic data, requiring the processing of large volumes of data. The BDTI is intended for processing big data, so it will support you in processing very large datasets such as these.</p><p>After training and running the machine-learning model using Jupyter Notebook, the output data is stored to create a dashboard using Apache Superset. The dashboard allows users to analyse savings that result from turning off public lighting when light is least needed. To determine when and where lighting is least needed, the predicted traffic levels have been used as a proxy for activity in a street. The lower the activity in a street, the less need there is for lighting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oNY0x-54yRK8uCRuFy8VBg.png" /></figure><p><strong>Conclusion</strong></p><p>This article delved into a use case that shows an example of how the BDTI’s capabilities and tools can be applied to create valuable insights from (open) data. The examples demonstrate that <strong>the BDTI platform offers a variety of open source tools, which can be used by people with different levels of data skills</strong>. For example, the data ingestion and visualisation examples do not require any coding skills, whereas the traffic prediction requires a certain level of data science skills. More example <a href="https://big-data-test-infrastructure.ec.europa.eu/resources/use-cases_en"><em>use cases</em></a> or real-life <a href="https://big-data-test-infrastructure.ec.europa.eu/success-stories_en"><em>success stories</em></a> are accessible via the <a href="https://big-data-test-infrastructure.ec.europa.eu/index_en"><strong><em>BDTI website</em></strong></a>.</p><p>Are you a civil servant interested in developing your own use case on the BDTI or a student and researcher eager to support a public administration? <strong>Apply for a BDTI pilot project </strong><a href="https://big-data-test-infrastructure.ec.europa.eu/apply-bdti_en"><strong><em>here</em></strong></a>, <strong>or reach out to the team via </strong><a href="/service/mailto:EC-BDTI-PILOTS@ec.europa.eu"><strong><em>EC-BDTI-PILOTS@ec.europa.eu</em></strong></a><strong>.</strong></p><p>Do you want to learn more about the Big Data Test Infrastructure (BDTI)? <a href="https://europa.eu/regions-and-cities/programme/2023/sessions/3006#close-modal"><strong><em>Join</em></strong></a><strong><em> </em>the dedicated session at the EU Week of Regions and Cities in Brussels on the 11th of October for an engaging workshop to learn more about the platform</strong>. Cannot make it? Then subscribe to the <a href="https://joinup.ec.europa.eu/collection/egovernment/solution/big-data-test-infrastructure-bdti/event/bdti-goes-european-week-regions-and-cities"><em>Joinup page</em></a> and newsletter to stay up to date with future events and initiatives.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=684115d1e0f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>