<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hatem Ben Yacoub on Medium]]></title>
        <description><![CDATA[Stories by Hatem Ben Yacoub on Medium]]></description>
        <link>https://medium.com/@hatem?source=rss-caab74aa7ddd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0YRdXxK4ogUd81K1fQuUCQ.jpeg</url>
            <title>Stories by Hatem Ben Yacoub on Medium</title>
            <link>https://medium.com/@hatem?source=rss-caab74aa7ddd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:45:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hatem/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Global Data Barometer first edition Released — Part 1]]></title>
            <link>https://medium.com/@hatem/the-global-data-barometer-first-edition-released-part-1-9b56591da485?source=rss-caab74aa7ddd------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b56591da485</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-economy]]></category>
            <category><![CDATA[global-data-barometer]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[open-data]]></category>
            <dc:creator><![CDATA[Hatem Ben Yacoub]]></dc:creator>
            <pubDate>Thu, 12 May 2022 00:00:00 GMT</pubDate>
            <atom:updated>2022-05-14T19:53:35.608Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Global Data Barometer first edition Released — Part 1</h3><p>State of the data in the MENA Region</p><p>The first edition of the global data barometer have just been released yesterday, a zoom webinar took place yesterday to introduce results of this first edition and talk about this amazing new experience. I was honored to work in this edition as researcher with two regional hubs : The Center for Continuing Education (CCE) at Birzeit University for the golf countries, and LDRI Kenya for North African countries. 109 countries have been surveyed in this edition to get the state of data for public good around the world !</p><p>The results in my opinion was very fair and reflect the reality of the data economy in the world. To be honest the research was much harder than previous barometers, as I worked previously in the ODB with the web foundation. In the GDB there are much more work and research, more details and more detailed modules !</p><p>There are always recommendations to improve the barometer, and I think most complains that I heard was about the political integrity module. Overall, the barometer for those who don’t know it is organized around four pillars or core areas of assessment: governance, capability, availability, and use and impact. The barometer include two core modules which correspond to data governance and capability, in addition to five thematic modules that examine data for the public good related to money, property, and power. The other two pillars of the Barometer, availability and use and impact, are assessed through the thematic modules.</p><p>So before diving in the MENA region, I want to point to few key findings in the barometer by module :</p><p><strong>Capabilities :</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/0*DUNRE-pmeWEsD_BO" /></figure><p>Capabilities is one of the four pillars, or areas of assessment, of the Global Data Barometer and will be examined as a core module given its foundational importance across all aspects of data activity. The module include 4 sub-modules : civil service training, open data initiative, Government support for reuse, and sub-national, which mean to what extent do city, regional, and local governments have the capability to effectively manage data.</p><p>Best score in capabilities goes to Estonia, with 92 out of 100. Followed by South Korea 88, then Span 82. Most probably sub-module that most countries are missing is civil service training, where Estonia scored 90.</p><p><strong>Climate action</strong></p><p>The climate action module investigates national and sub-national climate action data as a critical tool for increasing the number and diversity of climate actors and supporting the expansion and ambition of mitigation and adaptation efforts. In this module we have three pillars : Emission, biodiversity and vulnerability.</p><p>Best score in climate action goes to USA with a score of 94 out of 100. Followed by New Zealand 87 and Finland 81. What I find interesting in this indicator is that both USA and New Zealand got both top score (100) in the Emission indicator, while Finland got top score (100) in Biodiversity.</p><p>Data about climate vulnerability should integrate or otherwise address the two major strands of vulnerability approaches: the risks and hazards approach, which focuses on responding to natural hazards and extreme weather events; and the entitlements and livelihoods approach, which focuses on preventing undesirable outcomes by identifying where people have too few resources to withstand or recover from disaster-for example, in conjunction with poverty, gender, and marginalization.</p><p><strong>Company Information</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/0*XHw88A-ObpEmLav4" /></figure><p>The company information module has been developed in partnership with Open Ownership, explores data on companies as a key connecting element of a modern public data infrastructure, supporting trade, trusted business environments, and investigation and enforcement related to anti-corruption and anti-money laundering activities.</p><p>Best score in company information goes to Denmark with a score of 91 out of 100. Followed by UK and Ireland 83, and Latvia 81. Here it’s very interesting to notice the presence of a small country like Latvia among the top 3 in company information module ! Latvia’s economic freedom score is 74.8, making its economy the 18th freest in the 2022 Index of Economic Freedom.</p><p><strong>Governance</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/0*tiSQ01X2KKTKcG3n" /></figure><p>This section includes a number of country-level data governance indicators on the presence of regulatory regimes for data protection, right to information, and right to data, as well as identifying emerging frameworks for data sharing. In addition to ecosystem-specific governance indicators located within the different thematic modules covered in the Barometer. These indicators will explore the extent to which the collection, sharing, and secure management of specific datasets are safeguarded and optimized through laws, regulations, policies, infrastructure, and guidance.</p><p>Best score in Governance goes to South Korea with a score of 79 out of 100. Followed by New Zealand 78, and Spain 77. Scores are very close in this module, but what is most notable here is that the 3 first places share having a top score in one of the sub-modules of the Governance, which are Data management framework for South Korea (100/100), Data Sharing Framework for New Zealand (100/100) and Data protection law in Spain (100/100).</p><p><strong>Health and Covid19</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*mGWnnyc_hDlh1rcX" /></figure><p>The health &amp; COVID-19 thematic module is designed to cover research, public health work, and provision of primary healthcare, all of which draw substantially upon data. The COVID-19 pandemic has underscored the importance of reliable, accessible, and trusted health data to enable coordinated action. The governance and reusability of data in the health sector must combine respect for individual privacy rights with a focus on population and patient health. Data can be used to support access to healthcare across the many different healthcare systems of the world.</p><p>Best score in Health and Covid19 thematic module goes to United States with a score of 85 out of 100. Followed by Germany 83, and Slovakia 82. What we found interesting in this module is that most countries, or at least the top 50 countries, have all top score in Covid19 testing with 100/100 with few exceptions. Generally if the score is not 100 in Covid19 testing, it’s a zero. So it’s a feature either you have it all, or you don’t at all. However in vaccination score is very variable as there is not much details about vaccination published, if any.</p><p><strong>Land</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/0*yZ1XzV8kXK8A8eAz" /></figure><p>The land thematic module has been co-developed with the Land Portal research team to understand the state of data on land around the world and to contribute to overall Global Data Barometer assessments of data for the public good.</p><p>There are a wide range of sub-themes and datasets that might be covered by the topic of land data, and land issues are often highly interconnected, with consequences for many different fields including socioeconomic development, environmental protection, and anti-corruption efforts. The barometer focus specifically on land tenure and on land use, as well on uses of land data to support work on gender and inclusion.</p><p>Best score in Land thematic module goes to New Zealand with a score of 96 out of 100. Followed by Denmark 79, then United States 74. New Zealand marked top score in the Land tenure sub-module , which is not very frequent. Companies having top score in land tenure include also Estonia, Albania, and Germany. A very important indicator, which mean in these countries you can identify who holds rights over land. A lack of transparency is noticed overall in this module among all countries.</p><p><strong>Political Integrity</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/0*TYFnTViwqnKQOayy" /></figure><p>The political integrity module has been co-developed with the Open Government Partnership research team and Transparency International. To capture a breadth of information and data relevant to political integrity and to address key gaps in current data, it addresses five specific sub-themes: Data on political party finance; Data on political interest declarations; Lobbying registers; Data on public consultation in rule-making; and Data on right-to-information regime performance.</p><p>This is the largest thematic module in the Global Data Barometer 2021 edition and is being used to pilot an extended thematic module approach.</p><p>Best score in Political integrity thematic module goes to United States with a score of 82 out of 100. Followed by UK and Ireland 60, then Canada 58. Which is very notable in this module is that score is very low even less than average for most countries (92%).</p><p><strong>Public Finance</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*-Nwz6yb2eTB66kJf" /></figure><p>The public finance module has been co-developed with the Global Initiative for Fiscal Transparency to improve comparative understanding of the development of open data approaches to providing granular public finance information, and to contribute to overall Global Data Barometer assessments of data for the public good.</p><p>Best score in Public Finance thematic module goes to France with a score of 94 out of 100. Followed by Mexico and South Africa (93 both). Interesting to find in the top countries, names like Mexico, South Africa, but also Armenia 88, Republic of Korea 85, Indonesia 81, and Bulgaria 80; among names like Germany 87, Canada 84 and United States 83.</p><p><strong>Public procurement</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/0*-ZEEE0gS90GrTp7W" /></figure><p>The public procurement module has been developed in collaboration with the Open Contracting Partnership, built on the Open Data Barometer indicator on public contracts data and seeks to examine the connections between public procurement data and other key elements of public data infrastructure, including company information and public finance data.</p><p>Best score in Public procurement thematic module goes to Ukraine with 96 out of 100. Followed by Paraguay 91, then Kazakhstan 87. Ukraine is like Estonia have a top score in Public procurement (100/100), however Ukraine have the advantage of having extra score on procurement analytics which is the highest among other countries in this sub-module. Russia is ranked 8th in this module with a score of 82. Most interesting in this module that top countries are not necessarily top countries in the global ranking, or economies, but having the best data infrastructure for procurement.</p><p><strong>Conclusion</strong></p><p>In this first part I tried to cover all modules of the barometer one by one and mention some interesting insights from the results, to better understand score and recommendations. I’ll try to talk in the next part about the MENA region and key findings and insights in the region.</p><p>More details about the results of the 1st edition of the Global Data Barometer could be found at : <a href="https://globaldatabarometer.org/results/">https://globaldatabarometer.org/results/</a></p><p><em>Originally published at </em><a href="https://hbyconsultancy.com/blog/the-global-data-barometer-first-edition-released-part-1.html"><em>https://hbyconsultancy.com</em></a><em> on May 12, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b56591da485" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[WebUSB is dead]]></title>
            <link>https://medium.com/@hatem/webusb-is-dead-2629bfe294af?source=rss-caab74aa7ddd------2</link>
            <guid isPermaLink="false">https://medium.com/p/2629bfe294af</guid>
            <category><![CDATA[webusb]]></category>
            <category><![CDATA[embedded-systems]]></category>
            <category><![CDATA[usb]]></category>
            <category><![CDATA[chrome]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Hatem Ben Yacoub]]></dc:creator>
            <pubDate>Mon, 03 Feb 2020 19:54:13 GMT</pubDate>
            <atom:updated>2020-02-04T11:50:19.864Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kw1jotkRwYgYSUgP.png" /></figure><p>Two years ago I started playing with <a href="https://github.com/WICG/webusb">WebUSB</a>, a web API to connect hardware to the Web. The idea was crazy, especially that there is already an implementation supported by Chrome, but… unfortunately only few experiments worked fine. The same experiments that I saw two years ago.</p><p>Due to the lack of documentation, I quickly hit the limitation of the API and reached a dead-end. I posted few questions in stackoverflow, such as the <a href="https://stackoverflow.com/questions/48314830/webusb-limitation-controltransferout">controlTransferOut() limitation</a>, then I find out how to deal with it.</p><p>Then I posted more detailed code on how to use <a href="https://stackoverflow.com/questions/48136414/webusb-to-scan-documents">WebUSB to connect to a scanner</a> directly ! I was able to connect to my scanner, it was a Canon DR-M160II, but low level transfer code got stack somewhere. The issue seems to be related to webusb chrome implementation itself.</p><p>Emscripten wasn’t a solution too, as I quickly hit the limitation of the compiler too ! Sane was working correctly from my Ubuntu, so WebUSB should be working fine too !</p><p>Last but not least, WebUSB API lack of documentation until today. Not lots of people are using the API, 27 issues open today on github, 90 closed.</p><p>WebUSB is available in Chrome 61 for Android, Chrome OS, Linux, macOS and Windows. If you are able to connect successfully to your USB device from Chrome feel free to drop a message !</p><p>Notice that the last WebUSB API unofficial draft have been published on December 3rd 2019, by a team from Google ! The only implementation remain in Chrome… But I still thing that it’s dead, most probably because it was not able to leave Google.</p><p><em>Originally published at </em><a href="http://www.hbyconsultancy.com/blog/webusb-is-dead.html"><em>http://www.hbyconsultancy.com</em></a><em> on February 3, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2629bfe294af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Opening Up 2018 Tunisian Municipality Elections Data — Part 2]]></title>
            <link>https://medium.com/@hatem/opening-up-2018-tunisian-municipality-elections-data-part-2-b613f7b590e8?source=rss-caab74aa7ddd------2</link>
            <guid isPermaLink="false">https://medium.com/p/b613f7b590e8</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pdf]]></category>
            <category><![CDATA[tunisia]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[elections]]></category>
            <dc:creator><![CDATA[Hatem Ben Yacoub]]></dc:creator>
            <pubDate>Wed, 25 Apr 2018 09:10:09 GMT</pubDate>
            <atom:updated>2018-04-25T09:10:09.278Z</atom:updated>
            <content:encoded><![CDATA[<p>Just after publishing <a href="https://medium.com/@hatem/opening-up-2018-tunisian-municipality-elections-data-part-1-7025496beff0">my first post</a> converting Candidates lists in the 2018 Tunisian municipality elections from PDF to CSV, <a href="http://www.isie.tn/elections/elections-municipales-2018/candidatures/">ISIE published candidates names lists</a> ! More data ! This time it’s all in XLSX format ! Sounds good ? Not at all, I need to get all these files into a single document to be able to do any verification. Be aware, the process is not as easy as you may think ! Welcome to a new #OpenData challenge</p><p>First thing, I was not sure that there are the right number of files as it is organized in a tree. So let’s verify that there are 350 files :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/0*LEsWlQZx9bvPsrll.png" /></figure><p>Sounds good, I downloaded the 350 files. As usual you will find in the xlsx files title, sub-title, logo … just useless metadata before finding the data !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/0*ayPuKncvKG7Gqk_F.png" /></figure><p>So let’s convert everything to csv (sudo apt install gnumeric) if you don’t have ssconvert binary :</p><p>$ for i in *.xlsx; do ssconvert “$i” “$i.csv” ; done</p><p>Now we have list of csv, I will just move them to a separate folder :</p><p>$ mkdir csv</p><p>$ mv *.csv csv/</p><p>As I mentioned before the first 7 lines are not part of the data (in most files), so it should be removed (you will see later that it’s more than 7):</p><p>$ cd csv</p><p>$ for i in *.csv; do sed -i 1,7d “$i” ; done</p><p>Now I will try to combine them. Combining them with miller should fail if the files don’t have the exact format so it’s recommended to use instead of cat :</p><p>$ mlr — rs lf — csv sort -f date,code *.csv &gt; combined.csv</p><p>mlr: unacceptable empty CSV key at file “%D8%A8%D9%84%D8%AF%D9%8A%D8%A9%20%D8%A7%D9%84%D8%AC%D8%B1%D9%8A%D8%B5%D8%A9.xlsx.csv” line 1</p><p>Thank you, first error ! Let’s have a look on the error :</p><p>$head -5 %D8%A8%D9%84%D8%AF%D9%8A%D8%A9%20%D8%A7%D9%84%D8%AC%D8%B1%D9%8A%D8%B5%D8%A9.xlsx.csv</p><p>,,,,,,,</p><p>,,,,,,,</p><p>“الإدارة</p><p>الفرعية”,”الدائرة</p><p>البلدية”,”تسمية القائمة”,”طبيعة القائمة”,”لقب المترشح”,”إسم المترشح”,”رتبة</p><p>And you will notice that the two first lines are empty, I will remove the extra lines manually :</p><p>$ sed -i 1,2d %D8%A8%D9%84%D8%AF%D9%8A%D8%A9%20%D8%A7%D9%84%D8%AC%D8%B1%D9%8A%D8%B5%D8%A9.xlsx.csv</p><p>Then repeat miller command again, until getting the combined file. After three files miller still fail with “mlr: syntax error : unwrapped double quote at line 0”. Thanks miller we just hit your limitation, let’s move to something else :</p><p>$ cat *csv &gt; combined.csv</p><p>Combined file show more than 24 thousands records (that’s not the exact number), a quick look in openrefine and I found the files causing error. How to do it ? Easily look for known types like numeric, then facet/filter and look for Non-numeric/blank/errors :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/0*_MSxF6z2XcEl-ala.png" /></figure><p>Finally the error is due to files having more than 8 columns. As we are working with files created manually, always expect to find such issues.</p><p>But there is still one more problem to fix before copying the 8 columns. Notice in the result of the “head” command, the first line which should contain the header is in four or more lines ! We need to remove the newline character inside the columns. I did it my own way, copy header in a separate file, remove all first 4 lines from each csv file, then combine again using the header that we have already created. I will use any file since they are all the same, or almost.</p><p>$ head -4 %D8%A8%D9%84%D8%AF%D9%8A%D8%A9%20%D8%A7%D9%84%D8%AC%D8%B1%D9%8A%D8%B5%D8%A9.xlsx.csv &gt; header</p><p>I will edit the header file manually here, then :</p><p>$ for i in *.csv; do sed -i 1,4d “$i” ; done</p><p>Normally this should be okay, we did not delete any useful data. If any data will be deleted here by mistake we will notice that in the index not starting with 1. Now I will loop again on all files and copy the first 8 columns only :</p><p>$ mkdir copy</p><p>$ for i in *.csv; do cut -d “,” -f 1–8 “$i” &gt; “copy/$i” ; done</p><p>$ cd copy</p><p>$ cat *csv &gt; combined.csv</p><p>$ cat ../header combined.csv &gt; newcombined.csv</p><p>The last part is self explanatory, I just combined csv files and added the header on the top. Openrefine show again two more files creating issues, manual edit then my csv should be final :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b1E7aqIMtsD1Ah7n.png" /></figure><p>We have 45345 rows/records, which mean 45345 candidates ! Huge, how did the ISIE verified the candidature of each one of them ? Second thing I noticed is that some lists have up to 61 names ! And we can see in the next screenshot that there are 25 lists having between 40 and 61 candidates :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/0*CnCglTriaYyR_8DV.png" /></figure><p>Now that’s a file we can work with, feel free to download !</p><p><a href="http://www.hbyconsultancy.com/blog/opening-up-2018-tunisian-municipality-elections-data-part-2.html"><strong>CSV file available for download on the original article’s link</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b613f7b590e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Opening Up 2018 Tunisian Municipality Elections Data — Part 1]]></title>
            <link>https://medium.com/@hatem/opening-up-2018-tunisian-municipality-elections-data-part-1-7025496beff0?source=rss-caab74aa7ddd------2</link>
            <guid isPermaLink="false">https://medium.com/p/7025496beff0</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[tunisia]]></category>
            <category><![CDATA[elections]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pdf]]></category>
            <dc:creator><![CDATA[Hatem Ben Yacoub]]></dc:creator>
            <pubDate>Wed, 25 Apr 2018 08:57:40 GMT</pubDate>
            <atom:updated>2018-04-25T08:57:40.756Z</atom:updated>
            <content:encoded><![CDATA[<p>Publishing elections data in closed format become very common, in case any data have been published. Few days from the municipal elections we are still worried if it is going really to happen or not. Efforts come as usual from civil society and private sector to open up data and reuse it, and we saw few applications such as <a href="http://tunisieelection.org/lists-overview">TunisieElection.org</a> or <a href="http://www.targa-consult.com/2018/04/18/%D8%A7%D9%84%D8%A7%D9%86%D8%AA%D8%AE%D8%A7%D8%A8%D8%A7%D8%AA-%D8%A7%D9%84%D8%A8%D9%84%D8%AF%D9%8A%D8%A9-%D8%A7%D9%84%D8%AA%D9%88%D9%86%D8%B3%D9%8A%D8%A9-2018/">Targa’s analysis</a> … Big thanks to Abderrahman Gharsallah who did the first dataviz and released data in open format on <a href="https://github.com/hunter-x/All-Data">github</a>, a project supported by Democracy International and Mourakiboun.</p><p>One reason of doing this work is a comment from a friend telling me that there are some independent lists that are present in more than one region, and they could be not really independent as they claim ! I started googling the list name and found that it’s present in at least 6 regions. So I needed <a href="http://www.isie.tn/elections/elections-municipales-2018/candidatures/">official list of candidates</a> to check how many independent list is present in more than one region.</p><p>Unfortunately I have sometimes some bad habits to start from scratch, and that’s how I discovered that a file related to Tunis2 candidates have been removed from official ISIE website and replaced with a new one, nothing changed in data, it still in PDF but columns are in reverse order ! Now tables are in PDF ? First step to deal with this is <a href="http://tabula.technology/">Tabula</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*l87GmLRo_yIT1C3U.png" /></figure><p>I converted all PDF to CSV format :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/0*rzJw_j67BVzflvWr.png" /></figure><p>Then combined them in one file using “miller” :</p><p>mlr — rs lf — csv sort -f date,code *.csv &gt; combined.csv</p><p>Of course I make sure that I reversed columns order in Tunis2 file before combining files to be able to directly work with data. I was looking for independent lists, but that’s how I discovered that the whole list of names is messing !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*VFR_cRHqKDMSi30R.png" /></figure><p>Cluster analysis of data in Arabic is really hard, but it can help :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Qgw4jCcSxYjQDtqN.png" /></figure><p>So I started combining names to make sure that a same list have the same name in the whole dataset. The final result is quiet interesting : about 9% of independent lists have more than one list, six of them have more than ten lists ! Especially if you know that 21 parties are participating in this elections with one list only ! Independent lists seems to be more powerful than many well known political parties.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*fUvbYv14ZghtvvG_.png" /></figure><p>I will stop here, below the list of independent parties participating in this municipal elections and having more than 10 lists :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*_Trt7LBX3dfoC-z3.png" /></figure><p>It’s great to have independent lists participating in these elections, however lots of questions come to my mind when I saw these numbers. Are they really independent ? Who is behind these lists ? I think it’s for the ISIE to answer these questions in case municipal elections are going to happen this 6th May 2018.</p><p><a href="http://www.hbyconsultancy.com/blog/opening-up-2018-tunisian-municipality-election-data.html"><strong>CSV file available for download on the original article’s link</strong></a></p><p>Article originally posted on <a href="http://www.hbyconsultancy.com/blog/opening-up-2018-tunisian-municipality-election-data.html">http://www.hbyconsultancy.com/blog/opening-up-2018-tunisian-municipality-election-data.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7025496beff0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The power of Commodity Hardware]]></title>
            <link>https://medium.com/@hatem/the-power-of-commodity-hardware-76bb1a302cd0?source=rss-caab74aa7ddd------2</link>
            <guid isPermaLink="false">https://medium.com/p/76bb1a302cd0</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Hatem Ben Yacoub]]></dc:creator>
            <pubDate>Wed, 25 Oct 2017 07:51:18 GMT</pubDate>
            <atom:updated>2017-10-25T07:52:34.454Z</atom:updated>
            <content:encoded><![CDATA[<p>In this new era of Big Data, the term commodity hardware became very common, but it’s up to you to define how to use it. The term commodity hardware in general mean Computer hardware that is affordable and easy to obtain. Typically it is a low-performance system that is PC-compatible and is capable of running Microsoft Windows, Linux, or others operating systems without requiring any special devices or equipment.</p><p>Why do we need datacenters with special configurations to be able to run servers 24/7 : ACs, UPS, raised floors, networking, security…etc. While we used to have a desktop computer in every office. Imagine a company having 30 desktop computers having an i7 8cores 4GB ram 500GB hard drives each, that’s the equivalent of a 240 Core super computer with 120GB ram, and 15 TB or storage ! But obviously we don’t use them as servers, because these PCs fail frequently and they are not designed to run 24/7.</p><p>That’s exactly what commodity hardware are supposed to be, they fail ! Today we have very accessible technologies that allow us to run a large number of low-cost, low-performance commodity computers working in parallel instead of using fewer high-performance and high-cost computers, and we call it commodity cluster computing.</p><p>Startups should be the first to profit from these technologies, and for example in web applications development resources per computer could be easily shared in the network instead of having a local development environment per developer. The more your hardware is connected, the more powerful is your cluster.</p><p><strong>Some experiments :</strong></p><p>Having at home a development board which is the <a href="http://www.hbyconsultancy.com/blog/banana-pi-experiments-with-debian-wheezy.html">Banana Pi</a>, an A7 dual Core arm based board with 1GB ram, I installed debian jessie and make it join a Docker Swarm that I run on my own laptop using a simple network cable :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*qNltwwoQOKLhy37J.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*nmjW52mZERpSJor5.png" /></figure><p>That’s two different architectures, and very low-cost hardware, exactly what could represent commodity hardware. You can see it running Docker Swarm with portainer GUI giving a total of 4 CPUs and 9GB ram. Now I can easily run some code on the banana pi or in the cluster :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*55AOrQK9FYpw-Sbr.png" /></figure><p>But if I change my mind and decided to unplug the banana pi, or the hardware failed for any reason ? The two nginx containers will keep running on the available hardware. As simple as that !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*UKh9SpVA9smDNmi3.png" /></figure><p>Now just imagine the size the of the supercomputer that you have in front of you, and start thinking about what can you do with a supercomputer ?! How you can distribute it between daily tasks, and super computing or cloud computing tasks.</p><p>The power of Commodity Hardware October 25th, 2017Hatem Ben Yacoub</p><p>Tagged: <a href="http://www.hbyconsultancy.com/blog/tag/cluster-computing">Cluster computing</a>, <a href="http://www.hbyconsultancy.com/blog/tag/commodity-hardware">Commodity hardware</a>, <a href="http://www.hbyconsultancy.com/blog/tag/distributed-computing">Distributed computing</a>, <a href="http://www.hbyconsultancy.com/blog/tag/docker">Docker</a>, <a href="http://www.hbyconsultancy.com/blog/tag/swarm">Swarm</a></p><p><em>Originally published at </em><a href="http://www.hbyconsultancy.com/blog/the-power-of-commodity-hardware.html"><em>www.hbyconsultancy.com</em></a><em> on October 25, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76bb1a302cd0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sharing Economy en Tunisie, Analyse des données Airbnb]]></title>
            <link>https://medium.com/@hatem/sharing-economy-en-tunisie-analyse-des-donn%C3%A9es-airbnb-1774993bf888?source=rss-caab74aa7ddd------2</link>
            <guid isPermaLink="false">https://medium.com/p/1774993bf888</guid>
            <category><![CDATA[airbnb]]></category>
            <category><![CDATA[sharing-economy]]></category>
            <category><![CDATA[morocco]]></category>
            <category><![CDATA[airbnb-data]]></category>
            <category><![CDATA[tunisia]]></category>
            <dc:creator><![CDATA[Hatem Ben Yacoub]]></dc:creator>
            <pubDate>Mon, 15 Aug 2016 20:09:04 GMT</pubDate>
            <atom:updated>2016-08-20T11:55:15.347Z</atom:updated>
            <content:encoded><![CDATA[<p>Je ne suis pas vraiment fan de Airbnb mais j’avoue que j’aime bien le concept et ça me tente d’utiliser ce genre de service surtout si le rapport qualité prix est très compétitif. Et en parlant de Sharing Economy, vu que j’ai des données disponible sur le sujet, j’ai eu la curiosité d’analyser les listings Airbnb en Tunisie, et voilà ce que ça a donné :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*J1ij3Ee4ooVp9Bho.png" /></figure><p>En Tunisie y’a plus que 1500 listings inscrits, environ 5000 lits et une capacité globale de plus que 7500 personnes, avec une moyenne de 74 EUR la nuitée.</p><p>En regardant de plus près, 28% des listings sont nouveaux, soit une augmentation de la capacité d’accueil de 895 personnes et 624 lits.</p><p>Est ce que ces résidences permettent une réservation directe depuis le site ? 34% seulement le permettent, le reste ont d’autres source de clientèles, ou bien le responsable qui n’est pas disponible à plein temps ce qui nécessite une confirmation dans les deux cas. Par contre il est intéressant de remarquer que le site en premier lieu essaye de donner une distribution 50/50 ce qui peut favoriser les listings qui permettent la réservation directe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*DWESw00_zssfGoMW.png" /></figure><p>Si on veut voir le type de résidence qu’on peut trouver sur Airbnb, on trouve que la majorité c’est des appartements 51.2%, et y’a quand même 6% qui offrent Bed &amp; Breakfast et donc un service petit dej complémentaire. Donc grosso modo on a 83.5% des villas ou appartements indépendants, 15.5% des chambres privées, et seulement 1% des chambres partagées.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*9WrFP9ZSHOFo77e1.png" /></figure><p>Vous allez vous poser la question sur la qualité des services ? Alors, 35% ont reçus au moins un review, mais la totalité est très faible soit environ 2927 reviews. En ce qui concerne les étoiles, le minimum était de 2.5 étoiles et le maximum 5, avec une moyenne de 4.52 et seulement 1085 personnes qui ont votés pour ces résidences.</p><p>Pour les prix ça commence à partir de 10EUR la nuitée, avec une moyenne de 74.34EUR. Sachant que 16% seulement des prix sont supérieur à 100 EUR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*_8VQ6HlY2gWMHzwD.png" /></figure><p>Donc si on calcule avec le prix moyen de la nuitée et les 3000 reviews qui ont été enregistrées, on parle d’un montant d’environ 200 milles EUR seulement, soit environ 550 milles dinars. Mais je vous cache pas le chiffre réel est beaucoup plus bas et ne dépasse pas 125 milles EUR pour toutes les résidences.</p><p>Pour les villes où ces business sont entrain de se développer j’ai pas encore une analyse très détaillées, mais à priori par ordre on a Tunis, Sousse, Nabeul, Djerba… Par contre ce que vous ne pouvez pas voir clairement sur le site officiel de Airbnb, c’est qu’il y’a aussi des listings à l’intérieur du pays notamment dans Bizerte, Tabarka, Jendouba, Kairouan, Gafsa, Sfax, Gabes, Tataouine…etc.</p><p>C’est clair que Airbnb en Tunisie n’est pas aussi populaire que dans d’autres pays, et surtout que la législation Tunisienne n’a encore rien prévu en ce qui concerne l’économie collaborative “Sharing economy”. Mais c’est toujours intéressant de voir comment ce secteur est entrain de se développer en silence.</p><p>Source : <a href="http://www.hbyconsultancy.com/blog/sharing-economy-en-tunisie-analyse-des-donnees-airbnb.html">http://www.hbyconsultancy.com/blog/sharing-economy-en-tunisie-analyse-des-donnees-airbnb.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1774993bf888" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[#ODBarometer, how to turn 66% of Government data into OpenData ?]]></title>
            <link>https://medium.com/@hatem/odbarometer-how-to-turn-66-of-government-data-into-opendata-d370a53493b8?source=rss-caab74aa7ddd------2</link>
            <guid isPermaLink="false">https://medium.com/p/d370a53493b8</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[government]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Hatem Ben Yacoub]]></dc:creator>
            <pubDate>Sun, 24 Apr 2016 14:21:45 GMT</pubDate>
            <atom:updated>2016-04-25T05:50:24.844Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/855/0*9HAX9XdcFo-z4aTL.png" /></figure><p>First alarming result from the third edition of the Open Data Barometer is that only 10% of government data is available as Open Data, while 76.3% of these data is available online ! So, what’s missing to turn at least 60% of government data into Open Data ? Let’s have a look on the non-OpenData :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*NSLEMiOBwA4-pW_9QZspOA.png" /></figure><p>The graphics above show 66.1% of analyzed datasets are available but not open. You can imagine how the world will look like if at least these datasets were open ?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Y--Hzc0hA13VcWyQwYybUw.png" /></figure><p>Yes, almost like that ! But most of these data today are somehow locked, whether it require registration, or locked in unusable closed formats, etc…</p><p><strong>Analyzing the “closed data” :</strong></p><p>We’ll try to have a look deep inside all these datasets that are available online but not available as Open Data, and first let’s see where these datasets are located worldwide :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*m5w-gVeh-kYk9ithV04-FQ.png" /></figure><p>You may notice that this is happening all over the world and the non open datasets goes from 15 datasets in Japan which is ranked 13th in the ODB, to 2 datasets in UK which is ranked number one in this latest edition of the barometer. Below a table of the top 22 countries (Notice the column counter is for isOpen=0):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*c3LcRSbc1Nw7YlYW96orPA.png" /></figure><p>Even the number one is not having all its datasets published as Open Data, so to be realistic let’s lower the target of 76.2% to 62.9% which remain a great target to reach if we can publish all (13 of 15) the available datasets as Open Data.</p><p>So if we want to summarize here, we are talking about :</p><ul><li>1381 datasets worldwide</li><li>328 datasets not available</li><li>913 datasets available but not OpenData.</li></ul><p>Of course by “datasets” we mean all datasets related to one of the data categories of the ODB.</p><p><strong>The Machine readable datasets</strong></p><p>We are talking about datasets that are available online, but not open data. So, the first question, are these datasets available in machine readable format ? the map below summarize the answer to this question</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*dDL7oISb9d8yojL_TgEyFg.png" /></figure><p>The table below summarize the top 22 countries having datasets in machine readable formats but not open data :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*bdExu0WsvaYBxxA-6RcGbg.png" /></figure><p>You may notice that this list is a little different from the first one, and many datasets that are available online are available in a machine readable format. This remain not open data, but we are not so far. But how many online datasets are machine readable format ?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*V51_VgZo_vIfeWBDhAal-Q.png" /></figure><p>Only 45% of datasets available online are available in machine readable format, and this is the first criteria that we should consider to turn government data into Open Data.</p><p>We can also analyze if these datasets are available for free or not. We can see here that only 41.1% of the datasets available online are free. Which mean that most of the data is not available for free !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*JN-mCfLIqGim3mVXhc1aBw.png" /></figure><p>In the same way we can analyze the Open Data licenses availability for these datasets, and here you will be surprised that only 9.1% of the datasets published online include an open license which is too low !! The entire world seems not giving to legal issues related to the data published any importance, or limiting the usage of these data which does not make them really free for use and reuse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*21-SDicV9BbhJBR7NnQV5Q.png" /></figure><p><strong>Conclusion</strong></p><p>In this article I focused only on few criteria that, if respected, could lead to a higher amount of published open data worldwide up to 76% instead of the actual 10%.</p><p>Overall, we may find the same conclusion and recommendation in the ODB report, in addition to that the world should focus on making data machine readable, with open license, and free of charges to help the adoption of Open Data worldwide. The only remaining issue in the sustainability of the publication process which makes the data usually not updated, not consistent, and with very low quality.</p><p>Finally, we are still so far away from linked data, and probably if we focus more on linked data the process could be easier and the adoption will be faster, most important is that we’ll be promoting five stars grade Open Data, rather than promoting low level open data.</p><p>Tagged: <a href="http://www.hbyconsultancy.com/blog/tag/odbarometer">ODBarometer</a>, <a href="http://www.hbyconsultancy.com/blog/tag/open-data-barometer">Open Data Barometer</a>, <a href="http://www.hbyconsultancy.com/blog/tag/opendata">OpenData</a>, <a href="http://www.hbyconsultancy.com/blog/tag/web-foundation">Web Foundation</a></p><p><em>Originally published at </em><a href="http://www.hbyconsultancy.com/blog/odbarometer-how-to-turn-66-of-government-data-into-opendata.html"><em>www.hbyconsultancy.com</em></a><em> on April 24, 2016.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d370a53493b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Opening up company registers, then what ? #ODBarometer]]></title>
            <link>https://medium.com/@hatem/opening-up-company-registers-then-what-odbarometer-e98f5c83b11b?source=rss-caab74aa7ddd------2</link>
            <guid isPermaLink="false">https://medium.com/p/e98f5c83b11b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[panama-papers]]></category>
            <category><![CDATA[open-data]]></category>
            <dc:creator><![CDATA[Hatem Ben Yacoub]]></dc:creator>
            <pubDate>Tue, 12 Apr 2016 09:20:38 GMT</pubDate>
            <atom:updated>2016-04-12T09:45:04.148Z</atom:updated>
            <content:encoded><![CDATA[<p>The Web Foundation published yesterday a<a href="http://webfoundation.org/2016/04/odb-company-dataset/"> sneak preview on the 3rd edition of the Open Data Barometer</a> mainly related to the latest #PanamaPapers leak and the status of Open Data company register worldwide. The full barometer however will be fully published by April 21st.</p><p>According to the data published, we can see that Australia is ranked the first, followed by UK and Norway, but only Australia in the whole index is publishing company register in an open format ! And you can see below how the world looks almost all in red, except very small number of countries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/0*t8jPW3yjNz9F8S8Z." /></figure><p>There are always exceptions, and these exceptions are sometimes very interesting to study. For example in Africa if we look inside we will notice a small green point : Benin !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/0*1U_K8kIO1h0me3F7." /></figure><p>The datasets available in<a href="http://www.ccibenin.org/index.php/repertoire-entreprises.html"> Benin are from 2013</a>, in PDF format, not updated, but most notable thing is that these datasets are missing a lots of data, only few shareholders are mentioned, some companies do not even have a name. That’s the green point in Africa, so you can imagine how things are going in other countries. There are obviously some efforts to make these data available, and I can mention for example<a href="http://catalog.industrie.gov.tn/dataset/tn-persones-morales"> Tunisia that published last week the company register dataset in open data</a>, even if from first glance the datasets looks incomplete, but responsible confirmed that it will be updated regularly and it will be completed in the next few days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/0*MU1q9Ei8JBdJ4e29." /></figure><p>In Europe also the situation is not really green, however, there are a good number of countries that have a score of fifty and higher which mean that data is available in a machine readable format, however it remain locked for some reasons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/0*5hffFLIaRU6qO3sS." /></figure><p>In Asia you will be surprised as you can see countries such Kazakhstan or Indonesia in green with a score of 55 respectively, but all other countries seems to agree for keeping the same score of fifteen everywhere !</p><h4>What’s next ?</h4><p>The study found that only 1% of countries worlwide publish open company ownership data, that’s too low right ? But guess what will happen if 99% of countries will publish their company register as Open Data ?…</p><p>Nothing of course ! Because even if the company register is published, only top level information are included which will generally lead to nothing interesting.</p><p>For example shareholding information are not generally published, so we will get just a list of companies with creation date, location, etc. Even if we will be able to get shareholding details, owning a company in a tax haven is not necessarily illegal. It could, in certain circumstances, lead to some interesting discoveries, the case of #PanamaPapers, such as someone who passed away since 2005 and still creating companies and attending meetings !</p><p>The first barrier for making any discoveries with such information is privacy, since we are talking about personal information. Some countries will not make such data available, or possibly replace names with numbers.</p><p>Some persons have their names magically misspelled or changed, so even if you want to cross-check data between all these databases you won’t be able to find anything interesting. Unless if you are excepting from a thieves to let you track them easily. You should be dreaming !</p><h4><strong>So why Opening up company registers ?</strong></h4><p>In my humble opinion, opening up company register worldwide will not have any impact on transparency or anti-corruption around the world. However, it could have great impact on economy and that’s the most interesting part that countries should focus on it.</p><p>Once the data is there, the possibilities are unlimited, and that’s the right time to start talking about data-driven innovation. In that time OpenData will have a huge impact on the world economy, but only if the datasets published have very good quality.</p><p>So next thing that should be added to Open Data Barometer for example, is a measure of dataset quality. Which is not very hard to measure once the data is available as OpenData, but a research should be conducted in this purpose to define what to measure exactly and how to measure it.</p><p>Graphics used in this blog could be found<a href="https://docs.google.com/spreadsheets/d/1uGd3Z8bopLL7Vhf1sMAgwViKW_ErMTwDmre9w6SKBIA/edit?usp=sharing"> here</a>, and original document could be downloaded from the Web Foundation website<a href="http://webfoundation.org/wp-content/uploads/2016/04/Advance-of-the-OD-Barometer-3rd-edition-Companies-Registers.csv"> here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e98f5c83b11b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>