<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Louis Amon on Medium]]></title>
        <description><![CDATA[Stories by Louis Amon on Medium]]></description>
        <link>https://medium.com/@louis_amon?source=rss-b9599af8e715------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*TOj6lStc0I5SlDUr.jpg</url>
            <title>Stories by Louis Amon on Medium</title>
            <link>https://medium.com/@louis_amon?source=rss-b9599af8e715------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 14:09:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@louis_amon/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to build a machine-learning-powered record linkage workflow]]></title>
            <link>https://medium.com/@louis_amon/how-to-build-a-machine-learning-powered-record-linkage-workflow-b1890a0eb4ae?source=rss-b9599af8e715------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1890a0eb4ae</guid>
            <category><![CDATA[alignment]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Louis Amon]]></dc:creator>
            <pubDate>Mon, 07 May 2018 19:59:15 GMT</pubDate>
            <atom:updated>2018-05-08T09:54:46.703Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://en.wikipedia.org/wiki/Record_linkage">Record linkage</a> is one of those old problems resurfacing a lot these days due to the pervasiveness of data on the internet. It consists in linking several records to a same real-world object. Let’s consider wine bottles as an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*bI3AjI-GrTXEOgASH1Dctw.png" /></figure><blockquote><em>“Chateau Mouton Rothschild 1966”</em> and <em>“Mouton 1966”</em> refer to the same real-world item. You know that. A computer doesn’t.</blockquote><p>Linking several datasets can be a tricky problem because there often isn’t one static rule for translating one into the other.</p><p>Marketplaces &amp; other data aggregation platforms are struggling with this everyday, resulting in massive operational costs for database sanitization &amp; deduplication.</p><h4>Supervised classification</h4><p>Machine learning comes to the rescue with a class of <a href="https://en.wikipedia.org/wiki/Supervised_learning">supervised algorithms</a> called <a href="https://en.wikipedia.org/wiki/Multiclass_classification"><em>multiclass</em></a> <a href="https://en.wikipedia.org/wiki/Statistical_classification">classification algorithms</a>, which -in our previous example- would consists in computationally predicting the likeliness of any given term to be referring to a known wine. Awesome!</p><p>Assuming you have a “golden standard” dataset (a catalog you would consider to be the reference, usually your own), then what you need is a <strong>growing</strong> learning dataset of labeled data. This process is called <a href="https://en.wikipedia.org/wiki/Incremental_learning">Incremental Learning</a>.</p><blockquote>Using this technique, a small quantity of verified data can be used to automatically predict the rest of the unaligned data.</blockquote><p>Not only can you initially predict record linkages with the verified (labeled, in machine learning terms) data that you have at hand, but every time you correct a wrong prediction, you increase the accuracy of your model.</p><p>A smart way of starting is to use your own dataset itself to feed your model, as there’s a very good chance that you consider it clean enough and that it may already be similar-enough to the datasets you want to align on it.</p><blockquote>At some point, all you will need to do is a few checks here and there as most of the records will be correctly aligned by your model, hence drastically reducing your operational costs.</blockquote><p>The process consisting of a human (called “oracle” in machine learning terms) verifying data is called data labeling. For instance, someone will have to manually input that “Mouton 1966 (750ml)” is indeed a bottle of Chateau Mouton Rothschild 1966, at least for the first few records until the machine learning is confident enough to take over.</p><h4>Data labeling</h4><p>Data labeling is actually very common out there. As a matter of fact, you yourself have probably been doing all sorts of data labeling for Google in the past few years: <a href="https://techcrunch.com/2012/03/29/google-now-using-recaptcha-to-decode-street-view-addresses/">Google is using human information from solving Captcha &amp; reCaptcha</a> to feed their machine learning models &amp; improve their (proprietary) Google Books &amp; Google Maps databases.</p><p>Yes. Every time you solve a Captcha to access one of your favorite websites, you actually end up in a data labeling workflow very similar to the schema below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*ke5avOO3-UznO622EDnXfw.png" /></figure><p>Labeling can be a tedious task but it’s crucial for machine learning, so there already exists a bunch of outsourcing solutions, one of the most famous ones being <a href="https://www.mturk.com/mturk/welcome">Amazon Mechanical Turk</a>. Depending on your datasets and industry, it may however be best to use your own local resources if they’re trained for it — at least at the beginning.</p><p>Rather than the resources that you’ll use, what will be most impactful with scalability is building a proper data labeling interface, in which your staff can check the machine’s results and correct them easily. A well-built interface can be its own documentation &amp; training tool.</p><p>Let us focus on how we integrate our “oracles” into our workflow. First let’s lay the groundwork for a basic data labeling system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*Dnnaea3QrMQKwLAcjAxb6A.jpeg" /><figcaption>Data labeling workflow</figcaption></figure><p>You should note in the above picture that machine learning only produces semi-labeled data, in the sense that predictions are based on probabilities and cannot be blindly trusted. Human supervision is still necessary (to some extent) to help guide the model in it’s learning, until it eventually reaches 99% precision.</p><p>Another important thing to note is the right branch of the graph: always ensure that if a data is labeled once then it shouldn’t appear again in the labeling process. Exact matching should suffice, ensuring that the amount of human work always decreases linearly at least and exponentially at best, depending on the machine learning algorithm’s performance over time.</p><p>Now that we’ve outlined the basics of the process, let’s delve deeper into implementation itself.</p><p>These are the core technical items that you need to build in order to achieve a record linkage workflow:</p><blockquote>1) Machine learning framework</blockquote><blockquote>2) Server infrastructure dimensioned for machine learning</blockquote><blockquote>3) Some kind of model persistence infrastructure</blockquote><blockquote>4) Data labeling interface for oracles</blockquote><blockquote>5) A database for storing record linkages</blockquote><p>Let’s look into these in more details.</p><h4>Machine learning framework &amp; server infrastructure</h4><p>When it comes to dimensioning for machine learning, you’ll often read about cloud-based systems with incredible servers wielding impossible amounts of CPU and RAM.</p><p>It’s partly true that machine learning usually scales very vertically, meaning that you’ll often need few very powerful machines rather than a battlement of micro servers (which are more compatible with web processes for instance).</p><p>I would however like to note that, at the core, machine learning problems often boil down to solving a linear mathematical problem which should be highly distributable. Machine learning frameworks are still very young and there’s a lot of room for improvement — maybe one day all ML algorithms will be able to run on a server-distributed / GPU-distributed grid systems, who knows…</p><p>In the end, it all comes down to the algorithm your choose in order to build your model, and more specifically:</p><ul><li>Can the learning / prediction be distributed ?</li><li>Can the algorithm perform incremental learning (otherwise known as <a href="https://en.wikipedia.org/wiki/Online_machine_learning">Online Machine Learning</a>) or does it have to re-compute the model on the whole dataset every time ?</li><li>How much data do you need to process through it right now ? In the foreseeable future ?</li><li>What is the memory footprint of your algorithm ?</li><li>Is the precision of the algorithm satisfactory ?</li></ul><p>There exists a lot of machine learning softwares available on the shelf today. They come in all sorts of fashions and different languages, which your technical team may or may not master. My opinion on the matter is that you should try to find a framework that fits in the language requirements of your team(s), and if it also matches the language of the data labeling interface then you’ll save yourself a lot of pain as well. Performance comes second, as it cannot always be properly estimated from the get go, and there’s often a ton of tweaks that can drastically increase precision later on.</p><p>I’ve had decent success with Python’s <a href="http://scikit-learn.org/">scikit-learn</a>, although it takes a bit of research to find which algorithm to use, as per the above criteria and the graph below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LX2N0X8qeV4shqKx7rJoNQ.png" /></figure><p>Since we already know we’re looking at a multiclass classification problem, there’s actually very few algorithms that might fit so it’s not so much of a needle in a haystack. Still, if you find it difficult to determine the right algorithm, you should know that there also exists <a href="https://automl.info/">automated machine learning tools</a> that will find the right one for you based on your data.</p><p>Regarding the server itself, it doesn’t really matter if you use regular hosting or cloud-based solutions like Amazon AWS, Microsoft Azure or Google Cloud. All I can say is that as of today, none of the big cloud providers have a decent enough fully-integrated solution for this specific problem. The solution that came closest was <a href="https://aws.amazon.com/sagemaker/">AWS Sagemaker</a> that was <a href="https://techcrunch.com/2017/11/29/aws-releases-sagemaker-to-make-it-easier-to-build-and-deploy-machine-learning-models/">disclosed in AWS re:invent 2017</a> — which was too late at the time I built this project. At the end of the day, the best solution for me was to use a regular server and install everything myself. It’s a complex system so I needed to know what was going on, but if I had to do it again I might have done things differently.</p><blockquote>In my opinion, <strong>determining the algorithm and dimensioning the server that will host it are tasks that should be done together</strong>. Take it from me: failing to do so will result in a lot of unnecessary iterations in your project.</blockquote><p>Finally, I’d like to suggest to the most ambitious among you the possibility of <a href="https://en.wikipedia.org/wiki/Serverless_computing">serverless computing</a>: it is for example totally feasible to fit a ML framework in an <a href="https://aws.amazon.com/lambda">AWS Lambda</a> — with a few tweaks here and there. The benefit is mainly cost control &amp; scalability, but there’s also an architectural benefit in achieving a proper <a href="https://en.wikipedia.org/wiki/Service-oriented_architecture">service-oriented architecture</a>: making your own record-linkage-as-a-service API may be better in terms of process / ressources management.</p><h4>Model persistence</h4><p>When your model learns from your data, it has to store its “brain” somewhere. Well, long story short, here are the more common options:</p><ul><li>A <a href="https://en.wikipedia.org/wiki/Data_lake">data lake</a> (e.g. <a href="https://aws.amazon.com/s3/">AWS S3</a>)</li><li>A memory-based storage (e.g. <a href="https://redis.io/">Redis</a>)</li><li>An SSD-based storage (e.g. <a href="https://aws.amazon.com/efs/">AWS EFS</a>)</li></ul><p>Again, it’s an equation with multiple variables and unknowns: the server type chosen above, vertical and horizontal scaling requirements, cost control, throughput, DevOps proficiencies etc.</p><p>What I’ve had the best success with so far is AWS S3 (the data lake solution), but I’d like to note that the best speed can be achieved with memory-based storage.</p><p>NB: the data lake solution is also what AWS uses for its <a href="https://en.wikipedia.org/wiki/Platform_as_a_service">PaaS</a> solution for machine learning: AWS Sagemaker (cf. <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/API_S3DataSource.html">official documentation</a>), so it seems I came to the same conclusions at the AWS engineers who built it.</p><h4>Data labeling interface</h4><p>This is where you will actually spend 80% of your time in building this project: reducing the amount of human intervention is critical in quickly building a system that produces quality predictions.</p><p>Here are a few pointers which will hopefully save you some time:</p><ul><li>Always go for a web-based interface (extranet): it will allow you to delegate &amp; control permissions easily, especially if you’re outsourcing</li><li>Try to think about propagation: you may not need to label all the variants of a given item, maybe there’s a way to regroup items and propagate alignments on them</li><li>Dynamic interfaces &amp; autocompletions are huge time-savers, especially when your database starts to scale and your interface gets slightly less reactive</li><li>Start small, think scalable</li></ul><p>The solution I decided to go for is Python’s <a href="https://www.djangoproject.com/">Django</a> web framework, mainly because of its famous <a href="https://docs.djangoproject.com/en/2.0/ref/contrib/admin/">admin interface</a> (which now features autocompletion widgets) but also because it was highly compatible with the scikit-learn machine learning framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*8kezV3XMK9rceINmWYtjPQ.png" /><figcaption>Example of a data labeling interface achieved with Django</figcaption></figure><h4>Record linkage database</h4><p>This is where you have the most freedom. Any key-value store will do, relational or otherwise.</p><p>I went for <a href="https://www.postgresql.org/">PostgreSQL</a> because of:</p><ul><li>My experience with the technology</li><li>Its native integration with the data labeling interface above</li><li>Its native &amp; very performant CSV import/export feature</li></ul><p><a href="https://aws.amazon.com/dynamodb/">AWS DynamoDB</a> was a noteworthy contender at the time, and so it Microsoft Azure’s <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/introduction">CosmosDB</a> — which actually features a MongoDB interface, nifty!</p><p>From a system architect’s perspective, building a record linkage process is quite challenging because it involves and brings together a lot of different ecosystems:</p><ul><li>Machine learning / Big Data</li><li>Web / UX</li><li>Operations</li></ul><p>The endgame is worth the effort though, as there’s nothing more rewarding than seeing the smile on your team’s faces when the release goes online and they finally save hours of repetitive work thanks to this new system :)</p><h4>References</h4><ul><li><a href="https://en.wikipedia.org/wiki/Record_linkage">https://en.wikipedia.org/wiki/Record_linkage</a></li><li><a href="https://en.wikipedia.org/wiki/Bag-of-words_model">https://en.wikipedia.org/wiki/Bag-of-words_model</a></li><li><a href="https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a">https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1890a0eb4ae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Cloud Datalab vs Anaconda Cloud vs Domino Datalab]]></title>
            <link>https://medium.com/@louis_amon/google-cloud-datalab-vs-anaconda-cloud-vs-domino-datalab-4129114148e0?source=rss-b9599af8e715------2</link>
            <guid isPermaLink="false">https://medium.com/p/4129114148e0</guid>
            <category><![CDATA[anaconda]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-cloud-datalab]]></category>
            <dc:creator><![CDATA[Louis Amon]]></dc:creator>
            <pubDate>Wed, 03 Aug 2016 13:40:26 GMT</pubDate>
            <atom:updated>2018-05-08T09:55:42.628Z</atom:updated>
            <content:encoded><![CDATA[<h4>An overview of hosted notebook solutions</h4><p>When it comes to scientific experimentation and collaboration, people tend to need the same thing : an easy-to-use interface to hack and optimize their algorithms, a data i/o, support for their preferred language (R, Julia, Python…)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*33_uvt6YYaIDtIDBGeOLSQ.png" /><figcaption>Jupyter — IPython Notebook</figcaption></figure><p>A natural solution to these problems emerged in 2011 with the release of <a href="http://jupyter.org/">Jupyter</a>, a web application that allows you to create a “notebook file” that serves as:</p><ul><li>an interactive code interface for over 40 languages</li><li>a data visualization tool, that comes bundled with <a href="http://matplotlib.org/">matplotlib</a> and integrates with other graphing tools such as <a href="https://plot.ly/">plotly</a> and <a href="https://developers.google.com/chart/">Google Charts</a></li><li>a markdown editor to document algorithms, enabled for math formulas through Mathjax &amp; LateX</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*islZgwgB0l8sblclC32MxQ.png" /><figcaption>Math formulas in Jupyter</figcaption></figure><p>A few years back, people would work locally and email their collaborators with new versions of their algorithm once they’re satisfied of their improvements. A few (gifted) scientists would actually use version control softwares like Git or Mercurial…</p><p>Nowadays though, the scientific community is working more and more tightly with the computer science community. Version control is more of a norm now (thank God!) and people are looking towards cloud solutions for easier and more efficient collaboration.</p><h3>What are your options when it comes to Jupyter notebook collaboration ?</h3><p>Let’s start with the poor man’s solution : Google Drive / Dropbox. Hosting a notebook file on a cloud storage is virtually free these days, but it can prove quite challenging in terms of collaboration.</p><p>Git comes to the rescue with the infamous <a href="https://github.com/">Github.com</a> platform, which now natively supports .ipynb format (Jupyter Notebook) for visualization. It’s a cheap, great solution that introduces a much-needed layer of version control. Collaboration still isn’t as easy as it could be because each collaborator needs to have their own local Jupyter server to edit the notebook (e.g. using <a href="https://docs.continuum.io/anaconda/">Anaconda Navigator</a>).</p><p>A similar solution would be to use <a href="https://anaconda.org/">Anaconda Cloud</a>, which also allows notebook-sharing and representation but no online editor and cloud execution so far.</p><h3>What about Jupyter cloud solutions ?</h3><p>Aside from self-hosting a Jupyter dedicated server, there are (as far as I know) two main Jupyter cloud platforms out there.</p><h4><a href="https://www.dominodatalab.com/">Domino Data Lab</a></h4><p>A San-Francisco-based startup company providing a cloud infrastructure based on Amazon AWS.</p><p>Based on my experimentations, Domino’s solution lacks the following features.</p><ul><li>No on the fly editing</li></ul><p>Using DDL’s cloud solution works as a succession of “runs” that execute your whole notebook in one go. There is indeed a web-based IDE that allows you to edit your notebook, but you have to save it first then run the whole thing, which is much, much slower that hacking away at an algorithm locally using Jupyter.</p><p>DDL doesn’t seem to be intended towards algorithm development but more towards hosting and distributed execution.</p><ul><li>No upfront pricing</li></ul><p>You have to contact them directly to get a quote based on your projected needs.</p><ul><li>No version control</li></ul><p>Domino Datalab Doesn’t connect with any kind of version control as far as I know (except Github for package distribution). They do provide an “auto-save” feature for notebooks.</p><p>Concretely, their “run” system described above saves the output of each run (including produced/modified files). It still lacks the strength of an actual version control, including -but not limited to- branching.</p><h4><a href="https://www.dominodatalab.com/">Google Cloud Datalab</a> (beta)</h4><p>As of late 2015, Google also provides another Jupyter cloud solution : Google Cloud Datalab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MkWoqsZd1hYL_ag2BGNDQg.png" /><figcaption><a href="https://datalab.cloud.google.com/">https://datalab.cloud.google.com</a></figcaption></figure><p>I’ll go over how GCD actually works and some of the caveats I’ve encountered so far :</p><p>GCD isn’t actually a platform in itself, it’s a docker image built and maintained by the Google team <a href="https://github.com/googledatalab/datalab">on Github</a>.</p><blockquote>Note: Although Jupyter-based, GCD only allows Python language <em>so far</em>. It’s a beta though so I’m guessing other languages should be supported soon.</blockquote><p>The <a href="https://datalab.cloud.google.com/">installation</a> proposed so far is kind of minimalistic : it’s a basic OAuth2 procedure that requires your permission to deploy two servers. One Google Compute Engine instance (which serves only during installation and updates, so remember to shut it down afterwards) and one Google App Engine instance which is the actual Jupyter webserver you’ll be using.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CSMCuqWKzeO0BL-UmhYcoA.png" /><figcaption>GCD installation process</figcaption></figure><blockquote>Note: GCD requires a special type of Google App Engine instance called “flexible environment” (formerly “managed VM”). These are in beta and currently only available in the US regions.</blockquote><p>You’ll be able to see and manage both instances in your Google Cloud Console, including scalability.</p><p>Under the scene, GCD deployment also creates a Google Cloud Repository (beta) branch called “datalab” and that will serve as the underlying version control.</p><blockquote>Note: Google Cloud Repository is also a beta service so far. Although git-based as well, it is much inferior to Github for now. It does allow Github/Bitbucket mirroring which makes it more transparent but is nonetheless required by Google Cloud Datalab. That’s a pretty clear technical lock-in right there and a no-go for many CTOs. Do note also that git GUIs like SourceTree may not natively function with Google Cloud Repository due to Google’s OAuth-based authentication.</blockquote><p>One major strength that GCD does have though is on-the-fly online editing (finally!). You can indeed hack away at your algorithm online, extract data and represent it in graphs (including Google Charts, Plot.ly charts, matplotlib charts…). Code execution is handled by the underlying Google App Engine instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xQGkBy8fTt8Av8vISZilaA.png" /><figcaption>Combining Pandas dataframe with Plot.ly graphing in Google Cloud Datalab</figcaption></figure><p>Practically speaking, version control works but it’s kind of a mess to manage : although there’s an auto-save feature on the notebook itself, you still have to manually commit your online changes through the online interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*NA3OSNLPrw5iQmDLaWMuWg.png" /><figcaption>matplotlib works natively in Jupyter (and in GCD as well)</figcaption></figure><p>GCD’s greatest strength in my opinion is its native connection with <a href="https://cloud.google.com/bigquery/">Google BigQuery</a>. Where you used to work with CSV/JSON or a small MySQL database, you can now work with a big data infrastructure with very little additional cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OiGIbpl9PHRH2dLCo7UNIA.png" /><figcaption>Google Cloud Datalab + Google Big Query + Google Charts</figcaption></figure><p>Collaboration is pretty awful as well so far : you need to give “Project Owner” permission to your collaborators in order for them to simultaneously access Google Cloud Datalab’s GAE instance, Big Query and Google Cloud Repository (git)… that’s pretty lax. Given all that though, they should be able to hack away at your algorithm and commit their improvements to the underlying repository so that’s basically what you should be looking for.</p><h3>Conclusion</h3><p>Google Cloud Datalab is still far from being production-ready, but it’s a very promising solution. The only solution so far that comes close to the ease-of-use of a local Jupyter server with the added benefits of cloud hosting (CVS, scaling, sharing, etc.).</p><p>The proof of concept is indeed there, and it merely lacks some polish around the angles (more language support, easier notebook collaboration, better version control…)</p><p>With the recent release of <a href="https://www.google.fr/analytics/data-studio/">Google Data Studio</a> (beta), it seems Google aims to provide a complete data toolkit. Let us see if they indeed manage to create the final products they’ve set their sights on.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4129114148e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>