<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hugo Perrier on Medium]]></title>
        <description><![CDATA[Stories by Hugo Perrier on Medium]]></description>
        <link>https://medium.com/@hugoperrier?source=rss-ed54763e56b3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*zxRuCBnTCquVlmKv.</url>
            <title>Stories by Hugo Perrier on Medium</title>
            <link>https://medium.com/@hugoperrier?source=rss-ed54763e56b3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 05:08:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hugoperrier/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Python Package Management: A Guide to Avoid Dependency Conflicts]]></title>
            <link>https://medium.com/maif-data-design-tech-etc/python-package-management-a-guide-to-avoid-dependency-conflicts-0f0fe292f766?source=rss-ed54763e56b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f0fe292f766</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Hugo Perrier]]></dc:creator>
            <pubDate>Tue, 19 Mar 2024 12:47:07 GMT</pubDate>
            <atom:updated>2024-09-09T18:36:49.818Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Oops, the latest pandas version requires ‘numpy&gt;1.22’ but tensorflow requires ‘numpy~1.9’.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3tIEoFUoZvuggv9w8644fA.jpeg" /><figcaption>A pandas, a tensorflow and a numpy fighting each other with a sword in the style of a comic book.</figcaption></figure><h3>Table of contents</h3><ul><li>Motivations</li><li>What are conflicting dependencies?</li><li>Understand the issue with a code example</li><li>Design Patterns: Decoupling for Flexibility</li><li>Optional Dependencies: Tailored for Users</li><li>tox: Testing through the Maze of Dependencies</li><li>Conditional Test Execution: Making Tests Smarter</li><li>Wrap-up</li><li>Conclusions</li></ul><h3><strong>Motivations</strong></h3><p>As an open-source Python package maintainer and a Data Scientist, I’ve had the chance of witnessing the evolution of <a href="https://github.com/MAIF/melusine">Melusine</a>, a machine learning-powered email processing tool developed by MAIF.</p><p>Released in 2019, Melusine quickly became an integral part of our daily operations, accelerating our email processing workflows. But, as the package matured and new features were added, managing dependencies has become a real challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*6C6qyk0zt11HapzCVGQs-Q.png" /><figcaption>Melusine logo — OSS by MAIF</figcaption></figure><p>The ever-changing Python ecosystem demanded continuous updates and maintenance of Melusine’s dependencies. This manual process is time-consuming and error-prone. Outdated dependencies could introduce security vulnerabilities, compatibility issues, and disruptions in our internal systems.</p><p>Rather than attempting to patch up the existing Melusine codebase, we opted for a complete rewrite. This approach allowed us to address the dependency conflict challenges head-on, adopting a more modern and streamlined strategy.</p><h3>What are conflicting d<strong>ependencies?</strong></h3><p>As a Python developer, you’ve probably experienced dependency hell at some point. It’s the frustration of trying to keep your packages up to date without breaking your code. One package requires a newer version of a dependency, but another package requires an older version. It’s a mess.</p><p>In this article, I’ll show you how we used <a href="https://refactoring.guru/design-patterns/python">design patterns</a>, optional dependencies, and the package <a href="https://tox.wiki/">tox</a> to keep the Melusine package clean and up to date.</p><h3>Understand the i<strong>ssue with a code example</strong></h3><p>To illustrate the journey of rewritting Melusine, I’ll use a simple pseudo code example: a class to make a machine learning prediction using different types of models.</p><pre># predictor.py<br>import sklearn<br>import tensorflow as tf<br><br>class Predictor:<br>   def __init__(self, model):<br>     self.model = model<br><br>   def predict(self, data):<br>     # Tensorflow model<br>     if isinstance(self.model, tf.Model):<br>       result = self.model.predict(data)<br>  <br>     # Sklearn model<br>     elif isinstance(self.model, sklearn.TransformerMixIn):<br>       result = self.model.transform(data)<br>  <br>     # Unsupported model<br>     else:<br>       raise TypeError(<br>         f&quot;Object of type {type(self.model)} &quot;<br>         &quot;is not supported by the Predictor class&quot;<br>       )<br>  <br>     return result</pre><pre># run_prediction.py<br>from sklearn.ensemble import RandomForestClassifier<br>from my_package import Predictor<br><br>X = some_data<br>predictor = Predictor(model=RandomForestClassifier())<br>result = predictor.predict(data=some_data)</pre><p>The class uses an ML model to make a prediction, but depending on the type of model, the method to run predictions is different (transform or predict). There are a few weaknesses in this code block that could make it hard to maintain over time:</p><ul><li>The design forces you to modify the code and create a new <em>if</em> condition for each new type of model. This is particularly problematic if you don’t have the rights to modify the source code (using an open source package for example).</li><li>Both sklearn and tensorflow are imported in the module. This means that they both need to be installed, even if one is not used, and could lead to incompatibilities</li></ul><h3>Design Patterns: Decoupling for Flexibility</h3><p>The first step in our journey consisted in refactoring the code, leveraging the power of design patterns. Specifically, we adopted dependency injection, a technique that decouples our code from its dependencies, allowing us to swap out different dependencies without disrupting the overall system.</p><p>Let’s rewrite the code block using dependency injection. We start with defining an abstract class to fix a signature for all predictor objects.</p><pre># base_predictor.py<br>from abc import ABC, abstractmethod<br><br>class BasePredictor(ABC):<br>    @abstractmethod<br>    def predict(self, data):<br>        &quot;&quot;&quot;Execute a machine learning model prediction&quot;&quot;&quot;<br>        raise NotImplementedError()</pre><p>Then we define a class inheriting from BasePredictor for each type of model.</p><pre># sklearn_predictor.py<br>from base_predictor import BasePredictor<br>from sklearn.ensemble import RandomForestClassifier<br><br>class SklearnPredictor(BasePredictor):<br>    def __init__(self):<br>        self.model = RandomForestClassifier()<br><br>    def predict(self, data):<br>        &quot;&quot;&quot;Execute a machine learning model prediction&quot;&quot;&quot;<br>        return self.model.transform(data)</pre><pre># tensorflow_predictor.py<br>from base_predictor import BasePredictor<br>from tensorflow import SomeTensorflowModel<br><br>class TensorflowPredictor(BasePredictor):<br>    def __init__(self):<br>        self.model = SomeTensorflowModel()<br><br>    def predict(self, data):<br>        &quot;&quot;&quot;Execute a machine learning model prediction&quot;&quot;&quot;<br>        return self.model.predict(data)</pre><p>Finally, we instantiate a predictor object (it can be any class inheriting from BasePredictor) and use it to make a prediction.</p><pre># run_prediction.py<br>from my_package.sklearn_predictor import SklearnPredictor<br><br>X = some_data<br>predictor: BasePredictor = SklearnPredictor()<br>result = predictor.predict(data=some_data)</pre><p>This refactored code improves a lot the code maintainability:</p><ul><li>New types of model can be added easily without impacting the existing code. Users can just create a class inheriting from BasePredictor and use it right away.</li><li>Dependencies are independent from each other. The tensorflow package doesn’t have to be installed when running an SklearnPredictor (The SklearnPredictorand TensorflowPredictor are defined in different modules).</li></ul><h3><strong>Optional Dependencies: Tailored for Users</strong></h3><p>Instead of forcing all users to install all dependencies, Melusine provided optional dependency installation options. This allows users to choose the dependencies they needed based on their specific use cases, reducing the overall package size and simplifying the installation process.</p><p>In our example, we just want to install one of tensorflow or sklearn. Optional dependencies can be setup in the pyproject.toml file.</p><pre># pyproject.toml<br>[project]<br>name = &quot;melusine&quot;<br>dependencies = [&quot;pandas==2.0.0&quot;]<br><br>[project.optional-dependencies]<br>sklearn = [&quot;sklearn==1.3.2&quot;]<br>tensorflow = [&quot;tensorflow==3.2.0&quot;]</pre><p>Pandas is set as a mandatory dependency, it will always be installed when running pip install my_package , while sklearn and tensorflow are optional dependencies installed only when running pip install my_package[sklearn] and pip install my_package[tensorflow] respectively.</p><h3>tox: Testing through the Maze of Dependencies</h3><p>With the code refactored and dependency management streamlined, Melusine faced a new challenge — ensuring that the package works seamlessly with different dependency configurations. To address this challenge, the team turned to the tox testing framework.</p><p>Tox is a tool that can help you test your Python packages with different versions of dependencies. This can help you catch dependency conflicts before they cause problems for your users.</p><p>Once you have created and configured a tox.ini file, you can run tox to test your package with all of the specified versions of dependencies. If there are any dependency conflicts, tox will report them.</p><pre># tox.ini<br>[tox]<br>requires = tox&gt;=4<br>env_list = base, sklearn, tensorflow<br><br>[base]<br>description = run unit tests with the base dependencies<br>commands = pytest<br>deps = pytest<br><br>[sklearn]<br>description = run unit tests with the sklearn dependencies<br>commands = pytest<br>deps = pytest<br>extras = sklearn<br><br>[tensorflow]<br>description = run unit tests with the tensorflow dependencies<br>commands = pytest<br>deps = pytest<br>extras = tensorflow</pre><p>This file creates testing environments: base, sklearn and tensorflow.</p><h3>Conditional <strong>Test Execution: Making Tests Smarter</strong></h3><p>The last challenge we needed to tackle was to skip the tests requiring tensorflow when using the base and sklearn environments. This can be done simply with the pytest.importorskip command.</p><pre># test_tensorflow_predictor.py<br>import pytest<br>tensorflow = pytest.importorskip(&quot;tensorflow&quot;)<br>from my_package.tensorflow_predictor import TensorflowPredictor<br><br>def test_tensorflow_predictor():<br>    predictor = TensorflowPredictor()<br>    assert predictor.predict(some_data) == expected_result</pre><p>The pytest.importorskip command checks if the tensorflowpackage is installed and, if not, skips the following tests.</p><h3>Wrap-up</h3><p>The strategy we adopted to avoid dependency conflicts when rewritting melusine was:</p><ul><li>Reformat the code to use dependency injection</li><li>Set up optional dependencies in the package requirements</li><li>Configure tox to use multiple testing environments</li><li>Use pytest.importorskip to make test execution conditional to the installed packages</li></ul><h3>Conclusions</h3><p>Dependency conflicts are a real problem for package maintainers. However, using design patterns, optional dependencies, and tox, can help keep your Python packages clean and up to date.</p><p>The adoption of these enhanced code design principles in Melusine v3 has significantly transformed it’s maintainability. Developers can now focus on their specific areas of expertise, working independently on modules and components without the risk of impacting each other. This specialization has accelerated development and improved code quality.</p><p>Melusine’s journey from a complex codebase to a well-maintained open-source package highlights the importance of effective code design and modularity. By adopting a more structured and component-based approach, we have made it a significantly more robust and reliable for MAIF and the wider open-source community.</p><h3>About the author</h3><p><em>I grew up in the French Alps, studied Physics / Nuclear Engineering in Switzerland, Sweden and England, and I’ve been working as a Data Scientist since 2018 at Quantmetry and MAIF. I am also a big fan of wakeboarding and board games :)</em></p><p>Follow me on <a href="http://www.linkedin.com/in/hugoperrier">LinkedIn</a>.<br>Leave a star for <a href="https://github.com/MAIF/melusine">Melusine on GitHub</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f0fe292f766" width="1" height="1" alt=""><hr><p><a href="https://medium.com/maif-data-design-tech-etc/python-package-management-a-guide-to-avoid-dependency-conflicts-0f0fe292f766">Python Package Management: A Guide to Avoid Dependency Conflicts</a> was originally published in <a href="https://medium.com/maif-data-design-tech-etc">MAIF Data Design Tech etc.</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rewriting a python package: The journey towards Melusine v3.0]]></title>
            <link>https://medium.com/oss-by-maif/rewriting-a-python-package-the-journey-towards-melusine-v3-0-c57e2b6cee45?source=rss-ed54763e56b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c57e2b6cee45</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[technical-debt]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Hugo Perrier]]></dc:creator>
            <pubDate>Tue, 12 Mar 2024 14:25:30 GMT</pubDate>
            <atom:updated>2024-03-12T14:25:30.746Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KDnF7kfEclsORy-VMksCPg.jpeg" /><figcaption>Streamline your email processing workflow with Melusine.</figcaption></figure><p>Melusine is an email processing Python package developed by MAIF. Originally designed for email routing, Melusine has evolved over the years to encompass a wider range of email processing tasks.</p><p>As the Data Factory at MAIF matured, the need for a modern, maintainable, and extensible code base became apparent. This led to a <strong>complete rewrite</strong> of Melusine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*juyQn2rqay32uuWPQB7caw.png" /><figcaption>Evolution of automatic email processing at MAIF over the years (only the core functionalities are open-sourced, not the applications).</figcaption></figure><p>In this article, I’ll describe some of the pain points encountered while maintaining the open-source package and using it in a production environment.</p><h3>What is the problem ?</h3><p>If you have been running a software for several years, these situations may sound familiar to you:</p><blockquote><strong><em>Business analyst:</em></strong><em> “Could we add this simple rule to improve email qualification?”<br></em><strong><em>Data Scientist :</em></strong><em> “Sure, we will have to modify 30 files in five different repositories. It should take about 6 months”</em></blockquote><blockquote><strong><em>MAIF representative:</em></strong><em> “Why is this email marked as urgent?”<br></em><strong><em>Data Scientist:</em></strong><em> “Well it could be the model… or business rules… may be both? We are not sure”</em></blockquote><blockquote><strong><em>New developer: </em></strong><em>“What does this line of code do?”<br></em><strong><em>Confirmed developer:</em></strong><em> “It was there when I arrived and I’m affraid to delete it”</em></blockquote><p>Managing a large code base over time is full of challenges, but in the present article I’d like to focus on modernity, modularity, continuous integration and robustness.</p><h3>Modernity</h3><blockquote><em>“Bag-of-word, Tf-Idf, Word2Vec, GloVe, FastText, RNN, LSTM, Transformers, BERT, GPT, Llama, Mistral, …”</em></blockquote><p>With Melusine, <a href="https://www.deeplearning.ai/resources/natural-language-processing/">Natural Language Processing</a> (NLP) is used to process emails. For example, we can detect if someone is angry based on the content of the email. The challenge is that,</p><blockquote>since 2018, the NLP landscape has changed drasticaly!</blockquote><p>We went from data scientists running their custom preprocessing code and training models from scratch to largely available off-the-shelf large language models.</p><p>The new Melusine is designed to be deep-learning framework agnostic, ensuring compatibility with the latest advancements in NLP. The usage of proven design patterns made sure that users could easily switch from a machine learning package to another.</p><p>Experience it yourself with this <a href="https://maif.github.io/melusine/tutorials/07_BasicClassification/">tutorial</a> on emails Zero Shot Classification using Melusine and the HuggingFace transformers package.</p><p>We also focused on where Melusine shines: dealing with the specificities of email data with features like email conversation segmentation or signature text extraction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*7nX2jdYfdqLCRKFuaz3W9Q.png" /></figure><h3>Modularity</h3><blockquote><em>“I just had a call with the business, this year they would like to add 27 new use-cases of email automation”</em></blockquote><p>As the scope of Melusine’s functionality expanded to encompass various mailboxes and unique processing requirements, the existing code base has grown increasingly complex.</p><blockquote>Being able to deliver new features without breaking existing ones is paramount for Melusine.</blockquote><p>To address this challenge, while rewriting Melusine, we embraced the <a href="https://realpython.com/solid-principles-python/">Single Responsibility Principle</a> (SRP) as a guiding principle. The SRP dictates that each part of the code should have a single responsibility, performing a specific task and minimizing interdependencies.</p><p>In practice, the SRP has been implemented in Melusine v3.0 by organizing the code into a pipeline framework. Each pipeline step handles a distinct task, such as text cleaning or detection. For instance, one step could segment an email conversation, while another applies a machine learning model to identify dissatisfaction in an email.</p><p>This modular approach has yielded several benefits:</p><ol><li><strong>Facilitated Onboarding:</strong> New developers can work on a module directly without the need to dive into the entire code base.</li><li><strong>Secured Deployments: </strong>As coupling is minimized, the risk of breaking an existing feature while working on a new one is limited.</li><li><strong>Enhanced Collaboration: </strong>Code reviews are much simpler on a modular code and developers can spend more time discussing the general design rather than specific code blocks.</li><li><strong>Code Reusability:</strong> At MAIF, we use Melusine to process emails from different mailboxes. Building up on generic code blocks greatly simplifies the maintenance of out applications.</li></ol><p>Check out this <a href="https://maif.github.io/melusine/tutorials/00_GettingStarted/">tutorial</a> explaining how to build an email processing pipeline with Melusine.</p><h3>Continuous Integration</h3><blockquote><em>“We got a really cool external contribution to the open-source package. I’m just worried that it could impact our production code if we integrate it”</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_touDIDWPCfo-iJM0v6zPA.png" /><figcaption>Test automation with GitHub Actions</figcaption></figure><p>Data Science and NLP are highly dynamic fields, with new tools and updates being released on a daily basis. In this context, developers may be tempted to add every emerging functionality to their packages to chase the state of the art. However, this approach can lead to dependency conflicts, API breaking changes, and a constantly outdated codebase. To address these challenges, MAIF focused on three key aspects during the rewrite of Melusine:</p><ol><li><strong>Limited Mandatory Dependencies:</strong> To minimize the impact of dependency changes, Melusine v3.0 separates features requiring external packages from the core modules. These external dependencies are then made <a href="https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies">optional</a>, allowing users to select only what they need.</li><li><strong>Significant Test Coverage:</strong> To ensure the stability and reliability of Melusine, MAIF implemented a comprehensive testing strategy, achieving over 95% test coverage.</li><li><strong>Tested Tutorials:</strong> Instead of notebooks, the tutorials in Melusine v3.0 are written as unit tested python files and incorporated in the <a href="https://maif.github.io/melusine/">documentation</a>.</li></ol><p>With these measures in place, we are much more confident regarding the robustness and maintenability of the package over time and we will be able to focus our energy on evolutions rather than fixing bugs.</p><h3>Robustness</h3><p>In its early days, the MAIF Data Factory primarily relied on external developers to build its codebase. As these developers came and went, their varying coding styles introduced inconsistencies and made it challenging to onboard new developers. To address this issue, MAIF undertook a comprehensive overhaul of its Data Factory, focusing on four key aspects:</p><ol><li><strong>People: </strong>MAIF recruited a diverse team of skilled professionals, including data scientists, software developers, operations engineers, and project managers. New collaborators brought a much needed experience to the Data Factory.</li><li><strong>Technical Expertise: </strong>MAIF invested in developing its in-house NLP and data science expertise and kept learning from maintaining and developing open-source packages.</li><li><strong>Industrialization:</strong> The Data Factory’s deployment process transitioned from manual to highly automated workflows. This involved implementing rigorous test coverage, code quality checks, and pre-commit hooks.</li><li><strong>Organizational Structure: </strong>MAIF revamped its organizational structure, moving away from a startup-like approach with limited project management and sporadic business interactions. The Data Factory adopted a more structured organizational model, featuring agile teams with product owners, facilitators, and systematic interactions with end-users and business representatives.</li></ol><p>By implementing these transformative changes, MAIF instilled a culture of best practices and a more consistent coding philosophy within the Data Factory, laying a solid foundation for the development of the new Melusine.</p><h3>Conclusion</h3><p>In this article, we’ve presented the pitfalls of the maintainance of an open-source package and the challenges of running an application in production and we have shared the solutions we implemented to address them during Melusine’s rewrite.</p><p>Initiating a complete code rewrite is a daunting task but also a remarkable opportunity to enhance our coding abilities. The journey towards Melusine v3.0 has profoundly impacted MAIF DataFactory’s team and resulted in long lasting transformations.</p><p>MAIF is eager to share these valuable insights with the community by open-sourcing the <a href="https://github.com/MAIF/melusine">new version of Melusine</a>, paving the way for continued innovation and collaboration.</p><h3>About the author</h3><p><em>I grew up in the French Alps, studied Physics / Nuclear Engineering in Switzerland, Sweden and England, and I’ve been working as a Data Scientist since 2018 at Quantmetry and MAIF. I am also a big fan of wakeboarding and board games :)</em></p><p>Follow me on <a href="http://www.linkedin.com/in/hugoperrier">LinkedIn</a>.<br>Leave a star for <a href="https://github.com/MAIF/melusine">Melusine on GitHub</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c57e2b6cee45" width="1" height="1" alt=""><hr><p><a href="https://medium.com/oss-by-maif/rewriting-a-python-package-the-journey-towards-melusine-v3-0-c57e2b6cee45">Rewriting a python package: The journey towards Melusine v3.0</a> was originally published in <a href="https://medium.com/oss-by-maif">OSS by MAIF</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NLP : Automatisez votre boîte mail Outlook avec Melusine]]></title>
            <link>https://hugoperrier.medium.com/automatisez-votre-bo%C3%AEte-mail-outlook-avec-melusine-9a718535be3a?source=rss-ed54763e56b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a718535be3a</guid>
            <category><![CDATA[melusine]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[quantmetry]]></category>
            <category><![CDATA[email]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Hugo Perrier]]></dc:creator>
            <pubDate>Thu, 08 Jul 2021 06:31:46 GMT</pubDate>
            <atom:updated>2021-07-08T07:39:08.089Z</atom:updated>
            <content:encoded><![CDATA[<h3>NLP : Automatisez votre boîte mail Outlook avec Melusine</h3><p>Melusine Open-Source compte depuis peu une nouvelle fonctionnalité : le connecteur Outlook !</p><p>Pour vous présenter la nouvelle fonctionnalité, cet article met en scène une entreprise fictive SuperMel et trois de ses employés Anna, Miko et Ida.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*6C6qyk0zt11HapzCVGQs-Q.png" /></figure><h3>Introduction</h3><blockquote><strong>« Vous avez 2500 emails non lus dans votre boîte de réception »</strong></blockquote><p>Miko arrive au bureau de SuperMel un matin et trouve 2500 emails non lus dans la boîte mail centralisée de l’entreprise. Aux nombreux emails non-traités des jours précédents se sont ajoutés des nouvelles demandes clients, des emails internes, des spams et des invitations calendrier.</p><p>Cette situation est intenable, Miko aimerait pouvoir fluidifier la gestion des emails dans l’entreprise en automatisant :</p><ul><li>La priorisation des emails urgents</li><li>La clôture des emails sans importance et des spams</li><li>Le routage des emails directement vers les bons dossiers</li></ul><p>Il réuni deux collègues pour discuter liées à l’Intelligence Artificielle pour la gestion des emails. Anna, Data Scientist, s’occupera de l’entraînement de modèles d’IA alors que Ida, Data Software Engineer, s’occupera de l’intégration de la solution IA dans le système d’information de l’entreprise.</p><h3>Melusine pour le traitement automatique des emails</h3><p>En faisant quelques recherches, Anna découvre le package <a href="https://github.com/MAIF/melusine">Melusine OSS</a>. Ce package python développé par la MAIF — entreprise qui traite des milliers d’emails par jour — et son partenaire <a href="https://www.quantmetry.com/">Quantmetry</a> — cabinet de conseil en IA — permet, entre autres, d’automatiser le traitement des emails.</p><p>En se basant sur une extraction de 4 000 emails, Anna a pu utiliser Melusine pour entraîner un modèle de classification qui va router les emails reçus dans la boîte de réception vers des dossiers plus pertinents. Sur les données d’entraînement Anna constate que 92% des emails reçus sont correctement routés par l’IA !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R76OA95jMW89yE52" /><figcaption>Photo by <a href="https://unsplash.com/@chriskristiansen?utm_source=medium&amp;utm_medium=referral">Chris Blonk</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Pour l’instant, Anna travaille avec des données d’entrainement sur son ordinateur portable. Cela permet d’entraîner un modèle d’IA et de tester les performances du modèle, cependant, pour pouvoir traiter des données de production (les emails dans la boîte mail de l’entreprise) son modèle doit être industrialisé.</p><h3>Industrialisation du modèle d’IA</h3><p>Anna vient montrer ses résultats à Ida et, ensemble, elles se demandent comment industrialiser la solution développée. Les recherches d’Ida ont fait ressortir plusieurs options :</p><ul><li>Intégrer Melusine comme une « brique IA » dans une chaîne de traitement des emails</li><li>Utiliser un client Exchange python pour se connecter directement à une boîte aux lettres Outlook</li></ul><h4>Intégrer la « brique IA » dans une chaîne de traitement des emails</h4><p>Les emails peuvent avoir une importance critique dans certaines entreprises, notamment lorsqu’ils représentent l’interface d’échange privilégiée avec des clients. Dans ce type d’entreprise, le département informatique aura probablement mis en place une chaîne de traitement des emails, complexe et spécifique à l’entreprise, avec par exemple :</p><ul><li>Historisation des emails reçus dans des bases de données</li><li>Stockage des documents reçus en pièce jointe</li><li>Logging des actions réalisées suite à la réception d’un email</li><li>Visualisation des emails sur une interface spécifique à l’entreprise</li></ul><p>Pour intégrer une « brique Melusine » dans la chaîne de traitement global, on peut exposer les services de Melusine via une API. Une API permet de recevoir une requête avec le contenu de l’email, d’appliquer les traitements Melusine et de renvoyer un résultat (par exemple le dossier de destination d’un email).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nXIAgaD5PNqyttzwjKgAYw.png" /><figcaption>Fonctionnement d’une API</figcaption></figure><p>Cette option a l’avantage de rendre la brique IA indépendante du reste de la chaîne de traitement des emails. Un bug lors du traitement IA renverra une réponse vide qui sera ignorée dans la suite de la chaîne. Cette option permet également d’avoir une gouvernance, des équipes et des langages de programmation différents pour développer chaque maillon de la chaîne.</p><p>Dans le cas de l’entreprise SuperMel, il n’existe pas de chaîne de traitement des emails au sein de l’entreprise, Ida cherche donc une solution qui permet de connecter le code développé par Anna directement à la boîte aux lettres Outlook de l’entreprise.</p><h4>Intégration dans Outlook</h4><p>En explorant les fonctionnalités de la librairie Melusine, Ida et Anna découvrent l’<em>ExchangeConnector</em> de Melusine qui semble convenir à leur besoin. Basée sur la librairie <em>exchangelib</em> pour se connecter et interagir avec une boîte aux lettres Outlook, cette fonctionnalité de Melusine est développée spécifiquement pour le routage des emails. Elle permet notamment de réaliser des traitements automatiques sur les emails reçus dans la boîte de réception. On peut par exemple déplacer les emails vers différents dossiers en fonction des prédictions du modèle d’IA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3kKnD2O5G_oYaS5fe_ymCA.png" /><figcaption>Routage des emails</figcaption></figure><p>Ida décide donc d’utiliser Melusine pour se connecter à la boîte mail de l’entreprise et planifie l’exécution du code tous les matins à 6h. Ainsi, chaque matin, lorsque les employés de SuperMel arriveront au bureau, ils trouveront leurs emails bien organisés dans des dossiers et n’auront plus à dépiler une montagne d’emails divers et variés !</p><h3>Amélioration continue de la solution IA</h3><p>Quelques semaines après le déploiement de la solution IA chez SuperMel, des employés constatent des erreurs dans les traitements opérés par Melusine : certains emails ne sont pas routés vers les bons dossiers !</p><p>Heureusement, le package Melusine prévoit une méthodologie qui permet au modèle d’IA d’apprendre de ses erreurs. Les employés peuvent signaler les emails mal classés et Melusine pourra les récupérer et les ajouter au jeu de données d’entrainement du modèle. Anna pourra ainsi périodiquement réaliser des ré-entraînements du modèle pour en améliorer les performances.</p><h3>Conclusion</h3><p>Si les problématiques rencontrées par Anna, Miko et Ida chez SuperMel vous semblent familières, n’hésitez pas à tester vous-même le package <a href="https://github.com/MAIF/melusine">Melusine OSS</a>. Le répertoire GitHub contient même un <a href="https://github.com/MAIF/melusine/blob/master/tutorial/tutorial14_exchange_connector.ipynb">tutoriel</a> qui explique comment utiliser l’<em>ExchangeConnector</em>.</p><p>Si vous avez des idées pour contribuer au développement de Melusine, vous pouvez créer une <a href="https://github.com/MAIF/melusine/issues">Issue sur Github</a> ou encore contacter la MAIF ou <a href="https://www.quantmetry.com/">Quantmetry</a>.</p><p><a href="http://www.linkedin.com/in/hugoperrier">Hugo Perrier</a></p><p>Machine Learning Engineer @ Quantmetry</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a718535be3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we organized a Hackathon from scratch]]></title>
            <link>https://medium.com/data-science/https-medium-com-hugoperrier-how-we-organized-a-hackathon-from-scratch-75ef5d2c2780?source=rss-ed54763e56b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/75ef5d2c2780</guid>
            <category><![CDATA[imperial-college-london]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[event-planning]]></category>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Hugo Perrier]]></dc:creator>
            <pubDate>Wed, 13 Jun 2018 16:41:48 GMT</pubDate>
            <atom:updated>2018-06-13T16:42:17.636Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RRAL3NCIZvw1vsOD" /><figcaption>Photo by <a href="https://unsplash.com/@rawpixel?utm_source=medium&amp;utm_medium=referral">rawpixel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>With this article, I aim to demystify Hackathons and lower the barrier to entry for anyone considering organizing one.</p><p>The Hackathon I describe is focused on the trendy topic of “Data Science”, but, in my opinion, Hackathons are not limited to data or computer science.</p><blockquote>I encourage anyone to jump into the Hackathon adventure regardless of their background.</blockquote><h3>What is a Hackathon?</h3><p>The word Hackathon is a contraction of “Hack” and “Marathon”. “Hack” may be used in the sense of hacking into a computer system, but I prefer the broader sense of finding a quick solution (a “hack”) to solve a particular problem.</p><blockquote>The idea is to gather motivated people, form teams, give them a challenge to solve in a short time-frame (usually a day or a weekend) and let the magic happen!</blockquote><p>Examples range from technical tasks — such as finding an exploit in a security system — to open challenges — such as brainstorming on how to improve the commute of Londoners.</p><h3>How it all started…</h3><p>Email received on Wednesday, November 29th</p><blockquote>“Dear Hugo</blockquote><blockquote>I currently follow an MSc at Imperial College London and I am curious whether your society organizes a Data Science related Hackathon?</blockquote><blockquote>Best,</blockquote><blockquote>SK”</blockquote><p>Interesting… We wanted to organize a Hackathon last year but it didn’t happen… Should we try again?</p><p><strong>Yes, that’s how it all started for me! </strong>But before I describe my journey through Hackathon organization, let me introduce myself:</p><p>My name is Hugo Perrier and I grew up in the French Alps. After studying in Lausanne, Stockholm and Zürich, I ended up in London to pursue a PhD in Fluid Dynamics.</p><blockquote><em>At that time, my experience with “Hackathon-like” events was limited to a “Start-Up Weekend” in Bali, where I spent a weekend kick-starting a business idea.</em></blockquote><p>I must say, at first, organizing a Hackathon sounded intimidating to me. What if participants dislike the challenges proposed? What if no-one shows up?</p><h3>My early stage interrogations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Cif0KzlvN6RMIoZy" /><figcaption>Photo by <a href="https://unsplash.com/@kas923?utm_source=medium&amp;utm_medium=referral">Kyle Simmons</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Hackathon format</h4><p>A widespread format for Hackathons is as follows:</p><blockquote>Meet your team on Friday evening, work throughout the weekend and present to judges on Sunday afternoon.</blockquote><p>An event on three days is quite long! Can we keep participants motivated throughout the weekend? What if people drop-out on Friday night? Is it not too expensive?</p><h4>Type of challenges</h4><p>Data Science competitions are nothing new. With the online platform Kaggle, competitors are given the same problem to solve and there is a clear quantitative metric to rank the participants.</p><blockquote>Kaggle-style challenges are convenient to organize and to judge, but, they are too technical and quite far from the challenges faced by most businesses.</blockquote><p>Our target audience is diverse and includes students with a business background. To please everyone, we are considering more open-ended challenges. Provide participants with data relevant to a business and let them decide what they want to do with it.</p><p>As far as I am aware, this type of challenge is not usual in the field of Data Science. Do we have what it takes to organize a novel type of Hackathon?</p><h3>Forming a team and getting started</h3><p>The organization of a Hackathon sounds like a lot of work and I will surely need a team to help me. My first idea is to discuss the Hackathon with the student society of which I am Vice-President. Let me introduce ICDSS.</p><h4>The Imperial College Data Science Society</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kdQdm0Mnv4j5fG_IoLZkww.png" /></figure><p>The society has a large member base, efficient communication channels (mailing list, social media), a network of sponsors and experience organizing events on campus. There is a problem though… most of the committee members are already busy with other ICDSS activities and exams.</p><p>So here we are, with SK (from the email), SL (the head of marketing at ICDSS) and myself. That is a start but probably not enough.</p><blockquote>I think we need help !!</blockquote><h4>The Graduate Student Union (GSU)</h4><p>Email received on Wednesday, January 10th</p><blockquote>“Dear Postgraduate students,</blockquote><blockquote>The Graduate Students’ Union (GSU) solely works to represent all Postgraduate students at Imperial College London.</blockquote><blockquote>[…]</blockquote><blockquote><strong>Big Data Hackathon</strong></blockquote><blockquote>We have been in touch with several major companies spanning different industries (to be revealed). Companies will provide problems, related to Big Data, to work on. Students, organized in Multidisciplinary teams will compete for the best solution.</blockquote><blockquote>Date<strong>:</strong> 28th April 2018</blockquote><blockquote>[…]</blockquote><blockquote>LM President of GSU”</blockquote><p>Wow !! That is timely ! The Graduate Student Union (GSU) came up with the same idea. I don’t know anyone there but they seem to be ahead of us, let’s try to make contact!</p><h4>Highlight of the meeting with GSU</h4><ul><li><strong>Name</strong> : Big Data Hackathon — Imperial College Data Challenge — Data Day — etc.</li><li><strong>Target audience</strong> : Should we go for Postgraduate students only or include Undergraduates?</li><li><strong>Prospective numbers</strong> : ~80 students, 6–8 companies providing challenges (3 secured so far), get 2 helpers from each company.</li><li><strong>Finances</strong> : Cover the cost by asking a contribution from participating companies (500£/company). Use society money if necessary.</li><li><strong>Venue</strong> : Use university infrastructure (to be discussed).</li><li><strong>Marketing</strong> : Societies’ mailing list and social media, flyers, on-campus screens, etc (we’ll need more, let’s be creative!).</li><li><strong>Participants registration</strong> : Eventbrite ? Google forms ? (To be discussed)</li><li><strong>Date</strong> : Still not confirmed, risk of collision with Easter Break.</li></ul><p>Great ! The GSU team is super friendly and they are happy to collaborate. There is still a lot of work to do, but, now, we have enough people on-board and a road-map to follow. One of the priorities is to <strong>find companies to provide a challenge!</strong> This proved to be one of the most difficult parts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jVkwz0WRu1JeXd1SFp7ICw.jpeg" /><figcaption>The IC Data Challenge organizers</figcaption></figure><h3>Find companies to take part in the Hackathon</h3><p>We are organizing the very first edition of the IC Data Challenge (Yes, that is the name we picked). No one has ever heard of us, the format of the challenges is still undefined and, at that point, our ability to gather 70 students is uncertain.</p><p>Despite our own doubts, we need to convince companies that the Hackathon is happening and that they will benefit from it.</p><blockquote>How do we convince companies to join the adventure?!</blockquote><p>The short answer is recruitment. A Hackathon is a privileged environment for students and company employees to interact. Companies get the opportunity to present, in depth, their business to students from a top university.</p><p>At first, the task of contacting companies is frustrating: sending cold emails, getting no response, being confronted to administrative dead-ends. Even when we receive a positive response, it takes a lot of work to keep the communication going. Sometimes, we also need to adapt to companies’ specific needs.</p><blockquote>“Hey SK,</blockquote><blockquote>Thanks for reaching out.</blockquote><blockquote>We do not usually sponsor Hackathons. Is there a resume book with all the students involved? That way we can review and decide if this is something we want to consider.</blockquote><blockquote>Best,</blockquote><blockquote>Company X”</blockquote><blockquote>A resume book! Seriously? That particular response was scary!</blockquote><p>It took dedication and perseverance to get the first positive responses. After a few weeks, we secured the participation of a big name (a company that delivers food) and the team’s morale went up. In the following weeks, a few more companies joined the Hackathon, well done guys!</p><h3>Convincing students to participate in the Hackathon</h3><p>If you are like me, you have been to a few events — such as meet-ups, conferences, trainings, etc — and, thanks to targeted advertisement, you are now drowning in an ocean of event invitations.</p><blockquote>A talk on the future on Artificial Intelligence starring a co-founder of DeepMind? WOW, I am definitely registering for that !</blockquote><p>If you are like me, you also end up dropping out on half on the events you register to. There are only 24 hours in a day and we all need to get some sleep.</p><p>That is exactly what happens with the events we organize at the university. From our experience, the attendance to an event is about a third of the number of registrations. <strong>How can we secure the right number of participants?</strong></p><p>I won’t bore you with the details of the strategy we adopted but I wanted to give you a hint of the challenges encountered.</p><h4>Design of marketing material</h4><p>When it comes to advertising, a cool logo and a slick flyer can help a lot! Fortunately, ICDSS has skillful designers in its committee. I bet we’ll get quality marketing material very soon!</p><p>Message received on Slack on Friday, March 30th</p><blockquote>“Hi Team,</blockquote><blockquote>Many apologies for this but unfortunately my flat was raided two days ago.</blockquote><blockquote>A lot of our valuables got stolen including my laptop. Forensic team from police came and did a crime investigation. Until my new laptop arrives there’s not much I can do.</blockquote><blockquote>Luckily I saved everything to Dropbox so the files should still be there.</blockquote><blockquote>Good Luck,</blockquote><blockquote>SL (Head of Marketing at ICDSS)”</blockquote><p>It is sad to hear SL got raided… but he still managed to save the files on Dropbox. <strong>This guy is a Hero !</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t2lMFwQTJwOCRBJU" /><figcaption>Advertisement flyer for the IC Data Challenge designed by SL</figcaption></figure><p>Our advertising strategy worked well and we recorded 300 applications for the IC Data Challenge!</p><p>There are many other topics relevant to the organization of a Hackathon. Event logistics, formation of (balanced) teams, assigning teams to companies, judging (jury, criteria), prizes, etc…</p><p>For the sake of brevity, I’ll skip these topics and jump to the day of the Hackathon. Feel free to leave comments if you would like to know more.</p><h3>Imperial College Data Challenge 2018</h3><p>The Hackathon is finally happening and we need to address last minute developments: public transport disruption, a student showing up uninvited, Wi-Fi issues. At 8:00 a.m, we are ready to launch the event! We start with a presentation of the companies and the challenges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-M3CSVPeZP9p2slCrYp7Vg.jpeg" /><figcaption>IC Data Challenge participants listening to the company presentations.</figcaption></figure><p>Create algorithms to predict public transport delays. Analyze how events, such as goals or yellow cards, impact the odds of bets during a football game. Investigate food menus from a variety of restaurants and classify the items proposed. The range of the challenges proposed is broad.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7gRDk0LGDNgv_m-FbLGwnw.jpeg" /><figcaption>IC Data Challenge participants getting insight from their assigned industry supervisor.</figcaption></figure><p>For 12 hours, the challenges kept the participants busy. The challenges were intense but participants got help from company supervisors. Some teams were so focused on their challenge that they missed the food breaks!</p><p>The IC Data Challenge is a competition so each team presented their work to a jury. Two cash prizes were given to the best technical presentation and the best business presentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Le_S3r671woLl2DIBBHTOw.jpeg" /><figcaption>Team who won the best business presentation prize (500£).</figcaption></figure><p>Participants, industry helpers, judges, organizers, at the end of the day, we all looked exhausted. To relax and conclude a hectic day, we invited everyone for a drink at the university bar.</p><h4>Feedback from the participants</h4><blockquote>“I had never been to a Hackathon before. Working in a multidisciplinary team was new to me but it was very stimulating.</blockquote><blockquote>DM (IC Data Challenge participant)”</blockquote><p>Spending an entire day working on a challenge with a team, is a great way to make new friends. The participants enjoyed teamwork and the opportunity to meet people with different backgrounds.</p><p>There is a gap between the worlds of academia and companies. Hackathons are a great opportunity for students to get exposure to industry problematics and grow their business awareness. They also have a chance to develop valuable skills for their future job search.</p><p>Despite a lot of positive feedback, the last-minute disclosure of the challenges made some participants feel unprepared. A student mentioned that, given a few days notice, he could have familiarized with the technology involved. For future events, despite the risk of drop-outs, we should inform the participants about the challenges early on.</p><h4>Feedback from the companies</h4><p>Coming up with a challenge was not easy for companies, especially for start-ups. Most of them never participated in a Hackathon before. They had to collect, clean and anonymize the data required for the challenges. These tasks required a significant effort. They also had to brainstorm on the complexity of the challenges, is it too easy? too hard? too specific? too open?</p><p>The feedback we got from companies was encouraging though. They appreciated the opportunity to exchange with a variety of students. I wouldn’t be surprised if some participants get hired following the Hackathon!</p><blockquote>All the company said they would love to join the next Hackathon we organize!</blockquote><h3>To conclude…</h3><p>Organizing a Hackathon has been an intimidating, sometimes frustrating and often restless experience. However, the whole process has also been really exciting and, eventually, immensely rewarding!</p><p>I am confident that, thanks to the lessons we learned, the next Imperial College Data Challenge will be even better and I hope that my story will inspire others to go ahead and plan enriching events.</p><h3>Acknowledgment</h3><p>Thanks so much to all the Hackathon organizing team, GSU and ICDSS. You did a remarkable job and made this event possible!</p><p>Both the companies and participants showed a lot of enthusiasm which made the event very pleasant. This is the best reward I could expect!</p><p>I am also grateful to the people who gave me feedback to help me write this article. Feel free to send further critics, suggestions and questions in the comments section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75ef5d2c2780" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/https-medium-com-hugoperrier-how-we-organized-a-hackathon-from-scratch-75ef5d2c2780">How we organized a Hackathon from scratch</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>