<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Maisons du Monde - Medium]]></title>
        <description><![CDATA[A collection of articles published by Maisons du monde. - Medium]]></description>
        <link>https://medium.com/maisonsdumonde?source=rss----fa75f9cb0a3b---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Maisons du Monde - Medium</title>
            <link>https://medium.com/maisonsdumonde?source=rss----fa75f9cb0a3b---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:44:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/maisonsdumonde" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Transforming Data Science Workflows with GCP’s Vertex AI]]></title>
            <link>https://medium.com/maisonsdumonde/transforming-data-science-workflows-with-gcps-vertex-ai-91b76ee66b30?source=rss----fa75f9cb0a3b---4</link>
            <guid isPermaLink="false">https://medium.com/p/91b76ee66b30</guid>
            <category><![CDATA[sales-forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <dc:creator><![CDATA[Francesca Iannuzzi]]></dc:creator>
            <pubDate>Wed, 05 Jul 2023 07:10:54 GMT</pubDate>
            <atom:updated>2023-07-05T07:16:31.886Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Co-authored with Nicolas Gorrity, ML Engineer, and Melissa Cardinale Cortes, Data Scientist</em></p><p>A Data team has existed at Maisons du Monde since 2017. As discussed in other stories from this publication, our technical stack is built around GCP, Airflow, GitLab, and QlikSense. Over the years, the team has developed a solid foundation of Extract-Load-Transform workflows that feed into a comprehensive data warehouse<strong> </strong>featuring multiple levels of data enrichment. This robust backend has been complemented by extensive frontend BI development, enabling a range of business units and stakeholders to access and utilize data through dashboards and data marts. This, coupled with a steady investment in Data Governance, has revolutionized the way the company accesses and uses data.</p><p>This wealth of information opened the door to both descriptive analytics and early predictive use-cases. The latter set the stage for what later became a more deliberate strategy.</p><p>Indeed, as the team grew in maturity and foundational needs were met, it was time for a dedicated data science team to step in and address a broader array of use cases as part of a continuous effort.</p><h3>The Catalyst: Sales Forecasting POC</h3><p>The initial demands faced by the Data Science Team at Maisons du Monde were primarily around sales forecasting. In typical fashion, the need spurred from a context of urgency that translated into stringent time constraints. Indeed, towards the end of 2021, the volatility of post-Covid sales behaviors required close monitoring on the business side. We were asked for frequent, up-to-date forecasts to support decision-making.</p><p>We tackled this task as a proof of concept, aiming to provide immediate business support while simultaneously demonstrating the potential of a data-science driven approach to the problem. (Up until this point, sales forecasting was handled primarily on the business side via a blend of in-house resources and external proprietary solutions.)</p><p>This phase was characterized by an intensive use of notebooks, laptops, and GCP-based virtual machines for more time-consuming processes. We used <a href="https://facebook.github.io/prophet/">Prophet</a> as our primary algorithm as it is fast to set up, can factor in external variables, and provides interpretable results. (We later added other methods through the <a href="https://unit8co.github.io/darts/">Darts library</a>, but their discussion goes beyond the purpose of this article.) We retrained models every week to incorporate the latest available ground-truth data before making predictions. Feature engineering was continuously updated and, as a consequence, hyperparameters were frequently tuned.</p><p>By the end of the POC, we had developed dozens of models and generated nearly a hundred forecasts. This would not have been possible without concurrently developing a semi-automated workflow alongside our primary modeling objectives.</p><p>To this end, we set up a Python application that would handle this repetitive pipeline of tasks:</p><ul><li>Downloading the most recent data from BigQuery;</li><li>Formatting the data as a valid time series;</li><li>Fetching the hyperparameters used for the previous model (or new ones that we would have selected with hyperparameter optimization processes);</li><li>Training a new model;</li><li>Generating predictions; and</li><li>Uploading predictions into BigQuery.</li></ul><p>This Python app was dockerized and scheduled using Airflow, the orchestrator that was already in place for data engineering tasks. Every Monday, the necessary data would be automatically collected, processed, and modeled. The resulting forecasts became accessible via a QlikSense Dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9n3jxSzOpjmceJFX" /><figcaption><em>Fig. 1 — A simplified view of our first DAG, developed during the POC phase of the Sales Forecasting project. Data collection and preprocessing were triggered automatically every Monday. Following this, we would inspect the data, review our modeling approach if necessary, and then manually trigger the training and prediction stages.</em></figcaption></figure><h3>From POC To Project: New Challenges</h3><p>The successful results from the proof of concept led to sponsorship for a more ambitious project focused on sales forecasting. This required us to ensure the rapid deployment of a larger number of models, along with the establishment of strong technical foundations to support scalable growth. (In retail, forecasting can involve multiple target variables and potentially thousands of relevant time series at the most granular level.)</p><p>Our existing workflow served us well during the early stages of the project, supporting the first deployments. However, we understood that it needed to evolve to meet the increasing complexity of our tasks.</p><p>Our first concern was scalability. Using the standard infrastructure, originally designed for data engineering workloads, limited our flexibility in choosing the hardware configuration for the virtual machine that ran the retraining jobs. Retraining models may require to scale out (running multiple workloads running in parallel) or scale up (allocating more RAM, CPU or GPUs for particularly heavy models).</p><p>Another significant aspect was code maintainability and the ease of onboarding new team members on the project. Although our program was structured to minimize the amount of coding required by a data scientist (typically only a new forecasting model or a new data transformation function), as the complexity of our forecasting tasks increased, the deployment of new models became increasingly time-consuming. Under the existing setup, a team member needed to understand almost the entirety of the app to write configuration files correctly.</p><p>Lastly, we had to contemplate our long-term strategy, especially in relation to the production of future data science products, their costs, and speed of implementation. As the team expanded and started to tackle demands beyond sales forecasting, each new deployment phase felt like starting from scratch. Creating a new automated workflow for each project was time consuming and required DataOps skills beyond what could reasonably be expected from even our most versatile full-stack data scientists.</p><p>Beyond deployment, it was clear we had to revamp our development environment. It was time to transition away from local Python installations and the constant shuttling of information to and from virtual machines.</p><h3>Elevating The Stack With Vertex AI</h3><p>Over a year ago, we made the choice of developing our data science workflows by leveraging the features of GCP’s Vertex AI.</p><p>This was a somewhat natural choice, considering <a href="https://cloud.google.com/customers/maisons-du-monde">the rest of our stack heavily relied on GCP</a> already and given the pronounced computer-science / engineering skills in our data science team. Far from abstracting away the necessity of coding the details of the treatments, this platform enabled us to streamline the overall development, testing, and deployment processes.</p><p>In what follows, we are focussing on two main Vertex AI features: Pipelines and Workbench.</p><h3>Efficient Deployments with Pipelines</h3><p>Similar to the SQL-based treatments already in use by the data engineering team, we migrated to a tech stack where Vertex pipelines could be triggered by Airflow tasks.</p><p>With Vertex AI Pipelines, our data scientists design their workflows in terms of distinct steps, or <em>components</em>. Each component is a functional building block of the workflow — for example, it could be transforming a dataframe, training a model, or uploading data into BigQuery. These components ingest and output data objects known as <em>artifacts</em>. Interdependencies between components are formulated into a <em>pipeline</em> using a very Pythonic syntax.</p><p>In addition to migrating our deployments to Vertex AI Pipelines, we implemented some technical layers to further facilitate the process for data scientists.</p><ul><li>Once the Vertex pipeline is created, a CI/CD pipeline is automatically triggered to run the unit tests for the code, compile the pipeline into a template file using the Python SDK, and upload the compiled file to Google Cloud Storage.</li><li>We also developed a reusable Airflow operator that runs a dockerized app designed to trigger the execution of a Vertex pipeline simply given its name and desired parameters.</li></ul><p>As a result, data scientists are left to focus solely on coding the specifics of their use cases — the components and pipelines. They do not need to worry about Airflow operators, service accounts, or CI/CD pipelines. Transitioning to Vertex AI not only addressed the technical challenges we faced, but it also significantly improved the efficiency of our data scientists’ deployments and fostered an Agile approach in our workflow.</p><p>The creation, maintenance, and continuous evolution of this infrastructure is now the responsibility of the ML Engineer — a role that we grew internally in parallel to this process.</p><p>Upon triggering a pipeline, the Vertex AI Pipelines UI presents a visual representation of its execution, with components organized as a DAG and data artifacts coming in and out of components. An intuitive color code marks running components in blue, successful ones in green, and those that failed are marked in red.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uALCnDPLuJjxg5GN" /><figcaption><em>Fig. 2 — An excerpt from one of our Vertex Pipelines. Note how components (the rectangular boxes) are interconnected and artifacts are created and exchanged along the way. Also note the indication of execution status (the component at the top has failed).</em></figcaption></figure><p>The interface becomes a powerful tool for debugging, as we simply have to click on a component to visualize its logging messages. They are automatically captured by GCP from the standard output of the component execution and then uploaded into Google Cloud Logging. Identifying the cause of a failing pipeline only takes two clicks, and diagnosing the root issue is simplified as the workflow is divided into smaller, functional units: the components. This has been a huge time saver for us in performing model maintenance.</p><p>Finally, this is how the Airflow DAG has evolved from the one shown in Fig.1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*olVwMGf9J4qESizt" /><figcaption><em>Fig. 3 — A simplified view of our latest DAG. Note how each of the green Airflow tasks corresponds to an entire Vertex Pipeline, like the one shown in Fig 2.</em></figcaption></figure><h3>Using Workbench for Development</h3><p>As we improved our deployment process, it was equally crucial to upgrade our development environment. On this front, our team has had an insightful experience working with Workbench, a Jupyter notebook-based development environment part of the Vertex AI suite.</p><p>This platform allowed us to enhance our data science workflows and introduced several useful features.</p><p>A significant advantage is that each Workbench instance arrives pre-installed with numerous kernels containing common data science libraries. This eliminates the need for managing local Python installations and multiple virtual environments. Furthermore, users can create custom Python environments to install project-specific packages.</p><p>We primarily use Vertex AI’s Workbench for exploring and analyzing large datasets, as well as for tasks like creating models, conducting training and optimizing hyperparameters.</p><p>Indeed, this tool is particularly useful when running long jobs such as hyperparameter tuning, as it allows us to modify hardware on the fly. We can start with a smaller, less costly machine for preliminary testing and seamlessly switch to a more powerful one (with one or more GPUs) for resource-intensive tasks, boosting performance.</p><p>Despite all these advantages, our journey with Workbench was not without its challenges. We occasionally encountered operational issues such as prolonged startup times and unexpected machine shutdowns, which disrupted our workflows. Furthermore, the pre-installed Python version (3.7) didn’t always match our project requirements, something which we overcame by using Conda.</p><p>Overall, our experience with Vertex AI’s Workbench has been positive despite a few hiccups along the way. We will continue our journey, further exploring the features offered by the platform. Transitioning from entirely Google-managed to user-managed notebooks will be one of our next steps, allowing us to have more control over our environment.</p><h3>6 months on</h3><p>Today, our sales forecasting project features automated modeling of over 40 hierarchically arranged time series. Some of these series are tackled by multiple algorithms, leading to over 60 distinct batch predictions with each weekly run.</p><p>Should we decide to delve deeper into the hierarchy, generating forecasts for hundreds or even thousands of series, the primary considerations would be adopting the appropriate modeling approach and accepting the increase in computing costs. Deployment, in and of itself, is no longer an obstacle.</p><p>In parallel with this main project, we have migrated a product-assortment algorithm from custom code to a pipeline-based process on Vertex AI. Beyond the benefits already mentioned, the integration with other GCP components like Pub/Sub, and Cloud Run has allowed us to package this algorithm into a custom and interactive user interface, thereby opening up new possibilities for data products beyond our QlikSense dashboards.</p><p>Onboarding new team members has become more straightforward, thanks to a simplified code base and the development environment provided by Vertex AI’s Workbench. While we continue to work on initial proof-of-concept projects that can be a bit messy, operating within an integrated GCP environment overall provides a smoother experience.</p><p>As we continued to explore the potential of Vertex AI, we experimented with AutoML — state-of-the-art machine learning models that are easy to deploy and to customize with minimal effort. AutoML also offers valuable tools such as automatic feature engineering, feature importance analysis, and hierarchical forecasting that further simplify our work.</p><p>We also delved into Google’s latest generative AI model, PaLM2. We explored its functionalities through the PaLM API, specifically tuned for language tasks such as classification and summarization.</p><p>By integrating Vertex AI into our tech stack, we’ve built a foundation that accelerates our ability to offer complex treatments addressing diverse operational challenges. These treatments may leverage custom machine-learning models or glossy off-the-shelf AI functionalities, and the importance of a streamlined deployment avenue for such high-profile applications cannot be overstated. However, the advantages of using Vertex extend to the development and deployment of even more traditional algorithms. Indeed, in a company like Maisons du Monde, there are still simple data-powered treatments holding considerable potential to transform and optimize a variety of retail processes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91b76ee66b30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/maisonsdumonde/transforming-data-science-workflows-with-gcps-vertex-ai-91b76ee66b30">Transforming Data Science Workflows with GCP’s Vertex AI</a> was originally published in <a href="https://medium.com/maisonsdumonde">Maisons du Monde</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[L’Architecture event-driven au cœur du SI Maisons Du Monde]]></title>
            <link>https://medium.com/maisonsdumonde/larchitecture-event-driven-au-c%C5%93ur-du-si-maisons-du-monde-6a673903a556?source=rss----fa75f9cb0a3b---4</link>
            <guid isPermaLink="false">https://medium.com/p/6a673903a556</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data-centric]]></category>
            <category><![CDATA[google-pubsub]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <dc:creator><![CDATA[Emilien GARNIER]]></dc:creator>
            <pubDate>Wed, 01 Dec 2021 07:47:34 GMT</pubDate>
            <atom:updated>2021-12-02T10:35:47.513Z</atom:updated>
            <content:encoded><![CDATA[<p>Pour construire un SI robuste et évolutif, il convient de limiter le couplage entre les composants. Le style d’architecture <em>event-driven</em> offre une opportunité d’y parvenir.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fp_BL23yNGnUSHPzU-cfmA.png" /></figure><p>Nous vous proposons de découvrir la manière dont nous l’avons abordé pour construire le SI Maisons du Monde.</p><h3>Le système fonctionnel c’est quoi ?</h3><p>Nous sommes convaincus que la donnée doit être au centre de toutes nos préoccupations. A ce titre, nous avons organisé le SI autour de blocs fonctionnels cohérents, chacun d’entre eux portant un périmètre unique de données métier. On parle souvent de <em>produit</em> pour définir chacun de ces blocs fonctionnels, nous avons choisi le terme <em>système fonctionnel</em>, pour éviter tout amalgame lié à notre activité de <em>retailer</em>.</p><p>Chez Maisons Du Monde, les données sont gouvernées par l’équipe Data Gouvernance, en collaboration avec les métiers, puis documentées dans le dictionnaire de données.</p><blockquote><strong><em>On parle alors d’un SI data-centric.</em></strong></blockquote><h3>Le référentiel</h3><p>Le principe adopté pour implémenter notre SI et l’organiser en <em>systèmes fonctionnels</em>, est donc de créer des référentiels de données uniques pour chaque SF.</p><p>Chaque référentiel de données est responsable de :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/1*fFg2pTFfaorQTfzKZ1ucjA.png" /></figure><ul><li>Persister les données dont il est référent, en s’appuyant sur une base de données (n° <strong>1</strong> sur le schéma).</li><li>Exposer des services synchrones (API), pour permettre aux IHM et aux autres composants back-office du SI de les manipuler (n° <strong>2</strong> sur le schéma).</li><li>Informer le reste du SI lorsqu’une donnée est créée/modifiée, en publiant un message dans un topic (n°<strong>3</strong> sur le schéma).</li></ul><blockquote><strong><em>On parle d’émettre un event, dans une architecture event-driven (voir $ « Les topics »).</em></strong></blockquote><p>En aucun cas un composant du SI autre que celui qui porte l’API du référentiel n’est autorisé à interroger la base de données d’un référentiel en direct, il doit passer par une API du référentiel.</p><h3>Les fronts</h3><p>Un front porte la responsabilité de présenter les données au client et de lui permettre de suivre un parcours fonctionnel pour agir dessus.</p><p>Un système fonctionnel peut donc mettre à disposition des fronts qui interrogeront les référentiels du SF pour présenter les données (n°<strong>4</strong> sur le schéma).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ghF1uTs4tIZjQBlM-vTKeA.png" /></figure><p>Ces fronts pourront également s’appuyer sur les référentiels d’autres SF, uniquement en lecture, pour faciliter et améliorer l’expérience utilisateur.</p><p>Enfin, ces fronts pourront mettre à jour les données des référentiels du SF, en s’appuyant sur leurs API.</p><p>Prenons par exemple le système fonctionnel responsable du produit. Il dispose d’un front permettant de créer des produits, d’en modifier les caractéristiques (taille, poids, …). Il peut aussi présenter des informations connexes, portées par d’autres SF, telles que le stock du produit ou les photos du produit. Ce sont des infos utiles aux utilisateurs mais qui ne sont pas portées par le même système fonctionnel.</p><h3>Les orchestrateurs</h3><p>Certaines données métier possèdent un état qui évolue selon un workflow métier. Les orchestrateurs sont chargés de faire évoluer l’état de ces objets.<br>Chaque référentiel peut donc être équipé d’un ou plusieurs orchestrateurs.</p><p>Un orchestrateur peut réagir aux événements produits par d’autres référentiels (n°<strong>5</strong> sur le schéma).</p><ul><li>Exemple : Un orchestrateur de type <em>flow</em> qui mettra à jour le statut de la commande client en réagissant aux événements publiés par le référentiel transport.</li></ul><p>Un orchestrateur peut aussi être autonome pour faire évoluer l’état des objets métier du référentiel (n°<strong>6</strong> sur le schéma).</p><ul><li>Exemple : Un orchestrateur de type <em>batch</em> qui annulera une commande au bout de 10 jours si elle n’est pas payée.</li></ul><p>Un orchestrateur peut enfin recevoir des demandes pour déclencher une mise à jour de l’état d’un objet, on parle de <em>request</em> (ou de <em>command</em>), dans une architecture <em>event-driven</em> (n°<strong>7</strong> sur le schéma).</p><ul><li>Exemple : Un orchestrateur de type <em>flow </em>permettant de traiter des demandes d’impression asynchrones d’étiquettes de produits.</li></ul><blockquote><strong><em>On parle de réponse à un event, dans une architecture event-driven (voir $ « Les topics »).</em></strong></blockquote><p>Un référentiel maître de la donnée, des orchestrateurs pour faire évoluer les états de celle-ci, des fronts pour la présenter et interagir avec l’utilisateur et enfin des topics pour la communication : Ce sont tous ces éléments qui composent l’architecture Event Driven de Maisons Du Monde.</p><h3>Les Topics</h3><p>Les topics permettent la communication entre les différents <em>systèmes fonctionnels</em>. Dans une architecture <em>event-driven</em>, cette communication est massivement asynchrone, de manière à découpler au maximum les composants.</p><p>On distingue alors 2 types de topics, liés à ce style d’architecture et de communication :</p><ul><li>Les topics de type<em> event : </em>Chaque fois qu’un référentiel crée/modifie une donnée, il a la responsabilité d’en poster la photo (l’état complet de l’objet à l’instant t) dans un topic de type <em>event</em> (n°<strong>3.</strong> sur le schéma ci-dessus).<br><em>Exemple :</em> Le référentiel des commandes publie dans un topic chaque commande passée par un client pour que le SI puisse réagir à cet événement.</li><li>Les topics de type <em>request : </em>Les orchestrateurs qui proposent des services asynchrones le font au travers de topics de type <em>request</em> (n°<strong>7</strong> sur le schéma ci-dessus). Ce type de topic fait sens quand l’appelant a seulement besoin d’une confirmation de prise en compte de sa demande mais pas un besoin transactionnel sur le résultat.<br><em>Exemple :</em> Le référentiel des contacts propose un topic de type <em>request</em> permettant de désabonner un client aux e-mail marketing.</li></ul><p>Article co-écrit avec <a href="https://medium.com/u/dbec543a301b">Pierre FEVRIER</a></p><p>Illustrations signées <a href="https://medium.com/u/3c4d63443bed">Alexia Robinet</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a673903a556" width="1" height="1" alt=""><hr><p><a href="https://medium.com/maisonsdumonde/larchitecture-event-driven-au-c%C5%93ur-du-si-maisons-du-monde-6a673903a556">L’Architecture event-driven au cœur du SI Maisons Du Monde</a> was originally published in <a href="https://medium.com/maisonsdumonde">Maisons du Monde</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Qlik Sense under control]]></title>
            <link>https://medium.com/maisonsdumonde/qlik-sense-under-control-ab047c3b40a4?source=rss----fa75f9cb0a3b---4</link>
            <guid isPermaLink="false">https://medium.com/p/ab047c3b40a4</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[qlik]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[dataops]]></category>
            <dc:creator><![CDATA[Benjamin Berriot]]></dc:creator>
            <pubDate>Fri, 18 Dec 2020 16:09:45 GMT</pubDate>
            <atom:updated>2020-12-18T16:09:45.212Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t0fvIM7YZj7AbP4Z.jpg" /></figure><p>At Maisons du Monde, we are using Qlik Sense as our data vizualisation tool. We provide dashbaords for all the company. We also send hundreds of static reports every day, generated by Qlik NPrinting. This make Qlik one of our main tools by being the link between the data team and the business teams in the company. Hence it’s very important to keep Qlik under control.</p><h3>Qlik infrastructure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ncfp6tHoKeeDqKPl.jpg" /><figcaption>Google Data Center (<a href="https://www.youtube.com/watch?v=zDAYZU4A3w0">source</a>)</figcaption></figure><p>At Maisons du Monde, we have the following infrastruture for our Qlik environments:</p><ul><li>3 VM for <a href="https://www.qlik.com/us/products/qlik-sense">Qlik Sense</a>, oneVM is for development environment, onefor production environment.The last one, called master, is in charge of dashboard reloading, QVD (Qlik file format) creation and other Qlik administration under the hood.</li><li>1 VM for <a href="https://www.qlik.com/us/products/nprinting">Qlik NPrinting</a>. Qlik NPrinting is an application provided by Qlik, that can create static reporting and send report by email for example.</li></ul><p>All of these virtual machines are running Windows Server and are host on <a href="https://cloud.google.com/compute">Compute Engine in Google Cloud Platform</a>.</p><h3>Data pipelines V1: time based orchestration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O_PvgLH2Hbr2ygFz.png" /><figcaption>source: <a href="https://kinsta.com/fr/base-de-connaissances/tache-cron-wordpress/">https://kinsta.com/fr/base-de-connaissances/tache-cron-wordpress/</a></figcaption></figure><p>As we already explained in this <a href="https://medium.com/maisonsdumonde/migration-to-airflow-one-year-feedback-91a8a72cb4a1">article</a>, at the begining we were using Rundeck as the orchestrator of our data platform. Rundeck is a time based orchestrator. Qlik provide the possibility to schedule application reloading based on time. Our pipeline between our data platform (orchestrated by Rundeck) and Qlik (orchestrated by itself) was disconnected. This means that if a Rundeck tasks was failing or running wrongly, Qlik would still reload the dashboards with no up-to-date data or worse, wrong data. At Maisons du Monde, our pipelines are mainly excecuted on a daily basis, this is why having the data late or incorrect in the dashboards is a big issue for our business teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*GpD2YX3z3upFCPCBBJLNYA.jpeg" /><figcaption>It’s even worse with complex pipelines (source: author)</figcaption></figure><p>Moreover, using time based scheduling is not optimized in terms of landing times. Optimising these pipelines can be complicated. For users, it’s important to have their dashboards up-to-date in the morning, to take better and faster decision based on data.</p><h3>Data pipelines V1.5: hybrid orchestration</h3><p>In mid 2019, we switched from Rundeck to Airflow, but we focused our move on the data pipelines running inside Rundeck. During this period we kept dashbaord reloading scheduled in Qlik, still based on time. We were not experiencing any issues with this, because most of our pipelines running on Apache Airflow were ending earlier than with Rundeck. In fact most of our dashboard were reloaded every day without any issues or data out-dated.</p><p>However, this orchestration didn’t solve two main issues:</p><ul><li>Landing time: we had pipelines finishing early on Airflow but dashboards were reload later in the day (remember the security margin for Rundeck).</li><li>Observability: using two orchestration, we had to monitore two different pipelines to check if something went wrong.</li></ul><h3>Data pipelines V2: Direct Acyclic Graph (DAG)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*noJf5NstlhCbQAHJZn5-eQ.png" /><figcaption>Example of DAG (source: author)</figcaption></figure><p>In late 2019, we planned to trigger Qlik application reloading directly from Airflow. Luckily Qlik Sense provides an API to trigger various Qlik operations, such as reloading a dashboard. You can find the documentation to create Qlik Sense credentials <a href="https://help.qlik.com/en-US/sense-admin/November2020/Subsystems/DeployAdministerQSE/Content/Sense_DeployAdminister/QSEoW/Administer_QSEoW/Managing_QSEoW/export-certificates.htm">here</a> and a getting started <a href="https://help.qlik.com/en-US/sense-developer/November2020/Subsystems/RepositoryServiceAPI/Content/Sense_RepositoryServiceAPI/RepositoryServiceAPI-Getting-Started.htm">here</a>.</p><p>To trigger a task in Qlik Sense with the API we are using the following route: post /task/{id}/start/synchronous (<a href="https://help.qlik.com/en-US/sense-developer/April2019/APIs/repositoryserviceapi/index.html?page=1103">doc</a>). This route returns a session_id and you can check the task status with the route: GET executionresult/ful?filter=ExecutionId eq {session_id} (<a href="https://help.qlik.com/en-US/sense-developer/April2019/APIs/repositoryserviceapi/index.html?page=928">doc</a>). All these requests are send to http://&lt;your_server_ip&gt;:4242/qrs .</p><p>Bonus, this is the list of task result id and the corresponding status:</p><pre>all_status = {</pre><pre>0: &quot;NeverStarted&quot;,</pre><pre>1: &quot;Triggered&quot;,</pre><pre>2: &quot;Started&quot;,</pre><pre>3: &quot;Queued&quot;,</pre><pre>4: &quot;AbortInitiated&quot;,</pre><pre>5: &quot;Aborting&quot;,</pre><pre>6: &quot;Aborted&quot;,</pre><pre>7: &quot;FinishedSuccess&quot;,</pre><pre>8: &quot;FinishedFail&quot;,</pre><pre>9: &quot;Skipped&quot;,</pre><pre>10: &quot;Retry&quot;,</pre><pre>11: &quot;Error&quot;,</pre><pre>12: &quot;Reset&quot;,</pre><pre>}</pre><p><a href="https://medium.com/maisonsdumonde/migration-to-airflow-one-year-feedback-91a8a72cb4a1">As we discussed in our article about Apache Airflow</a>, we are using KubernetesPodOperator as operator. The stack in the data team is Python, so we started to develop a Python code that interacts with the Qlik Sense API. Nothing really complicated right here, we are using <a href="https://fr.python-requests.org/en/latest/">requests</a> to make API calls.</p><p>At the begining of 2020, we extended the code to start Qlik NPrinting tasks. You can find the getting started tutorial <a href="https://help.qlik.com/en-US/nprinting/November2020/Content/NPrinting/Extending/NPrintingAPI-Getting-Started.htm">here</a>. Authentification with this API is a little bit tricky, because this API uses NTLM Auth. To achieve the authentification you need to create a user in Qlik NPrinting.</p><p>Our Python code is still using <a href="https://fr.python-requests.org/en/latest/">Requests</a>, but we added <a href="https://pypi.org/project/requests_ntlm/">requests-ntml</a> for authentification. First, login with this route GET /login/ntml (<a href="https://help.qlik.com/en-US/nprinting/November2020/APIs/NP+API/index.html?page=9">doc</a>). To start a NPrinting task we are using this route POST /tasks/{task_id}/executions (<a href="https://help.qlik.com/en-US/nprinting/November2020/APIs/NP+API/index.html?page=51">doc</a>), you can extract the execution id in the response under data/id. Then to check the task status, we are usingGET /tasks/{task_id}/executions/{execution_id} (<a href="https://help.qlik.com/en-US/nprinting/November2020/APIs/NP+API/index.html?page=49">doc</a>).</p><p>With Qlik Sense and Qlik Nprinting triggered by Airflow, we improved landing time, data dependencies and observability. As an example, in our pipeline V1, we had a dashboard which was reloading at 3pm. After the full switch to Airflow, this same dashboard is reloaded at 11am just by changing the orchestration. That is a huge improvement for user experience.</p><h3>Data lineage</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*MYGKk77FYN8hExjJ.jpg" /></figure><p>One of the big issue we are experiencing with Qlik Sense, is data lineage. To add data into Qlik Sense dashboard, we need to write a loading script, in which we define connection to the datasource, load data and make some transformation. For example this is a basic script that loads data from a database into Qlik Sense, with SQL:</p><pre>LIB CONNECT TO &#39;DataSource&#39;;</pre><pre>LOAD *;</pre><pre>SQL SELECT * FROM `Sales_data`;</pre><p>This scripts is written by the developer of the dashbaord, directly inside the Qlik Sense application. When the dashboard is published, the script can’t be edited or viewed. You need to duplicate the app in your development environment to edit/view and republish the app. One of the problem with this process is when you want to deprecate or update a datasource, and you want to check all dashboards using this datasource.</p><p>In october 2020, I found this repository on Github (<a href="https://github.com/ptarmiganlabs/butler-spyglass">https://github.com/ptarmiganlabs/butler-spyglass</a>) where someone developped a code in Javascript to get the Qlik loading script of an app through an API. I search in Qlik documentation, and finally found it, the <a href="https://help.qlik.com/en-US/sense-developer/November2020/Subsystems/EngineAPI/Content/Sense_EngineAPI/introducing-engine-API.htm">Qlik Engine JSON API</a> . With this websocket you can dump the loading script of the application into a file. We implemented a Python version of this code (inspired by this code: <a href="https://gist.github.com/clintcarr/d0a8bb6e15cb6c9a3472d424c344ce73">https://gist.github.com/clintcarr/d0a8bb6e15cb6c9a3472d424c344ce73</a>).</p><p>To make the results open for everybody in our team, we are uploading all loading scripts generated by our code to Gitlab every night. We can now find easily which data source is used by which dashboards, and have versioning of every scripts. As a git repository you can clone it on your computer and with this VSCode extension (<a href="https://marketplace.visualstudio.com/items?itemName=Gimly81.qlik">https://marketplace.visualstudio.com/items?itemName=Gimly81.qlik</a>) you can now open all scripts files, have syntax colors (save files as .qvs) and perform a “search in all files: XXX”.</p><h3>Performance monitoring</h3><p>Our Qlik infrastructure is running on 4 VM as described above, so we have a performance monitoring of each machine. We are monitoring basic informations such as CPU, RAM, I/O with a Grafana dashboard. This monitoring is really useful to know if all VM’s are ok. One monday morning we had Qlik down/really slow, when we saw on the monitoring that one of our machine had no more ram we understood the problem. We simply rebooted all machines and the problem was gone. We investigated the root cause and found that every monday morning the ram usage was really high. So we looked for a specific dashboard loading every monday morning, we found it, optimised it and this problem never occurred again.</p><h3>Conclusion</h3><p>We have developed some tools to improve data pipelines and data lineage with Qlik Sense. There is still some issues to address such as alerting when dashboard reload will be delay, data quality and dashboard versioning. With automated extraction of Qlik loading scripts, Airflow and Google Cloud Logs we are close to perform a full data lineage from Google Storage to Qlik Sense.</p><p>We are looking forward for your comments, hoping that this feedback was useful and feel free to reach us for further discussions and experience sharing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab047c3b40a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/maisonsdumonde/qlik-sense-under-control-ab047c3b40a4">Qlik Sense under control</a> was originally published in <a href="https://medium.com/maisonsdumonde">Maisons du Monde</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migration to Airflow: One year feedback]]></title>
            <link>https://medium.com/maisonsdumonde/migration-to-airflow-one-year-feedback-91a8a72cb4a1?source=rss----fa75f9cb0a3b---4</link>
            <guid isPermaLink="false">https://medium.com/p/91a8a72cb4a1</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Ismail Addou]]></dc:creator>
            <pubDate>Mon, 01 Jun 2020 13:41:21 GMT</pubDate>
            <atom:updated>2020-06-01T13:54:27.620Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GPSUuyMJi8jbpC6xKT4SlQ.png" /></figure><p>At Maisons Du Monde, we used to create and schedule our data pipelines with Rundeck, which is an automation server like Linux Cron for the veterans. Rundeck had been available on our company’s network, so we simply used it to get the pipelines started.</p><p>At the beginning, this solution was convenient as long as workflows finished in time. But <em>“winter is coming !”</em> The platform kept on growing, the pipelines became increasingly bigger and much more complex. So Rundeck’s maintenance has become much harder, because of an unconvenient user experience and some performance limits on the local server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*JFN4mYQZXzmDWPLLm1jBbw.jpeg" /></figure><blockquote>That’s why we moved to Airflow !</blockquote><p>In this article, we will express our feedback after one year of Airflow usage, comparatively to Rundeck, in the data team of Maisons Du Monde.</p><p>In the first section of this article, we will discuss about the main issues that we have encountered with Rundeck.</p><p>In the second section, we will explain how did we integrated Airflow to satisfy our needs. <em>The best solution for us, is the one which is sustainable and user friendly !</em></p><h3>Why we left Rundeck ?</h3><p>In this automation system, our tasks were organized over tree folders, similarly to files over an OS system. Moreover, the scheduler executes jobs inside folders <strong>sequentially</strong>, following a <a href="https://en.wikipedia.org/wiki/Depth-first_search">depth-first search</a> execution priority. So, there is no concrete dependencies, only some execution rules in the task scope.</p><p>However, separate folders could be run in parallel, but without dependencies. If needed, we shifted adequately the start time for the dependent workflows, with a safety margin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*HoJwEYPEUPVRhG6LbUWpUw.png" /><figcaption>Time based dependency: Fernando Sucre would start his job at 3 am</figcaption></figure><blockquote>Fernando here doesn’t know if Michael has reached the nursery. He’s just expecting it, and puts faith on the plan to have a chance escaping from the prison. It is 3 am now, Fernando shall get started. Hopefully Michael didn’t get caught.</blockquote><p>So this is making the user experience harder for exploring or troubleshooting the workflows. Thus, catching up the failing jobs tends to be a nightmare when it comes to replay the time dependent workflows.</p><p>But there were many other reasons to quit Rundeck, here are some examples:</p><ul><li>Time loss due to safety margins and sequential execution inside folders</li><li>Frequent errors due to the overwhelming Rundeck’s server</li><li>Troubleshooting and incident recovery is too manual</li><li>Difficulty to add new pipelines and dependencies</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F5799547&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fview%2Fnetwork-gif-5799547&amp;image=https%3A%2F%2Fmedia.tenor.com%2Fimages%2F6209e758eb8d5fe72aaf2209a2fb4179%2Fraw&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/01537b852d88830540f9f495f2b5c779/href">https://medium.com/media/01537b852d88830540f9f495f2b5c779/href</a></iframe><h3>Airflow from our perspective</h3><p>Apache Airflow is a popular open-source solution with an active community around it. It came up with a new spirit of automation suited for data use cases, also it could be integrated on a scalable architecture.</p><p>The solution could resolve all the previously mentioned issues and <a href="https://airflow.apache.org/docs/stable/">Airflow’s documentation first lines</a> bring some answers to that. To sum up, here is some good reasons that led us to choose Airflow: it uses DAGs, has a rich user interface for monitoring and troubleshooting, could be scalable, workflows are Python coded and last but not the least, Google Cloud Platform — which is our cloud provider — has a managed service for Airflow named Composer. It helped us a lot to quickly study and deploy Airflow on our infrastructure.</p><h4>Directed Acyclic Graphs (DAG)</h4><p>Moving from folders to DAGs was a relief, simply because a DAG is less restrictive: it unlocks the possibility to make new dependencies and to multi-task.</p><p>We took this advantage to rethink the existent pipelines and dependencies. As a result:</p><ul><li>Workflows fit accurately to the reality</li><li>Troubleshooting doesn’t require a background knowledge about tasks dependencies</li><li>Time saving thanks to multi-tasking and accurate dependencies</li></ul><h4><strong>User interface</strong></h4><p>Airflow has a rich user interface to handle troubleshoot and to manually catch up tasks over <em>time</em> and over <em>dependencies</em>.</p><h4>Extensible Python framework</h4><p>This is a very attractive aspect of Airflow. Each DAG in Airflow has an equivalent python code which has three steps:</p><ul><li>Instantiate a DAG</li><li>Instantiate tasks: operators and sensors</li><li>Make dependencies between tasks</li></ul><p>And here is the best part, with Airflow plugins you could create your own operators or sensors by overriding the classes <em>BaseOperator</em> or <em>BaseSensor.</em></p><p>Before discussing about Airflow integration, I would like to share with you some useful operators and sensors: <em>LatestOnlyOperator, BranchOperator, ExternalTaskSensor, KubernetesPodOperator </em>and also the <em>DummyOperator tha</em>t could be useful even if it is a dummy.</p><h3>Airflow integration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MGSi7xA0TAH3FZda_1e6Tg.png" /></figure><blockquote>We use Google Cloud Composer as a managed Airflow’s service.</blockquote><p>Google Cloud Composer instance has a specific bucket dedicated to put Airflow’s code in production. So the deployment is operated by Gitlab using CI/CD pipelines in two steps:</p><ul><li><em>Testing step:</em> to detect the syntactical issues in the code to be deployed</li><li><em>Deployment step: </em>to put in production using a dedicated bucket in Google Cloud Storage, which is accessible from Airflow workers thanks to <a href="https://cloud.google.com/storage/docs/gcs-fuse?hl=fr">Cloud Storage FUSE</a>.</li></ul><p>Once CI/CD pipelines are achieved, it takes only few seconds to notice the changes in Airflow’s web-server.</p><h4><strong>Local development</strong></h4><p>An Airflow docker image was developed by the team for this purpose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*HS02PCJBi45_tajxr6G7Sw.png" /></figure><blockquote>A volume is mounted, to make the container access to the Airflow’s git local repository.</blockquote><p>Airflow’s configuration inside the container is similar to the production configuration. Thus, if the workflow works locally, technically, it should be working fine on production as well.</p><p>Furthermore, we have developed a <em>“dry run”</em> feature for a testing purpose, that could be set to True or False through the UI. With an active “<em>dry run</em>” mode, the execution would be simulated by just logging the steps instead of launching their execution.</p><p>It is possible to mount a volume on the context folder (~/. kube/config) with the running container in order to test the pipelines in a Kubernetes environment. This could be useful to test the new developed plugins before their releases.</p><h4><em>KubernetesPodOperator</em></h4><p>We are using <em>KubernetesPodOperator</em> because most of our python projects are containerized and could be run in Composer’s GKE cluster. <em>KubernetesPodOperator’s</em> input parameters are quiet similar to a Kubernetes job’s <em>Yaml</em> file. Here is an example of a program which runs a transformation in the datalake.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9a5551757a000f3b7637116d261a6ac/href">https://medium.com/media/b9a5551757a000f3b7637116d261a6ac/href</a></iframe><p>So now you will not only require some Airflow skills, but also some additional Kubernetes skills for any job deployment. That’s why it was important to create custom plugins to apply some minimalism to the operators.</p><p><em>Do we need to change all the Kubernetes parameters anytime we would like to reuse a specific program like a transform operation ? Luckily not!</em></p><p>Actually, we only have three changing parameters for a transform operation which corresponds to its Docker entrypoint:</p><ul><li>dataset_id: equivalent of schema in BigQuery</li><li>table_id: table name in BigQuery</li><li>reference_date: which is the contextual date by default</li></ul><p>So an Airflow plugin at Maisons Du Monde will expose only the changing parameters as inputs, and will preset the environment configurations. For example, the preset would contain a hard-coded Docker image URI and a predefined version value, “<em>Latest” </em>by default. Moreover, we used plugins to automatically define <em>task_id </em>by combining the required inputs, to generate a task_id which looks like:<em> “OPERATOR_NAME-my_dataset.my_table” .</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d500151c614b019e5d9751609935fc18/href">https://medium.com/media/d500151c614b019e5d9751609935fc18/href</a></iframe><blockquote><em>Let’s take a look under the hood !</em></blockquote><h4><strong>Overriding KubernetesPodOperator</strong></h4><p>For the record, the purpose of our plugins is to have the fewest required inputs in order to use our custom operators. So, every Docker image we use has a specific plugin, which is basically a tiny class extending the abstract class <em>MdmKubeOperator</em>, this one is a child class of the original <em>KubernetesPodOperator</em>. The middle class <em>(MdmKubeOperator)</em> is very important to define some common methods and attributes that could be reused over all the plugins.</p><p>Here is a class diagram representing, among others, two of our plugins: <em>QueryLauncherOperator</em> for transform operations and <em>DatabaseCollectOperator</em> for databases collect operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*-SIEbe2x8K6JGEsRuNYnTQ.png" /></figure><h4>DAG’s organization</h4><p>Now that we know how Airflow could fit on our infrastructure, it remains to normalize the separation between DAGs in order to define some common task regrouping rules. In simple words, what defines a DAG ?</p><p>We ended up choosing the following organization:</p><ul><li><strong>Collect DAGs (level 1): </strong>the rule is one DAG per data source. Each DAG contains only collect tasks and cleaning transformations <em>without external dependencies</em>.</li><li><strong>Use case DAG (level 2): </strong>this kind of DAGs merges workflows coming from different sources, in order to construct a topic workflow. Hence, it is often at this level that we calculate our most clean and reusable tables. For example: Users, Transactions, Traffic, Emailing…</li><li><strong>Output DAG (level 3): </strong>for workflows which prepare data to be exposed outside the datalake. Output tasks could refresh an API, reload a Dashboard, train a model or transfer data.</li></ul><p>As Airflow doesn’t allow regrouping DAGs inside folders, we prefix every DAG by its level in order to see the collects first in the UI, then the central tables, then the output DAGs.</p><blockquote>We didn’t end up with this solution at the first step of the integration. At the beginning, we had migrated only few workflows from Rundeck, in order to test &amp; learn. So we went for a “DAG per use case” separation for a couple of months but we quickly found out that it wasn’t really sustainable. For example, a collected table that is used in two use cases DAGs could be problematic to place, hence the necessity to define what is a DAG in your company ! However this transition period was very crucial to raise our knowledge about Airflow operators and tips.</blockquote><h4><strong>Workflows Implementation</strong></h4><p>DAGs dependencies could be implemented using the <em>ExternalTaskSensor</em>. It periodically checks for another task’s execution state and turns on to green when it succeeds. This wasn’t possible with Rundeck, remember ?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*mxRbcYDBvqmYe99ODIIxjA.png" /></figure><blockquote>Fernando has a cell phone now to check. He could precisely start right after the first job is done by Scofield.</blockquote><p>However, we don’t recommend using sensors to directly listen on the tasks one by one, because the dependencies will get complicated, like too many octopus hugging each others.</p><p>Instead we do recommend to put some dummy tasks as checkpoints, to create tasks clusters inside each DAG. As a result, we listen on clusters instead of elementary tasks. In our case, checkpoints are based on execution duration.</p><p>So we have agreed on 3 duration levels inside each DAG:</p><ul><li><strong>level 1:</strong> “quick” dependencies</li><li><strong>level 2:</strong> “average” dependencies</li><li><strong>level 3:</strong> “slow” dependencies</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*pEdzD5syYg4sG6z0v3NMMw.png" /></figure><blockquote>In the second DAG, the tasks A3, A4 and A5 depend on the first DAG’s level 1 tasks, which refer to the quick dependencies ! So this separation allows to execute urgent workflows first.</blockquote><h3>After one year of Airflow usage</h3><p>More than one year after Airflow’s integration, only time could prove the final solution’s sustainability through real scenarios.</p><p>Indeed, it was a success because the current system satisfied our initial requirements:</p><ul><li>It resolved the basic issues encountered with Rundeck.</li><li>Workflows succeeded most of the time, and the most urgent workflows could finish before 8 am.</li><li>Troubleshooting became practical and catching up dependencies, now, requires only few clicks</li><li>Scheduling jobs on production (Kubernetes cluster), now, requires only some <em>minimum</em> Airflow skills, but <em>no</em> Kubernetes skills</li></ul><p>Integration success key is based on good decisions and scenarios anticipation which takes into consideration the infrastructure constraints. This required the entire team’s engagement to avoid forgetting any scenario.</p><p>So after more than one year of Airflow’s integration, the system was exposed to real situations which have revealed some strengths and some weaknesses as well.</p><h4>New system strengths</h4><p>Let’s take a look to a real scenario that witnessed the following key strength points: A trigger campaign sending a welcome email to the new Maisons Du Monde’s clients, scheduled on Airflow.</p><p><strong>Scheduling easiness: </strong>Every member of the team is able to know how and where to schedule just by applying the rules. For the trigger campaign example, the task should be set in an output DAG which depends on two DAGs.</p><ul><li><em>2_Transactions:</em> to check what did the client bought</li><li><em>2_Users:</em> to target only the contactable clients</li></ul><p>The interesting thing is that selecting dependencies remains abstract and keeps workflows simple.</p><p><strong>Troubleshooting convenience: </strong>It is handy to identify and manage incidents for workflows execution. Let’s assume that our email campaign has failed, we know that it depends on <em>2_Transactions</em> and <em>2_Users</em>. So we are able to find and fix the root cause error and its dependencies easily through the user interface. However, most incidents were related to network problems or data sources themselves, hence the interest of having separate DAGs per collected source.</p><p><strong>Responsibilities separation: </strong>every team member could enrich the production pipelines by adding new tasks to the workflow that runs on Kubernetes. However, Kubernetes skills are not required in the pipelines making. The interesting point, is that responsibilities could be divided between two types of users:</p><ul><li><em>Plugins creators: </em>they <em>f</em>ocus on Airflow’s and Kubernetes preset parameters, like variable environment, secrets, volume … to create a custom operator that only requires the functional parameters.</li><li><em>DAG managers:</em> for workflows monitoring and DAGs creation using the existing operators and plugins.</li></ul><p><strong>Time gain: </strong>the pipelines were optimized thanks to the fitting workflows and sensors which have reduced dependencies waiting time. As a result, this trigger campaign is most of time ready at 8 am instead of 5pm before: <em>9 hours optimization.</em></p><h4>The system weaknesses</h4><p>It is True that Airflow made our scheduling routines easier since we have began using it. Nevertheless, some weaknesses should be highlighted.</p><p><strong>The scheduler: </strong>As you might now, the scheduler manages DAG-runs and orchestrates tasks executions across the workers and pods. So for every heartbeat, the scheduler loops over all the DAGs tasks and query the Airflow’s database, to orchestrate all the executions. But when the scheduler is down, Airflow is out of service meanwhile Composer will be trying to pop up a new scheduler to recover. So there is two recommendations to minimize scheduler’s incidents:</p><ul><li>Reducing the scheduler’s heartbeat</li><li>Avoid executing codes inside DAG python files, because it would be executed on the scheduler pod instead of the workers.</li></ul><p><strong>Zombies tasks: </strong>Some tasks keep in a running state on Airflow even if they have been completed. That happens likely for three reasons, when Airflow database is overwhelmed, when an Airflow worker is down or when we use a sub-dag (that we strongly advise you not to use). The only way we found so far to recover is to re-run the task manually.</p><p><strong>Sub-DAGs: </strong>We used to encapsulate the reusable workflows inside sub-DAGs. It simplifies more the pipelines but it have caused so many bugs that we have abandoned using this feature.</p><p><strong>DAG folders:</strong> DAGs are listed on a paginated list, and it is not possible to organize them on folders for a better navigation on the UI.</p><p><strong>Timezone handling: </strong>We have encountered some issues to handle timezone, like the time inconsistencies between the UI and the execution context, or the summer time switch. Even though it is configurable in “airflow.cfg”, we managed to programmatically adjust the context’s timezone to<em> Europe/Paris</em> inside plugins, in the parent class <em>MdmKubeOperator</em>, because it doesn’t work fine in our current version 1.10.0 of Airflow<em>. </em>Fortunately, this problem was tackled by the community since the version 1.10.10 of Airflow.</p><p><strong>Trigger task with input parameters: </strong>It was the missing features that we had with Rundeck before. So a new Airflow page was developed internally, to launch a task with custom parameters to fill in the form. To know more details about this internal feature, check this Maisons Du Monde’s article to do the same. <a href="https://medium.com/maisonsdumonde/road-to-add-form-for-airflows-dag-1dcf2e7583ef">https://medium.com/maisonsdumonde/road-to-add-form-for-airflows-dag-1dcf2e7583ef</a></p><h3>Looking further</h3><p>Beyond the features which are currently developed in this project, we have a to-do list of new features to improve the user experience a little more. However, we stay alert to <a href="https://cwiki.apache.org/confluence/display/AIRFLOW/Roadmap">Airflow’s Roadmap</a> and the new versions changelogs for a better prioritization.</p><h4>Sensor task’s redirection</h4><p>We want to add a new button in the <em>ExternalTaskSensor</em>’s window, which will redirect to the corresponding DAG.</p><h4>DAG dependencies overview</h4><p>We want to automate the creation of a top level DAG named “<em>0_ALL_DAGS</em>”, which displays all the relationships between DAGs. Each task of “<em>0_ALL_DAGS</em>” is actually the same sensors which have been used for DAGs dependencies. So dependencies between DAGs would become concretely represented and we will avoid the pitfall of creating cycles between DAGs.</p><h3>In the end</h3><p>Airflow is a really good tool that could fit, basically, on every architecture. However, there is no common integration standards to integrate your solution. Furthermore, we noticed that companies have a different integration logic, but a lot of them have encountered the same pitfalls. I hope that this feedback will inspire you. Your solution would be definitely different but remember: the better solution you have, is the one which is <em>“sustainable”</em> and <em>“user friendly”</em>.</p><p>I would like to thank you for your reading, a special thanks to the entire data team that made this success story happen, also to Benjamin Berriot and Guillaume Duflot for this article review.</p><p>We are looking forward for your comments, hoping that this feedback was useful and feel free to reach us for further discussions and experience sharing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91a8a72cb4a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/maisonsdumonde/migration-to-airflow-one-year-feedback-91a8a72cb4a1">Migration to Airflow: One year feedback</a> was originally published in <a href="https://medium.com/maisonsdumonde">Maisons du Monde</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gitlab-CI/CD and GitLab PyPi Repository]]></title>
            <link>https://medium.com/maisonsdumonde/gitlab-ci-cd-and-gitlab-pypi-repository-4916a51d22eb?source=rss----fa75f9cb0a3b---4</link>
            <guid isPermaLink="false">https://medium.com/p/4916a51d22eb</guid>
            <category><![CDATA[gitlab-ci]]></category>
            <category><![CDATA[pypi]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[gitlab]]></category>
            <dc:creator><![CDATA[Benjamin Berriot]]></dc:creator>
            <pubDate>Thu, 30 Apr 2020 10:43:16 GMT</pubDate>
            <atom:updated>2020-04-30T10:43:15.997Z</atom:updated>
            <content:encoded><![CDATA[<p>At Maisons du Monde we are using Gitlab as our main and only code management platform. On the Data team we have two main types of repositories: applications (API or data pipeline) and components (generic code that can be reused by multiple applications). Our components are private and we can’t upload them on PyPi. Luckily, a few days ago, April 22nd, Gitlab released their last version 12.10 (<a href="https://about.gitlab.com/releases/2020/04/22/gitlab-12-10-released/">https://about.gitlab.com/releases/2020/04/22/gitlab-12-10-released/</a>) introducing PyPi Repository. In this article, we will see how to create our package, to upload it into Gitlab and to automate the process with Gitlab CI/CD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/0*DEHDcJsrVm97P8IN.jpg" /></figure><h3>How to create and publish a python package on Gitlab PyPi Repository?</h3><p>The first step is to create a repository including a valid python package inside. At the root of our repository, we need to add a setup.py looking like that:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c99e9a4279c4e7504141e64922b5d8c/href">https://medium.com/media/0c99e9a4279c4e7504141e64922b5d8c/href</a></iframe><p>You can check if the package is valid by running this command at the root of your project:<br>python setup.py bdist_wheel</p><p>This command will create a <em>dist</em> folder, containing a .whl file:<br>&lt;package-name&gt;-&lt;version&gt;-py3-none-any.whl</p><p>You can now install the package by running:<br>pip install &lt;package-name&gt;-&lt;version&gt;-py3-none-any.whl</p><p>Now that we have a functional package, we want to share it across our different users. We can achieve that by publishing the package on Gitlab.</p><p>In order to help us publishing the package on Gitlab, we used Twine (<a href="https://pypi.or/project/twine/">https://pypi.or/project/twine/</a>), a tool used to upload packages on PyPi or other repositories. Let’s install Twine :<br>pip install twine</p><p>Let’s create a file ~/.pypirc. This file is used to add more repositories to Twine. By default Twine upload packages on PyPi. But in our case we want to upload our packages on our Gitlab project. To achieve that we need to add this file ~/.pypirc with the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/058a44790248ffc87d49dcf0f012bff5/href">https://medium.com/media/058a44790248ffc87d49dcf0f012bff5/href</a></iframe><p>Your <strong>project_id</strong> can be found on the home page of your project, or in<em> Settings &gt; General</em>. To create a personal access token you can follow this guide: h<a href="https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html">ttps://docs.gitlab.com/ee/user/profile/personal_access_tokens.html</a>. The personal access token needs the <strong>API </strong>scope.</p><p>Now it’s time to upload our package on Gitlab, with the following command: <br>python -m twine upload --repository gitlab dist/&lt;package-name&gt;-&lt;version&gt;-py3-none-any.whl</p><p>Side note, you can manage multiple repositories in your .pypirc file, like that:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/312648e8588189ffd521861ceaf42d5f/href">https://medium.com/media/312648e8588189ffd521861ceaf42d5f/href</a></iframe><p>You can for example upload your package on testpypi with the command:</p><p>python -m twine upload --repository testpypi dist/&lt;package-name&gt;-&lt;version&gt;-py3-none-any.whl</p><p>Getting back to Gitlab: if you go in your project, in the tab <strong>Packages</strong>, you can now find your freshly created package, ready to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OqeTV0s_hg12NRe5m3z6hQ.png" /></figure><h3>Now it’s time to automate the process with Gitlab CI/CD</h3><p>Firstly go inside <strong>Settings &gt; CI/CD &gt; Variables </strong>and add a new variable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*ldnKvaPm_8HzxaMc6yZAig.png" /></figure><p>The value of the variable PYPIRC is the value of the file .pypirc created above. <strong>Note: the type of the variable is File</strong>.</p><p>Now create a file .gitlab-ci.yml at the root of your project as follow:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0fc9d202b043ac6d521f9ba78f457d85/href">https://medium.com/media/0fc9d202b043ac6d521f9ba78f457d85/href</a></iframe><p>The goal of this file is to define our Gitlab pipeline. In this case we are created a step named build-package in our pipeline. It will build our python package and upload it to Gitlab thanks to Twine. You can find more information about Gitlab-ci here: <a href="https://docs.gitlab.com/ee/ci/">https://docs.gitlab.com/ee/ci/</a></p><p>This pipeline will be triggered as soon as a tag is created, and we will use the tag value in our pipeline as a parameter with ${CI_COMMIT_TAG}.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KHP-v0P-MZgF6gD16IQtg.png" /></figure><p>In this case in the command python -m twine upload --repository gitlab dist/mypackage-${CI_COMMIT_TAG}-py3-none-any.whl --config-file /tmp/.pypirc , the value of ${CI_COMMIT_TAG} will be 0.0.1, which mean we are looking for the file mypackage-0.0.1-py3-none-any.whl.</p><p>Warning: if you use the tag value in your pipeline (like in this article) the tag name must match with the version provided in the file setup.py . Which mean, when you release a new version of you package, before tagging, update the file setup.py.</p><h3>How to use our new package?</h3><p>We have an application running inside a container, and we want to install our freshly created package. Update your requirements.txt file like that:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e31ad93f1b2edd94bc9d280046f4e61/href">https://medium.com/media/9e31ad93f1b2edd94bc9d280046f4e61/href</a></iframe><p>The --extra-index-url option to pip install makes pip looking in official PyPi package repository before looking in our Gitlab repository. This is useful when you have multiple packages in your requirements.txt, some on PyPi and others in your private repository. An other option exists in pip, --index-url , but this option force pip to look in one specific repository. If you don’t use requirements.txt file (or you have only packages from one repository in your requirements.txt file), you can install your package with the command: <br>pip install --index-url https://__token__:&lt;personal_access_token&gt;@gitlab.com/api/v4/projects/&lt;your_project_id&gt;/packages/pypi/simple your_package_name</p><p>If we go back to our application, the Dockerfile is looking like that:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e749c26c3cceb29e0283676b1002e2c1/href">https://medium.com/media/e749c26c3cceb29e0283676b1002e2c1/href</a></iframe><p>We can build the container by running: <br>docker build --build-arg GITLAB_PIP_TOKEN=mysupertoken .</p><p>In this case the value of variable GITLAB_PIP_TOKEN will be replaced by the value mysupertoken , and when we perform the pip install, the --extra-index-url in our requirements.txt file, is looking like that: <br>--extra-index-url https://__token__:mysupertoken@gitlab.com/api/[...] .</p><h3>Final words</h3><p>We are really happy that this feature is now implemented in Gitlab. It’s really simple to use and close to what we do when publishing a package on PyPi. This feature also works if you have your own Gitlab repository: you simply need to replace “gitlab.com” by your custom gitlab url within the different files.</p><p>To build our container, on Maisons du Monde’s Data team, we are using Google Cloud Builder. We create a container-builder.yml:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05b38e44ea541021fc0a494e212edf0c/href">https://medium.com/media/05b38e44ea541021fc0a494e212edf0c/href</a></iframe><p>And we submit the build using this command: <br>gcloud builds submit --config=container-builder.yml \ <br>--substitutions=_GITLAB_TOKEN=&quot;mysupertoken&quot;</p><p>You can find more informations about Google Cloud Builder here: <a href="https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values">https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values</a></p><h3>References:</h3><ul><li><a href="https://docs.gitlab.com/ee/user/packages/pypi_repository/index.html">GitLab PyPi Repository</a></li><li><a href="https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html">Personal access tokens | GitLab</a></li><li><a href="https://packaging.python.org/tutorials/packaging-projects/">Packaging Python Projects - Python Packaging User Guide</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4916a51d22eb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/maisonsdumonde/gitlab-ci-cd-and-gitlab-pypi-repository-4916a51d22eb">Gitlab-CI/CD and GitLab PyPi Repository</a> was originally published in <a href="https://medium.com/maisonsdumonde">Maisons du Monde</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TensorFlow Roadshow Paris 2019]]></title>
            <link>https://medium.com/maisonsdumonde/tensorflow-roadshow-paris-2019-11d312ac69e9?source=rss----fa75f9cb0a3b---4</link>
            <guid isPermaLink="false">https://medium.com/p/11d312ac69e9</guid>
            <category><![CDATA[google]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Benoit Pimpaud]]></dc:creator>
            <pubDate>Wed, 04 Mar 2020 13:04:56 GMT</pubDate>
            <atom:updated>2020-03-04T13:04:56.644Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MGjh5Lemsxa8acOmRNauJQ.png" /></figure><p>When Google released TensorFlow as an open-source library in 2015, it was mainly a tool designed for researchers or data-scientists facing specific machine learning models.</p><p>At Maisons du Monde data team, we are fully working with Google Cloud Platform. Back in 2018, we used TensorFlow 1.8 to develop some models.</p><p>The fact is that it was very painful to develop and then maintain this code while TensorFlow was more about a low-level API than a user-friendly interface.</p><p>Meanwhile, other libraries such as Keras were developed over the TensorFlow API to help the creation of deep-learning models with simple methods. We mainly owe this library to François Chollet, a French engineer working at Google.</p><p>Since the end of 2018, teams working on the TensorFlow project succeeded to integrate the Keras environment into the core of TensorFlow: 2.0 version was born.</p><p>Beyond the integration of Keras as a high-level interface, simplification of code routines and changes of low-level APIs are making TF 2.0 a very interesting tool in order to develop and put machine learning models into production.</p><p>Early in 2019, we had the chance to attend the TensorFlow Roadshow in Paris. Here is a summary of the most interesting talks.</p><h3>François Chollet’s keynote</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_5Z_9oA04rZ5E8qR" /><figcaption>François Chollet during his presentation.</figcaption></figure><p>Before any technical subject, François presented how industries are changing because of the amount of data we are now able to collect and transform. Nothing new to the big picture: there is more and more data, processed by more and more computation power, resulting in a lot of progress in deep-learning fields.</p><p>Among interesting projects showed by the French engineer is <a href="http://www.inf.uniroma3.it/db/icr/">“Codice Ratio”: a project for automatic transcriptions of the Vatican manuscripts</a>. There is a lot of manuscripts in Vatican libraries that has not been studied since their creation. This project aims to develop new methods and tools to support content analysis and knowledge discovery from large collections of historical archives.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fv-FgOACRgfs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dv-FgOACRgfs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fv-FgOACRgfs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e1b3fec4a87872941f7a7a7b68efad5/href">https://medium.com/media/7e1b3fec4a87872941f7a7a7b68efad5/href</a></iframe><h3>tf.distribute()</h3><p>A new enhanced feature in TensorFlow 2.0 is the <a href="https://www.tensorflow.org/guide/distributed_training">code distribution routines</a>. One of the big advantages of TensorFlow is its ability to perform computation on many engines at the same time.</p><p>In this new version, several strategies appeared or have been improved such as MirroredStrategy or MultiWorkerStrategy. tf.data.prefetch allowing preprocessing and train data on distributed workers and tf.data.interleave to process many input files concurrently are also good added features.</p><h3>New NLP routines</h3><p>Missing in the previous version, some NLP basics are coming in the new one. The endpoint tf.text brings classic features such as tokenization, offsetting, n-grams or sliding windows methods, it will be easier to use TensorFlow in text processing pipelines.</p><p>Hence we had a quick presentation on how to train and execute BERT model on cloud TPUs — very powerful chipsets developed by Google for deep-learning models.</p><p>Though not particularly related to NLP, the arrival of tf.Ragged - dimension variable tensor - could enhance some processes too.</p><h3>TensorFlow Lite</h3><p>Doing machine learning on embedded devices can be difficult due to constraints on power and resources. Whilst first versions of TensorFlow had already advanced features in comparaison to other frameworks in this domain, the new version brings these developments to enhance moves from classic TensorFlow to <a href="https://www.tensorflow.org/lite">TensorFlow Lite</a>. For example, it is possible to build your machine learning model on a powerful computer and then deploying it on your powerless devices.</p><h3>MLIR</h3><p><a href="https://www.tensorflow.org/mlir">MLIR project — standing for Multi-Level Intermediate Representation</a> — comes from a casual fact: in the last twenty years, we have multiplied our computer resources by a one million factor. This project aims to create and unify (not only TF) a compilation infrastructure allowing machine learning pipeline execution on any hardware.</p><h3>Model optimization</h3><p>Altough there are more and more computer resources, it is still smart to optimize data processing. In that way, it could be interesting to apply low-level optimization. Here are two mentioned examples:</p><p><strong>Quantization</strong>: Transforming 32bit float to 8bit integer. It comes at the price of precision. In many cases, losing a bit of information can be negligible, but it can save you a lot of time and memory on a high volume of data. For example : transforming 32bit float to 8bit integer.</p><p><strong>Pruning</strong>: Reducing model complexity by eliminating unnecessary values in model weights. This optimizes memory and accelerates processes at the cost of a slight loss on precision.</p><h3>Airbus Defence &amp; Space and ISS</h3><p>One of the Airbus Defence and Space data team presented a use case on the International Space Station. They developed a LSTM autoencoder to detect anomalies and prevent security breaches in order to make sure astronauts are safe up there.</p><p>Sensors deployed on the ISS are recording more than 17 000 data events per second: big data takes on its full meaning. But more than the scientific issues, it was interesting to learn about the infrastructure behind these use cases.</p><p>Due to contracts involving several government organizations and constraining privacy policies, they can’t operate with a public cloud provider such as Google Cloud Platform or Amazon Web Services. Thus they work with a cloud solution hosted on private servers to manage these large amounts of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*672s6i2nD8DRHcbX" /></figure><h3>TensorFlow Hub</h3><p><a href="https://tfhub.dev/">This repository</a> contains pre-trained models for several tasks. Indeed, it is often better to reuse already trained models on high-quality resources rather than trying to reach the same performances on your infrastructure and starting from scratch. Why would you reinvent the wheel ?</p><h3>OLX</h3><p><a href="https://www.olx.com/">The OLX website</a> hosts advertising that are published free of charge by users all over the world. Their main markets are Brazil, India, Indonesia, Pakistan, Portugal, Romania and South Africa. Hence they have to process more than nine million images per day!</p><p>To do so, they built a pipeline on Amazon Web Services to upload images, extract features and then serve all these data throughout the entire company, allowing every team to work on these pictures.</p><p>One of the use case is the product categories classification thanks to image similarity computations — all done with Keras, the upper hand TensorFlow API. Moreover, they use the <a href="https://github.com/horovod/horovod">Horovod distributed framework</a> on top of <a href="https://www.docker.com/">Docker</a> and <a href="https://kubernetes.io/">Kubernetes</a> infrastructure.</p><p>As a conclusion, we can say that it was a very pleasant event. We felt how Google wants to keep the lead on this machine-learning frameworks and how they are supporting the open-source on the development of TensorFlow</p><p>With this new TensorFlow release, several actors who had doubts about the complexity to work with TensorFlow, either for research and development or for production pipeline, should probably give another chance to this framework.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11d312ac69e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/maisonsdumonde/tensorflow-roadshow-paris-2019-11d312ac69e9">TensorFlow Roadshow Paris 2019</a> was originally published in <a href="https://medium.com/maisonsdumonde">Maisons du Monde</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Road to add form for Airflow’s dag]]></title>
            <link>https://medium.com/maisonsdumonde/road-to-add-form-for-airflows-dag-1dcf2e7583ef?source=rss----fa75f9cb0a3b---4</link>
            <guid isPermaLink="false">https://medium.com/p/1dcf2e7583ef</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[plugins]]></category>
            <dc:creator><![CDATA[Benjamin Berriot]]></dc:creator>
            <pubDate>Mon, 24 Feb 2020 12:48:16 GMT</pubDate>
            <atom:updated>2020-02-24T13:07:11.342Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sDqMG4TKa0z6y66oD2HVnw.png" /></figure><h3>Road to trigger Airflow’s dag through a form</h3><p>At Maisons du Monde data team, we are using Airflow as our main datalake’s job orchestrator. Hence, all our jobs used for collecting, transforming and exposing data are running via Airflow (which is deployed on Google Cloud Composer).</p><h3>What we want</h3><p>We want to be able to trigger a dag with arguments filled through a form on Airflow UI. This can be useful when we want to launch a machine learning pipeline or an ETL job with specific arguments (that we often do in our daily job).</p><p><strong>Disclaimer</strong> : this is probably not the best way for doing this because it uses low level API that can change at any moment. The following code is tested for <strong>Apache Airflow 1.10.2</strong> and python 2 (I know…). Airflow is running on Google Cloud Composer v1.8.2.</p><h3>How to trigger a dag from the UI?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*F1y2DwADZuNuMjab.gif" /></figure><p>First, in Airflow there is a button that can trigger the dag on demand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/246/1*UdNSVkGcCnQ7JIN0pJBzlw.png" /><figcaption>First button on the left</figcaption></figure><p>Let’s search inside the Airflow code how this button works. We start by finding where the button is implemented within the html template file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d47bede4c030fef005c91dfc75e70d2a/href">https://medium.com/media/d47bede4c030fef005c91dfc75e70d2a/href</a></iframe><p>The button is linked to airflow.trigger and it’s sending to the function the dag_id. Then, let’s check in the file, views.py, to find the route /trigger.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b1eafa0d687f7884a834d17ffa7d83e7/href">https://medium.com/media/b1eafa0d687f7884a834d17ffa7d83e7/href</a></iframe><p>Perfect, now we have the code to launch a specific dag with its dag id.</p><h3>How to pass arguments to the dag execution?</h3><p>This is the most important thing, we want to send information to the dags through a form. We will use the Airflow jinja templating feature to achieve that (<a href="https://airflow.apache.org/docs/stable/concepts.html#id1">https://airflow.apache.org/docs/stable/concepts.html#id1</a>), in particular, we will use dag_run variable. When we create a new dag_run we can send a conf dict with the argument conf.</p><pre>dag.create_dagrun(<br>    run_id=run_id,<br>    execution_date=execution_date,<br>    state=State.RUNNING,<br>    conf=run_conf,    # here<br>    external_trigger=True<br>)</pre><p>We just need to parse all arguments sent using the form, fill this dict and then create a new dag_run with these arguments, pretty simple.</p><h3>Create the plugin</h3><p>Now that we have all pieces of code needed for doing what we want. It’s time to build the final code.</p><p>Important point, we want to keep the plugin as simple as we can and to make easy to add a new form. So we decided to use a json file to describe the form.</p><p>The tree of the plugin is as follow:</p><pre>plugins/<br>  custom_launcher/<br>    custom_launcher.py<br>    config.json<br>templates/<br>  custom_launcher/<br>    main.html</pre><p>We can start by creating the dag that we want to launch. Here is an example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c8156b41827748e615de73e429b3f40/href">https://medium.com/media/2c8156b41827748e615de73e429b3f40/href</a></iframe><p>It’s a simple dag with one task that launches a query on BigQuery: the operator needs a dataset id, a table id and a date.</p><p>Then, we create the json file (config.json) to describe the form that we want.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d0476d134ca5ff504a01b1919127532/href">https://medium.com/media/0d0476d134ca5ff504a01b1919127532/href</a></iframe><p>The first entry in the file (here: “Z_QUERY_LAUNCHER”) is the dag_id we want to trigger. Then, all entries correspond to the form input, with some additional information like, if the field is required, the placeholder etc.</p><p>Now we need to create the html template for the form.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/17bd207812cd2ea6d2453e3f7f67c0d4/href">https://medium.com/media/17bd207812cd2ea6d2453e3f7f67c0d4/href</a></iframe><p>After, the most important file, the python file that parses the config file, renders the html and triggers the dag.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b762cc8333487ec6366c4841720d191/href">https://medium.com/media/5b762cc8333487ec6366c4841720d191/href</a></iframe><p>The plugin exposes two endpoints, the root endpoint ‘/’ that renders the html and ‘/trigger’ that triggers the dag.</p><p>Finally the plugin class:</p><pre># Defining the plugin classclass AirflowCustomLauncher(AirflowPlugin):<br>    name = &quot;custom_launcher&quot;<br>    admin_views = [CustomLauncher(category=&quot;Custom Launch&quot;, name=k, endpoint=k) for k in form_config]<br>    flask_blueprints = [bp]</pre><p>The argument ‘admin_views’ adds new views to Airflow and if we want to use one class but different endpoint (one endpoint by form) we specify the argument ‘endpoint=endpoint_name’.</p><h3>Final result</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1dou48HoxTQfDJwzlzhO9g.png" /></figure><p>We can see the new tab in the menu containing the form created following the config.json file.</p><p>When we click on the submit button, a new dag_run is created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4MEmSCdPxx2ZpWD9CtX-6A.png" /></figure><p>If we go to the rendered tab, we have for each dag_run all arguments sent to the dag:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pwXa6IRQoBKtrR0DkUl5QA.png" /></figure><p>Et voila, we can launch our dag through a form!</p><h3>Production state</h3><p>Time for production, we deployed the code on Google Cloud Composer and we experienced some bugs:</p><ul><li>config.json file not found, the path on our local Airflow and composer was not the same. When we put config.json in plugin file, we replace the path by “conf.get(‘core’, ‘plugins_folder’) + /custom_launcher/config.json” instead of the hard coded path(never a good solution) “/home/airflow/plugins/custom_launcher/config.json”</li><li>After we deployed on composer we had a high CPU usage on our Kubernetes cluster. The problem was really hard to identify, but we saw a rise of network usage at the same moment as the rise of CPU usage. So we were looking for code that uses network. We found that the instruction “dagbag = models.DagBag(settings.DAGS_FOLDER)” was outside the class CustomLauncher and was called every time Airflow was parsing the file, creating high network and CPU usage. So we moved this instruction inside the class and the problem was fixed.</li></ul><h3>Conclusion</h3><p>This feature is really useful for our team and as we can see on some Airflow meetups it’s a missing feature for a lot of users, this is why we are sharing our work. Digging inside Airflow code was very interesting and helped us to understand how DAGs are triggered in Airflow.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1dcf2e7583ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/maisonsdumonde/road-to-add-form-for-airflows-dag-1dcf2e7583ef">Road to add form for Airflow’s dag</a> was originally published in <a href="https://medium.com/maisonsdumonde">Maisons du Monde</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>