<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matthieu Rousseau on Medium]]></title>
        <description><![CDATA[Stories by Matthieu Rousseau on Medium]]></description>
        <link>https://medium.com/@modeoAI?source=rss-86eaa22b0b15------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JrhEckFoVDh_XxjsGSJmKQ.png</url>
            <title>Stories by Matthieu Rousseau on Medium</title>
            <link>https://medium.com/@modeoAI?source=rss-86eaa22b0b15------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 00:42:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@modeoAI/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[GenAI for data teams: future ways of working, shifting roles and limits]]></title>
            <link>https://medium.com/data-science/genai-for-data-teams-future-ways-of-working-shifting-roles-and-limits-3a9394b7f46d?source=rss-86eaa22b0b15------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a9394b7f46d</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Matthieu Rousseau]]></dc:creator>
            <pubDate>Sun, 10 Aug 2025 09:59:09 GMT</pubDate>
            <atom:updated>2025-08-22T12:35:07.432Z</atom:updated>
            <content:encoded><![CDATA[<h3>Impact of AI on Data Teams : a new way to build Data Platforms (1/3)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3EorP6xTQDeR_OLAvZYTSg.png" /></figure><h3>Intro</h3><p>The rapid rise of Generative AI (GenAI) invites us — collectively — to rethink how Data teams should work, organize themselves, and create value.</p><p>The last major revolution in the Data world dates back to 2019: the emergence of the <strong>Modern Data Stack</strong>. At that time, this new paradigm represented a major breakthrough in how Data projects were designed, with many promises:</p><ul><li>Greater autonomy for business teams, thanks to easier adoption of tools and higher-quality data</li><li>Shorter time-to-insight</li><li>Improved data quality through governance and observability solutions</li><li>More flexible, scalable architectures powered by the Cloud</li><li>Better pipeline maintainability through a SQL-first approach</li></ul><p>It was in this context that we created Modeo, with a clear mission: <strong>help our clients successfully deliver ambitious projects by leveraging this new stack.</strong></p><p>To make this vision concrete, we built deep expertise in Data Engineering, grew a team of more than 30 Data Engineers, and supported over 100 clients of all sizes and across many industries. That’s how we became the French reference for the Modern Data Stack.</p><p>These projects have taught us what does — and does not — work within a Data team to build high-value use cases involving multiple stakeholders. They also helped us identify the main challenges to address, whether technical, organizational, or process-related.</p><p>Now, the question is: <strong>how will this new GenAI revolution reshape Data teams?</strong> This is why I decided to write this series of three articles:</p><ol><li>A new way to build Data Platforms.</li><li>To what extent will Data teams be transformed? Should we imagine a world where Data teams no longer exist?</li><li>How should a Data team leverage GenAI — and what risks should they avoid?</li></ol><blockquote><strong>Disclaimer:</strong> This vision is based solely on my personal opinion and internal discussions at Modeo. Our understanding of how Data teams operate is rooted in a European context. This is not meant to be an absolute truth, nor directly applicable to every organization. Every Data context is unique, and the reflections shared here must be adapted to each company’s specific reality.</blockquote><h3>How will the Data Platforms of tomorrow be built?</h3><h3>Key innovations</h3><h4>Vibe Coding &amp; Copilots</h4><p><strong>What is vibe coding / a copilot?</strong><br> Vibe coding is a new way of developing, made possible by integrating AI into data tools. In practice, it means a development experience where the Data Engineer is assisted by intelligent copilots that understand context, generate code, propose fixes or suggestions, and make the workflow faster with less friction.</p><p>You’re no longer starting from a blank page: you describe what you want to do, the tool drafts a first version, and you iterate. All within an interface designed to facilitate experimentation, testing, and iteration. This isn’t a change of job — it’s a change of pace: more fluid, more intuitive, and more business-logic driven.</p><p><strong>Example IDEs enabling vibe coding:</strong> <a href="https://getnao.io/">Nao</a>, <a href="https://cursor.com/">Cursor</a>, <a href="https://windsurf.com/editor">Windsurf</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qmxJ_9rmkQkfslgc" /></figure><h3>MCP</h3><p><strong>What is MCP?</strong><br> The Model Context Protocol (MCP) is an open standard that allows AI systems (such as copilots) to easily access an enterprise’s data — whether stored in Google Drive, Slack, GitHub, or a database. Instead of building a specific integration for every tool, MCP offers a single protocol to connect AI to all relevant sources.</p><p><strong>How does it work?</strong><br> The AI doesn’t connect directly to your tools or databases — it interacts with an MCP Server, which acts as a single entry point to access data.<br> The MCP Server standardizes the connection method through a universal protocol: it describes the available data in JSON format, exposes possible actions (read a file, query a table, etc.), and lets the AI understand what it can do, within what limits, and with what permissions. The AI reads this context file — often JSON enriched with schemas and metadata — which serves as an access map to various APIs or business resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*ezXxynvuwMVk0XOZ" /></figure><p>From there, the AI can:</p><ul><li>Read business objects or documents via REST or GraphQL APIs</li><li>Ask contextual questions (e.g., “What are the latest anomalies in this monitoring dashboard?”)</li><li>Generate custom queries (SQL, API calls, transformations) based on the exposed structure<br> All within a secure, controlled, and interoperable framework.</li></ul><p>More: <a href="https://modelcontextprotocol.io/overview">https://modelcontextprotocol.io/overview</a></p><h3>AI Agent</h3><p><strong>What is an AI Agent?</strong><br> An AI Agent is an autonomous AI system connected to your data and tools, capable of understanding a business requirement, retrieving the right data, transforming it, and delivering actionable results — without relying on fixed pipelines or pre-written transformations.</p><p>An agent activates specialized “micro-skills” (also called agents) depending on the task:</p><ul><li>A parsing agent reads raw files (CSV, JSON)</li><li>A cleaning agent handles missing values or duplicates</li><li>A transformation agent calculates aggregates or reformats data</li><li>An enrichment agent adds external data (e.g., geolocation)</li><li>A mapping agent aligns different data schemas</li></ul><p>These agents can work in coordination, like an autonomous team building a processing pipeline on the fly, adapting to context rather than following a rigid set of predefined rules.</p><h3>Self-Healing Pipelines</h3><p><strong>What are self-healing pipelines?</strong><br> These are data workflows capable of automatically detecting an error, identifying its probable cause, and executing a corrective action without human intervention.<br> It goes beyond basic alerting or monitoring — the pipeline can take action itself: restart a failed step, fix inconsistent values, switch to an alternate source, or disable a problematic transformation — while logging every action taken.</p><h3>Multi-Agent Platforms</h3><p><strong>What is a multi-agent platform?</strong><br> It’s an environment where multiple autonomous AI agents collaborate on a complex process, each responsible for a specific part.</p><p>In such a platform:</p><ul><li>Agents share a common objective defined by the user or system</li><li>Tasks are explicitly divided according to roles and expertise</li><li>Agents exchange intermediate outputs, such as partial results or shared context/data</li><li>Agents adapt their behavior based on others’ actions and results</li><li>A central orchestrator coordinates the process, tracks each agent’s state, and ensures progress toward the goal</li></ul><p>This can be asynchronous or sequential collaboration — but always with clear context and boundaries for each agent.</p><h3>What Will Change: layer-by-layer analysis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h7LzuFnCdpG0QOnQuZouFA.png" /><figcaption>6 layers in a Modern Data Platform</figcaption></figure><h3>Data Ingestion</h3><p>AI will change how ingestion pipelines are built through vibe coding. Smart tools will auto-generate code, validated by the developer.<br>To integrate into a Data Platform, this generated code must follow company-defined standards — otherwise, you’ll end up with a patchwork of inconsistent snippets. The Data Platform / DataOps / Center of Excellence team will be critical in defining and enforcing these standards.</p><p>Regarding MCP, I’m not convinced it will replace traditional ingestion methods. Its value isn’t in replacing ETL — it still essentially:</p><ol><li>Connects to a system and retrieves data</li><li>Transforms the data</li><li>Outputs it in the desired format</li></ol><p>Today’s Data teams favor ELT over ETL largely to centralize transformation in a semantic, “medallion”-style architecture that locks in business logic — born from iterations between business and tech. The modern data stack was built by isolating each layer for simplicity, flexibility, and scalability. A full return to ETL would be counterproductive.</p><p>However, software vendors (Salesforce, HubSpot, Google Analytics…) will adapt their products for MCP-like compatibility to improve UX — a clear competitive edge. This isn’t entirely new; APIs and GUIs already allow it. But MCP makes it more accessible to business users for ad-hoc queries — e.g., “Give me the average basket size for offer X.”</p><p>MCP also delivers fresh data with minimal latency, but it can’t lock in analyses or complex business rules (e<a href="https://blog.dataexpert.io/p/how-i-made-airbnb-millions-with-this">.g., Airbnb’s definition of “availability”</a>).<br> Data teams will need to compete with ad-hoc analyses by providing <strong>high-quality, fresh, high-value data with great UX</strong> — and ensure such analyses are supervised to avoid siloed chaos.</p><p><strong>AI-impacted KPIs:</strong></p><ul><li>Pipeline robustness → fewer incidents</li><li>Time to integrate a new data source</li><li>Code standardization</li></ul><h3>Transformation</h3><p>The goal is to make raw data reliable, understandable, and actionable for business teams — this is where KPIs are built.</p><p>Like ingestion, data transformation scripts will be greatly accelerated through vibe coding:</p><ul><li><strong>Exploration &amp; understanding:</strong> Before pipeline construction, AI can identify missing values, data distributions, and metadata, and answer natural-language questions like “Are there metadata documenting the source and types for each column?”</li><li><strong>SQL generation &amp; autocomplete:</strong> Task-specific LLMs can generate SQL from natural-language questions, optimize queries, and improve readability.</li></ul><p>Teams will use smart IDEs like Nao to generate DBT-style transformations. Producing code up to <a href="https://www.linkedin.com/posts/olaslett_we-can-ship-10x-faster-with-ai-were-also-activity-7353744784915587076-7H1m"><strong>10x faster</strong></a> (Olivier Laslett, CTO of Lightdash) will speed up business–Data iterations, enabling more personalized, impactful Data products with leaner teams.</p><p><strong>AI-impacted KPIs:</strong></p><ul><li>Development speed / Time-To-Insights</li><li>Quality of delivered analyses</li><li>Business team autonomy</li><li>Standardization of development practices</li></ul><h3>Storage</h3><p>The dominant Data Warehouse model — centralized, rigid, and expensive to adapt — has shown its limits with the growth of volumes, diversity of use cases, and the explosion of data sources, formats, and consumption patterns.<br> The emergence of the <strong>Data Lakehouse</strong> has already paved the way toward more unified platforms, able to handle both traditional analytics and ML/AI workloads, while offering much greater customization potential.</p><p>By 2025, many organizations still running SQL-based warehouses will hit a wall:</p><ul><li>Engines unfit for modern analytics and AI use cases</li><li>Poor developer experience</li><li>Lack of scalability</li></ul><p><strong>The core platforms powering Data &amp; AI projects will be:</strong></p><ul><li><strong>Snowflake</strong> — Cloud-native SaaS data platform at the heart of composable architectures, combining dynamic scalability, compute-storage separation, support for open formats like Iceberg, and hybrid integrations. Ideal for high-performance analytics with growing AI and real-time capabilities.</li><li><strong>Databricks</strong> — Lakehouse platform optimized for data science and ML, built for handling raw and semi-structured data, large-scale distributed training, and native Delta Lake integration. Favored for complex AI projects.</li><li><strong>ClickHouse</strong> — Ultra-fast, columnar analytics engine designed for interactive queries at massive scale. Ideal for real-time BI, monitoring, observability, and operational analytics with sub-second query times.</li><li><strong>Trino + Iceberg</strong> — A distributed query engine (Trino) paired with an open table format (Iceberg) to query data where it lives, without duplication — maximizing flexibility and interoperability while reducing costly data movement.</li></ul><p><strong>Key selection criteria:</strong></p><ul><li>Cost efficiency</li><li>Compliance (GDPR, AI Act, sensitive data handling)</li><li>Sustainability</li></ul><p><strong>AI-impacted KPIs:</strong></p><ul><li>Storage optimization</li><li>Storage cost per project</li><li>Business team autonomy</li></ul><h3>Orchestration</h3><p>Orchestration coordinates data processing scenarios — deciding execution order, checking dependencies, allocating compute (CPU, memory, GPU), monitoring progress, and handling errors to ensure workflows complete successfully. In modern, distributed architectures, orchestration is strategic to align ingestion, transformation, analytics, and AI workloads.</p><p><strong>Short term:</strong> Orchestrators will explain the root cause of incidents, making debugging easier for Data teams.</p><p><strong>Long term:</strong> Orchestrators will use AI to build <strong>dynamic</strong>, not static, processing flows. Agents will:</p><ul><li>Continuously analyze pipeline and system health</li><li>Detect anomalies proactively</li><li>Adjust priorities or resource allocation in real-time</li><li>Trigger self-healing actions</li></ul><p><strong>Examples of AI-powered orchestration features:</strong></p><ul><li><strong>Self-healing pipelines</strong> — Automatic diagnosis (source down, transformation failure, bad data), corrective action (rerun task, switch source, fix value, pause step), and full audit logging.</li><li><strong>DAG generation via vibe coding</strong> — Teams describe workflows in natural language, get auto-generated DAGs, plus optimization and readability suggestions.</li></ul><p><strong>Safeguards needed:</strong><br> A strong DataOps foundation with development standards, pre-approved DAG templates, CI/CD, and automated tests — ensuring generated pipelines remain reliable, observable, and environment-compatible.<br> Agents may even open pull requests and propose code reviews, but each org must define the right human oversight level.</p><p>It remains to be seen whether orchestrator vendors (Airflow, Dagster) will embed these AI capabilities in paid tiers or leave them to in-house development.</p><h3>Data Activation &amp; Analytics</h3><h4>Evolving analytics tools</h4><ul><li><strong>Legacy dashboard tools</strong> like Tableau, Power BI, Looker now add chat-based querying in natural language, returning both the answer and a relevant visualization. Some go further with AI copilots generating code for ad-hoc analysis.</li><li><strong>Next-gen analytics platforms</strong> like Hex or Mode enable cross-data exploratory analysis directly on the warehouse — shifting from descriptive queries (“List sales by country”) to explanatory/predictive (“Identify factors driving Spain’s Q4 sales”).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9Ib9ubFFrIrMPSL4" /><figcaption>HEX for self-service Analytics</figcaption></figure><h4>Levels of analytics autonomy (car analogy)</h4><ul><li><strong>Level 0 — No automation</strong>: user must know the data model and write queries manually</li><li><strong>Level 1 — Query automation</strong>: AI generates queries from natural language</li><li><strong>Level 2 — Guided insights</strong>: AI suggests relevant visualizations or analysis paths</li><li><strong>Level 3 — Proactive analytics</strong>: AI flags anomalies, trends, or opportunities without prompts</li><li><strong>Level 4 — Autonomous decision support</strong>: AI recommends actions based on all available data and business goals</li></ul><p><strong>Caveat:</strong> These tools are still maturing, and LLM-based reasoning for advanced analytics is not yet fully proven.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/0*-NGnzIkRrZwHxNPk" /></figure><h4>Prerequisites for impact</h4><p>A solid <strong>semantic layer</strong>:</p><ul><li>Central, manageable scope with core KPIs</li><li>Cleaned, pre-calculated, documented data for quality and consistency</li><li>Rich business definitions, lineage, and calculation methods for correct interpretation</li></ul><p>Self-service BI adoption depends heavily on breaking complexity barriers. AI-driven analytics could remove these roadblocks, giving business teams unprecedented autonomy — if the semantic layer is robust.</p><p><strong>AI-impacted KPIs:</strong></p><ul><li>Business team autonomy</li><li>Data quality</li><li>Data team impact</li></ul><h3>Governance &amp; Observability</h3><p>Governance defines the rules and mechanisms for effective, reliable, maintainable data management, across:</p><ul><li><strong>Business</strong> — clear KPI definitions to avoid ambiguity</li><li><strong>Strategic</strong> — alignment between data use and company priorities</li><li><strong>Legal</strong> — GDPR/AI Act compliance and confidentiality</li><li><strong>Sustainability/Ethics</strong> — responsible collection, storage, and usage</li><li><strong>Technical</strong> — best practices for architecture, security, quality, and performance</li></ul><p>Observability measures and explains system health and data quality — going beyond monitoring to quickly answer <em>why</em> a pipeline failed, <em>where</em> data got corrupted, and <em>how</em> to fix it.</p><p><strong>AI transformations in governance &amp; observability:</strong></p><ol><li><strong>Faster, more transparent log analysis</strong> — Natural-language queries like “Show the last three critical errors for pipeline X and probable causes.” AI agents consolidate logs, pinpoint failures, propose fixes, and speed root cause analysis. Security teams can also query logs to check data access patterns.</li><li><strong>Self-healing pipelines</strong> — Automatically detect, diagnose, fix, and log incidents — reducing downtime and repetitive manual fixes.</li><li><strong>Natural-language quality test generation</strong> — Describe a rule (“Check monthly sales totals match the billing table within 1% tolerance”), and AI generates relevant, maintainable tests.</li><li><strong>Dynamic, auto-completed documentation</strong> — Automatically extract and update metadata, maintain business glossaries, lineage docs, and live data catalogs.</li></ol><p>This shifts governance/observability from:</p><ul><li>Reactive control → Continuous monitoring</li><li>Manual documentation → Living documentation</li><li>Reactive monitoring → Preventive management</li></ul><p><strong>AI-impacted KPIs:</strong></p><ul><li>Data quality improvement</li><li>Business team data adoption</li><li>Compliance with legal/security standards</li></ul><h3>Conclusion</h3><p>GenAI does not simply introduce new tools — it redefines the architecture and functioning of Data Platforms from the ground up. Each layer — ingestion, transformation, storage, orchestration, analytics, governance — is being reshaped by copilots, autonomous agents, and dynamic approaches. The potential benefits are significant: fewer manual efforts, faster time-to-insight, and greater business autonomy.</p><p>Yet these gains come with equally structural challenges. The financial and energy costs of intensive AI use will increase, requiring stronger <strong>FinOps</strong> practices and sustainability considerations. And while Data teams will be able to “do more with less,” this will only be viable if underpinned by a <strong>robust DataOps foundation</strong> that enforces standards, governance, and coherence. Otherwise, the risk is building unstable, costly platforms — or worse, a new technological “spaghetti factory.”</p><p>In short, this first GenAI-driven revolution places Data teams before a dual imperative: harness AI’s power to boost impact, while keeping complexity and sustainability under control.</p><p>In the next article in this series — <strong>Impact of AI on Data Teams (2/3): Shifting roles and responsibilities</strong> — we will explore how these transformations directly affect team organization and skillsets, and what capabilities will be needed to thrive in this new ecosystem.</p><p>For our French readers who want to know more about Modeo, check out our blog: <a href="https://www.modeo.ai/articles">https://www.modeo.ai/articles</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a9394b7f46d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/genai-for-data-teams-future-ways-of-working-shifting-roles-and-limits-3a9394b7f46d">GenAI for data teams: future ways of working, shifting roles and limits</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lancement de l’AWS Community Builders Program]]></title>
            <link>https://medium.com/@modeoAI/lancement-de-laws-community-builders-program-600c1423c152?source=rss-86eaa22b0b15------2</link>
            <guid isPermaLink="false">https://medium.com/p/600c1423c152</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[saas]]></category>
            <dc:creator><![CDATA[Matthieu Rousseau]]></dc:creator>
            <pubDate>Tue, 11 Aug 2020 12:36:24 GMT</pubDate>
            <atom:updated>2020-08-11T12:40:04.273Z</atom:updated>
            <content:encoded><![CDATA[<p>…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EQiVtcsyYtTRjJ6qQZkyww.png" /></figure><p>Amazon Web Services (AWS), division du groupe Amazon spécialisée dans les services de cloud computing pour les entreprises et les particuliers, a annoncé le lancement le 30 juillet dernier de l’AWS Community Builders Program, un programme proposant des ressources techniques, des programmes de mentorat et des opportunités de networking à des membres de la communauté AWS. Parmi ces membres, Modeo AI est la première entreprise en France à rejoindre la phase bêta du projet.</p><h3><strong>Un leader du cloud computing</strong></h3><p>Utilisée par des millions de clients et des milliers de partenaires à travers le monde, AWS est une plateforme cloud lancée en 2006 et permettant la mise à disposition de ressources informatiques à la demande via Internet. Elle permet aux startups, entreprises et organisations du secteur public d’accéder à des services technologiques (puissance de calcul, stockage, bases de données, analyse/reporting, Machine Learning, etc.) sans avoir besoin de posséder des serveurs et des centres de données physiques.</p><p>Bénéficiant de la première communauté de clients et partenaires parmi lesquels Expedia, Shell, Airbnb, Lyft ou encore l’Agence américaine des produits alimentaires et médicamenteux (FDA), AWS dispose de l’infrastructure cloud la plus complète, fiable et sécurisée couvrant 7 zones dans 24 régions géographiques à travers le monde.</p><h3><strong>La transformation du modèle d’exploitation des services informatiques</strong></h3><p>Véritable révolution dans la gestion des services informatiques, le cloud computing permet en particulier aux ETI/PME de bénéficier de réduction de coûts en externalisant le stockage et les environnements de travail sans avoir à investir dans le développement de logiciels ou l’achat de matériel informatique. Les entreprises ont de plus accès à des outils évolués à des prix abordables grâce à une tarification basée sur l’utilisation réelle. Elles peuvent ainsi se focaliser sur leur activité et travailler dans un environnement sécurisé qui offre une plus grande flexibilité et mobilité à leurs collaborateurs. En 2020, 78% des petites entreprises devraient ainsi avoir recours au cloud pour héberger leurs environnements informatiques et rester compétitives dans un univers de plus en plus concurrentiel.</p><p>Dans un marché mondial des services de cloud public en croissance de près de 6,3% en 2020 et qui devrait atteindre plus de 364 milliards de dollar en 2022 selon Gartner, le cloud computing apparaît comme la nouvelle alternative répondant à la demande croissante des entreprises pour des services réellement adaptés à leurs besoins et utilisation et facilitant le travail collaboratif, en particulier dans le contexte de la crise liée à la pandémie de Covid-19.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bi-DNuHDx1oFuUt--S1k9Q.png" /></figure><p><em>Source : Gartner</em></p><h3><strong>Les community builders</strong></h3><p>Avec son programme AWS Community Builders<em>, </em>Amazon regroupe des membres de la communauté AWS. Ces <em>community builders,</em> sélectionnés parmi des membres actifs de la communauté à travers le monde proposant du contenu (contributions open source, articles, blogs, etc.), sont répartis dans différentes catégories et thèmes (Machine Learning, Stockage, Data, etc). Ils bénéficient ainsi d’un accès exclusif à des webinars avec les ingénieurs AWS en charge du développement des nouveaux produits et services, de crédits promotionnels AWS pour la création de contenu, de systèmes de mentorat avec des experts AWS ainsi que d’opportunités de networking.</p><p>L’ambition de ce programme est de développer et d’organiser la communauté des utilisateurs autour des solutions AWS et de mettre un place un mécanisme d’échange et de partage d’information au sein de cette communauté et avec les ingénieurs du géant américain.</p><p><strong>A propos de nous : </strong>Modeo AI est une société de développement de solutions intégrant de l’Intelligence Artificielle à destination des PME &amp; ETI. Nous sommes experts en IA, développement logiciel et data engineering.</p><ul><li>Site web : <a href="http://modeo.ai/">https://modeo.ai/</a></li><li>Adresse : 7 rue Piccini, 75116, Paris France</li><li>Linkedin : <a href="https://www.linkedin.com/company/modeo-ai/">https://www.linkedin.com/company/modeo-ai/</a></li><li>Twitter : <a href="https://twitter.com/modeoai">https://twitter.com/modeoai</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=600c1423c152" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can AWS Sagemaker AutoPilot be used to predict the outcome of a tennis match?]]></title>
            <link>https://medium.com/@modeoAI/can-aws-sagemaker-autopilot-be-used-to-predict-the-outcome-of-a-tennis-match-127d62fe98e1?source=rss-86eaa22b0b15------2</link>
            <guid isPermaLink="false">https://medium.com/p/127d62fe98e1</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[sagemaker]]></category>
            <category><![CDATA[automl]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Matthieu Rousseau]]></dc:creator>
            <pubDate>Fri, 17 Jul 2020 23:18:20 GMT</pubDate>
            <atom:updated>2020-07-31T07:55:46.686Z</atom:updated>
            <content:encoded><![CDATA[<h3>Can Amazon Sagemaker AutoPilot be used to predict the outcome of a tennis match?</h3><p>How Sagemaker AutoPilot works on real life use case ? How to take full advantage of Sagemaker AutoPilot? Are the performances as good as those obtained by a datascientist? We will answer these questions in this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*730i02vP40NfteLa.jpeg" /></figure><ol><li>Introduction to AutoPilot</li><li>The use case : Machine Learning &amp; Tennis</li><li>AutoPilot without any preprocessing</li><li>AutoPilot with processed dataset</li><li>Conclusion</li></ol><h3><strong>I. Introduction to AutoPilot</strong></h3><h3>What is SageMaker AutoPilot ?</h3><p>Auto-ML (AutoPilot) is an integrated tool in SageMaker, announced by AWS at Reinvent:2019 which we were able to attend. The main goal of Auto-ML is to simplify model creation to a point where it can be done by non-technical people. We just provide SageMaker with a dataset, specify the type of problems and the target attribute. Auto-ML will be able to build models accomplishing the given task.</p><h3>How SageMaker AutoPilot works ?</h3><p>Auto-ML perform three different actions when an experiment is started :</p><ol><li>Data exploration</li><li>Candidate selection</li><li>Model Tuning</li></ol><h4>Data exploration</h4><p>Auto-ML will make some basic analysis on the provided dataset such as statistical analysis or NaN values handling. The objective of this part is to understand what the data represents (continuous values, discret values, categorical values etc).</p><h4>Candidate selection</h4><p>Once SageMaker understood data, it can select what it calls candidates. Each candidate is a combination of a preprocessing script and a Machine Learning Model. SageMaker will select a bunch of candidates based on the observation it has done on the data. The Machine Learning models can be for example xgboost or logistical regression. Preprocessing scripts are combinations of previously developed preprocessing strategy such as RobustImputer (which converts missing values to nan), ThresholdOneHotEncoder or RobustStandardScaler (all of those scripts can be found on Amazon SageMaker public Github).</p><h4>Model Tuning</h4><p>Now that SageMaker have selected candidate, it will lunch some fine tuning tasks on those candidates. Basically it will simultaneously run models training with slightly different hyper parameters combinations.</p><h4>Model deployment</h4><p>One very interesting thing with Auto-ML is the model deployment. You can deploy your model in 1 click and AutoPilot returns an endpoint reachable via a REST API.. You can chose the type of instance you want to use and how many of them. This is where we see the real power of connected AWS services.</p><h3>II. The use case : Machine Learning &amp; tennis</h3><p>You can easily find on the web AutoML example using really basic datasets or what we commonly call the “Hello World”s datasets (Titanic Dataset, MNIST, iris-dataset). The question is : how it works on real life use case. At Pronoo.ai we are working on sports results prediction such as tennis matches.</p><h4>Our data</h4><p>We achieved a huge data scrapping on tennis matches from 1970 to 2020 and records on the first 250 best tennis players. Today we have around 75000 matchs records compiled with both players stats.</p><p>Here is the dataset we use (see Github link for detailed dataset -&gt; <a href="https://github.com/matrousseau/AWS-AutoPilot-on-real-life-use-case-">https://github.com/matrousseau/AWS-AutoPilot-on-real-life-use-case-</a>) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9johbX9bhKN0ifdB19E0kQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*khaGEZxAClI1Ai6m2HkM1Q.png" /></figure><p>To evaluate model performance, we are training our models on all the matches before 2020, and we test our models on 2020’s matches. With ou self-made model et data pipeline we can reach 76.6% of accuracy on our test set. The goal here is to see if AutoPilot can beat that score or at least reach it.</p><h3>III. AutoPilot without any preprocessing</h3><p>For this first experience, we use our raw dataset, without any processing. So we don’t have to do the feature selection phase and we don’t have to deal with the features engineering phase either.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a6hssHetZmoqirt3r4fnlQ.png" /></figure><p>As you can see, the results are very surprising. The precision is almost 100% on the training and validation set (remember that with our own models we get 76.6% accuracy).</p><p>When we deploy the model and test with a test set, the accuracy is 50%. Even worse, when we make a prediction with training data the accuracy is 45%.</p><p><strong>Our interpretation : </strong>For a large number of columns and no feature engineering, AutoPilot has difficulty converging models. It is therefore preferable to make your own feature selection of columns before the training.</p><h3>IV. AutoPilot with processed dataset</h3><p>Auto-ML didn’t succeed in understanding our scratch dataset. This is mainly due to the fact that all of the features are duplicated (as there is two players in a match). Let’s see if we can help SageMaker understand beter the problem.</p><h4>Data preprocessing</h4><p>To help SageMaker we will perform a bunch of hand made pre-processing :</p><ul><li>Remove useless features</li><li>Convert columns to numerical values</li><li>One hot encode categorical columns</li><li>Compute difference between both players on their stats</li></ul><p>We know have a much cleaner dataset :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DZXO6vo8lxFetAp2SmO2Sw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qyChJSQNme2Xc-SGc4P-PA.png" /></figure><p>With this version of the dataset, AutoPilot is able to build models that reach 73% (maximizing accuracy) and 74% (maximizing F1-score) of accuracy on predicting the games of 2020..</p><h4>Scaling data</h4><p>We can know try to add a scaller to our own pre processing to see if it helps Auto-ML increasing its score. The answer is no, actually it’s pretty much the opposite. Adding a scaller is not recomended as Auto-ML will most likely select a candidate with a Scaller inside. Scalling already scalled datas with a different methode will result in weird value distribution and bad results on our model.</p><h3><strong>V. Conclusion</strong></h3><p>Finally, it appear that AutoPilot is able to understand pretty well complexe problems. But hand crafted pre processing still have a valuable impact on results.</p><p>The use of AutoPilot also shows how important it is to use a human eye and business logic on feature selection. For the purely technical part, the tool is extremely powerful and saves a lot of time. It is therefore a tool to use when you have a very short time to market and a very good knowledge of your data.</p><p>Co-authored by Jérémie Basso (<a href="https://www.linkedin.com/in/j%C3%A9r%C3%A9mie-basso-498144165/">https://www.linkedin.com/in/j%C3%A9r%C3%A9mie-basso-498144165/</a>)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=127d62fe98e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[L’impact du Covid-19 sur la démocratisation des agents conversationnels intelligents]]></title>
            <link>https://medium.com/@modeoAI/limpact-du-covid-19-sur-la-d%C3%A9mocratisation-des-agents-conversationnels-intelligents-99da38e7ca31?source=rss-86eaa22b0b15------2</link>
            <guid isPermaLink="false">https://medium.com/p/99da38e7ca31</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[speech-to-text-api-market]]></category>
            <dc:creator><![CDATA[Matthieu Rousseau]]></dc:creator>
            <pubDate>Wed, 15 Jul 2020 15:57:20 GMT</pubDate>
            <atom:updated>2020-07-16T09:39:01.235Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VdOrCTTLU8c3cgsemcfqFA.png" /></figure><p>La crise du COVID-19 s’est accompagnée d’un nombre d’appels record vers les hôpitaux (+ 160 %). Cet important flux d’appels a été éreintant pour le SAMU et pour les malades qui pouvaient attendre jusqu’à une heure avant de trouver un interlocuteur.</p><p>Les régulateurs du SAMU ont dû suivre une procédure de premier contact basique et répétitive consistant à poser des questions prédéfinies. La majorité de ces questions requéraient une réponse binaire : “oui” ou “non”.</p><p>Exemple de question :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rFUj1Vgv3DBWFO82wxkxuA.png" /><figcaption>Source: <a href="https://www.maladiecoronavirus.fr/se-tester">https://www.maladiecoronavirus.fr/se-tester</a></figcaption></figure><p>Dans un contexte nécessitant des délais de réponse extrêmement réduits, l’intelligence artificielle se présente comme une solution permettant aux professionnels de la santé de réduire leur temps passé sur des tâches répétitives et de se concentrer sur la prise en charge des patients infectés.</p><p>Nous verrons dans un premier temps quelles initiatives ont été mises en place pour simplifier les premiers contacts avec les patients à risque, puis nous proposerons notre propre approche à ce problème, basée sur le connecteur Modeo Speech-To-Text.</p><p><strong>Les initiatives mises en place par l’Etat</strong></p><p>Dès mars, l’OMS a mis en place un chatbot sur WhatsApp. L’Etat français s’est lui aussi doté de plusieurs chatbots issus d’éditeurs privés : Covid-bot, AlloCovid, ISA, Chatbot TPLS Covid.</p><p><strong>COVID-BOT</strong> est le chatbot du gouvernement français. Il utilise des règles décisionnelles basiques et il est donc difficile dans ce cas de parler d’IA. Nous avons testé son autodiagnostic. La conversation est guidée par le bot, qui vous pose au minimum 21 questions. Les huit premières questions concernent vos symptômes, le reste votre état de santé générale afin de déceler vos risques de comorbidité. Après avoir recueilli suffisamment de données pour un premier diagnostic Covid-19, le questionnaire vous demande votre code postal. L’objectif avec cette dernière question est d’associer la probabilité qu’une personne est le Coronavirus et la zone géographique afin de prédire la formation d’un nouveau cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J2JPntMdl1NDdciP" /><figcaption>Covid-Bot</figcaption></figure><p><strong>Notre avis : </strong>CovidBot est intéressant pour sa simplicité de mise en oeuvre. En effet, il est relativement simple de créer un agent conversationnel basé sur des règles décisionnelles. Cependant, il est dommage d’utiliser une interaction manuscrite car elle rend d’une part l’expérience utilisateur plus compliquée auprès des personnes âgées. D’autre part, elle ne permet pas au patient d’évoquer toutes les informations qu’il souhaite transmettre à l’agent conversationnel.</p><p><strong>AlloCovid</strong> est un bot vocal, issu d’une coopération entre partenaires académiques et privés : l’Inserm et l’université de Paris, e.Voyageurs SNCF et Allo Media. Il s’agit du premier cas d’utilisation de Chatbot vocal au service de la santé publique selon Xavier Jouven, porteur du projet, qui dirige l’unité Inserm Epidémiologie intégrative des maladies cardiovasculaires.</p><p>C’est un service gratuit, disponible 7j/7, 24h/24 et capable de traiter 1000 appels en parallèle.</p><p>Le bot pose des questions au patient puis utilise un système de compréhension du langage naturel (NLP) pour analyser les réponses et les classifier selon leur gravité. Après avoir posé une succession de questions et analysé chaque réponse, AlloCovid vous donne un diagnostic rapide, vous incitant ou non à consulter un médecin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*SmNSXwXMrd0bpGp6.png" /></figure><p><strong>Notre avis : </strong>AlloCovid possède toutes les caractéristiques pour se substituer à un médecin pour un premier contact. Cependant, il est difficile de savoir à quelle point le modèle de transcription est performant, quand on voit la simplicité des questions posées. On regrette aussi un petit tutoriel de pré-diagnostic pour expliquer au patient comment fonctionne l’agent conversationnel et comment lui parler.</p><p><strong>Le concept d’agent conversationnel que nous proposons</strong></p><p>L’intelligence artificielle (IA) permet d’analyser n’importe quel type de données y compris les données vocales. En partant d’un fichier audio, nous pouvons :</p><ul><li><strong>Convertir des audios en texte </strong>: permet d’archiver les appels en les anonymisant pour rétrospectivement en extraire des informations pertinentes sur des gros volumes de données (tendance au niveau des symptômes, âge moyen des patients etc…), anticiper des signes précurseurs.</li><li><strong>Mots/expressions clés</strong> : possibilité d’extraire d’un appel des données importantes : âge de la personne, symptômes, date d’apparition des symptômes etc…</li><li><strong>Niveau d’affaiblissement/émotion : </strong>analyse du ton de la voix pour en déduire le niveau d’affaiblissement et l’émotion du patient.</li><li><strong>Génération d’une réponse adaptée :</strong> Association de la question et des mots/expressions identifiés à la réponse la plus adaptée pour ensuite orienter le patient vers un médecin ou lui recommander de rester chez lui.</li><li><strong>Classification des patients : </strong>Segmenter les patients selon les niveaux de symptôme pour concentrer le travail des médecins sur les cas les plus importants.</li></ul><p>Le schéma ci-dessous illustre la façon dont nous pourrions automatiser le traitement des appels téléphoniques grâce à l’IA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*dD5EmQqwQ_hmFGZ7r2VBqQ.png" /><figcaption>Exemple d’architecture d’agent conversationnel intelligent en réponse au Covid-19</figcaption></figure><p><strong>Fonctionnement de la solution</strong></p><ol><li>Les appels des patients sont réceptionnés via un système d’acquisition et de traitement en temps réel d’audio, comme AWS Connect (<a href="https://aws.amazon.com/fr/connect/">https://aws.amazon.com/fr/connect/</a>).</li><li>Un <em>Load Balancer</em> redirige le flux d’appels entrant vers des machines virtuelles utilisées pour transcrire la voix en texte avec le model Speech-To-Text Modeo puis analyser le niveau d’affaiblissement et d’émotion de la voix.</li><li>Une fois l’audio converti en texte, on applique à la transcription le modèle custom Modeo de NLP spécialement entrainé sur des données médicales, pour extraire les mots clés : fièvre, symptômes, âge etc…</li><li>On utilise ensuite un modèle de détection d’informations sensibles pour anonymiser le texte, puis on fait un chiffrement des données avant de les envoyer dans une base de données sécurisée.</li><li>Enfin, on prédit la réponse la plus adaptée, basée sur la transcription et un ensemble de réponses possibles. Il ne reste plus qu’à envoyer la réponse la plus adaptée avec un système de Text-To-Speech.</li></ol><h3><strong>Conclusion</strong></h3><p>La gestion des premiers contacts au téléphone avec les patients est un enjeu majeur dans la lutte contre le Covid-19. D’une part, ils mobilisent du personnel médical qualifié sur des tâches répétitives et relativement simples. D’autre part, ils sont une source d’informations fiables pour les patients, limitant ainsi la propagation de Fake News. C’est également une source d’informations pour les médecins. En effet, ces échanges contiennent de nombreuses données, extrêmement utiles pour suivre l’évolution de l’épidémie et mieux comprendre le virus</p><p>Lors de la première vague du Covid-19, plusieurs initiatives ont été mises en place par l’Etat et par des centres de recherche privés ou publics pour simplifier la prise en charge des patients utilisant différents agents conversationnels. Ces systèmes ont su faire leurs preuves, en comptant plusieurs centaines de milliers d’utilisateurs. Il s’agit d’un premier pas vers la démocratisation de l’utilisation des agents conversationnels dans notre société, dans le secteur de la santé ou autres (retail, transport, télécommunications, e-commerce etc…).</p><p>Cependant, les agents déployés pendant la première vague restent relativement basiques et de nombreuses autres fonctionnalités auraient pu être développées pour tirer pleinement profit de ces agents conversationnels (analyse de l’affaiblissement, segmentation des patients les plus à risque, prédiction de la formation d’un nouveau cluster etc…).</p><p><strong>A propos de nous : </strong>Modeo AI est une société de développement de solutions intégrant de l’Intelligence Artificielle à destination des PME &amp; ETI. Nous sommes experts en IA, développement logiciel et data engineering.</p><ul><li>Site web : <a href="http://modeo.ai/">https://modeo.ai/</a></li><li>Adresse : 7 rue Piccini, 75116, Paris France</li><li>Linkedin : <a href="https://www.linkedin.com/company/modeo-ai/">https://www.linkedin.com/company/modeo-ai/</a></li><li>Twitter : <a href="https://twitter.com/modeoai">https://twitter.com/modeoai</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99da38e7ca31" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 tips to make your data science code cleaner and more efficient]]></title>
            <link>https://medium.com/data-science/10-tips-to-make-your-data-science-code-cleaner-798ba398fcbe?source=rss-86eaa22b0b15------2</link>
            <guid isPermaLink="false">https://medium.com/p/798ba398fcbe</guid>
            <category><![CDATA[ds-in-the-real-world]]></category>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Matthieu Rousseau]]></dc:creator>
            <pubDate>Mon, 27 Apr 2020 11:57:32 GMT</pubDate>
            <atom:updated>2020-04-30T15:46:26.975Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/towards-data-science/data-science-in-the-real-world/home"><strong>Data Science in the Real World</strong></a></h4><h4>Most Moocs and university courses don’t teach you how to professionally code in Python, this article will give you some tips on how to do it.</h4><p>[Note: This article is regularly updated by following your feedback. Feel free to add your remarks in the comments.]</p><p>First of all, I would like to note that this list is not exhaustive but that it is the result of my many experiences and reading :</p><ul><li>My experiences working on data science projects, doing code review and developing Machine Learning pipelines in which many data scientists were involved (Data scientist / Co-Founder at Modeo.ai, CTO at Pronoo.ai, Freelance data scientist and AiLab Leader at GarageISEP).</li><li>My work on the AWS DeepRacer, the autonomous car championship organized by Amazon Web Services, where I won the French edition and participated in the world final in Las Vegas in December 2019 (see: <a href="https://medium.com/@matthieurousseau_16015/what-is-the-aws-deepracer-league-and-how-did-we-win-the-french-edition-facae656680c">https://medium.com/@matthieurousseau_16015/what-is-the-aws-deepracer-league-and-how-did-we-win-the-french-edition-facae656680c</a>)</li><li>The books I’ve read covering this subject (I highly recommend):</li></ul><ol><li>Robert C. Martin. <em>Clean Code: A Handbook of Agile Software Craftsmanship</em>. Pearson, 2009.</li><li>Aurélien Géron. <em>Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems</em>. O’Reilly, 2017.</li><li>Wes McKinney. <em>Python for Data Analysis</em>. O’Reilly, 2017.</li><li>Robert C. Martin. <em>Clean Architecture: A Craftsman’s Guide to Software Structure and Design</em>. Pearson, 2017.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U6Xf_MEn3MPdra95.png" /><figcaption>Robert C. Martin. <em>Clean Code: A Handbook of Agile Software Craftsmanship</em>. Pearson, 2009, <a href="https://www.amazon.fr/">https://www.amazon.fr/</a>.</figcaption></figure><p>Despite my years spent in French university learning data science, most of my knowledge was acquired through online courses on Coursera and Udemy (I guess I’m the cliché of the modern data scientist but thanks Andrew Ng). However, I have to say that there is a huge gap between the way code is taught in online courses and how the code has to be developed to be used in real-world applications.</p><p>I don’t pretend to say that my way of coding is the best and I don’t think that coding properly is an exact science either, but by following these 10 tips, you can considerably improve the readability of your code.</p><h3><strong>1. One level of abstraction per function</strong></h3><p>To make the code more readable, each function must be dedicated to a single task. It is better to have a lot of functions rather than functions that are too long, do several things and are therefore confusing. To be sure that a function does only one thing, it is important to limit the function to a single level of abstraction.</p><p>What is a level of abstraction? The purpose of this function is to retrieve data from a database and then apply preprocessing to the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AV2vrAuX1sPc9GxNq3VbuQ.png" /></figure><p>In the image below you can see three levels of abstractions. This is actually about the code level. Do we treat a “High level” function as process_data_from_db, do we directly manipulate a Pandas DataFrame like renaming columns?</p><p>Here’s what we can do to make this code more readable :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kUV4kGW6ujVnFc0oBCOdSA.png" /></figure><h3><strong>2. Use type hints to improve readability</strong></h3><p>Since our goal is to manipulate data, it is usual for our functions to take different data types as arguments (list, tuples, dict, Pandas DataFrame, Numpy Array etc…). It is therefore mandatory to specify the type of each argument as well as the type of the returned objects.</p><p>To do this, it is recommended to use hints to specify the type of each argument and each returned object. This makes the code easier to read and maintain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nDwKZ_RF7HWghBVmnrAXug.png" /></figure><p>See <a href="https://www.bernat.tech/the-state-of-type-hints-in-python/">https://www.bernat.tech/the-state-of-type-hints-in-python/</a></p><h3>3. <strong>Arguments</strong></h3><p>Arguments should be limited as much as possible. Surprising, right? Let me tell you why.</p><p>There are several types of functions, niladic (0 argument), monadic (1 argument), dyadic (2 arguments), triadic (3 arguments) and polyadic (more than 3 arguments). Monadic are acceptable, dyadic should be avoided and more than 3 arguments are way too much.</p><p>The main reason is that they take a lot of conceptual effort and are therefore hard to understand, especially in pipelines with complex workflows. A second reason concerns unit tests. When you perform unit tests on your function, you have to test all possible combinations between each argument, which makes this phase of testing much more complex.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zPryneaVeguo_5YBD5prPA.png" /></figure><h3><strong>4. Blocks and indenting</strong></h3><p>The functions must be small. No one wants to have to look for an error in a 2000-line function. To make the function readable, try to have a minimum of lines for each statement (if, else, while…).</p><p>Ideally, try to keep only one line for each statement. To be as clear as possible, this line can be used to call a function with an explicit name. When you don’t have a choice, try to limit yourself to a maximum of two levels of indentation so as not to affect the readability of the code.</p><p>Also, Python allows loops to be made in a single line, which makes for greater clarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*diqTvEsqQ7DJU2Nlq7dPrw.png" /></figure><h3><strong>5. Error handling</strong></h3><p>Error management is not the most exciting part of developing data science pipelines, but it is essential. It will help you to supervise the execution of your pipelines and make the right decisions in case of error.</p><p>First of all, a good reflex is to start your top-level scripts with a try, except, finally, as they define a scope for the execution of your program. If the execution failed, it will resume in the catch so you will be able to take the right decision depending on the type of the error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C4nDyH8tB7X6Y6L19GjxJA.png" /></figure><p>Another way to make the most of exceptions is to develop your own.</p><p>Imagine that you are retrieving data via an API. Most likely, you don’t have control over what the API returns to you, which can impact the proper execution of your program.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yM7Hm6kduXIzMBg2Uom7aw.png" /></figure><p>See : <a href="https://jeffknupp.com/blog/2013/02/06/write-cleaner-python-use-exceptions/">https://jeffknupp.com/blog/2013/02/06/write-cleaner-python-use-exceptions/</a></p><h3>6. <strong>Use logger, not print</strong></h3><p>Prints are good for debugging and testing but when it comes to putting your code into production, a logger is much more appropriate.</p><p>Prints are more complicated to format, less readable and above all contain little information. On the other hand, loggers contain more information, are directly formatted and have a level of information that makes it possible to identify the importance of the information (INFO, WARNING, ERROR etc.).</p><p>On the image below you can see the display of the execution of a training script. See <a href="https://docs.python.org/3/howto/logging.html">https://docs.python.org/3/howto/logging.html</a> for more details about logging in Python.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GSUH0_LkxwpxkGJOZVXmtQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nOPHoOUqkG3WKeSaLm56tQ.png" /></figure><h3>7. <strong>Docstring</strong></h3><p>Docstring is an essential element in Python. It must appear for any statement declaration (function, class, or method definition). They make the code more explicit and serve as documentation for the object.</p><p><em>“A universal convention supplies all of maintainability, clarity, consistency, and a foundation for good programming habits too. What it doesn’t do is insist that you follow it against your will. That’s Python!”</em></p><p><em>— Tim Peters on comp.lang.python, 2001–06–16</em></p><p>There are two types of docstrings, single line docstring and multiple lines docstring.</p><p><strong>Single-line docstring</strong>s are used for the most obvious cases, they fit on a single line. The description is concise and provides additional information to that provided by the name of the function. As stated in Python documentation :</p><ul><li>Triple quotes are used even though the string fits on one line. This makes it easy to later expand it.</li><li>The closing quotes are on the same line as the opening quotes. This looks better for one-liners.</li><li>There’s no blank line either before or after the docstring.</li><li>The docstring is a phrase ending in a period. It prescribes the function or method’s effect as a command (“Do this”, “Return that”), not as a description; e.g. don’t write “Returns the pathname …”.</li><li>The one-line docstring should NOT be a “signature” reiterating the function/method parameters (which can be obtained by introspection).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RdsWMzWGvmG6qQ-Ae7lAZw.png" /></figure><p><strong>Multi-lines docstring</strong>s are used for more complex cases. The description is more elaborated and it’s followed by a blank line. It is important that it fits on one line and is separated from the rest of the docstring by a blank line. The summary line may be on the same line as the opening quotes or on the next line. The entire docstring is indented the same as the quotes at its first line (see example below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rVITV_lJUKulus2As3cUJg.png" /></figure><p>See : <a href="https://www.python.org/dev/peps/pep-0257/#abstract">https://www.python.org/dev/peps/pep-0257/#abstract</a></p><h3>8. Unit test</h3><p>Python testing is a very broad topic and can be very complex, but they’ re essential for making your python code cleaner and more robust.</p><p><em>“In computer programming, </em><strong><em>unit testing</em></strong><em> is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use”. </em>Kolawa, Adam, Huizinga, Dorota (2007). <a href="http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470042125.html"><em>Automated Defect Prevention: Best Practices in Software Management</em></a>. Wiley-IEEE Computer Society Press. p. 75.</p><p>A good tip for effective testing is to use the three Laws of TDD and the F.I.RS.T principle (which apply to Python as well as other programming languages).</p><p><strong>The Three Laws of TDD:</strong></p><ol><li>You can’t write any production code until you have first written a failing unit test.</li><li>You can’t write more of a unit test than is sufficient to fail, and not compiling is failing.</li><li>You can’t write more production code than is sufficient to pass the currently failing unit test.</li></ol><p><strong>F.I.R.S.T</strong></p><ol><li><strong>Fast</strong>: Test should be fast and run quickly, as you have to test them frequently. Otherwise, you will not use them to their full potential as they will waste your time.</li><li><strong>Independent: </strong>Test should not depend on each other, they should not set up conditions for the next test. You should be able to run them in the order you want.</li><li><strong>Repeatable: </strong>The tests must be able to be run in any environment, whether in production phase or in prototyping, on any machine. Thus, the environment used does not impact the success of the tests.</li><li><strong>Self-Validating: </strong>The output of the test should be boolean. Either the test is passed or it is not passed but there is no intermediary. Otherwise, passing the tests becomes subjective.</li><li><strong>Timely: </strong>Unit tests must be developed at the right time, i.e. during the test phase and not in production.</li></ol><p>See: <a href="https://docs.python.org/3/library/unittest.html">https://docs.python.org/3/library/unittest.html</a></p><h3>9. <strong>Comments</strong></h3><p>Python comments are very useful when used properly, but can be a major source of confusion and over-information when misused. We use them when the code we develop is not explicit enough which is normal as it is not always possible to explain clearly what our code does.</p><p>There are a few types of <strong>good</strong> comments :</p><ol><li><strong>Legal comments</strong>: The Copyright, the license etc…</li><li><strong>Informative comments</strong>: When you need to specify the format of a date you’re parsing.</li><li><strong>Intent explanation</strong>: The indentations often reflect a complex logic, it can be used to briefly explain what they are doing.</li><li><strong>Clarification: </strong>Your code can be hard to understand. It is sometimes useful to clarify the use of an argument or variable returned.</li><li><strong>Warning of consequences</strong>: Executing certain code bricks can have important consequences. Comments can be used to alert another code user.</li><li><strong>Amplification</strong>: They can be used to amplify the importance of code that might go unnoticed.</li></ol><p>But there are also several types of bad comments:</p><ol><li><strong>Unclear</strong>: When you’re lazy about writing well and the commentary doesn’t provide clear information.</li><li><strong>Redundant</strong>: When you repeat what is said in the docstring or in previous comments.</li><li><strong>Noise information</strong>: You don’t have to comment on everything, or it means your code is unclear.</li><li><strong>Commented-Out Code: </strong>It is better to delete the old unused code than to comment it out.</li></ol><h3>10. <strong>Tools to reformat code</strong></h3><p>Concerning the formatting of the code, the structure of Python requires us to be very meticulous in order to format our code properly. Indeed, the number of lines to be skipped varies according to what is declared in the code (do we declare a class, a method, a variable?).</p><p>Lists should not be too long, otherwise, you have to go back to the line, the same goes for line length, space rules etc… In short, there are a lot of rules to follow and it is much too long to follow them meticulously by hand (see <a href="https://www.python.org/dev/peps/pep-0008/">https://www.python.org/dev/peps/pep-0008/</a>).</p><p>Fortunately, some libraries automatically format your code and the time saved is considerable. The best-known libraries are Black and Yapf and I highly recommend them. They integrate easily into an IDE like PyCharm and the result is really good.</p><p>Also, a good IDE can highlight major style errors and may suggest modifications. For my part, I use PyCharm and the correction system is very efficient.</p><p>The code below shows you how to transform badly formatted ugly code into well-formatted ugly code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qZC1U1w-GQ9vJtN6QmnbTg.png" /><figcaption>Before</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cW7sK_OU1E1Qa8PWBK5-1w.png" /><figcaption>After Black &amp; Pycharm</figcaption></figure><p>See : <a href="https://opensource.com/article/18/7/7-python-libraries-more-maintainable-code">https://opensource.com/article/18/7/7-python-libraries-more-maintainable-code</a></p><h3>Conclusion</h3><p>I hope you will have found this article interesting and that it will have given you some clues to improve your Python code.</p><p>I also recommend you to have a look at this GitHub repository which includes the concepts mentioned in this article with other code examples: <a href="https://github.com/davified/clean-code-ml">https://github.com/davified/clean-code-ml</a>.</p><p>If you have any comments about the tips mentioned, please feel free to comment on this article or send them to me by email: <a href="/service/mailto:matthieu@modeo.ai">matthieu@modeo.ai</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=798ba398fcbe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/10-tips-to-make-your-data-science-code-cleaner-798ba398fcbe">10 tips to make your data science code cleaner and more efficient</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build your own recommendation system chatbot with Facebook Messenger and Node.js (Part 1)]]></title>
            <link>https://blog.chatbotslife.com/build-your-own-recommendation-system-chatbot-with-facebook-messenger-and-node-js-part-1-5853ca61be51?source=rss-86eaa22b0b15------2</link>
            <guid isPermaLink="false">https://medium.com/p/5853ca61be51</guid>
            <category><![CDATA[bots]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Matthieu Rousseau]]></dc:creator>
            <pubDate>Fri, 03 May 2019 14:55:07 GMT</pubDate>
            <atom:updated>2019-05-07T16:44:50.242Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*fTNZuasgqUDPw47xFVb-XQ.jpeg" /></figure><p>The purpose of the tutorial is to give you a guideline to develop your own recommendation system. We’re going to build a system which performs technical monitoring. In our case we will focus on technical media. By following this tutorial, you will be able to develop your own <a href="https://chatbotslife.com/">chatbot</a> and recommendation system with the entire process of data extraction, data processing and return to users.</p><p>This tutorial is based on the DeepNews.ai project, a chatbot that carries out technical monitoring in the field of artificial intelligence.</p><p>Github : 👉 <a href="https://github.com/matrousseau/DeepNews.ai">https://github.com/matrousseau/DeepNews.ai</a> (it’s important if you want to follow the tutorial).</p><p>Chatbot : 👉 <a href="https://www.facebook.com/DeepNewsai-766106250456126">https://www.facebook.com/DeepNewsai-766106250456126</a></p><figure><a href="https://www.eventbrite.com/e/chatbot-ai-conference-tickets-55380774387"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ac4Wm94IFfL37KAAXnEUXQ.png" /></a></figure><p>This article is divided in three parts :</p><ul><li>How to extract data and build a recommendation system ?</li><li>How to build a messenger <a href="https://chatbotslife.com/">bot</a> which send daily messages and collect data to improve the recommendation ?</li><li>How to improve the recommendation system as soon as we have enough data ?</li></ul><h3>Trending Chatbot Tutorials</h3><blockquote><a href="https://chatbotslife.com/build-simple-chatbot-with-rasa-part-1-f4c6d5bb1aea">1. Build simple ChatBot in Python with RASA — Part 1</a></blockquote><blockquote><a href="https://chatbotslife.com/build-simple-chatbot-with-rasa-part-2-16726357b72c">2. Build a simple ChatBot in Python with RASA — Part 2</a></blockquote><blockquote><a href="https://chatbotslife.com/how-i-developed-my-own-learning-chatbot-in-python-from-scratch-and-deployed-it-on-facebook-88bc828be0a8">3. How I developed my own ‘learning’ chatbot in Python</a></blockquote><blockquote><a href="https://chatbotslife.com/chatbot-conference-2019-in-nyc-8b010058f8f9">4. Chatbot Conference 2019 in NYC</a></blockquote><h3>First part : The recommendation system</h3><ol><li>Collect data</li><li>Clean Data</li><li>Predict articles</li></ol><h3>1. Collect data</h3><p>To manage all this section, we used python. First, build your environnement, you can use Anaconda ( <a href="https://www.youtube.com/watch?v=pVME6JvdD5g&amp;t=67s">https://www.youtube.com/watch?v=pVME6JvdD5g&amp;t=67s</a>) or VirtualEnv. In our case, we used VirtualEnv with PyCharm for the IDE.</p><p>The methode we use to collect the data is the scraping, using the library Scrapy. This is the first lib you need to install.</p><p><strong>Collect data with Scrapy :</strong></p><ol><li>Create a project : scrapy startproject NameOfTheProject</li><li>cd /pathToYourProject</li><li>scrapy genspider NameOfTheCrawler theLinkOfASiteYouWantToScrap.anything</li></ol><p>A file “NameOfTheCrawler” should have been created. Open this file and look at the code.</p><pre># -*- coding: utf-8 -*-<br>import scrapy</pre><pre>class NameofthecrawlerSpider(scrapy.Spider):<br>    name = &#39;NameOfTheCrawler&#39;<br>    allowed_domains =       [&#39;<a href="https://thelinkofasiteyouwanttoscrap.anything&/#39;">https://theLinkOfASiteYouWantToScrap.anything&#39;</a>]<br>    start_urls =  [&#39;<a href="http://https//theLinkOfASiteYouWantToScrap.anything/&#39;">http://https://theLinkOfASiteYouWantToScrap.anything/&#39;</a>]</pre><pre>def parse(self, response):<br>   pass</pre><p>Now we’re going to fetch the webpage you want to scrap and select the appropriate data : title of the articles, link of the articles and we will add a date for later analysis.</p><ol><li>Open a scrapy shell with command : scrapy shell</li><li>Fetch the website (we will scrap medium for the example) : fetch(‘<a href="https://medium.com/topic/artificial-intelligence">https://medium.com/topic/artificial-intelligence</a>&#39;)</li><li>If it output 200 it means that is works, then you need to search for the data you want</li></ol><p>You can select data by using many informations : id, tag, h1, h2 etc… using response.xpath(‘YourXpath’). Check the doc if you want more explanation : <a href="https://docs.scrapy.org/en/xpath-tutorial/topics/xpath-tutorial.html">https://docs.scrapy.org/en/xpath-tutorial/topics/xpath-tutorial.html</a></p><p>Here is an example of the command we use to scrap Medium :</p><pre># -*- coding: utf-8 -*-<br>import scrapy<br>from datetime import datetime</pre><pre>class MediumaiSpider(scrapy.Spider):<br>    name = &#39;MediumAI&#39;<br>    allowed_domains = [&#39;<a href="https://medium.com/topic/artificial-intelligence&#39;">https://medium.com/topic/artificial-intelligence&#39;</a>]<br>    start_urls = [&#39;<a href="https://medium.com/topic/artificial-intelligence/&#39;">https://medium.com/topic/artificial-intelligence/&#39;</a>]</pre><pre>def parse(self, response):<br>        titles = response.xpath(&#39;//*[<a href="http://twitter.com/id">@id</a>=&quot;root&quot;]/div/section/section[1]/div[3]/div[1]/section/div/section/div[1]/div[1]/div[1]/h3/a/text()&#39;).extract()<br>        link = response.xpath(&#39;//*[<a href="http://twitter.com/id">@id</a>=&quot;root&quot;]/div/section/section[1]/div[3]/div[1]/section/div/section/div[1]/div[1]/div[1]/h3/a/@href&#39;).extract()<br>        now = datetime.today().strftime(&#39;%Y-%m-%d&#39;)<br>        clean_approx = []</pre><pre>  for sentence in titles:<br>            clean_approx.append((sentence.replace(&quot;,&quot;, &quot;&quot;)))</pre><pre>yield {&quot;Title&quot;: clean_approx, &#39;Link&#39;: link, &#39;Date&#39;: now}</pre><p>While you have edit your python file with your corrects xpath, you will be able to crawl the data by typing “scrapy crawl NameOfTheCrawler -o NameOfTheCrawler.csv” in a command shell at the root of your project.</p><p>Then, create a bat file where you activate your environnement and then you run every crawler.</p><pre><a href="http://twitter.com/echo">@echo</a> off<br>cmd /k &quot;D:\IA\Projets\VenvManager\DeepNews.ai\Scripts\activate &amp; cd IACrawler &amp; scrapy crawl MediumCrawler -o MediumCrawler.csv &amp; scrapy crawl actuia -o actuia.csv &amp; scrapy crawl dscentral -o dscentral.csv &amp; scrapy crawl itsocial -o itsocial.csv &amp; scrapy crawl MediumML -o MediumML.csv &amp; scrapy crawl MediumDS -o MediumDS.csv &amp; scrapy crawl MediumAI -o MediumAI.csv &amp;  scrapy crawl mediumtech -o MediumTech.csv &amp; scrapy crawl approximatelycorrect -o approximatelycorrect.csv &amp; scrapy crawl kdnuggets -o kdnuggets.csv &amp; cd .. &amp; python Main.py&quot;</pre><p>It should store all your csv files in the same folder. Now we have to clean all this data to make predictions.</p><h3>2. Clean data</h3><p>Since all the csv are stored in the same folder, we can load them and clean them in order to create a simple csv file with all the articles/links/dates.</p><p>This task is done using the class DailyData. There are two functions to clean the csv files because differents websites return data with differents structures.</p><p>Then, in the <strong>main.py</strong> file, we load every file to add it in a huge csv file and then we save this file locally by calling it ‘AI_articles_dataset.csv’.</p><pre># 1. Chargement de la base de données</pre><pre>load_data()</pre><pre># 2. Chargement des données de la veille et entrainement</pre><pre>init_users()</pre><pre># 3. Réentrainement du model et selection des 3 meilleurs articles</pre><pre>train_and_predict()</pre><pre># 4. Actualisation des articles sélectionnés sur S3</pre><pre>update_files_S3()</pre><pre># 5. Préparation des fichiers pour l&#39;entrainement de demain</pre><pre>end_daily_task()</pre><h3>3. Predict articles</h3><p>Once we have our articles of the day cleaned, we need to select 3 articles for each user depending on his preferences.</p><p>In a firt time, we need to collect many data to build a recommendation system. Since we start with no data, there is a problem called the “Cold Start problem”, which mean we have to give to users articles without knowing their preferences.</p><p>We will first use a basic neural network with transfer learning (with ELMO) for the embbeding layer to make a classification between articles. Each user has his own file with his history of what he has read. We initialize it with a default file and then each day we add the previous day’s data depending on whether or not he has read his articles.</p><p>The file <strong>TrainModel.py</strong> works in several steps :</p><ol><li>Add the label from the previous day</li><li>Preprocessing on the daily data</li><li>Retraining of the model with new data</li><li>Predict 3 best articles and upload it on AWS S3</li></ol><p>As soon as we have the 3 predicted articles uploaded on S3, the messenger <a href="https://chatbotslife.com/">bot</a> is able to send the customized articles for each user.</p><p>Next step : create the messenger bot 👉 LINK : <a href="https://medium.com/@pacearthur/build-your-own-recommendation-system-chatbot-with-facebook-messenger-and-node-js-part-2-2cdb17e32926">https://medium.com/@pacearthur/build-your-own-recommendation-system-chatbot-with-facebook-messenger-and-node-js-part-2-2cdb17e32926</a></p><h3>Don’t forget to give us your 👏 !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*JwE3-qj29nhGKMoh" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F3e59ef%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F3e59ef&amp;image=http%3A%2F%2Fapi.screenshotlayer.com%2Fapi%2Fcapture%3Faccess_key%3Dfe59908dad3baab69ffab249a2224b03%26viewport%3D1024x612%26width%3D1000%26url%3Dhttps%253A%252F%252Fupscri.be%252F3e59ef%253Fscreenshot&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/7078d8ad19192c4c53d3bf199468e4ab/href">https://medium.com/media/7078d8ad19192c4c53d3bf199468e4ab/href</a></iframe><figure><a href="https://chatbotslife.com/bot-communities-mastermind-group-d2dae9876709#.53x0py6ou"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*6XUspT9JOSq0w0Fi35HIaA.png" /></a></figure><figure><a href="https://m.me/ChatbotsLife"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*c1LDMH5vbnIz9rmAka8Hwg.png" /></a></figure><figure><a href="https://chatbotslife.com/chatbot-development-hire-top-ai-chatbot-developers-c8b45ef7f207"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*D0Jf3dI6ZThtqcfwDYY7mg.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5853ca61be51" width="1" height="1" alt=""><hr><p><a href="https://blog.chatbotslife.com/build-your-own-recommendation-system-chatbot-with-facebook-messenger-and-node-js-part-1-5853ca61be51">Build your own recommendation system chatbot with Facebook Messenger and Node.js (Part 1)</a> was originally published in <a href="https://blog.chatbotslife.com/">Chatbots Life</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is the AWS DeepRacer League and how did we win the French Edition ?]]></title>
            <link>https://medium.com/@modeoAI/what-is-the-aws-deepracer-league-and-how-did-we-win-the-french-edition-facae656680c?source=rss-86eaa22b0b15------2</link>
            <guid isPermaLink="false">https://medium.com/p/facae656680c</guid>
            <category><![CDATA[autonomous-cars]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deepracer]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[Matthieu Rousseau]]></dc:creator>
            <pubDate>Tue, 23 Apr 2019 18:34:14 GMT</pubDate>
            <atom:updated>2019-06-16T11:58:29.594Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is the AWS DeepRacer League and how did we win the French Edition ?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHH4H0AYj0q_xB1ujgU24g.jpeg" /></figure><p>On April 2, 2019, the Amazon Web Services summit of Paris took place. AWS is Amazon’s cloud service, used by millions of developers worldwide. AWS offers more than 90 services, including computing, storage, network, database, data analysis, application services etc etc..<br>This year AWS created a brand new competition: the “AWS DeepRacer League”. In this article, we will first explain the concept of this contest and then we will analyse how our approach enabled us to win!</p><h3><strong>I. Concept of the AWS Deep Racer League</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*vYUuQ7LJlWAaiAYShMKmPw.png" /></figure><p>The AWS DeepRacer League is a worldwide competition for developers. The goal is to develop an algorithm that let a small autonomous car drive on a track. Candidates have to use reinforcement learning algorithms (the concept will be explained later) to train their artificial intelligence. The winner is the one who completes a complete round fastest.</p><p>The competitions take place at the AWS summit, there are 20 of them in the year in the world’s largest cities and each winner is invited by AWS to the world final in Las Vegas in 2019!</p><p>You can also attend to virtual competitions, one per month and win your place in Las Vegas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MWzgxXJw1tIvXwyetFzrOg.jpeg" /></figure><h3>II. How does the training of the model works ?</h3><p>AWS provides two methods to train the algorithm :</p><ul><li>Use the dedicated service on AWS: the developer focuses on the reward function and tunes the deep Neural Network to train the model</li><li>Three services: SageMaker, RoboMaker, S3: you have access to a larger number of variables to train the model but the platform is more difficult to handle and the results are often worse than with the first method according to what we have observed during the different summits.</li></ul><p>In our case, we had to use the second method because the first service was not yet released.<br>To understand how training works, it is first essential to make a short parenthesis on the way reinforcement learning works (This article is not intended to be technical so we will not dive into too much detail).</p><p><strong>How does reinforcement learning works ?</strong></p><p>There are three major areas of machine learning:</p><ul><li>Supervised Learning: data is labelled which means that we have previously annotated each row of data to indicate the class to which each element of the dataset belongs. Thus, the algorithm tries to learn which patterns belong to each type of data. For example, we could teach to a car how to drive by controlling it with a remote, save the remote’s inputs and train the model by merging this data with images collected from a camera.</li><li>Non-Supervised Learning: the data is not labelled which means that we try to learn patterns depending on the relation between data.</li><li>Reinforcement Learning: The data is not labelled. An agent (the car) interacts in an environment (the track). Depending on its policy, the car gets a reward. Then, we update the policy in order to maximize the reward function.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pa29l9_vcQbZWBklZJuJ7Q.png" /></figure><p>The car has a limited range of N actions (for example speed up, move left slightly, move right strongly etc…). A Deep Neural Network (DNN) predicts which of these N actions is the most appropriate depending on an image as an input, captured from the onboard camera.</p><p>To train the network, the car makes a large number of attempts in a virtual simulator. After each attempt, we give the car a reward. If, for example, she stayed in the middle of the track and she went far, she will obtain more points than if she went off the track directly. Then, we backpropagate in the DNN to update every weight every k steps.</p><p>As you can see below, AWS provides a virtual environment to train the car. Having a virtual environment gives us access to a large number of variables: distance from the centre of the track, speed, wheel inclination, etc… Knowing that we reward the policy of the car depending on these variables, using a virtual environment is essential. On the right, you can see a plot which represents the evolution of the reward depending on the step. When there is a full bar, it means the car has completed a full lap.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FP7cfuOMMDmM%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DP7cfuOMMDmM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FP7cfuOMMDmM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2482b25d7afc2574178072507306f884/href">https://medium.com/media/2482b25d7afc2574178072507306f884/href</a></iframe><p>You can see below one of our tours during the Paris summit.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F5hMWD3-JBao%3Fstart%3D77%26feature%3Doembed%26start%3D77&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D5hMWD3-JBao&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F5hMWD3-JBao%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ceb31cec29f6f9b12b02a79f229d1d06/href">https://medium.com/media/ceb31cec29f6f9b12b02a79f229d1d06/href</a></iframe><h3><strong>III. </strong>What we learned from the race</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X1DU13sIioh8Poyc_j0T-w.jpeg" /></figure><p>We have noticed that some parameters have a big impact on the quality of the model :</p><ul><li>You have to find the right balance when you choose the training time of your model : if you train your model you will overfit the track and if you don’t train it enough, the model won’t be strong enough.</li><li>Don’t focus on the speed but on the reliability of your model, if it can behave well on the track, You can accelerate the car on the tablet.</li><li>It’s all about the trajectory, working on angles has improved our model a lot.</li></ul><p>We will not reveal more details about our solution as we are currently working on a new algorithm for the final in Las Vegas in December 2019. 😉</p><p>I hope you have been able to learn more about the DeepRacer League through this article. If you have any questions, do not hesitate to contact Arthur (pacearthur.github.io) or myself (matrousseau.github.io) and we will be happy to answer you.</p><p>Thank you ! 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=facae656680c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>