<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[iNex Blog - Medium]]></title>
        <description><![CDATA[iNex develops digital solutions for the recycling and renewable energy industry. We bring valuable data through open-data and machine learning. - Medium]]></description>
        <link>https://medium.com/inex-blog?source=rss----47e9073a4591---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>iNex Blog - Medium</title>
            <link>https://medium.com/inex-blog?source=rss----47e9073a4591---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:13:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/inex-blog" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How we aligned the team with OKR]]></title>
            <link>https://medium.com/inex-blog/how-we-aligned-the-team-with-okr-c94e9e407b0b?source=rss----47e9073a4591---4</link>
            <guid isPermaLink="false">https://medium.com/p/c94e9e407b0b</guid>
            <category><![CDATA[okr]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[autonomy]]></category>
            <dc:creator><![CDATA[Pierre Beuret]]></dc:creator>
            <pubDate>Sun, 05 Jan 2025 12:57:45 GMT</pubDate>
            <atom:updated>2024-05-15T14:00:25.665Z</atom:updated>
            <content:encoded><![CDATA[<p>During our first years at iNex, we made lots of product and service experimentations to find our market-fit. When time to scale came, we focused and chose a clear positionning and had to make choices.</p><h3>We felt a need for aligning everyone</h3><p>Nobody likes to spend time on something that does not bring value to the company and its clients. When we were a bunch of people, it was easy to make sure everyone was on the same page. As our team grew up to 20 people, we saw dysfunctions appear : engineers spent time on improvments our clients were not interested in andsales targeted a market that was not our priority.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*d2Wv96iQpiWmwPATeEj-EQ.png" /><figcaption>Not being aligned</figcaption></figure><p>With a top-down management structure, this could be solved easily through a very detailed planification from managers. But at iNex we want teams to be autonomous, deciding what they are going to work on and taking their own decisions.</p><h3>OKR was the perfect match</h3><p>We found what we were looking for with the OKR methodology. I will not get into details with the theory as lots of thing have been written on the topic : see at the end “Inspirational Resources”, especially the tremendous one from Gitlab.</p><p>In a nutschell, the idea is to define 3 aspirationnal objectives and for each one 3 key indicators measuring their success. We like it because it is simple :</p><ol><li>iNex cofounders define OKR every quarter before starting a new one</li><li>Each team codesigns its OKR</li><li>OKR are reviewed and validated with cofounders</li><li>OKR are tracked every month to measure progress</li></ol><p>It really empowers the teams as they are responsible for defining and tracking their own objectives. And everyone can see these in our collaborative tool Asana whenever they want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*V57zlG-dxsjtxiDo81jWyQ.png" /><figcaption>Product OKR examples</figcaption></figure><h3>How we set it up</h3><p>At iNex, we always test things fast and then iterate. So we started testing it with my cofounder at company level, during 6 months.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MMoibSP2wFApsFmk5QuU9w.png" /><figcaption>Defining targets</figcaption></figure><p>When we thought we were able to set up relevant OKR at company level, we designed a methodology to codesign teams OKR with teams: Business, Customer Relations, Product, Engineering. We basically asked team contributors to prepare 2 objectives each and planned a quaterly 1h30 meeting :</p><ul><li>Last quarter OKR evaluation debate (45mn)</li><li>Review and redefinition of next quarter Objectives suggestions &amp; vote to select 3 (30mn)</li><li>Codesign of Key Results for each of the 3 Objectives (15mn)</li></ul><h3>The impact</h3><p>We clearly saw a direct impact on employees’ motivation as they now understand how they fit in the company strategy and roadmap. And most of all, they are empowered to define their own team objectives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zzCencNePpdHxiQwoni8VA.jpeg" /><figcaption>Motivation increased</figcaption></figure><p>It also forced us as cofounders in regularly defining and tracking quaterly objectives. This became also clearer for our board.</p><h3>There is a long way to go</h3><p>We started that process 1 year ago, and we still have a lot to learn. Do not hesitate to reach us if you got idea on how we could solve these questions :</p><ul><li>Is it relevant to have different OKR for Product and Engineering ?</li><li>How to make sure we set the best OKR at company level ?</li><li>How to better codesign OKR within each team ?</li><li>How to set up a tracking routine without spending too much time on it ?</li></ul><h3>Inspirational resources</h3><ul><li>iNex Team OKR Guidelines (in french) : <a href="https://inex.slite.com/app/docs/oFUYQvVFF5YLC4">Définition</a></li><li>OKR Guidance step by step : <a href="https://www.whatmatters.com/get-started">https://www.whatmatters.com/get-started</a></li><li>Github internal handbook on OKR : <a href="https://handbook.gitlab.com/handbook/company/okrs/">https://handbook.gitlab.com/handbook/company/okrs/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c94e9e407b0b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inex-blog/how-we-aligned-the-team-with-okr-c94e9e407b0b">How we aligned the team with OKR</a> was originally published in <a href="https://medium.com/inex-blog">iNex Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why you need to migrate from pandas to Polars ?]]></title>
            <link>https://medium.com/inex-blog/why-you-need-to-migrate-from-pandas-to-polars-255efb96d1d8?source=rss----47e9073a4591---4</link>
            <guid isPermaLink="false">https://medium.com/p/255efb96d1d8</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Benoit de Menthière]]></dc:creator>
            <pubDate>Tue, 16 Jul 2024 14:11:27 GMT</pubDate>
            <atom:updated>2024-07-16T14:36:34.897Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why you need to migrate from pandas to Polars ?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*iMVwh9m981Gwu2ce56gwaA.png" /></figure><p>In the ever-evolving landscape of data processing, choosing the right tools can significantly impact efficiency and productivity. After careful consideration, we decided to migrate to Polars, a DataFrame library designed for speed and ease of use. Here are the key reasons behind our decision:</p><h3>Leveraging Rust’s Capabilities</h3><p>Polars is crafted in Rust, a language celebrated for its low-level memory management without sacrificing safety, thanks to its compile-time memory safety checks. Rust’s approach to concurrency is also noteworthy; it enables safe and efficient parallel data processing, which is pivotal for handling voluminous datasets swiftly. These characteristics contribute significantly to Polars’ performance, making it an excellent tool for time-sensitive data operations.</p><p>At the core of Polars’ architecture lies its integration with <a href="https://arrow.apache.org/docs/index.html">Apache Arrow</a>, an in-memory data format designed for efficient data processing. While pandas 2.0 now supports <a href="https://arrow.apache.org/docs/python/index.html">pyarrow</a>, it is still built on top of numpy which is less efficient for non-float data types, leading to performance bottlenecks. Polars uses Arrow’s columnar memory format natively. This alignment drastically reduces memory overhead and speeds up data operations.</p><h3>Lazy</h3><p>Polars excels in computational efficiency through intelligent query optimization. It minimizes redundant computations and reorders operations to enhance execution speeds. This proactive optimization ensures that data processing is not only faster but also more resource-efficient.</p><p>For instance, when dealing with <a href="https://www.data.gouv.fr/fr/datasets/base-sirene-des-entreprises-et-de-leurs-etablissements-siren-siret/">Sirene files </a>containing 38 million rows, traditional tools like pandas struggle to handle the data in memory. Our previous solution involved loading data in batches, which was not scalable and made some operations, like a simple group-by median, very complicated. Polars, with its efficient data handling, eliminates these issues.</p><pre><br>pl.scan_csv(<br>    path_sirene_file,<br>).filter(<br>    pl.col(&quot;trancheEffectifsEtablissement&quot;).is_in(list(dic_tranche_employes))<br>).group_by(<br>    pl.col(&quot;activitePrincipaleEtablissement&quot;).str.slice(0, length=5)<br>    .alias(&quot;code_ape&quot;)<br>).agg(<br>    pl.col(&quot;trancheEffectifsEtablissement&quot;)<br>    .replace(dic_tranche_employes, default=None)<br>    .median()<br>    .alias(&quot;median_employes&quot;)<br>).explain()<br><br>output:<br>AGGREGATE<br> [<br>col(&quot;trancheEffectifsEtablissement&quot;)<br>.replace([Series, Series, null])<br>.median().alias(&quot;median_employes&quot;)<br>] <br>BY [<br>col(&quot;activitePrincipaleEtablissement&quot;).str.slice([0, 5]).alias(&quot;code_ape&quot;)<br>] <br>FROM<br><br>    Csv SCAN /home/data/sirene/StockEtablissement_utf8.csv<br>    PROJECT 2/53 COLUMNS<br>    SELECTION: col(&quot;trancheEffectifsEtablissement&quot;).is_in([Series])</pre><p>Only required lines and rows are loaded ! Their <a href="https://pola.rs/posts/benchmarks/">benchmark</a> gives a good idea of the optimization.</p><p>Fail fast is a best practice in data engineering; we don’t want to wait hours before encountering an issue like a missing column. Polars’ LazyFrame only evaluates the result when it is needed, deferring execution to the last minute. This approach has significant performance advantages and is why the Lazy API is preferred in most cases. For example, it won’t load unnecessary columns for calculations and raises a missing column exception immediately, without waiting for the entire calculation to complete. The stricter Polars schema also reduces schema-related bugs.</p><h3>API</h3><p>Polars offers an API that combines expressiveness with simplicity, enabling users to perform complex data manipulations with fewer lines of code and greater clarity. This not only improves code readability but also reduces the potential for errors.</p><p>Polars expressions are quite powerful and flexible, so there is much less need for custom Python functions compared to pandas. Check their <a href="https://docs.pola.rs/api/python/stable/reference/expressions/index.html">documentation</a> for the list of available expressions.</p><h3>Consistency</h3><h4>Missing data</h4><p>Polars provides a more consistent approach to handling missing data, distinguishing clearly between null and NaN, which simplifies data cleaning and preprocessing steps.</p><h4>Syntax</h4><p>Finally, I never understood pandas names, sometimes there is an underscore sometimes not, same for “s”: nunique, is_unique, dropna, hasnans, drop_duplicates. Polars syntax is more consistent, allowing faster developments.</p><p>Also, the fact that there is no index, makes things simpler, no more need to reset_index and less errors !</p><h3>Conclusion</h3><p>As you can see, Polars has many advantages compared to pandas. One of the few reasons I see to keep pandas is its rich ecosystem as it is compatible with many libraries including visualization and machine learning. For instance, we use a lot <a href="https://geopandas.org/">geopandas</a> to handle geospatial data efficiently. While <a href="https://geopolars.org/latest/">geopolars </a>is still under development, we use <a href="https://duckdb.org/">duckdb </a>and its <a href="https://duckdb.org/docs/extensions/spatial">spatial extension</a> to accelerate our geospatial data processing. But if you really need to use pandas, you can still easily <a href="https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.to_pandas.html#polars.DataFrame.to_pandas">convert</a> Polars DataFrame to pandas DataFrame. For those interested in trying Polars, we recommend <a href="https://github.com/PacktPublishing/Data-Analysis-with-Polars/">this exercice</a>, which we have used internally.</p><p>sources:</p><ul><li><a href="https://docs.pola.rs/">https://docs.pola.rs/</a></li><li><a href="https://blog.jetbrains.com/pycharm/2024/07/polars-vs-pandas/">https://blog.jetbrains.com/pycharm/2024/07/polars-vs-pandas/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=255efb96d1d8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inex-blog/why-you-need-to-migrate-from-pandas-to-polars-255efb96d1d8">Why you need to migrate from pandas to Polars ?</a> was originally published in <a href="https://medium.com/inex-blog">iNex Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we use GPT to improve data quality and entity extraction]]></title>
            <link>https://medium.com/inex-blog/how-we-use-gpt-to-improve-data-quality-and-entity-extraction-b35e21b05ef4?source=rss----47e9073a4591---4</link>
            <guid isPermaLink="false">https://medium.com/p/b35e21b05ef4</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Benoit de Menthière]]></dc:creator>
            <pubDate>Thu, 22 Jun 2023 13:37:04 GMT</pubDate>
            <atom:updated>2023-12-19T18:29:41.398Z</atom:updated>
            <content:encoded><![CDATA[<p>Data cleaning is an essential yet time-consuming task in any data-driven project. It involves transforming and standardizing data to ensure accuracy and consistency, but it often consumes valuable time and resources. However, with advancements in natural language processing (NLP) and the availability of powerful AI models like GPT, part of data cleaning can now be automated efficiently.</p><p>As an innovative company that aggregates hundreds of different data sources, <a href="https://sourcing.inex-circular.com/">iNex</a> must rely on accurate and reliable data to develop effective solutions that promote sustainability and environmental impact. In this article, we will explore the various ways in which <a href="https://www.askmarvin.ai/">AskMarvin.ai</a> an OpenAI API wrapper, helps us to save significant time and effort.</p><blockquote>Marvin leverages <strong>“functional prompt engineering”</strong> to build a battle-tested and typesafe interface to LLMs. We expose our API through drop-in decorators, creating an unparalleled developer experience so that every developer can reach for LLMs when it’s the right tool.</blockquote><h3>Standardizing</h3><p>One of the most common challenges in data cleaning is dealing with inconsistent or misspelled strings. With GPT API, you can leverage its text generation capabilities to standardize names effortlessly. By providing a list of names in different formats, you can ask the model to generate standardized versions. This eliminates the need for manual review and editing, saving valuable time and ensuring consistency across the dataset. Let’s look at a first example extracted from <a href="https://www.askmarvin.ai/guide/ai_functions/data/#standardization">askmarvin documentation</a>.</p><pre>from marvin.ai_functions.data import standardize<br>survey_data=[<br>    &quot;(555) 555 5555&quot;,<br>    &quot;555-555-5555&quot;,<br>    &quot;5555555555&quot;,<br>    &quot;555.555.5555&quot;,<br>    &quot;55&quot;<br>    ]<br># converts all values to (555) 555-5555<br>standardize(survey_data, format=&quot;US phone number with area code&quot;)<br><br># Result:<br># [&#39;(555) 555-5555&#39;, &#39;(555) 555-5555&#39;, &#39;(555) 555-5555&#39;, &#39;(555) 555-5555&#39;, &#39;&#39;]</pre><p>It is very easy to define your own cleaning function, for instance to correct some spelling errors:</p><pre>from marvin import ai_fn<br>@ai_fn<br>def fix_spelling_errors(data: list[str]) -&gt; list[str]:<br>    &quot;&quot;&quot;<br>    Fix all spelling errors in a list of string<br>    &quot;&quot;&quot;<br><br>fix_spelling_errors([<br>    &#39;St. Albans&#39;,<br>    &#39;St.A1bans&#39;,<br>    &#39;St Albans&#39;,<br>    &#39;St.Ab1ans&#39;,<br>    &#39;St.albans&#39;,<br>    &#39;St. Alans&#39;,<br>    &#39;S.A1bans&#39;,<br>    &#39;St..A1bans&#39;,<br>    &#39;S.A1bnas&#39;,<br>    &#39;St.A1bnas&#39;,<br>    &#39;St.A1 bans&#39;,<br>    &#39;St.Algans&#39;,<br>    &#39;Sl.A1bans&#39;,<br>    &#39;St. Allbans&#39;,<br>    ])<br><br># Result:<br># [&#39;St. Albans&#39;, &#39;St. Albans&#39;, &#39;St. Albans&#39;, &#39;St. Albans&#39;, &#39;St. Albans&#39;, &#39;St. Albans&#39;, &#39;St. Albans&#39;, &#39;St. Albans&#39;, &#39;St. Albans&#39;, &#39;St. Albans&#39;, &#39;St. Albans&#39;, &#39;St. Albans&#39;, &#39;St. Albans&#39;, &#39;St. Albans&#39;]</pre><h3>Zero Shot Classification</h3><p>Data cleaning often involves categorizing or classifying data based on predefined criteria. This can be a tedious and error-prone process, especially when dealing with large datasets. By providing a simple docstring you can build a classifier with zero NLP knowledge.</p><pre>@ai_fn<br>def classify_sentiment(tweets: list[str]) -&gt; list[bool]:<br>    &quot;&quot;&quot;<br>    Given a list of tweets, classifies each one as<br>    positive (true) or negative (false) and returns<br>    a corresponding list<br>    &quot;&quot;&quot;</pre><p>The library also provides some useful functions for these kinds of tasks:</p><pre>from marvin.ai_functions.data import map_categories, categorize<br><br>colors_data = [<br>    &quot;teal&quot;, <br>    &quot;cyan&quot;,<br>    &quot;peach&quot;,<br>    &quot;salmon&quot;,<br>    &quot;red-orange&quot;,<br>    &quot;lime&quot;,<br>]<br><br># assign categories from a known list<br>map_categories(colors_data, categories=[&#39;red&#39;, &#39;orange&#39;, &#39;yellow&#39;, &#39;green&#39;, &#39;blue&#39;, &#39;indigo&#39;, &#39;violet&#39;])<br><br># Result:<br># [&#39;blue&#39;, &#39;blue&#39;, &#39;yellow&#39;, &#39;red&#39;, &#39;red&#39;, &#39;green&#39;]</pre><p>If you don’t know in advance your categories, just provide a description:</p><pre># describe possible categories in natural language<br>categorize(colors_data, description=&#39;the colors of the rainbow&#39;)<br><br># Result:<br># [&#39;blue&#39;, &#39;blue&#39;, &#39;yellow&#39;, &#39;red&#39;, &#39;red&#39;, &#39;green&#39;]</pre><h3>Handling missing data</h3><p>Dealing with missing data is another time-consuming aspect of data cleaning. GPT API can assist in this process by imputing missing values based on the available data. This automated approach saves significant effort compared to traditional manual imputation methods.</p><pre>import pandas as pd<br>from marvin.ai_functions.data import context_aware_fillna_df<br><br><br>survey_responses = pd.DataFrame(<br>    [<br>        [&quot;NY, NY&quot;, None, None],<br>        [&quot;Boston, Massachusetts&quot;, None, None],<br>        [&quot;Boston MA&quot;, None, None],<br>        [&quot;NYC&quot;,  None, None],<br>    ],<br>    columns=[&quot;response&quot;, &quot;city&quot;, &quot;state&quot;],<br>)<br><br><br>context_aware_fillna_df(survey_responses)<br><br><br># Result:<br>#                 response      city          state<br># 0                 NY, NY  New York       New York<br># 1  Boston, Massachusetts    Boston  Massachusetts<br># 2              Boston MA    Boston  Massachusetts<br># 3                    NYC  New York       New York</pre><p>Here is the function used behind the scene. Once again, you can easily customize instructions to fit your needs:</p><pre>@ai_fn<br>def context_aware_fillna(data: list[list], columns: list[str] = None) -&gt; list[list]:<br>    &quot;&quot;&quot;<br>    Given data organized as a list of rows, where each row is a list of values,<br>    and some missing values (either `None` or `np.nan`), fill in any missing<br>    values based on other data in the same row. Use the `columns` names and<br>    other data to understand the likely data model. Returns the original data<br>    with the missing values filled in.<br>    &quot;&quot;&quot;</pre><h3>Entity extraction</h3><p>AI Models in Marvin, built on <a href="https://docs.pydantic.dev/latest/">pydantic</a>, offer a revolutionary approach to data processing by converting unstructured text data into structured formats. This empowers us to interrogate our data through our model schema, combining the reasoning capabilities of AI with the type boundaries set by pydantic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9pRc-xbGRrUse4ZZ.png" /><figcaption>The model understands that we refer to Chicago and fills in the data based on its global knowledge.</figcaption></figure><p>Consider the example of parsing resumes in an applicant tracking system. Previously, data scientists had to create custom models and regular expressions for each data feature, triggering a development cycle with every new requirement. However, with marvin, we can employ pydantic to shape our data model and enhance it with the @ai_model decorator. This endows our pydantic model with the ability to handle unstructured text seamlessly. Let&#39;s delve into a practical example:</p><pre>import datetime<br>from typing import List, Literal, Optional, Union<br>import pydantic<br>from marvin import ai_model<br><br>class Institution(pydantic.BaseModel):<br>    name: str<br>    start_date: Optional[datetime.date]<br>    end_date: Union[datetime.date, Literal[&#39;Present&#39;]]<br><br>@ai_model<br>class Resume(pydantic.BaseModel):<br>    first_name: str<br>    last_name: str<br>    phone_number: Optional[str]<br>    email: str<br>    education: List[Institution]<br>    work_experience: List[Institution]<br><br>Resume(&quot;&quot;&quot;<br>Ford Prefect<br>Contact: (555) 5124-5242, ford@prefect.io<br><br>Education:<br>- University of Betelgeuse, 1965 - 1969<br>- School of Galactic Travel, 1961 - 1965<br><br>Work Experience:<br>- The Hitchhiker&#39;s Guide to the Galaxy, Researcher, 1979 - Present<br>- Galactic Freelancer, 1969 - 1979<br>&quot;&quot;&quot;).json(indent = 2)<br># result<br>{<br>  &quot;first_name&quot;: &quot;Ford&quot;,<br>  &quot;last_name&quot;: &quot;Prefect&quot;,<br>  &quot;phone_number&quot;: &quot;(555) 5124-5242&quot;,<br>  &quot;email&quot;: &quot;ford@prefect.io&quot;,<br>  &quot;education&quot;: [<br>    {<br>      &quot;name&quot;: &quot;University of Betelgeuse&quot;,<br>      &quot;start_date&quot;: &quot;1965-01-01&quot;,<br>      &quot;end_date&quot;: &quot;1969-01-01&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;School of Galactic Travel&quot;,<br>      &quot;start_date&quot;: &quot;1961-01-01&quot;,<br>      &quot;end_date&quot;: &quot;1965-01-01&quot;<br>    }<br>  ],<br>  &quot;work_experience&quot;: [<br>    {<br>      &quot;name&quot;: &quot;The Hitchhiker&#39;s Guide to the Galaxy&quot;,<br>      &quot;start_date&quot;: &quot;1979-01-01&quot;,<br>      &quot;end_date&quot;: &quot;Present&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;Galactic Freelancer&quot;,<br>      &quot;start_date&quot;: &quot;1969-01-01&quot;,<br>      &quot;end_date&quot;: &quot;1979-01-01&quot;<br>    }<br>  ]<br>}</pre><p>🤯🤯🤯 Mind blowing !</p><p>This makes our scrappers much simpler and smarter, as they don’t need to be adapted to new websites.</p><h3>Limitations</h3><p>Keep in mind that GPT is not perfect and can sometimes be wrong. The most difficult task is to write efficient prompt, that’s why marvin library is very interesting as all prompts are included and optimized for you. I recommend you to lower the temperature parameter, in order to reduce model creativity and have <a href="https://www.askmarvin.ai/guide/concepts/ai_functions/#deterministic-ai-functions">deterministics results</a>.</p><p>Also, the current implementation requires an OpenAI account/key. OpenAI is doing an awesome job, but it can also be expensive. <a href="https://www.askmarvin.ai/guide/concepts/mapping/#caching-results">Caching results</a> avoids re-processing items / paying for duplicate AI calls.</p><p>Finally, Marvin is still under active development and is likely to change.</p><h3>Conclusion</h3><p>This was just a first introduction to the library, but you can do a lot more including slackbot, connect to internet thanks to plugins and smart document retrieval thanks to the <a href="https://docs.trychroma.com/usage-guide">chroma </a>wrapper. iNex successfully harnesses the power of LLM to improve data quality, perfects its scraper and classification algorithms. I really encourage you to take a look at the documentation and have fun!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b35e21b05ef4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inex-blog/how-we-use-gpt-to-improve-data-quality-and-entity-extraction-b35e21b05ef4">How we use GPT to improve data quality and entity extraction</a> was originally published in <a href="https://medium.com/inex-blog">iNex Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to improve geospatial data manipulation with mongoDB]]></title>
            <link>https://medium.com/inex-blog/how-to-improve-geospatial-data-manipulation-with-mongodb-c58876fd68e5?source=rss----47e9073a4591---4</link>
            <guid isPermaLink="false">https://medium.com/p/c58876fd68e5</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[environment]]></category>
            <category><![CDATA[geospatial-data]]></category>
            <dc:creator><![CDATA[Letiziapichon]]></dc:creator>
            <pubDate>Thu, 01 Dec 2022 08:44:20 GMT</pubDate>
            <atom:updated>2022-12-01T08:44:20.719Z</atom:updated>
            <content:encoded><![CDATA[<p>iNex helps the construction industry by developing digital solutions that aggregate a wide variety of land data on a single platform.</p><p>As a data engineer, part of our job is to create pipelines that will retrieve opendata, such as <a href="https://www.geoportail-urbanisme.gouv.fr/">Geoportail de l’urbanisme</a>, process it and then store it on our database. The majority of this data is geospatial such as a buildings or plots of land. And we use MongoDB to deal with it.</p><p>How to best manage and process geospatial data while maximizing MongoDB capabilities ?</p><h3>The advantages of handling geospatial data with MongoDB</h3><p>MongoDB is a great tool to store this kind of data in a geojson format as a “Point”, a “Polygon” or a “Multipolygon”. Its main advantage is that it provides tools through specific keywords to query this type of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*lF-CkevNm5PgcSBcPP77IQ.png" /><figcaption>Geojson on MongoDB Compass</figcaption></figure><p>In order to use these queries, mongodb needs to know that the field ‘geometry’ is a geometric variable. To do that, we need to create an index on this field.</p><h4>Creating a 2dsphere Index</h4><p>To create an index, you must first create a collection and insert at least one element in it. For geometric data you have to make sure that the inserted data is in a valid geojson format (example photo above).</p><p>Once our element is inserted, we can go to the “Indexes” tab and click on “CREATE INDEX”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*bqPyxdKELaLWskq2des4Pw.png" /><figcaption>Indexes Tab on MongoDB Compass</figcaption></figure><p>Then set the index definition to “geometry” and the index type ‘2dsphere’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*Wc2hdUVwrNivjhk248KUng.png" /><figcaption>Create an index on MongoDB Compass</figcaption></figure><p>If an error occurs during the creation of the index it means that either the inserted geojson is not in the right format or one of the geometries is incorrect. For example MongoDB will not accept the geometry in the image below because two segments are crossing. Later in this article you will find a way to correct the geometries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*rsiUvpCyscE5qrFVDEmFSA.png" /><figcaption>Exemple of an incorrect geometry</figcaption></figure><h4>Geospatial queries</h4><p>Now that our index is created we can use the following queries.</p><p><strong>GeoWithin</strong></p><p>This MongoDB keyword allows us to select elements that are entirely contained in the given geospatial data. Therefore, if an element is partially present in the defined bounds, it will not be selected. This allows us to perform the same task as the within function of the geopandas library in python.</p><pre># exemple of a query with geoWithin<br>mongo_collection.find(<br>  {&quot;geometry&quot;: {&quot;$geoWithin&quot;: {&quot;$geometry&quot;: geojson_to_cross}}}<br>)</pre><p><strong>GeoIntersects</strong></p><p>GeoIntersects performs a query close to Geowithin. Indeed, it returns the same geometries as the function ‘GeoWithin’, but it also returns the geometries that intersect the bounds set by our geojson (similar to the ‘intersect’ function in Geopandas).</p><pre># exemple of a query with geoIntersects<br>mongo_collection.find(<br>  {&quot;geometry&quot;: {&quot;$geoIntersects&quot;: {&quot;$geometry&quot;: geojson_to_cross}}}<br>)</pre><p><strong>Near</strong></p><p>The Near keyword returns elements from nearest to farthest from the specified point. In geopandas, we can have the same result by calculating the distance for each geometry. The process is therefore longer with geopandas than with mongodb.</p><pre># exemple of a query with Near<br>mongo_collection.find(<br>  {&quot;geometry&quot;: {&quot;$near&quot;: {&quot;$geometry&quot;: geojson_point , &quot;$maxDistance&quot;: 200}}}<br>)</pre><p>The ‘$maxDistance’ argument is optional and express in meters</p><h3>Correting data with incorrect geometry</h3><p>When collecting open data, the geometry might be incorrect. As we mentionned when creating the index, an erroneous geometry cannot be inserted on mongo.</p><p>There are two main types of errors. We can use the code in python below to correct them.</p><h4>How to detect the error type</h4><p>To correct the error we first need to identify it. To achieve this, we need to intercept the error coming from mongo when inserting data with python.</p><p>We can recreate an error by building an index on the following multipolygon:</p><pre># Exemple of a geojson with incorrect edges in the middle of the geometry<br>{<br>  &quot;properties&quot;: {},<br>  &quot;geometry&quot;: {<br>    &quot;type&quot;: &quot;Polygon&quot;,<br>    &quot;coordinates&quot;: [[<br>      [2.52460760194092, 50.95605670516066],<br>      [2.529651020828509, 50.95202942954249],<br>      [2.529628005044639, 50.95202375369679],<br>      [2.528513959639505, 50.95174915427584],<br>      [2.525168256108671, 50.95477944337451],<br>      [2.525256056398724, 50.95469992655801],<br>      [2.525310248343852, 50.95473395382295],<br>      [2.525230124934766, 50.95480651845794],<br>      [2.52460760194092, 50.95605670516066]<br>    ]]<br>  }<br>}</pre><p>MongoDB will return this error:</p><blockquote>Edges 3 and 5 cross. Edge locations in degrees: [2.5285140, 50.9517492]-[2.5251683, 50.9547794] and [2.5252561, 50.9546999]-[2.5253102, 50.9547340]</blockquote><p>You can use the following snippet to intercept the MongoDB error in python and to find the problematic edges in our geometrical figure.</p><pre># try to insert geometry in our collection on MongoDB<br>try:<br>    collection_mongo.insert_one(geometry)<br>except WriteError as e:<br>    # intercept mongo error<br>    error_message = e.details.get(&quot;errmsg&quot;, &quot;&quot;)<br>    # Find the edges causing the error <br>    edges = re.findall(&quot;Edges ([0-9]+?) and ([0-9]+?) cross&quot;, error_message)<br>    <br>    # retrive all the coordinates from error message<br>    list_coord = literal_eval(<br>            re.search(&quot;Loop is not valid:(.*?) Edges&quot;, error_message).group(1).replace(&quot; &quot;, &quot;&quot;)<br>        )<br>    <br>    edges = [int(ed) for ed in edges[0]]</pre><h4><strong>Incorrect edges in the middle of the geometry</strong></h4><p>As we could see with the multipolygon and the error above, the edges that cause the error are in the third and fifth place. They are therefore in the middle of our geometry.</p><p>We can correct this geometry with the following piece of code. This code will allow us to make sure that our geometry is in the right format (list of coordinates) and to remove the edge that causes the error (in our example edge 4).</p><pre># Fix polygon with incorrect edges<br>if row_to_insert[&quot;geometry&quot;][&quot;type&quot;] == &quot;Polygon&quot;:<br>    for idx in range(len(row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;])):<br>        if list(map(<br>            lambda x: list((float(f&quot;{x[0]:.15f}&quot;), float(f&quot;{x[1]:.14f}&quot;))),<br>            row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;][idx]<br>        )) == list_coord:<br>            <br>            row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;] = list(row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;])<br>            row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;][idx] = list(row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;][idx])<br>            <br>            # delete edge crossing for each polygon<br>            row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;][idx].pop(edges[0] + 1)</pre><h4><strong>Incorrect edges in the beginning of the geometry</strong></h4><p>The crossing edges can also be the first and the last edge as for the following multipolygon:</p><pre># Exemple of a geojson with incorrect edges in the beginning of the geometry<br>{<br>  &quot;geometry&quot;: {<br>    &quot;type&quot;: &quot;Polygon&quot;,<br>    &quot;coordinates&quot;: [<br>      [<br>        [2.52460760194092, 50.95605670516066],<br>        [2.524954170439705, 50.9562501457054],<br>        [2.525206527677853, 50.95636241761319],<br>        [2.526022919458763, 50.95672561783388],<br>        [2.526276711255924, 50.95675973555245],<br>        [2.526574232640361, 50.95660049785125],<br>        [2.526962910362348, 50.95618996171248],<br>        [2.527654732520641, 50.9555109582227],<br>        [2.524831826656184, 50.95546504708952],<br>        [2.524886628450981, 50.95568258355751],<br>        [2.52488667542939, 50.95569733043608],<br>        [2.524879567054954, 50.95571137647318],<br>        [2.524640852055885, 50.95601182009512],<br>        [2.524645905230251, 50.9560146406423],<br>        [2.52460760194092, 50.95605670516066]<br>      ]<br>    ]<br>  }<br>}</pre><p>To correct this error, the code is slightly different as we need to delete the first and last edge. We then proceed to close the geometry by duplicating the first edge and placing it at the end.</p><pre># Fix polygon with incorrect edges<br>if edges[1] - edges[0] != 2:<br>  list_coord = [list_coord[edges[0]], list_coord[edges[1] + 2]]<br><br>  if row_to_insert[&quot;geometry&quot;][&quot;type&quot;] == &quot;Polygon&quot;:<br><br>    if len(row_to_insert[&quot;geometry&quot;][&#39;coordinates&#39;][0]) == edges[1] + 3:<br><br>      for idx in range(len(row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;])):<br>        coord_list = list(map(lambda x: list((float(f&quot;{x[0]:.15f}&quot;), float(f&quot;{x[1]:.14f}&quot;))),row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;][idx]))<br>        if [coord_list[edges[0]], coord_list[edges[1] + 2]] == list_coord:<br>          row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;] = list(<br>            row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;]<br>          )<br>          row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;][idx] = list(<br>            row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;][idx]<br>          )<br>          #delete first edge<br>          row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;][idx].pop(0)<br>          # delete last edge<br>          row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;][idx].pop(-1)<br>          # duplicate first edge<br>          row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;][0].append(<br>            row_to_insert[&quot;geometry&quot;][&quot;coordinates&quot;][0][0]<br>          )</pre><h3>Conclusion</h3><p>These corrections allow us to retrieve open data, which is not always reviewed and can comprise errors like the ones listed above. This methodology improves our ability to provide the best available data to our clients with the best performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c58876fd68e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inex-blog/how-to-improve-geospatial-data-manipulation-with-mongodb-c58876fd68e5">How to improve geospatial data manipulation with mongoDB</a> was originally published in <a href="https://medium.com/inex-blog">iNex Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we scaled a french open-data product to an international level]]></title>
            <link>https://medium.com/inex-blog/how-we-scaled-a-french-open-data-product-to-an-international-level-af2f618d65c1?source=rss----47e9073a4591---4</link>
            <guid isPermaLink="false">https://medium.com/p/af2f618d65c1</guid>
            <category><![CDATA[environment]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[recycling]]></category>
            <category><![CDATA[international]]></category>
            <dc:creator><![CDATA[Pierre Beuret]]></dc:creator>
            <pubDate>Tue, 08 Nov 2022 16:18:11 GMT</pubDate>
            <atom:updated>2022-11-08T16:16:46.332Z</atom:updated>
            <content:encoded><![CDATA[<p>Our Saas product<strong> iNex Waste Sourcing</strong> can detect waste producers in a few clicks and give information about their flows and contacts. Industrial companies use it to source waste locally for their recycling plants or biodigester units.</p><p>It relies on geolocalised producers databases, waste models per producers profiles and a deep work of data treatment. After offering the product mainly in France during a few years, it is now available in around 10 countries in Europe, North America and India.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SMOO41S45lgvplMCe3xbdQ.jpeg" /><figcaption>Map in our office in Paris where available countries are flagged</figcaption></figure><p>Our product relies on open-data and paid database what differs a lot between countries. <strong>Ho how were we able to deploy our product with a good data quality ?</strong></p><h3><strong>The data exploration phase is key</strong></h3><p>When we open a new country, <strong>we spend nearly 25% of our time on that phase</strong>. We have experienced that it is crucial in order to be 100% sure the data we finally produce matches our clients needs.</p><p>Data availability differs a lot between countries, whether it is open-data (seee below) or data quality from our global data suppliers. <strong>Finding the best data suppliers locally requires anticipation.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*UYhz_4Voqt8NjCGGEIV8bg.png" /><figcaption>A great tool we use to pre-analyse corporate data availability : <a href="http://registries.opencorporates.com/">http://registries.opencorporates.com/</a></figcaption></figure><p>Our data analyst have developed <strong>detective skills !</strong> They dig again and again in the internet to discover sources and methods to use it. We for example identified a way in India to detect big farmers through mecanisation incentives.</p><p>Last but not least,<strong> a deep sample check </strong>is conducted to select best data sources and make sure the results will be exhaustive and liable. We run an analysis, for example on the following criterias : site exhaustivity, size availability, geolocalization accuracy, sectorial information availability.</p><h3><strong>Our data pipelines were strongly adapted</strong></h3><p>To get data in very poor availability context, we had to find new ways to transform data. We <strong>designed a pipeline that can transform PNG into SHP</strong>. It was for example used in germany to produce statistical layers on agriculture from the data source Geoportal.DE (see an example below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*QiXjOPvfqir4L7i74MUecA.png" /><figcaption>Colza land repartition, PNG print screen, Geoportal DE</figcaption></figure><p>Where we needed 1 source in France, we needed sometimes to aggregate 3 different sources. We greatly <strong>improved our matching algorithms</strong> based on adresses, names and activities.</p><p>For the same reason, we had to <strong>strenghten our qualification pipelines</strong> that for example :</p><ul><li>Detect the activity of a potential waste producer through we pages data scrapping</li><li>Geolocalize a site through Google Maps, Open-Street Map or API Gouv in France</li></ul><h3><strong>Targeting the exact same quality as in France would be a mistake</strong></h3><p>We realized our skill as a leader in environmental data providing was to adapt to the local context. The value of our service clearly is more important when data is difficult to find.</p><p>We for example learned that :</p><ul><li>Statistical layers data can fulfill our clients needs where they used to need geolocalized data site by site</li><li>Getting an high quality data on priority targets can have more value than exhaustive data</li></ul><h3><strong>Conclusion</strong></h3><p>It tooks us time to find the right recipe to scale our data to other countries : exploration, dedicated pipelines, adaptation to local constraints. But thanks to the lessons learned, we are proud to target to open 10 countries in the next year !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af2f618d65c1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inex-blog/how-we-scaled-a-french-open-data-product-to-an-international-level-af2f618d65c1">How we scaled a french open-data product to an international level</a> was originally published in <a href="https://medium.com/inex-blog">iNex Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to detect the solar potential of industrial sites]]></title>
            <link>https://medium.com/inex-blog/how-to-detect-the-solar-potential-of-industrial-sites-a5f07f0126e9?source=rss----47e9073a4591---4</link>
            <guid isPermaLink="false">https://medium.com/p/a5f07f0126e9</guid>
            <category><![CDATA[solar-energy]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[open-data]]></category>
            <dc:creator><![CDATA[Benoit de Menthière]]></dc:creator>
            <pubDate>Mon, 19 Sep 2022 12:58:54 GMT</pubDate>
            <atom:updated>2022-11-09T09:39:45.087Z</atom:updated>
            <content:encoded><![CDATA[<p>iNex develops digital solutions for the recycling and renewable energy industry. We bring valuable data through open-data and machine learning.</p><p>Thanks to various open-data sources and internal pipelines, we have been able to link industrial sites to their parcels, buildings and parking lots all around France. This interests particularly the solar industry that is looking urgently for areas in which to implement solar panels.</p><h4>Create a custom data visualization and cleaning interface</h4><p>Our first challenge was to visualize and correct the data from our pipelines. We needed an internal tool, and for this we used the <a href="https://streamlit.io/">streamlit</a> library.</p><blockquote><em>Streamlit turns data scripts into shareable web apps in minutes.</em></blockquote><blockquote><em>All in pure Python. No front‑end experience required.</em></blockquote><p>Thanks to the <a href="https://github.com/PablocFonseca/streamlit-aggrid">Agrid </a>component, we can easily select the industrial sites to visualize.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RLvvFxAKJE5ebyB_nvkwDQ.png" /><figcaption>Agrid component allows to select a site by clicking on it</figcaption></figure><p>Once selected, we visualize the plots, buildings and parking lots identified by our pipelines using the <a href="https://plotly.com/">plotly</a> library.</p><pre>def get_fig(company, type_color, token, geojson_object=None, opacity=0.5):<br>    fig = px.scatter_mapbox(company, lat=&quot;latitude&quot;, lon=&quot;longitude&quot;, zoom=zoom)<br>    mapbox = {&quot;accesstoken&quot;: token}<br>    if geojson_object:<br>            mapbox[&quot;layers&quot;] = [<br>                dict(<br>                    sourcetype=&quot;geojson&quot;,<br>                    source=feature,<br>                    type=&quot;fill&quot;,<br>                    color=type_color[feature.get(&quot;properties&quot;, {}).get(&quot;type&quot;, &quot;unknown&quot;)],<br>                    opacity=opacity,<br>                )<br>                for feature in geojson_object.get(&quot;geojson&quot;, {}).get(&quot;features&quot;, [])<br>            ]<br>    fig.update_layout(<br>        margin={&quot;r&quot;: 0, &quot;t&quot;: 0, &quot;l&quot;: 0, &quot;b&quot;: 0}, mapbox_style=&quot;satellite-streets&quot;, mapbox=mapbox<br>    )<br>    return fig</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*GUz3U52zoAdJgfb55m6m-w.jpeg" /></figure><p>If errors are identified, we can select the features to correct:</p><pre>types = [&quot;parcelles&quot;, &quot;buildings&quot;, &quot;parkings&quot;]<br>with st.sidebar:<br>    with st.form(&quot;columns&quot;, clear_on_submit=False):<br>        bools = [st.checkbox(t) for t in types]<br>        st.form_submit_button(label=&quot;Select geojson types&quot;)</pre><p>This brings up a new interface created with <a href="https://leafmap.org/foliumap/#leafmap.foliumap.Map">leafmap</a> on which we have the possibility to delete existing results, or add new ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*8jOh8fegCOCr-j3UmFJUoA.png" /></figure><p>Let’s take the example of the parcels: on the interface, we can see all the parcels returned by the IGN API on the geographical area.</p><p>To add a parcel to a company, there are 2 possibilities:</p><ul><li>Place a marker on a geojson returned by the API.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*OklnyorxQCcOSR9n9iE_cg.png" /></figure><ul><li>Drawing polygons by hand.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*JtmcGvfpGgUBVpvJu9MFeQ.png" /></figure><h3>Calculate indicators of solar potential of sites</h3><p>Many indicators are used together to measure the solar potential of a site: peak power, energy production, distance to the connection station, etc. Let’s take the example of energy production.</p><p>Once these previous corrections are made, we can simply calculate the surface from the <a href="https://geopandas.org/">geopandas</a> library, taking care to convert the coordinate system.</p><pre>import pyproj<br>LAMBERT93 = pyproj.CRS(&#39;EPSG:2154&#39;)<br>WGS84 = pyproj.CRS(&#39;EPSG:4326&#39;)</pre><pre>gdf[&quot;surface&quot;] = gdf.geojson.to_crs(LAMBERT93).area</pre><p>The area is only calculated on areas without environmental and urban planning constraints. iNex thus recovers about a hundred criteria and excludes certain areas (e.g. protected areas).</p><p>We then retrieve the sunshine for each of the detected areas thanks to the API <a href="https://re.jrc.ec.europa.eu/pvg_tools/fr/">https://re.jrc.ec.europa.eu/pvg_tools/fr/.</a> With all these data, we calculate the energy potential of the different sites. It only remains to visualize the result.</p><h3>FO visualization</h3><p>In order for our customer to quickly identify a high potential site, the sites are represented on our platform <a href="https://www.inex-circular.com/">https://www.inex-circular.com/.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cmTwaf-JSoMjXj1rzjbpWA.png" /></figure><p>Thus, in 2 clicks, we can filter and sort by peak power, energy production, distance to the substation or any other criteria.</p><p>When you select a site, all the details appear: general data, energy indicators, land details and other environmental data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MhXEOB3nlYl_A0-U44BtxA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Bqmfh9YqcQ463JplxT9nQ.png" /></figure><h3>Conclusion</h3><p>iNex enables its clients to detect the potential of solar panel implantation by aggregating hundreds of sources, enriched by business or statistical predictions.</p><p>Based on complementaries technologies, iNex also analyzes land to target the installation of recycling plants, energy renovation of buildings or companies subject to certain environmental regulations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5f07f0126e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inex-blog/how-to-detect-the-solar-potential-of-industrial-sites-a5f07f0126e9">How to detect the solar potential of industrial sites</a> was originally published in <a href="https://medium.com/inex-blog">iNex Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comment on a construit l’équipe data]]></title>
            <link>https://medium.com/inex-blog/comment-on-a-construit-l%C3%A9quipe-data-ea28eaabfbee?source=rss----47e9073a4591---4</link>
            <guid isPermaLink="false">https://medium.com/p/ea28eaabfbee</guid>
            <category><![CDATA[organization]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[green-tech]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Pierre Beuret]]></dc:creator>
            <pubDate>Fri, 16 Sep 2022 13:32:18 GMT</pubDate>
            <atom:updated>2022-08-24T13:50:15.653Z</atom:updated>
            <content:encoded><![CDATA[<h4>En 18 mois</h4><h4>La donnée, au cœur de la mission d’iNex</h4><p>Face à la crise environnementale, la transition énergétique s’accélère. Mais les industriels de l’environnement sont freinés par le manque de données pour faciliter les implantations d’usines et le sourcing de leurs clients. On a donc décidé de développer des applications digitales pour leur fournir des informations sur-mesure.</p><p>Après plusieurs preuves de concept et un virage stratégique, on a vendu notre premier pilote en 2019. On s’est lancé dans le développement d’un premier outil de sourcing des zones d’implantations d’usines de méthanisation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/0*N3Gc-Qb1Ot22MgLY.png" /></figure><p>La donnée, c’est La matière première d’iNex, son or vert. On s’est confronté à 3 défis complexes :</p><ul><li>Collecter des millions de sources de données en open-data</li><li>Les nettoyer et analyser automatiquement</li><li>Prédire les gisements de déchets générés par chaque entreprise française</li></ul><h3>Une équipe complémentaire de Data Engineers et Data Analysts</h3><p>Pour développer une technologie capable d’aiguiller des décisions d’investissement industriel, on a construit une équipe de talents à 2 profils :</p><ul><li><strong>Data Engineer : </strong>création de l’architecture serveur, création et orchestration des pipelines, algos de prédiction</li><li><strong>Data Analyst :</strong> modélisation des flux environnementaux, data visualisation, collecte et analyse de sources spécialisées</li></ul><p>Les compétences de chacun diffèrent mais chacun doit être aussi bien capable de manipuler des grosses quantités de données que de prendre du recul face à des données environnementales.</p><p>On a besoin de collaborateurs qui savent travailler en équipe pour imaginer des process, trouver des solutions à des problèmes complexes et grandir ensembles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/0*KVT-OZQSzr_-aq6Y.png" /></figure><h3>Notre quotidien</h3><p>On a développé notre méthode de gestion de projet par sprint d’1 semaine avec des mêlées tous les deux jours. Toute la gestion de projet passe par l’outil collaboratif Asana qui nous a sauvé des limbes d’Excel. Même l’équipe commerciale s’y est mis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/0*MdE2WyEuJuM7l5_I.png" /></figure><p>On a identifié trois principes clés que chacun a constamment en tête :</p><ul><li><strong>Planifier intelligemment</strong></li></ul><p>C’est l’étape la plus importante d’une tâche. On a appris à prioriser selon 3 axes : possibilité de délivrer de premiers résultats rapidement, impact du résultat pour le client, impact du résultat en gain d’efficacité. On découpe au maximum un problème et ses solutions. Et c’est parti.</p><ul><li><strong>Délivrer rapidement</strong></li></ul><p>Tout ce qui n’est pas essentiel pour sortir rapidement des nouvelles données et des nouvelles fonctionnalités est laissé de côté. L’objectif c’est d’obtenir des retours clients le plus vite possible pour développer des produits utiles.</p><ul><li><strong>Automatiser tout ce qui peut l’être</strong></li></ul><p>Pour fournir une information de qualité à un prix abordable, pas de secret. L’équipe pense chacune des solutions dans une logique de réplicabilité. Et on les remet constamment en question.</p><h3>On est à un virage technologique</h3><p>On a construit une techno qui fonctionne et délivré une donnée de qualité à une cinquantaine de clients industriels (ENGIE, Veolia, GRDF, Evergaz et bien d’autres) dans 10 pays dans le Monde.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*POMZIv7YDLTEp4ht.png" /></figure><p>Depuis la rentrée 2022, on fait face à 3 nouveaux challenges :</p><ul><li>Mettre à jour nos données en temps réel, automatiquement sur de grosses volumétries</li><li>Continuer de développer notre technologie sur de nouvelles verticales de l’environnement (solaire, hydrogène, …)</li><li>Ajouter des nouvelles briques technologiques pour apporter plus de valeur à nos clients : analyse images satellites, analyse sémantique</li></ul><p>L’équipe est jeune, la moyenne d’âge avoisinant les 28 ans, avec une motivation à tout épreuve. Le QJ d’iNex c’est 150m2 avec verrière à Paris au cœur des Batignolles. On a aussi une grosse culture du télétravail avec 2j par semaine par personne.</p><p>Si tu veux venir relever ces défis avec nous, contacte-moi directement ton profil Linkedin ou va jeter un œil à nos offres du moment sur Welcome to the Jungle.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea28eaabfbee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inex-blog/comment-on-a-construit-l%C3%A9quipe-data-ea28eaabfbee">Comment on a construit l’équipe data</a> was originally published in <a href="https://medium.com/inex-blog">iNex Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>