<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[OVRSEA - Medium]]></title>
        <description><![CDATA[Freight tech to step ahead in your supply chain - Medium]]></description>
        <link>https://medium.com/ovrsea?source=rss----91ee87b4533d---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>OVRSEA - Medium</title>
            <link>https://medium.com/ovrsea?source=rss----91ee87b4533d---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 11:26:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/ovrsea" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using PostgreSQL views to ensure backwards-compatible, non-breaking migrations]]></title>
            <link>https://medium.com/ovrsea/using-postgresql-views-to-ensure-backwards-compatible-non-breaking-migrations-017288e77f06?source=rss----91ee87b4533d---4</link>
            <guid isPermaLink="false">https://medium.com/p/017288e77f06</guid>
            <dc:creator><![CDATA[Moaad FATTALI]]></dc:creator>
            <pubDate>Mon, 08 Jan 2024 17:21:02 GMT</pubDate>
            <atom:updated>2024-01-08T17:21:02.839Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y4ANPlujWgE9t752J0UJqQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@karlpb?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Karl Paul Baldacchino</a> on <a href="https://unsplash.com/photos/brown-elephant-walking-on-brown-field-during-daytime-A3Tp9XbR3BU?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><h3>TL;DR</h3><p>You can use PostgreSQL views to rename tables and columns in a backward-compatible way.</p><h4>Renaming a table from foo to bar:</h4><h4>Step 1:</h4><pre>CREATE VIEW bar AS SELECT * FROM foo; <br>-- creates an &quot;alias&quot; for the table: <br>-- you can now address it as foo (real table) or bar (table alias)<br>-- it will point to the same table</pre><h4>Step 2:</h4><p>Change the code to use bar instead of foo and deploy</p><h4>Step 3:</h4><p>Once you have decommissioned the old code that references the table as foo , you can run a (transactional!) migration with</p><pre>DROP VIEW bar;<br>ALTER TABLE foo RENAME to bar;</pre><p>Works also with renaming columns in a similar fashion !</p><h3>General principle for non-breaking / backwards-compatible / no downtime migrations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CV2HMtzi9aRqAew6n_drzw.png" /></figure><p>Why is it important to have backwards-compatible migrations ?</p><p>The alternative is to wait for ongoing processes to finish and then shut off your app, run your migration, and then restart your app. While this may be acceptable for non-critical, low traffic systems, it is generally not something you can afford to do.</p><p>Even if you <em>can </em>reasonably<em> </em>introduce downtime during your migrations, doing so discourages you to adopt continuous deployment practices: if your deployments degrade user experience by causing downtime, you’ll be incentivized to deploy less.</p><p>Additionally, backwards-compatible migrations are much less stressful to deploy, even if they require a little more upfront work. In the chance that something goes wrong, it’s less likely to break production.</p><h3>Introduction to Views</h3><p>One can think of views as <em>named queries</em>.</p><p>Instead of having to explicitly having to write this query everytime:</p><pre>SELECT column1, column2 FROM some_table LEFT JOIN some_other_table ON condition;</pre><p>You can create a view:</p><pre>CREATE VIEW my_complex_query AS SELECT column1, column2 FROM some_table LEFT JOIN some_other_table ON condition</pre><p>and operate on my_complex_query directly.</p><p>We can also see my_complex_query as a computed table !</p><h3>Renaming a table</h3><p>Let’s consider a simple case where we had a table named invoices — containing billing data of our clients. This name was fine until we started handling the invoices from our suppliers in our system ! To avoid confusion, we must now express the separation between client invoices and supplier invoices. This requires us to rename our old invoices table ;).</p><h3>Naive way</h3><p>We can do so by running a simple RENAME:</p><pre>ALTER TABLE invoices RENAME TO client_invoices;</pre><p>But this would break old, still running code :(</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GsB49Qk8jPP7NLwV0GA9aQ.png" /></figure><h3>Using views</h3><p>This time, we won’t simply rename — we will also create a view to keep the old code working:</p><pre>ALTER TABLE invoices RENAME TO client_invoices;<br>CREATE VIEW invoices AS SELECT id, ... FROM client_invoices;</pre><p>That way, we can address this table as invoices OR client_invoices, and postgreSQL makes sure the two are perfectly in sync!</p><p>Make sure this migration is transactional, so your database is always in the initial state or the final state ! (i.e client_invoices is always defined either as a view or as the new name of the table)</p><p>When we finally decommission the old code, the app completely forgot about the invoices table. We can run a migration to safely remove the View:</p><pre>DROP VIEW invoices;</pre><p>You have now successfully renamed table invoices to client_invoices without downtime or breaking changes !</p><h3>Renaming a column</h3><p>Using the exact same principle, we can rename columns — the downside is we will be forced to rename the table as well (Take the opportunity to give it a more meaningful name, or use the technique again to rename it back, choice is yours!)</p><h4>Naive way</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_dj_7AKms1NZOS8FrEFiYA.png" /></figure><h4>Using views</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q0XmkZK180iSayuXUHeihw.png" /></figure><h3>Conclusion</h3><p>We’ve seen two simple examples where PostgreSQL Views can help us refactor our database structure without introducing breaking changes from the code’s perspective.</p><p>You can see a code illustration of the first example (renaming table) <a href="https://github.com/surizom/migrations-with-views/blob/main/viewMigrations.test.ts">here</a></p><p>For the sake of simplicity, the code in the examples was written in SELECT. For simple views such as the ones we used and for PostgreSQL 9.3+ — UPDATE, INSERT, DELETE statements also work !</p><p>See <a href="https://www.postgresql.org/docs/current/sql-createview.html">postgreSQL doc</a> for more info:</p><blockquote><em>A view is automatically updatable if it satisfies all of the following conditions:</em></blockquote><blockquote>The view must have exactly one entry in its FROM list, which must be a table or another updatable view.</blockquote><blockquote>The view definition must not contain WITH, DISTINCT, GROUP BY, HAVING, LIMIT, or OFFSET clauses at the top level.</blockquote><blockquote>The view definition must not contain set operations (UNION, INTERSECT or EXCEPT) at the top level.</blockquote><blockquote>The view’s select list must not contain any aggregates, window functions or set-returning functions.</blockquote><p>Keep in mind that views are just virtual tables that can be defined by any SELECT query. Depending on the use case, you can use them to handle much more complex refactoring operations:</p><ul><li>splitting tables,</li><li>regrouping them,</li><li>moving tables or columns around,</li><li>organizing them in domain-bounded schemas</li></ul><p>and make your database schema (almost!) as malleable as your code.</p><p>There are many other use cases to views I find useful that I didn’t mention in this article, for example:</p><ul><li>We can use views as a “database API”, either to simplify a complex database model for a read use case or to expose a clean interface for downstream data consumers and avoid coupling</li><li>We can use views for frequently used queries which we want to name with minimal overhead</li><li>In read use cases where we can afford eventual consistency, we can use materialized views for read use cases to greatly enhance performance on expensive reads. If you have a very complex query with a lot of joins that takes a long time to load, you can cache the result in a materialized view and put as many relevant indexes as you’d like !</li></ul><p>So I can only encourage you to use them liberally :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=017288e77f06" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ovrsea/using-postgresql-views-to-ensure-backwards-compatible-non-breaking-migrations-017288e77f06">Using PostgreSQL views to ensure backwards-compatible, non-breaking migrations</a> was originally published in <a href="https://medium.com/ovrsea">OVRSEA</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Ops development: Moving from a tool-based to a conceptual-centered approach.]]></title>
            <link>https://medium.com/ovrsea/ml-ops-development-moving-from-a-tool-based-to-a-conceptual-centered-approach-4d2e8254d723?source=rss----91ee87b4533d---4</link>
            <guid isPermaLink="false">https://medium.com/p/4d2e8254d723</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[concept]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Paul Couturier]]></dc:creator>
            <pubDate>Thu, 06 Apr 2023 08:07:34 GMT</pubDate>
            <atom:updated>2023-04-06T08:07:34.098Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Mj05nZfTknKGa_6cIgVpw.jpeg" /><figcaption>Picture from <a href="https://unsplash.com/@dancristianpaduret?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Dan Cristian Pădureț</a> on <a href="https://unsplash.com/fr/photos/noOXRT9gfQ8?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>TL;DR: The strong need of dedicated technologies in ML Ops is often the base of mis-understanding of what ML Ops is really about. While ML stack represents only the field of possibility, holistic concepts such as monitoring or alerting for instance, act as fundamental bricks that drive the development of an entire ML project plan. These concepts bring the right questioning without constraining your vision to the ML Ops tool you considered. By envisioning our ML projects from a concept &amp; need perspective, we are able to deliver great ROI even without state-of-the-art tools.</p><h3>1. ML concept approach</h3><p>ML Ops is one of these disciplines where everything changes from one company to another depending on the stack. Entirely focusing on putting the model in production and monitoring it afterward, ML ops practice is fundamentally backed up by hardwares and tools that ensure the model survival. This strong need for dedicated technologies can often lead to a complete misunderstanding of what ML Ops is really about. The job is not really to use state-of-the-art tools to handle the model in production but to deploy &amp; monitor it adequately to retrieve the best ROI of it. Thus the predominance of tools importance in ML ops is undermining the ROI estimation process of such a model. Some companies tend to use the most trendy technologies without considering the concrete need of the project.</p><p>But thinking simply in terms of need seems also a bit shortening in terms of project development. Indeed, the need, while being at the root of the project, is never the right ally when it comes to project roadmap, and this is where ML concepts intervene.</p><p>What I meant by ML concept is ideas that should be holistic and not project-dependent and that drives your ML development plan. They are fundamental bricks that guide you from the R&amp;D to the production. Contrary to specific technologies use, concepts are always relevant no matter the project and they trigger the right questioning. These questions will guide your development while the project’s fundamental needs will provide the answers.</p><blockquote><strong><em>The ML concept is your question and your need will provide the answer.</em></strong></blockquote><p>ML concepts include monitoring, alerting, accuracy metrics, user adoption, and health checks for instance, among others! I tried to give a quick and partial overview in the following schema.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sg9MlZe3HzC7Lcd20MHURA.png" /><figcaption>Non-exhaustive schema of some ML concepts and the questions they triggered.</figcaption></figure><p>They often come from (bad) experiences or best practices. Thus, at each postmortem, try to really define what concept you missed before starting the project and what impact it had afterward.</p><p>For example we could think of a conversation between a Data Scientist (DS) and a Product Manager (PM) such as:</p><ul><li>DS: How do you want to monitor the ML model?</li><li>PM: Hum, didn’t think of that… Well for this model I would need a to have ROI information pushed to me every month as this ML model tends to generate quarterly-based revenue.</li></ul><p>And here you will notice that nothing is about tech. We don’t talk notebooks, MLflow or BI dashboard with relevant metrics. We don’t evoke them because probably the right solution will not be among them. We don’t think in terms of solutions. On the other hand the monitoring concept in itself is enabling the Data Scientist to ask the right question to its PM and avoid a pitfall such as model drifting.</p><p>As the PM is often closer to the Business team, he will be able to specify what he needs. And then and only then, based on the PM answer, the Data Scientist will be able to look for the right tool to implement the monitoring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*doO_czN504-5sAoTwnzHeg.png" /><figcaption>Conceptual approach in ML development</figcaption></figure><h3>2. Ramp-up on technologies following your need</h3><p>Tech stack represents the field of possibilities. And obviously, you want to know what you can actually do, compared to what you want to do regarding your needs. In addition tech stack is also valuable to help you discover new concepts. Indeed specific tools are responding to specialized needs that you might not have forecasted.</p><p>So yes I won’t say that you should not use dedicated ML Ops tool at all. However what pushes you to use it, is key. Are you really using a tool based on experience and need, and do you know why it represents value for your business?</p><p>In all teams and businesses, you have to ramp-up. You are starting small with a data team of one or two members, and hopefully you will end up with a lot of people and sub-teams specialized in certain ML fields.</p><p>By following concepts, you will arrive to the right tech maturity that you need in order to solve your case. And probably, while your company will grow bigger and bigger, more maturity will be needed. However, there is no added value in targeting state-of-the-art when you don’t need it. Indeed if you have a too high level of technology for your maturity or need, it will certainly:</p><ul><li>cost you more money</li><li>require more maintainability</li></ul><p>So don’t be afraid of not being perfect. Yes, ML Ops is really different from what you can learn on in uni. You don’t have always the resources you need or develop in an ideal dev environment. While data science part is often done in a very controlled environment, ML in production can be dirty and messy. You have to handle the computing resources, the alerting channel, the monitoring possibility, the feature store… It seems that you will face many issues and this is where it is tempting to use a state-of-the-art tool per issue and decide that this will “solve” your problem. But it will not! Solving is not about answering perfect, but enabling ROI. So, really focus on what matters and mature the technology in phase with your need.</p><h3>3. Use case: Health check alerting with Metabase</h3><p>Let me give you a quick use case where concept over tool was quite a decisive approach in one project at Ovrsea.</p><p>We had just put a model into production that was giving our operational staff some weekly recommendations. This model was very sensitive and yet it was only a first draft version. We were working with a product manager to know how we should monitor this system.</p><p>While checking for our need, we found that the most important factor for success in this project is the ability to provide dynamic recommendations, which is the key in this project to maximize return on investment (ROI). Indeed here recommendation are only here to challenge the ops, and so we didn’t so much care about the actual figure given as recommendation, but the fact that recommendation itself was updated every week.</p><p>So we had to make sure that this ML model was working well and was being able to push new recommendations every week.</p><p>We could have thought of conventional alerting and monitoring system such as: Graphana &amp; Prometheus, <a href="http://neptune.ai/">Neptune.AI</a> or Amazon SageMaker alerting for the end point. However, we don’t use much these technologies for now and our need for this first version of our health check alerting was very simple. Moreover, this model is based on multiple bricks: feature store, end point model, and additional input to handle from users. Its architectural complexity is quite high and adapting state-of-the-art tech would have taken ages. At the same time, we know that while complexity was high due to interface between multiple bricks, the bricks itself were very simple and so errors would be easy to identify if we had a global alert.</p><p>That’s why we decided to put an <a href="https://www.metabase.com/docs/latest/questions/sharing/alerts">alert system on Metabase</a> which is our BI tool. This alert would warn us if a certain amount of recommendations were not predicted and pushed each week. In addition to this alert, we created a very simple dashboard with key BI metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EFKwo-gKSTImUhl9TpN9aQ.png" /><figcaption>Alert system on Metabase: the goal line represents the minimum number of updated recommendation that is required per week for the alert not to be triggered. Here we know that our system should at least push 2 400 recommendations per week.</figcaption></figure><p>If an alert is raised, then it worked as a health check and we received both the PM and the DS an email and a slack message. Then we only need to dig a bit among the different bricks in production to identify the error. In the meantime this alert also guarantees the project impact as dynamic recommendation update is the main key to unlock ROI.</p><p>I agree, this is neither a fancy solution nor a ML ops specific tool as we are using our BI tool to do the monitoring of a ML model. We also might not have the degree of observability we could obtain with other systems. But yet it was:</p><ul><li>very easy to put in place</li><li>cover our fundamental need regarding health-check monitoring and ROI estimate.</li></ul><p>We followed concept and best practices of monitoring that lead us use a modest solution that perfectly fills our requirements.</p><h3>Conclusion</h3><p>I think Data teams would gain in considering the problem with a paper and pen and try to focus on what really matters through concept and need. Once they have that, ML tools will obviously be needed to fulfill the task, but it should not polarize the debate.</p><p>However, while being very much more in favor of being pragmatical and ROI focus on project, I think that having a healthy routine of tech discovery outside project is key to be up-to-date and be able to know what is your field of possibilities. This would have mostly to do with having a healthy pedagogy culture in your data team. So be focus on ROI in your project, but also mature outside of it and ramp up with your need!</p><p>Thank you for reading and feel free to reach out! We will be more than happy to discuss and debate ML Ops best practices with other companies!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d2e8254d723" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ovrsea/ml-ops-development-moving-from-a-tool-based-to-a-conceptual-centered-approach-4d2e8254d723">ML Ops development: Moving from a tool-based to a conceptual-centered approach.</a> was originally published in <a href="https://medium.com/ovrsea">OVRSEA</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Analytics framework in Python: from scientific approach to actionable implementation]]></title>
            <link>https://medium.com/ovrsea/data-analytics-framework-in-python-from-scientific-approach-to-actionable-implementation-d47737382769?source=rss----91ee87b4533d---4</link>
            <guid isPermaLink="false">https://medium.com/p/d47737382769</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[scientific-method]]></category>
            <category><![CDATA[project-management]]></category>
            <dc:creator><![CDATA[Paul Couturier]]></dc:creator>
            <pubDate>Tue, 31 Jan 2023 09:48:45 GMT</pubDate>
            <atom:updated>2023-01-31T10:04:05.046Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6_qNWA2AXijOHyhzsvhESQ.jpeg" /><figcaption>Photo from Scott Graham, on <a href="https://unsplash.com/fr/photos/5fNmWej4tAA">unsplash</a></figcaption></figure><p>When I started doing analytics at Ovrsea, we were really at the start of our journey to understand what actionable data analytics is.</p><p>Indeed the first project I have conducted in data analytics at Ovrsea, I have used an online jupyter notebook with raw data stored as a csv on cloud. And that was for sure not a great idea!</p><p>We had many struggles on that project due to the way we organized and considered the project. The three main reasons that pushed me rethink our entire way to do data analytics were :</p><ul><li>Not reliable enough</li><li>Not agile enough</li><li>Not reusable enough</li></ul><ol><li>Not reliable enough</li></ol><p>Without testing and by simply using notebooks, our code could contain some undetectable errors and when we would discover ones, it would create severe pressure on all the team as all the results of the analytics could be rigged by this error.</p><p>2. Not agile enough</p><p>The thing with analytics is that often you don’t really know what you are looking for. Generally, you have a global question that you need to investigate but when you start the analysis you don’t know where it is going to lead you. And that’s completely fine as it’s part of such a project nature! But then when one of the stakeholders requires to have new features in the dataset, you need to be agile and this is hard when your data project is only based on a downloaded csv.</p><p>3. Not reusable enough</p><p>When we think analytics we often think one time job. And most of the time, that’s false. In the case of projects having some direct impacts on the company, it will trigger actions that can go from human process to algorithm implementation on your platform. Personally, I worked on a project that ended up being deployed as a custom deterministic algorithm, with a code that was not up to the standard of quality we want for deployed code. So you should never think of analytics as a one-shot project, that’s not the way to go!</p><p>Considering these issues, I have tried to tackle them by working on two axes : <strong>an analysis approach</strong> and <strong>a python framework</strong>.</p><h3>1. Analysis approach</h3><h4>Business &amp; Sciences</h4><p>Data analytics is a very complex discipline. It requires to have a scientific approach while being in a business context. Questions and challenges of stakeholders are needed but at the same time are hardly fitting in a scientific approach that requires to follow statistical precepts to prevent biases.</p><p>In fact, it’s all about finding the balance between having goals and outcomes in mind and yet being open minded to new ideas. We often talk about business being data driven, but <strong>we rarely estimate the quality of the analytics done by companies</strong>.</p><p>In an analysis, without any methods and with simple visual and unordered stacked jupyter notebooks, you will definitely face some of these issues:</p><ul><li>confirmation biases</li><li>p-value hacking</li><li>apophenia</li><li>confusion between correlation and causality</li></ul><p>Of course multiple statistical technics have been designed to overcome these issues. For example, well-defined statistical tests help you distinguish between mild visual hint and a significant result. However, even used properly, they could lead to multiple testing and hence p-hacking due to multiple interaction with stakeholder. Indeed, stakeholders will encourage you to search in all directions and frequently change conditions, and you will test twice the same thing even if you are strongly informed of such pitfall.</p><p>This is why I have been reflecting on proper project steps and a management framework to overcome these challenges.</p><h4>Project structure</h4><p>First we need a project structure that should help us:</p><ol><li>Explore the data</li><li>Without having to face data-dredging</li><li>And finally draw accurate conclusion</li></ol><p>To achieve that, <strong>the architecture I have recently adopted is EDA / pre-registration / CDA</strong>. It’s a three steps analytics project plan that helps you use data to its full potential while preventing biaises. For the record this approach doesn’t come from me, but from the very interesting paper of Jebb, A. T.</p><p>Step 1: EDA</p><p>Most of you might know the EDA, which stands for “Exploratory Data Analysis”. This has been developed by the famous statistician John Tukey in the 70s. It is the part of the analysis that aims at discovering your data using general mathematical tools. In this phase you can explore your data as much as you want to be able to understand it!</p><p>However at this stage you should not work on the final significant insights you will communicate about. EDA is only here to help you figure out what the data is about and what could be the hypothesis you could work on. But it is way too soon to draw conclusions.</p><p><em>Note: To keep in mind that EDA is just here to gather generic assumptions on the data I often only use 1/3 of the data at this stage. It allows to keep in mind the probabilistic aspect of the trends that we can spot at this stage of the analysis.</em></p><p>Step 2: Pre-registration</p><p>Once you have defined the possible assumptions you want to check and understood your dataset, then comes the pre-registration step. In scientific research, pre-registration is the phase where you define clearly the data experimentation you will conduct and register it for your peer to have a testimony of the experiment. It enables scientific research to avoid as much as possible biaises.</p><p>In a business context, this phase can be done by gathering all the stakeholders of the analysis and explaining them the hypothesis you are going to test and have their feedback before the experiment. The EDA already helped you understand the dataset and lead you toward the tests you would like to run. Stakeholders input will help you double check your strategy providing possible insights.</p><p>Step 3: CDA</p><p>This is a phase that is often less known than the EDA and it stands for Confirmatory Data Analysis. In this phase, you are only going to conduct all the tests you have defined with your stakeholder in the pre-registration ones and draw conclusions from them. This is how you will obtain maximal accuracy by avoiding re-designing these tests and simply following the ones you defined in the pre-registration phase.</p><p>In conclusion this approach of EDA/pre-registration/CDA is maximizing your impact and use of the data as you are able to :</p><ol><li>First explore your dataset during the EDA</li><li>Then define a clear scope in pre-registration with the stakeholders</li><li>Finally apply few accurate tests to draw significant results from your data in CDA.</li></ol><h4>Project management</h4><p>But even with the best architecture, your analysis might be a waste of time if not followed by well-defined project management guidelines. Indeed, two things are required to have an impactful analysis:</p><ul><li>Stakeholder should be informed and involved in the project. Analytics is all about communication and will not have any impact on the company if not guided and followed by people that will be able to take the right measure or process depending on the outcome of the project. We often consider data analysts as people sitting at a corner, and drawing beautiful graphs that will never be considered by anyone. This is definitely something to avoid as stakeholder will never be able to apply conclusions they didn’t take part in drawing.</li><li>Small scope needs to be defined. Start small, and then if needed, add more complexity. But doing the opposite is taking the bet that with more graphs of poorer quality you will be able to draw something meaningful. This seems like a losing strategy…</li></ul><h3>2. Python project framework</h3><p>Once you have a clear project management plan, you still need a python framework that will help you apply the guidelines previously defined smoothly.</p><p>I am not going to give you turnkey python structure project as I think this is will definitely not help you designing your own framework. Indeed, each framework must always be adapted to the size and the timeline of the project. For example, on my side I have recently tried to use <a href="https://kedro.org/">Kedro</a> which is the framework of QuantumBlack to design a python pipeline for analytics. After studying it, while being very interesting, it turns out that it was far too complex for my use and I recreated a python template specifically for this project.</p><p>That being said, some guidelines are common to any project and this is the ones you should care as they will help you respect all the principles we have seen above and avoid the pitfalls.</p><h4>Source your data directly in the project from the database</h4><p>An analytics project is often long. So you want to be sure to always stay agile considering the raw data you are using. Being directly connected to your database and sourcing your data using SQL requests stored in your analytics project is a very important practice. It will allow you:</p><ul><li>more flexibility considering new feature demands</li><li>being consistent with the parent database and hence enable an easy deployment if needed</li><li>to benefit for more freshness as you can update your data at any time</li></ul><p>This is a game changer, and it will only take to set-up quickly the database connection.</p><h4>Feature engineering should be done in TDD using pytest</h4><p>Test Driven Development — aka TDD — is a software development practice that relies on coding first the test and then create the function in an iterative way. I will not dig deeper in the TDD concept in this article as it would be way too long but let’s simply say that designing function in python that you can reuse is a must have during an analysis and TDD helps you do that. Moreover it gave you a safety net during addition of features as your development test will help you watch over your features to avoid breaking older ones.</p><p>What part of your code should you develop as functions with TDD ? Mostly the code of your feature pipeline as you will reuse it very often.</p><p>Remember, data analytics is often long, and considering it as a one-shot is the best way to eventually obtain no insight at all. Hence, by insuring your code quality and building pure function using TDD, you will be sure of the code you produced.</p><h4>Store your datasets corresponding to pipeline steps</h4><p>This is fairly unexpected that you only need one type of dataset during your entire study. This is due to the fact that you might need new view of the same dataset: aggregated, pivoted, summarized, in time series etc.</p><p>So don’t delete your tables, but rather store your data at each step and try to organise it in specific folders. This will enable you to be way more agile, while taking very few time to do so.</p><p>A typical convention you could choose to store these tables is the one specified by Bruce Philp and Guilherme Braccialli <a href="https://towardsdatascience.com/the-importance-of-layered-thinking-in-data-engineering-a09f685edc71">here</a> and supported by Kedro framework.</p><h4>Separate EDA and CDA in two different Notebooks</h4><p>EDA and CDA are analytical phases that often require visualization. Don’t try to conduct them on a terminal interface. Jupyter notebook while being “only” useful for visual purpose is still an amazing tool and you should not just stop using it. Once your features are computed using TDD in a well organised script, you can work on your visualisation and stats tests in jupyter notebooks. Still you should implement EDA and CDA in two separate notebooks to be sure that you will not have an exploratory phase mixed with statistical tests.</p><h3>Conclusion</h3><p>Data analysis is really about conducting a proper scientific research inside a business environment. Thus, you need to structure your project as well as your python framework to be sure to explore as much as possible your data, and still draw accurate conclusions guided by your stakeholder advice and knowledge.</p><p>Thank you for reading and feel free to reach out! I will be more than happy to learn how you lead your data analyses in python and what we could do to even improve our current framework.</p><h4>Source</h4><p>Jebb, A. T., Parrigon, S. &amp; Woo, S. E. Exploratory data analysis as a foundation of inductive research. <em>Human Resource Management Review</em> <strong>27</strong>, 265–276 (2017). <a href="https://www.researchgate.net/publication/306130819_Exploratory_data_analysis_as_a_foundation_of_inductive_research">https://www.researchgate.net/publication/306130819_Exploratory_data_analysis_as_a_foundation_of_inductive_research</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d47737382769" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ovrsea/data-analytics-framework-in-python-from-scientific-approach-to-actionable-implementation-d47737382769">Data Analytics framework in Python: from scientific approach to actionable implementation</a> was originally published in <a href="https://medium.com/ovrsea">OVRSEA</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to detect unused GraphQL operations]]></title>
            <link>https://medium.com/ovrsea/how-to-detect-unused-graphql-operations-2b9c5bc48afe?source=rss----91ee87b4533d---4</link>
            <guid isPermaLink="false">https://medium.com/p/2b9c5bc48afe</guid>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Guillaume Munsch]]></dc:creator>
            <pubDate>Fri, 02 Dec 2022 10:29:30 GMT</pubDate>
            <atom:updated>2023-04-13T13:55:35.603Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*yj4WZhs7LzJz-r-i" /></figure><p>At <a href="https://www.ovrsea.com/">Ovrsea</a>, we use GraphQL for our API. After several years, our codebase got bigger and bigger and we ended up encountering an issue: how to detect if queries or mutations are still being used.</p><p>In general, we want to remove code that’s not used anymore to avoid maintaining it for nothing. This is costly and useless.</p><p>With GraphQL, this is not an easy task because GraphQL queries are not used via direct imports. This leads us to the following question:</p><blockquote>How can you know if an operation is still being used or not?</blockquote><p>First of all, let’s define that. We’ll call <strong>“operation”</strong> a query or a mutation.</p><p>After refactors, optimizations, product evolutions, … removing the unused operations from your codebase is a good practice. You could do this manually for each of them, digging through the codebase and watching its usage on Apollo Studio, but wouldn’t it be nice to have it done automatically?</p><h3>Let’s create a script that does exactly this</h3><p>🙃 <strong>TL;DR: The final version of this script is available via an </strong><a href="https://www.npmjs.com/package/@ovrsea/graphql-detect-unused-operations"><strong>NPM Package</strong></a><strong> and all the code is open sourced </strong><a href="https://github.com/ovrsea/graphql-detect-unused-operations"><strong>HERE</strong></a><strong> on Github.</strong></p><p>This process is split in 3 steps:</p><ol><li>Find your GraphQL schema’s operations</li><li>Find all operations usages inside the codebase</li><li>Compare them 👍</li></ol><p>This script will be written in TypeScript.</p><h3>1. Find your GraphQL schema’s operations</h3><p>Here we’ll first generate a .json version of our schema which is way easier to parse. We’ll do this with the NPM package graphql-codegen. You will need the following devDependencies:</p><pre>&quot;devDependencies&quot;: {<br>    &quot;@graphql-codegen/cli&quot;: &quot;^2.12.1&quot;,<br>    &quot;@graphql-codegen/introspection&quot;: &quot;^2.2.1&quot;,<br>    &quot;@graphql-codegen/typescript&quot;: &quot;^2.7.3&quot;,<br>    &quot;@graphql-codegen/typescript-operations&quot;: &quot;^2.5.3&quot;,<br>    &quot;@graphql-codegen/typescript-resolvers&quot;: &quot;^2.7.3&quot;<br>  }</pre><p>Now, let’s create graphql-codegen config file and name it config.yml.</p><pre>#config.yml<br>overwrite: true<br><br>generates:<br>  ./graphql.schema.json:<br>    schema: &quot;./tests/schema.graphql&quot;<br>    plugins:<br>      - &quot;introspection&quot;<br>    config:<br>      apolloClientVersion: 3</pre><p>To generate the graphql.schema.json run the following command.</p><pre>npx graphql-codegen -config config.yml</pre><p>Now we can parse this json to extract the list of all available operations.</p><pre>const schema = require(&quot;path/to/my/graphql.schema.json&quot;);<br><br>const findOperationsInSchemaByType = (<br>  schema: Schema,<br>  type: string<br>): SchemaField[] =&gt; {<br>  const schemaFields = find(<br>    schema.__schema.types,<br>    (schemaType) =&gt; schemaType.name === type<br>  );<br>  return schemaFields?.fields ?? [];<br>};<br><br>const parseSchema = (schema: Schema): string[] =&gt; {<br>  return [<br>    ...findOperationsInSchemaByType(schema, &quot;Query&quot;),<br>    ...findOperationsInSchemaByType(schema, &quot;Mutation&quot;),<br>  ].map((elem) =&gt; elem.name);<br>};<br><br>const schemaOperationsList = parseSchema(schema);</pre><h3>2. Find all operations usages inside the codebase</h3><p>This isn’t an “easy” thing to achieve, at least manually. We’ll need to define all the files we want to check. For this we used <a href="https://www.npmjs.com/package/glob">Glob</a> with the following pattern: &quot;**/*.{ts,tsx,js}&quot; .</p><pre>const files = glob.sync(&quot;**/*.{ts,tsx,js}&quot;, { cwd: &quot;&quot; });</pre><p>And now it gets a little bit tricky. To extract the operations from these files, we feared a manual and laborious parsing of each file’s AST. Thankfully, after digging through GraphQL Tools documentation, we found a CodeFileLoader that’ll do all this job for us 🎉. By “job” I mean, opening the file, reading it, parsing its AST, extracting only the GraphQL related elements. Read more about it <a href="https://the-guild.dev/graphql/tools/docs/api/classes/loaders_code_file_src.codefileloader">HERE</a>.</p><p>Basically this allows us to create such a function that will extract the operations used in a file.</p><pre>const loadAndParseFile =<br>  async (filePath: string): Promise&lt;Operations&gt; =&gt; {<br>try {<br>      const sources = await loadDocuments(filePath, {<br>        loaders: [new CodeFileLoader()],<br>      });<br>      const fileOperations = sources<br>        .filter(({ document }) =&gt; document !== undefined)<br>        .map(({ document }: Source) =&gt; {<br>     // We&#39;ll explain this function right after<br>     // &quot;DocumentNode&quot; is a type from &quot;graphql&quot;<br>          return parseDocumentNode(document as DocumentNode);<br>        })<br>        .reduce((acc, currentOperations) =&gt; {<br>     // &quot;union&quot; comes from lodash and just removes the duplicates<br>          return union(acc, currentOperations);<br>        }, []);<br>      return fileOperations;<br>    } catch (error) {<br>      return [];<br>    }<br>  };</pre><p>There we have it. We just need to write the method that parses the DocumentNode to extract the operations.</p><pre>// DefinitionNode and OperationDefinitionNode come from &quot;graphql&quot;<br>const isOperationDefinitionNode = (<br>  definitionNode: DefinitionNode<br>): definitionNode is OperationDefinitionNode =&gt;<br>  definitionNode.kind === &quot;OperationDefinition&quot;;<br><br>const parseDocumentNode = (<br>  node: DocumentNode<br>): string[] =&gt; {<br>  const resolvedOperations =<br>    node.definitions<br>      .filter(isOperationDefinitionNode)<br>      .map((definition: OperationDefinitionNode) =&gt; {<br>        return definition.selectionSet.selections<br>          .filter(isOperationDefinitionNode)<br>          .map((selection) =&gt; {<br>            return selection.name.value;<br>          });<br>      }) ?? [];<br> // &quot;flatten&quot; comes from lodash as well. <br> return flatten(resolvedOperations)<br>};</pre><p>We now have a fully functional method called loadAndParseFile that allows us to extract all the operations used in a file.</p><p>We now just need to call this method for each file. We can do it like this:</p><pre>const getAllOperations = async (<br>  files: string[],<br>) =&gt; {<br>  const parsedFiles = await Promise.all(files.map(loadAndParseFile));<br><br>  return parsedFiles.reduce(<br>    (acc, currentOperations) =&gt; {<br>        return union(acc, currentOperations);<br>      },<br>    []<br>  );<br>};<br><br>const allUsedOperationList = await getAllOperations(files);</pre><p>And there you have it, we have all the operations used in our codebase 🎉.</p><h3>3. Compare them 👍</h3><p>For this final step, we decided to simply use the difference method from Lodash that detects the differences between two arrays.</p><pre>const unusedOperations = difference(<br>      schemaOperationsList,<br>      allUsedOperationList<br>    );</pre><p>Good job, you can now use this tool directly in your CI for example 🙂.</p><h3>How did we use it?</h3><p>As explained a little bit earlier in this article, at <a href="https://www.ovrsea.com/">Ovrsea</a> we worked on this problem and went a bit further. You can read about it on <a href="https://github.com/ovrsea/graphql-detect-unused-operations">the package’s Github page</a>.</p><p>We implemented several other options like the detection of unused fragments, a whitelist of operations that we do not want to remove (for ongoing development for instance) or even patterns to ignore some specific folders. Contributions are more than welcome 🙂.</p><h3>How it helped us</h3><p>Adding this package to our CI allowed us to remove 20 unused operations and 4 unused fragments right when we implemented it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b9c5bc48afe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ovrsea/how-to-detect-unused-graphql-operations-2b9c5bc48afe">How to detect unused GraphQL operations</a> was originally published in <a href="https://medium.com/ovrsea">OVRSEA</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why you should use jest-when to enhance your test mocks]]></title>
            <link>https://medium.com/ovrsea/why-you-should-use-jest-when-to-enhance-your-test-mocks-e29287d70717?source=rss----91ee87b4533d---4</link>
            <guid isPermaLink="false">https://medium.com/p/e29287d70717</guid>
            <category><![CDATA[mock]]></category>
            <category><![CDATA[jest]]></category>
            <category><![CDATA[unit-testing]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Maud]]></dc:creator>
            <pubDate>Tue, 04 Oct 2022 12:30:14 GMT</pubDate>
            <atom:updated>2022-10-04T12:30:13.968Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zWJKG_9cX0gUzejSdSgF2Q.jpeg" /></figure><h3>TL;DR</h3><p><strong><em>jest-when</em></strong> is a powerful, easy-to-use library that allows to specify return values depending on the arguments passed to the mocked function.</p><p>Here is why you should use it:</p><ul><li><strong>it overcomes some limitations of jest’s native mock methods</strong>; notably, the need to specify the correct invocation order of the mocked function — which can be a nightmare in the case of asynchronous calls;</li><li><strong>it allows to better decouple your tests from the implementation</strong>; notably by becoming independent of the invocation order of the mocked function;</li><li><strong>it improves the readability of your tests</strong> by making them more concise and comprehensible;</li><li><strong>mastering it requires a minimal investment of your time </strong>thanks to the similarity of its API to that of jest.</li></ul><p>Here is what you have to do to use jest-when within a test:</p><pre>// 1. Specify the function to mock<br>jest.mock(&quot;./path/to/functionToMockFile&quot;, () =&gt; ({<br>  <strong>functionToMock</strong>: jest.fn(),<br>}));</pre><pre>// 2. Specify the return value based on the arguments received <br><strong><em>when</em></strong>(functionToMock)<br>  .calledWith(matchedArguments)<br>  .mockReturnedValue(returnValue);</pre><pre>// NB: mockReturnedValue may be replaced by any jest method: mockResolvedValue, mockImplementation, mockResolvedValueOnce...</pre><p>Mocking is a technique consisting of replacing dependencies with pre-programmed data, thereby allowing to isolate the test subject. In JavaScript, one of the most popular framework for tests is <a href="https://www.npmjs.com/package/jest">jest</a>. Handily, jest comes natively with plenty of very handy mock methods that allow to intercept calls and set controlled return values. Consequently, its mock methods are widely used by the community.</p><p>However, in certain cases, these built-in methods scarcely manage to do the trick and can become delicate to handle without confusion.</p><p>In this article, we will go through an example to uncover such limitations of the native jest mock methods and show how <strong><em>jest-when</em></strong>, another JavaScript library complementing jest, can overcome some of these weaknesses.</p><p>Let’s begin with a simple example.</p><p>Say we want to compute the euclidian distance between two locations. As a reminder, the <a href="https://en.wikipedia.org/wiki/Euclidean_distance">euclidian distance</a> between two points p and q of cartesian coordinates (p1, p2) and (q1, q2) is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QKI8VXkrXNCXcxjy-glG_A.png" /></figure><p>This can be implemented as follows in a function named computeDistance:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c8a9b565d38afb8475bf5f6cb79001c/href">https://medium.com/media/6c8a9b565d38afb8475bf5f6cb79001c/href</a></iframe><p>Here, the computeDistance function takes two objects of type Location as arguments and calculates the euclidian distance between these two locations by :</p><ol><li>fetching the coordinates of the two locations via successive calls to the fetchCoordinates function to retrieve their coordinates;</li><li>applying the euclidian distance formula to the fetched coordinates.</li></ol><p>The implementation of the fetchCoordinates function is of no real interest in our case study. The only thing we need to know about this function is its interface: it takes a location parameter and returns its coordinates as drafted hereunder.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ece3704bcc89d53eb21517dc6609f795/href">https://medium.com/media/ece3704bcc89d53eb21517dc6609f795/href</a></iframe><h3>Standard way of mocking: jest native methods</h3><p>If we want to make sure our computeDistance function is correctly implemented, we would probably write a test along the lines of:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71051340f4f98991571d7c0c8089001a/href">https://medium.com/media/71051340f4f98991571d7c0c8089001a/href</a></iframe><p>However, given that fetchCoordinates encapsulates a call to an external API, you would most likely need to <strong>mock </strong>this function.</p><p>The function tested (computeEuclidianDistance) makes two distinct calls to fetchCoordinates and, to have the test pass, we need it to return two distinct values (first, the coordinates for Rome and second, those for Tirana). Luckily, there is a native jest mock method allowing to return a given resolved promise value at each different invocation: <a href="https://jestjs.io/docs/mock-function-api#mockfnmockresolvedvalueoncevalue">mockResolvedValueOnce()</a> (or, equivalently, <a href="https://jestjs.io/docs/mock-function-api#mockfnmockreturnvalueoncevalue">mockReturnValueOnce()</a> in the case a synchronous function is being mocked).</p><p>In practice, prepending the following lines to the test above will suffice:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ce5916db0164c64716334e810a8904f/href">https://medium.com/media/0ce5916db0164c64716334e810a8904f/href</a></iframe><p>With this piece of code, the first time fetchCoordinates is invoked, it returns the coordinates of Rome (0, 0) and the second time it returns those of Tirana (0, 1).</p><p>If you need to run a second test with mocks returning different values in a different order, the following piece would work without any problem:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3fa1910ad7148d1d1410486b845e33de/href">https://medium.com/media/3fa1910ad7148d1d1410486b845e33de/href</a></iframe><blockquote>This is reasonably simple and allows tests to pass without any problem. Nonetheless, let’s keep in mind that this requires to <strong>set mocked values in the exact order in which the function is called</strong>.</blockquote><p>Now, let’s spice things up a bit !</p><p>Let’s assume we want to find out which distance is the shortest between a list of possible locations and a reference point. In that case, we may define a function computeMinDistance that would look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a27a51c5c6238979ab760f3753cb85a6/href">https://medium.com/media/a27a51c5c6238979ab760f3753cb85a6/href</a></iframe><p>Here, computeMinDistance starts by computing all distances between each location of the input list on the one hand and the reference point on the other hand, and then selects the minimal distance among the lot.</p><p>If we wanted to find the minimal distance between a given reference point (say, Rome) and two other locations (say, Tirana and Berlin), here is the test that we would implement:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7cfcb76a49b7b2edab1bcef931402b87/href">https://medium.com/media/7cfcb76a49b7b2edab1bcef931402b87/href</a></iframe><p>This code is straightforward, but there is one thing missing: we need to define the successive outputs that fetchCoordinates returns. In this case, the function is called 4 times:</p><ol><li>to fetch Rome coordinates in the Rome-Tirana distance computation;</li><li>to fetch Tirana coordinates in the Rome-Tirana distance computation;</li><li>to fetch Rome coordinates in the Rome-Berlin distance computation;</li><li>to fetch Berlin coordinates in the Rome-Berlin distance computation.</li></ol><p>Now, here is the tricky part: <em>given our implementation, in which order are these four calls made ? </em>Actually, this order <strong>depends on the speed at which each promise is processed by the backend side</strong> (see <a href="#49df">Appendix below for a proof of this result</a>).</p><blockquote>Therefore, in our test implementation,<strong> the order in which the mocked values are specified is tightly coupled to the implementation of the function that invokes the mocked one</strong>. As a consequence, even though native jest mock methods such as <em>mockResolvedValueOnce(), mockReturnValueOnce()</em> or <em>mockImplementationOnce()</em> are very handy to make mocks, they can become tricky to use when the invocation order is not crystal clear (as within <em>Promise.all</em> groups).</blockquote><p>In addition, in such cases where the invocation order is a bit blurry, tests become much less readable and comprehensible for other developers reading the code (they would need to understand in depth in what order JavaScript processes each line of code).</p><h3>Mock return values based on arguments received</h3><p>To avoid the trouble of determining the exact order in which the mocked function should return each successive value, the easiest way would be to mock the returned values based on the arguments passed to the mocked function.</p><p>This is exactly what <a href="https://www.npmjs.com/package/jest-when">the <strong><em>jest-when</em></strong> library</a> is made for.</p><p>Hereafter is what we would write to adapt our example using <strong><em>jest-whe</em>n</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af1a132af2899fbcf6632b02201d8c44/href">https://medium.com/media/af1a132af2899fbcf6632b02201d8c44/href</a></iframe><p>As you can see, the syntax for <strong><em>jest-when</em></strong> is pretty simple:</p><pre><strong><em>when</em></strong>(functionName)          // 1. Wrap your function with when()<br>  .calledWith(arguments)    // 2. Add the arguments to match<br>  .mockResolvedValue(value) // 3. Specify the return value</pre><p>Once this is done, anytime the mocked function is called with the passed arguments, it returns the specified value. As such, there is no more need to find out in which order the mocked function is called: the mere arguments it receives determine the return value.</p><p>Another advantage of <strong><em>jest-when</em></strong> is the similarity of its API with that of jest. As such, all usual jest behaviors and features are conserved in <strong><em>jest-when</em></strong>:</p><ul><li>The scope of the mocked function is bounded to the block where it is defined. In other words, if you need to setup a default return value for given arguments, define it at the top of the test file; but if you need different return values with the same argument in two distinct tests, define your mock within the adequate describe or test block.</li><li>It supports most existing features of jest such as: chaining and replacement of mock trainings, support for single calls as well as resolved and rejected promises, resetting between tests, use of most jest matchers, etc…</li></ul><p>You can find out all the existing features of this library on <a href="https://www.npmjs.com/package/jest-when">the <strong><em>jest-when</em></strong> documentation page</a>.</p><h3>Conclusion</h3><p>In this article, we uncovered one limitation of jest’s native mock methods: namely, the need to specify mocked return values in the correct order, which can be tricky notably in the case of promises resolved concurrently.</p><p>Next, we have shown that <strong><em>jest-when</em></strong>, a lightweight JavaScript library complementing jest, succeeds in overcoming this limitation by matching mocked returned values with specific call arguments. Using this library to mock your dependencies also makes your tests tremendously more comprehensible, robust and modulable. Last, given the similarity of its API with that of jest, you will need to invest a very negligible amount of your time to learn to use it. All in all thus, jest-when is a great library to use to enhance your test mocks ; now you don’t have any reason not to be using it anymore !</p><h3>Appendix — Proof that the order in which promises are fulfilled is not preserved on the backend side</h3><p>In this appendix, we will bring proof that the order in which promises are fulfilled is <strong><em>*not*</em></strong> preserved on the back-end side. Let’s come back to our computeMinDistance function which can be tested with the following piece of code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7cfcb76a49b7b2edab1bcef931402b87/href">https://medium.com/media/7cfcb76a49b7b2edab1bcef931402b87/href</a></iframe><p>Now, let’s define how fetchCoordinates should be mocked. We thus need to define the successive outputs that fetchCoordinates returns. In this case, the function is called 4 times:</p><ol><li>to fetch Rome coordinates in the Rome-Tirana distance computation;</li><li>to fetch Tirana coordinates in the Rome-Tirana distance computation;</li><li>to fetch Rome coordinates in the Rome-Berlin distance computation;</li><li>to fetch Berlin coordinates in the Rome-Berlin distance computation.</li></ol><p><em>Given our implementation, in which order are these four calls made ?</em></p><p>Calls <strong>1</strong> and <strong>2</strong> on the one hand (Rome-Tirana distance computation), and <strong>3</strong> and <strong>4</strong> on the other hand (Rome-Berlin distance computation) are made sequentially within the computeDistance calls. Hence, these calls are necessarily made in the following order:</p><ul><li><strong>1</strong>&lt;<strong>2</strong> on the one hand (read ‘&lt;’ as ‘precedes’ or ‘is called before’);</li><li><strong>3</strong>&lt;<strong>4</strong> on the other hand.</li></ul><p>Since the two invocations of the computeDistance function are grouped within a Promise.all call, they are run concurrently, from which ensues that the invocation order should be: <strong>1</strong>~<strong>3</strong> &lt; <strong>2</strong>~<strong>4</strong>.</p><p>In this case, we would thus need to set the mocks in the following order:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2f1b7a6d06b45d45f9d10e44c2f503c/href">https://medium.com/media/d2f1b7a6d06b45d45f9d10e44c2f503c/href</a></iframe><p>However, this order depends on the speed at which each promise is processed by the backend side. Indeed, if a more complex implementation ensued a longer time to execute computeDistance with the first arguments (Rome/Tirana) than the second ones (Rome/Berlin), the correct mock order would be different.</p><p>We can prove this simply by creating a timeout wrapper (hereunder named slowDown) and modify the computeMinDistance function so as to wrap it around the first execution of computeDistance. Hereunder are these modifications:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef95565a25b28a78ca41c6ff9bb576df/href">https://medium.com/media/ef95565a25b28a78ca41c6ff9bb576df/href</a></iframe><p>In this case, if we keep the mocks defined in the order previously set (<strong>1❤</strong>&lt;<strong>2</strong>&lt;<strong>4</strong>), the test fails because fetchCoordinates are invoked in a different order here. Indeed, because of the lag of 100ms preceding the first computeDistance call (with Rome/Tirana locations), the second computeDistance call (with Rome/Berlin locations) is invoked first, thus modifying the required mock order to: <strong>3</strong>&lt;<strong>4</strong>&lt;<strong>1</strong>&lt;<strong>2</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/343d1cea1f3ece78fda0fd6d58c65618/href">https://medium.com/media/343d1cea1f3ece78fda0fd6d58c65618/href</a></iframe><p>With the mocks defined in this order, the test finally passes.</p><blockquote>All in all, thus, <strong>the order in which the promises are received by the backend is *not* preserved</strong> as they may be processed at different speed (this is in stark contrast with the behavior on the <a href="https://www.geeksforgeeks.org/javascript-promise-all-method/">the client-side where the order is maintained</a> — since the order of the resolved elements in the resulting array matches the order of the promises passed to <em>Promise.all</em>).</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e29287d70717" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ovrsea/why-you-should-use-jest-when-to-enhance-your-test-mocks-e29287d70717">Why you should use jest-when to enhance your test mocks</a> was originally published in <a href="https://medium.com/ovrsea">OVRSEA</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Manifesto on data testing at Ovrsea: an architectural design]]></title>
            <link>https://medium.com/ovrsea/manifesto-on-data-testing-at-ovrsea-an-architectural-design-3c814af33be0?source=rss----91ee87b4533d---4</link>
            <guid isPermaLink="false">https://medium.com/p/3c814af33be0</guid>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[test]]></category>
            <category><![CDATA[stategy]]></category>
            <category><![CDATA[philosophy]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Paul Couturier]]></dc:creator>
            <pubDate>Fri, 16 Sep 2022 10:27:16 GMT</pubDate>
            <atom:updated>2022-09-16T11:50:41.047Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LBcx2Mp0GUGjUPYqIv8z9A.png" /><figcaption>Image 1: Photo by Sven Mieke of a blueprint — unsplash.</figcaption></figure><blockquote>TL;DR<em>: At Ovrsea we have been working for some weeks now on how we could improve our way to test and give the right data to our users. And we finally came up with a philosophy that supports and drives strongly our data testing implementation process.</em></blockquote><blockquote><em>We have created a data testing philosophy that relies on three mantras: “Ownership matters”, “Perfect accuracy is no goal”, “Process &amp; culture over project”.</em></blockquote><blockquote><em>Using these guidelines, we have defined an implementation strategy that is based on two types of test: sanity check &amp; data quality test.</em></blockquote><h3>Context</h3><p>Data testing is a pain point for any tech company. Never satisfied by the accuracy and always looking for the “right” coverage, these challenges lead any data team to endless discussions on test implementation…</p><p>At Ovrsea we have been working for some weeks on how we could improve our way to test and give the right data to our users. And we finally came up with a philosophy that supports and drives strongly our data testing implementation process.</p><p><strong><em>Ok but before starting what exactly is data testing ?</em></strong></p><p>Generally speaking, testing is the process of executing a program to find errors. In the data context,<strong> tests aim at finding logic flaws that should not happen for the data to be coherent</strong>. What is not data testing ? It is not a way to monitor data shift or outlier. Incoherent data is different from outlier as it should have 0 chance of happening.</p><p>In a sense you can think of data testing as … architectural design.</p><p>The <strong>blueprint is what you design conceptually and hope to see build</strong> and the <strong>building itself is your actual schemas and data pipeline</strong>. Testing is simply checking on the field how your building actually looks like and if the structure may not have any flaw that would cause some heavy consequences for people using this building.</p><p><strong><em>But why are we testing again ?</em></strong></p><p>We are testing because no one expects an organisation with hundreds of tables and dozens of data sources from not having any flaw in it. As a data team, our role is to provide accurate data to any internal or external user. BI as well as sensitive reporting or ML projects depend on the accuracy of our data. Thus testing is a way to avoid missing new mistakes and to get people to trust our data by alerting them even before they noticed the error. As a building never exactly looks like its blueprint, you need to check and find the incoherencies and potential threat to its stability.</p><h3>1. Data testing philosophy</h3><p>One of the main goals at the data team is to provide accurate data to all our users internally and externally. After reviewing literature and facing concrete implementation challenges, we have been defining our data testing philosophy around three major pillars.</p><h4>“Ownership matters”</h4><p>In general, tests should be owned and designed by those who are in charge of either its creation or its use. Indeed they are the ones knowing the logic that data should follow. If you have not designed the blueprint or don’t understand exactly what are the overall implications between system interactions in the building, it is very hard to say if the actual system is coherent or not.</p><p>In the case of Ovrsea two majors stakeholders intervene in this situation :</p><ul><li><strong><em>Tech team</em></strong> who are producing the data used in our plateform</li><li><strong><em>Business teams </em></strong>(Revops, OpsExc, Billing<strong><em>)</em></strong> who are in charge of BI or external softwares</li></ul><p>Both teams have a role to play. Tech teams are extremely good at understanding links between tables, front &amp; back etc while business teams know the exact output they should expect with a high level of abstraction from the code itself. Both should be mainly the owner and designer of the tests while the data team should only help implementing these tests.</p><p>However as a data team we are still in charge of some part of the data creation. Indeed we are doing transformations to enable the tech practical data tables to become readable and usable by anyone non tech related at Ovrsea. These transformations are also being tested and here the data team should take the lead.</p><h4>“Perfect accuracy is no goal”</h4><p>Test considering your ROI: if testing is taking longer than needed or more expensive than the consequences of an error, then STOP. Indeed if the building is not exactly looking like the blueprint it doesn’t mean it is catastrophic, it simply depends on the consequences of it.</p><p>Ok but how can we do that ? Concretely to improve you ROI you would need to focus on high coverage of specific zones. These zones should be defined by their usage frequency or their criticality level.</p><p>For example in your data pipeline these zones could typically be specific layers of transformation. Once you found the layer to test you should focus your tests on what you think could break or be wrong. Testing is not a reassuring task and should never be. If you are only focusing on data that is easy and unlikely to be inconsistent (break old word), you will never find any errors and still believe you pipeline is perfect. Testing is an appetite for finding errors. When errors are found while you are testing then don’t be disappointed, this means you have done your job! As we can’t expect our pipeline to be perfect, we should be grateful for finding errors such as we improve our accuracy. So test until it breaks!</p><h4>“Process and culture over project”</h4><p>As a data team you are mainly in charge of providing the test framework and the process to be able to implement the tests as quickly as possible. But more than that, you need to insufflate the culture of data testing in all the others teams! Testing is often laborious if the wrong routine has been put in place. This will never be a one time project but it should be a wise and well defined process that should be improved each time a lack has been found. As you are not going to check only once your building is completely finished, you should give process to follow the construction all along its lifecycle.</p><h3>2. Test process &amp; strategy at Ovrsea</h3><p>A philosophy is nothing without implementation. Before defining this philosophy, we have been searching and benchmarking on all the ways to do data testing on our pipeline considering our current stack. You can found all this information in <a href="https://medium.com/@paul.couturier/data-testing-framework-with-python-and-dbt-e80b9002ced3">our previous article</a>.</p><p>So in our process and implementation strategy we need to mix and create the perfect match between : enabling owners to design tests, considering the ROI of test and providing an efficient process and framework to insufflate testing culture at Ovrsea.</p><p>To answer these three pillars we have been defining a strategy relying on two kinds of test that we have called :</p><ul><li><strong>Sanity checks</strong> : requested by other teams (handled by the dbt macros test framework)</li><li><strong>Data quality test</strong> : defined by the data team itself (handled by the dbt generic test framework and pytest)</li></ul><p>These two categories of test answer different needs to cover the three pillars defined above:</p><ol><li>“Ownership matters”</li><li>“Perfect accuracy is no goal”</li><li>“Process and culture over project”</li></ol><h3>Sanity checks</h3><p>These tests are designed upon request from other teams and implemented by the data team. Because we are not in contact with the logic of code nor the business intelligence, we cannot design and think of these tests by ourself. These tests are covering general purpose of error between the fabrication of tables by tech and the final use by BI.</p><p>This should be complex tests specifically asked by external team to check a behaviour that is hard to check otherwise due to the layer of complexity from raw table to business understanding.</p><p>Dbt macros test offer us a simple framework to implement these tests as SQL statement using singular test concept.</p><h4><strong>Example of a BI sanity check</strong></h4><p>Our BI team needed to check that if the status of a client was either “open”, “demo” or “first quotation”, then the variable “number_of_opportunity_won” should be equal to null. It allows them to test for inconsistencies in the customer journey. After the request was made by our BI team, we implemented a dbt singular test using SQL to check this statement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJXeSK9-nd3qqp_95sFh5A.png" /><figcaption>Image 2: Code of a dbt singular test asked by one of our BI team to check the client journey consistency</figcaption></figure><h4><strong>Process</strong></h4><p>The test demand is done through a specific ticket to the data team. Partly to enforce the use of tests, we are using the <a href="https://about.gitlab.com/handbook/business-technology/data-team/direction/data-champion/">Data Champion organisation</a>. Each data champion participates in general monthly meeting as well as a weekly 121 meeting. In these meetings we can talk about the test definition and strategy for each team. This is a way to communicate efficiently with them.</p><h3>Data quality test</h3><p>However to answer the need to have a quick and yet efficient safety net and to handle the fact as a data team we are also creating some data in our pipeline, we need to have very easy implemented tests of our own. We called these tests the data quality test.</p><p>Data quality tests are different from sanity checks as they are only superficial tests that aim at seeing huge inconsistencies. In fact they fulfill the need for general checking of the pipeline transformation and could catch up errors few days before a human would.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*Bc2yg119Wdy7Bi1dEwT7BQ.png" /><figcaption>Image 3: Code of a dbt generic custom test triggered on the column “created_at” to check the freshness of this column in hours.</figcaption></figure><p>These are simple tests designed by us using the custom generic test framework of dbt (or the pytest framework for test on python function). We have created 5 templates of custom generic dbt tests that we can apply anywhere in the pipeline on any table or column in one sec. Implementing this test should not require any intense business reflexion. If it does, it means it is a mistake. They should be transformed into a sanity check test and the data team should not be the owner of this test.</p><p>In regards to the pillar “Perfect accuracy is no goal” even if these tests are very quick to implement we are still prioritizing high value layer to test: we have started by testing the most used tables in self-service and the sensitive exposures.</p><h4><strong><em>Process</em></strong></h4><p>Two processes are applied to implement tests :</p><ul><li><strong>At each merge request:</strong> if you have modified a table, you need to review the tests of this table to see directly any update you could do. This avoids having to write tests weeks after updating the table, which would mean wasting time understanding the table again.</li><li><strong>Testing day:</strong> when we need to target a new layer or work on technical debt regarding test, all our Data team is dedicated to implement or update tests during either half day or an entire day. This is also a good way for the team to be up to date on schema modifications.</li></ul><h3>Conclusion</h3><p>From a benchmark of testing possibilities on our stack to creating a philosophy and finally transforming it into a concrete strategy, the road has been long and hard.</p><p>The final framework gives us very good satisfaction right now but we need to keep on improving and modifying it as new challenges appear. Moreover one task remains : How to be sure to trigger actions from other teams when tests are failing ? Is the alerting system downstream adapted enough ? Well, we are still working on this final part but an article should be coming soon so stay tuned !</p><p>Thank you for reading and feel free to reach to us! We will be more than happy to discuss and debate on data testing best practice with other companies!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c814af33be0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ovrsea/manifesto-on-data-testing-at-ovrsea-an-architectural-design-3c814af33be0">Manifesto on data testing at Ovrsea: an architectural design</a> was originally published in <a href="https://medium.com/ovrsea">OVRSEA</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data testing framework (with python and dbt)]]></title>
            <link>https://medium.com/ovrsea/data-testing-framework-with-python-and-dbt-e80b9002ced3?source=rss----91ee87b4533d---4</link>
            <guid isPermaLink="false">https://medium.com/p/e80b9002ced3</guid>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[test]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Paul Couturier]]></dc:creator>
            <pubDate>Fri, 16 Sep 2022 10:27:04 GMT</pubDate>
            <atom:updated>2022-09-16T10:02:47.549Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>TL;DR: This article presents our data testing possibilities regarding our dbt/python pipeline stack. It provides a regard on how, what &amp; when we can test. Understanding our own stack possibilities regarding tests is a crucial step to create a data testing philosophy &amp; strategy!</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G3eNpcdMrDvh_-eDiKD3nA.jpeg" /><figcaption>Image 1: Photo of code by Markus Spiske</figcaption></figure><p>In this short article we are going to present our data testing framework regarding our stack. This technical article comes from a need for us as a Data team to benchmark the possibilities we have when it comes to data testing. Indeed, data testing strategy is somehow a balance between the philosophy you have developed and the technological possibilities you have. So let’s dive right into our stack!</p><h3>1. Description of the stack</h3><h4>Pipeline transformation</h4><p>In the Ovrsea data team, we are in charge of transforming raw data into actionable data either for analytics or to provide external software with data. The various sources of data, as well as the multitude of tables force us to structure our pipeline.</p><p>The principe of pipeline design is clear : defining a general task by layer and parallelizing specific transformations on tables in each layer.</p><p>Our main data pipeline is constituted :</p><ol><li>first by <strong>a raw data</strong> schema that stores every database (with a dump and restore)</li><li>then a first <strong>cleaning process</strong> is done with soft delete and simple cleaning tasks executed</li><li>after that comes the <strong>company definitions</strong> to respect the “single source of truth” guideline and to enable clear definitions of any indicator shared at a company level.</li><li>some general resolving are then done in the <strong>intermediate tables</strong></li><li>finally, <strong>ready-to-use tables</strong> and tables for <strong>exposures</strong> are computed.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I7sI1wqdZs9NYlp6fr2mQg.png" /><figcaption>Schema 1: Description of our main data transformation pipeline at Ovrsea</figcaption></figure><h4>Pipeline stack</h4><p>To perform these transformations we are taking advantage of the power of three huge and well known technologies. Transformation is done using a mix between python and dbt while the pipeline is orchestrated by Airflow: details of each use is recalled in the schema below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tY7OP-xxcsjHQPGJms6wbg.png" /><figcaption>Schema 2: Details of the use of dbt, python and framework in our stack</figcaption></figure><p>So this means that without any additional software to our pipeline <strong>we can only use python and dbt as framework to implement test!</strong></p><h3>2. Data testing types</h3><p>One thing to get is that when talking about data you can test two things:</p><ul><li>The data itself</li><li>The transformations of the data</li></ul><p>Here are our possibilities to test these two elements considering our stack.</p><h4>D<strong>bt test</strong></h4><p>Dbt framework enables the user to implement simple tests on any raw or transformed data tables using SQL. Dbt is a powerful allie when it comes to test as it provides two very different types of test:</p><ul><li><a href="https://docs.getdbt.com/docs/building-a-dbt-project/tests">Singular test</a> : which are one-shot SQL statements in macros to test a specific behaviour of a specific table. This category of test enables complex solution-oriented test at a medium to low cost. Indeed the test is an SQL statement that adresses one complex check that you want to do on a specific table.</li><li><a href="https://docs.getdbt.com/docs/building-a-dbt-project/tests">Generic test</a> : are reusable dbt tests that can still be customized by parametrizing inputs and can be apply many times on tables using the dbt documentation yaml. These test are often simple but have the advantages of being very easy to add on any table, any columns in a sec. By creating new generic test — through <a href="https://docs.getdbt.com/guides/legacy/writing-custom-generic-tests">the custom feature test</a> — you can design your testing plan and address generic issue.</li></ul><h4><strong>Pytest library</strong></h4><p>Pytest handles the test on the python functions. This is simply unit test where you provide specific input and the expected output and see if you function works as expected. This is a very common framework for dev but here it will be used to test the python function that directly transform our data to generate our pipeline.</p><h3>3. Data testing orchestration</h3><p>The last question we need to answer is when are we testing ? In fact we have three possibilities:</p><ul><li>Testing at each merge request (as part of our CI/CD)</li><li>Tests embedded in the main data transformation airflow pipeline</li><li>Tests on general data using an other specific pipeline (launched once a day)</li></ul><p>These possibilities are very interesting as they involved diverse opportunities in terms of test and alerting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SUYfynwJfKNHtXtGT1o-2w.png" /><figcaption>Schema 3: Description of the different possible test triggers and their implications</figcaption></figure><h4>Testing at each merge request</h4><p>During a PR we will be more interested in testing things that do not depend on data and only need to be tested once the code is ready; that is: python functions.</p><p>Indeed, the test on these functions will not be impacted by the real data as the test are relying on “dummy” data designed to test the specific behaviour of the function using pytest. Thus testing them only when the code changes during PR makes perfect sense!</p><p>Note: we are also working on triggering an airflow pipeline during the CI/CD to check that the entire data pipeline is computable with the code modified. However this is still a work in progress.</p><h4>Testing embedded in the data transformation pipeline</h4><p>During the data pipeline the things that we want to test will be the tables themselves and in particular to trigger critical tests. Indeed the interest in testing during the pipeline is to stop the pipeline if the test fails. So we have to consider dbt test that are very crucial for the good health of the pipeline. Generating downstream tables wouldn’t make sense if these tests fail and pushing this rigged data to our BI tools or integrated softwares would be risky.</p><h4>Tests on general data during a specific pipeline</h4><p>Finally on all generated and stored tables, we want to test the dbt tests that are not critical, but still important. For the least critical tests, it still makes sense to avoid enabling this test to break the pipeline. Alerts will enable us act before the next data pipeline runs.</p><p><em>Note: To trigger this test once a day we are defining another Airflow pipeline independent from the data pipeline.</em></p><p>In fact triggering a test during the data transformation pipeline or on a different dedicated pipeline is really about finding balance between freshness and accuracy. Blocking the entire pipeline for an error is really impacting for BI so we should choose wisely what should block the pipeline, and what should raise our attention without breaking the pipeline.</p><p><strong>Here are two examples of test of either breaking or non breaking test.</strong></p><p><strong>A breaking test</strong>: “There is a mismatch between currency index and prices in our quotations” This could have massive impact and should be breaking to avoid mis-BI use.</p><p><strong>A non breaking test:</strong> “The ETD date of a shipment is inaccurate” This should still trigger an alert, but as no massive or critical data is affected, we should still provide data to our BI team, as we believe this could be a specific case to deal with.</p><h3>Conclusion</h3><p>Dbt mixed with python is a powerful and efficient engine to handle testing framework during the entire pipeline. This first study was very important for us as it enables us implement our testing strategy regarding our philosophical drivers. To know more about data testing philosophy at Ovrsea check this other article!</p><p>Thank you for reading and feel free to reach to us! We will be more than happy to discuss and debate on data testing best practice with other starts up! Improving and challenging is key when it comes to testing!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e80b9002ced3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ovrsea/data-testing-framework-with-python-and-dbt-e80b9002ced3">Data testing framework (with python and dbt)</a> was originally published in <a href="https://medium.com/ovrsea">OVRSEA</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Serverless event-driven application with SNS and Node.js]]></title>
            <link>https://medium.com/ovrsea/building-a-serverless-event-driven-application-with-sns-and-node-js-ab0240616982?source=rss----91ee87b4533d---4</link>
            <guid isPermaLink="false">https://medium.com/p/ab0240616982</guid>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[serverless-framework]]></category>
            <dc:creator><![CDATA[Eric Cabrel Tiogo]]></dc:creator>
            <pubDate>Thu, 30 Jun 2022 09:19:49 GMT</pubDate>
            <atom:updated>2022-06-30T12:15:33.027Z</atom:updated>
            <content:encoded><![CDATA[<h3>Build a Serverless event-driven application with SNS and Node.js</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sb6ZtefBDr4M7o1pEipafA.jpeg" /><figcaption>Photo by Vincent Lin on Unsplash</figcaption></figure><p>At <a href="https://www.ovrsea.com/">Ovrsea</a>, we run on a microservice architecture that is fully Serverless. Microservices trigger many events in destinations to others. Most of these events are handled asynchronously, reducing the coupling and improving system availability.</p><p>Here are the benefits of having an event-driven architecture:</p><ul><li>Low coupling between microservices.</li><li>Easier collaboration between teams since each one can work on their microservice.</li><li>Improvement in the request-response time because microservices can handle some actions asynchronously.</li><li>Your system can handle scalability better because you can batch events to be consumed by a service.</li><li>Your system is more fault-tolerant since the messages are stored in a queue; you can replay failed events.</li></ul><p>On AWS, there are many services to build event-driven applications like SNS, SQS, Kinesis, Event Bridge, etc.</p><p>This post will focus on using SNS to communicate between two microservices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*2lYlCzRtbZMgC-Ub_6d1kA.jpeg" /><figcaption>Example of microservices subscribing to an SNS Topic</figcaption></figure><h4>What will build</h4><p>To demonstrate the use of SNS, we will have two microservices:</p><ul><li><strong>User management</strong>: expose an endpoint through the API gateway for registering a new user. Once the user is saved, and the event is published to notify other services</li><li><strong>Emails Sender</strong>: This microservice listens to many events and sends an email to the corresponding recipient</li></ul><p>Here is what our architecture will look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*EgHew5Gd0cu6eOQINmy4eQ.jpeg" /><figcaption>The architecture of the system to build</figcaption></figure><h4>Prerequisites</h4><p>You must install these tools on your computer to follow this</p><ul><li>Node.js 14+ - <a href="https://nodejs.org/en/download/">Download link</a></li><li>AWS CLI - <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">Installation guide</a></li><li>AWS IAM configured - <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html">Configuration guide</a></li><li>Serverless Framework - <a href="https://www.serverless.com/framework/docs/providers/fn/guide/installation">Installation link</a></li></ul><h4>Create the SNS Topic</h4><p>We use the SNS topic to connect two or many applications; one can publish an event, and all the applications subscribed to this topic will receive that event and act according to the payload.</p><p>We can create a topic from the AWS console or through the AWS CLI; here is the command to run:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c134e8a01cbfef4a23ef03859818d478/href">https://medium.com/media/c134e8a01cbfef4a23ef03859818d478/href</a></iframe><p>The <strong>TopicArn</strong> will be printed in the console, copy it and keep it somewhere; we will use it later:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AZsKoqn3brisjnwMK0mIrQ.png" /><figcaption>Create an SNS Topic with the AWS CLI</figcaption></figure><p>Press CTRL + C to exit</p><p><strong>Note:</strong> You must create one topic per environment to avoid sending events to the wrong services.</p><h4>Set up the project</h4><p>I prepared a project with the two microservices already built, but they are not communicating together yet. We will update them to make that possible.</p><p>Run the commands below to clone the project and set up the user management microservice:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7945b8a95e313e3b7b668443c6317b55/href">https://medium.com/media/7945b8a95e313e3b7b668443c6317b55/href</a></iframe><p>We get the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ktZ74N-Lm8Pb38aeKraJWw.png" /><figcaption>Locally invoke the Lambda function for user signup</figcaption></figure><p>Run the commands below to set up the emails-sender microservice:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6637beaf67eab2266b463419d380da12/href">https://medium.com/media/6637beaf67eab2266b463419d380da12/href</a></iframe><p>We get the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G4fO8w4MCU59sQLD_ICzWw.png" /><figcaption>Locally invoke the Lambda function for emails sending</figcaption></figure><h4>Publish a message in the SNS Topic</h4><p>To publish a message on a topic, with need the <a href="https://aws.amazon.com/fr/sdk-for-javascript/">AWS SDK for JavaScript</a> that provides an API to make this call. Let’s install it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44c2776c4a8c16743a040c3948083e87/href">https://medium.com/media/44c2776c4a8c16743a040c3948083e87/href</a></iframe><p>Publishing a message with the SDK required two elements:</p><ul><li><strong>The topic ARN</strong>: we created the topic earlier.</li><li><strong>The message</strong>: It’s a JSON string with the information the others will use to identify the action to perform. The maximum content size is 256KB.</li></ul><p>Update the file user-management/src/functions/signup/handler.ts with the code below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c659905856cc90c0b2c705c4a1d71ec3/href">https://medium.com/media/c659905856cc90c0b2c705c4a1d71ec3/href</a></iframe><p>Let’s permit the Lambda function to publish in the SNS Topic. Update the property provider of the file user-management/serverless.ts by adding the code below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46449fa37d206c7eedfbb9fffe24bba9/href">https://medium.com/media/46449fa37d206c7eedfbb9fffe24bba9/href</a></iframe><h4>Subscribe to a message in the SNS Topic</h4><p>The user-management service publishes an event on the topic, but no service has subscribed to this topic yet. Let’s configure the emails-sender service to subscribe to it.</p><p>We need to indicate the event that triggers this Lamdba function which is a message in an SNS Topic. Update the file emails-sender/src/functions/main/index.ts with the code below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c9f7247e6a50490b80facfd3e891604/href">https://medium.com/media/8c9f7247e6a50490b80facfd3e891604/href</a></iframe><p>The only remaining thing is to give the SNS subscription to the Lambda function. Update the property provider of the file emails-sender/serverless.ts by adding the code below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98af5f71fdc8beea371fccf5b9a80d66/href">https://medium.com/media/98af5f71fdc8beea371fccf5b9a80d66/href</a></iframe><p>That is it! The only remaining thing to do is update the Lamdba handler to send the email, but we will not cover it for brevity.</p><h4>Deploy in production and test</h4><p>Locally, we can only test the two services independently; but to test the whole system, we need to deploy it on AWS. Thank the <a href="https://www.serverless.com/">Serverless Framework</a> that makes it easier for us.</p><p>Run the commands below to deploy the two services:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3344eedf690259db7e4a66ded047718e/href">https://medium.com/media/3344eedf690259db7e4a66ded047718e/href</a></iframe><p>Wait for the deployment to complete, copy the API Gateway URL printed in the terminal of the user-management service</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zJ5RTdZRkogaOlLE-tu7BA.png" /><figcaption>Deploy the Lambda function on AWS</figcaption></figure><p>Use an HTTP client to send a request to this endpoint and see the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hc9il9k_v-jgnVUUyLoB6A.png" /><figcaption>Test the user registration with Postman</figcaption></figure><p>In the AWS console, go to the CloudWatch Logs of the emails-sender function; you will see the event log.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MSkX2w6BqgZKcKtKaUrEYg.png" /><figcaption>CloudWatch logs of the Lambda function that send the emails</figcaption></figure><p>The two services communicate together as expected 🎉</p><p><strong>Wrap Up</strong></p><p>You must think about event-driven architecture for better low-coupled and fault-tolerant services.</p><p>AWS provides many services to build event-driven applications. You should pick the one that suits your needs.</p><p>Find the source code of this project on the <a href="https://gitlab.com/tecogill/event-driven-sns-serverless/-/tree/final">GitLab repository</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab0240616982" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ovrsea/building-a-serverless-event-driven-application-with-sns-and-node-js-ab0240616982">Building a Serverless event-driven application with SNS and Node.js</a> was originally published in <a href="https://medium.com/ovrsea">OVRSEA</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Parsing poor-context documents with AI, a major challenge for the freight industry]]></title>
            <link>https://medium.com/ovrsea/parsing-poor-context-documents-with-ai-a-major-challenge-for-the-freight-industry-a046faba1d57?source=rss----91ee87b4533d---4</link>
            <guid isPermaLink="false">https://medium.com/p/a046faba1d57</guid>
            <category><![CDATA[document]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[research-and-development]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[freight-forwarding]]></category>
            <dc:creator><![CDATA[Paul Couturier]]></dc:creator>
            <pubDate>Thu, 30 Jun 2022 08:12:06 GMT</pubDate>
            <atom:updated>2022-06-30T08:12:06.837Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iCF_eyw39wyQtUYo8Hjtaw.jpeg" /><figcaption>Image 1 : Photo of administrative documents by <a href="https://unsplash.com/@golfarisa?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Arisa Chattasa</a> on <a href="https://unsplash.com/s/photos/document?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><blockquote>TL;DR: Freight is generating several legal documents for every shipment. They are both mandatory and full of useful information. Being able to retrieve automatically this data would make the experience of freight smoother and more insightful for our customers. Enabling to handle customs, doing analysis on transport cost per product or giving a forecast of stock are jobs that we could do for our customers by parsing these legal documents.</blockquote><blockquote>We have studied the general framework needed to parse such poor-context documents using machine learning. This framework focuses on three aspects of this challenge : recognizing the text from the image, understanding the layout and getting the meaning of each word.</blockquote><blockquote>Following this long study, we are very proud and happy at Ovrsea to deliver our first feature with the automated reading of commercial invoices for the stock vision!</blockquote><h3>The high interest of parsing documents</h3><p>In the freight industry, a bunch of documents are often generated alongside a transport. You will deal with a Bill of Lading or an Air waybill, customs, packing lists, commercial invoices … the list is quite long. And all these documents will allow your container or goods to check every requirement to arrive safely at your warehouse.</p><p>They are both necessary and full of useful data such that being able to retrieve information automatically would make the experience of freight smoother and more impacting for our customers. We could think of many use cases that would benefit them:</p><ul><li>Provide a stock vision to the buyer by reading the packing list</li><li>Simplify the customs by reading the HS codes of every product</li><li>Give an analysis of shipment price cost per item by reading the commercial invoices</li></ul><p>According to the feedback collected by our Product team from our customers, automatically retrieving these pieces of information from freight documents is an amazing feature!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*27sXV4vi2g4mKIae.JPG" /><figcaption>Image 2: Bill of Lading from Wikipedia</figcaption></figure><p>But if it is mandatory data, when the freight is going to be fully digital we could get rid of all these documents, right? Well, it is not that easy. Or at least not going to happen anytime soon.</p><p>As described in most of the articles we have read about this topic, the entire freight eco-system is still very late in digitalisation and while many actors try to catch up the race, we can’t really expect the market to stop using pdf document soon [1] [2].</p><h3>The challenges of extracting information on poor-context documents</h3><p><strong><em>Notion of poor-context document</em></strong></p><p>But what exactly are these documents ? In fact the freight documents are pdf often stored as images. They are not written in full sentences and their layout is often built around approximate tables or forms that contain the minimal viable information. Thus each word is not surrounded by others that would explain what it actually means contrary to a standard sentence. And tables may be very complex with multiple entries as shown in the <a href="https://cdn-images-1.medium.com/max/1600/0*27sXV4vi2g4mKIae.JPG">Image 2</a>. <strong>This is why we call them poor-context documents.</strong></p><p>So, how can we extract information across diverse documents with various templates and structure these information in a database to enable our customers to use them?</p><p>Let’s say we are able to transform the image into a text — we will talk later about it. Since most of the data we want to retrieve are in a table, we might think this should be a child’s play. Well … not exactly. Due to the way the machine learning technologies have developed, this is in fact much more complicated to retrieve information from several tables than from text with semantic. Let’s try to popularise a bit!</p><p><strong><em>Artificial intelligence on text</em></strong></p><p>The Natural Language Processing (NLP) is one field of AI that aims at understanding the semantic behind the text. Two majors breakthroughs have been done in this field by Google and OpenAI (supported by Microsoft) respectively with the BERT transformer and the GPT in the last 10 years [3] [4]. By relying on massive quantity of data, they have been building models that are -kind of- understanding our languages.</p><p>I will not dig deeper into this topic, but if you want to take a look, you can try OpenAI API or even … simply write a query on google search ! This is how Google is able to understand when you are typing your search query as it was a question you would ask a friend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*grUybRu3cKQxtU8yhD6wsw.png" /><figcaption>Image 3: Example of an OPEN AI question we have asked to OPEN-AI [3]</figcaption></figure><p><strong><em>Artificial intelligence on poor-context documents</em></strong></p><p>In our case, we can’t rely directly on such technologies. Indeed, poor-context documents don’t contain sentences that would rely on semantic syntax. The information is much more diluted across three main factors:</p><ul><li>The text itself as a word without context</li><li>The geographical position of the text in the document</li><li>Lines and geometry that define layout</li></ul><p>Now that we have seen why parsing document is a challenge, let’s explain how we tried to solve it!</p><h3>AI tools to deal with poor-context document</h3><p>As this problem is a complex one, we need to rely on multiple technologies. As explained <a href="https://medium.com/ovrsea/data-b2b-how-to-launch-a-high-impact-ml-project-when-your-core-product-is-not-ai-centric-67eaf9959490">in our previous article</a>, our goal at Ovrsea’s Data team is to define the right level of R&amp;D between home-made and off-the-shelf technologies. While working on customs solutions, we have also been exchanging with different start-ups to define our own solution.</p><p>We are going to present you the general tools to tackle such a challenge. Notice that they are many ways to solve it and this framework may not cover every one of them!</p><h4>1. Recognize the text with Optical Character Recognition (OCR)</h4><p>The first brick of every possible solution should be OCR. Indeed, <strong>the input here is an image</strong>. Depending on the type of pdf, some can be read directly as xml file, but most of them will only be uploaded as images like png or jpg that should then be transformed into actual text. <strong>OCR that stands for</strong> <strong>Optical Character Recognition</strong> is a technology that is <strong>able to detect and automatically read text from an image</strong>. Many OCR solutions exist on the market, either free or not such as Amazon Textract or Tesseract (sponsored by Google). From the OCR benchmark we have been doing, they are still very much all focusing on latin languages.</p><p>By analyzing the document and the characters, they are able to reconstitute the text and give its absolute position on the document. The output of OCR is often a bounding box that embeds the text of the document (<em>see bounding box in Image 4</em>). But the OCR needs training to be able to recognize coherent groups of words as one token. For example it should not mix the title with the first sentence of the text, and be able to create two bounding boxes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zKLHai4PHNNGJydJX3U4_w.png" /><figcaption>Image 4: Output of the DiT layout algorithm applied on a scientific article paper through Hugging Face with the 4 labels recognized: table, text, title, other [10] (Left) Output of the LayoutLM V2 algorithm applied on invoice through Hugging Face with the 4 labels recognized: question/answer, header, and title [8] (Right).</figcaption></figure><p>However, even when you have the text and its position, it doesn’t yet help you to fully understand it. For that, you need to get what is the underlying structure.</p><h4>2. Understanding the layout with Deep learning and statistics</h4><p>In poor-context document, <strong>the understanding of the layout is one of the key information we need to get</strong>. Here we will be defining the layout analysis as any information that helps us understand the structure of the document and the relationship between tokens of word recognized but the OCR.</p><p><strong>The general desired output for layout understanding is often to be able to label the box surrounding each token of text</strong> to define its possible type — see Image 4. Is it the answer to a question? A general information over the document (date, reference number etc)? A table of interest?</p><p>If the document doesn’t contain too much diverse templates, then a simple statistics analysis with a basic machine learning algorithm such as random forest could help us get the information we need. For example by looking at specific words in a pre-defined position of the document, we might be able to retrieve a specific information such as a due date for instance.</p><p>However, in many situation, this is not sufficient as documents’ structures are very complex and diverse. A statistical analysis may not be able to understand it enough. This is where deep learning comes into play. Over the last two years, a lot of huge breakthroughs have been done in this field. This topic — known as Document layout Analysis — is still way behind traditional NLP, but by relying on the same mechanism, deep learning algorithms are almost able to understand the document structure.</p><p><strong><em>The foundations of deep-learning document layout analysis : Transformers and attention</em></strong></p><p>These deep learning algorithms often rely on the transformer architecture that has been created with the release of the algorithm BERT in 2017 by Google. This architecture is trained to encode and decode information (it can be images, texts, …), by using “attention” which is a mechanism that enables your algorithm to understand the link between your inputs. We will not dig deeper into this technology since it is not the purpose of this article, but if you are interested, I advise you to read “Attention is all you need” [5].</p><p>The original transformer architecture has been adapted since 2017 and is very much used for layout understanding. To give you some examples :</p><ul><li>Microsoft is currently working on the algorithm LayoutLM that is both using OCR and computer vision for images to be able to understand them [6] [7] [8]. Just to give you an overview, a scheme of its architecture is shown below.</li><li>Another algorithm developed by Microsoft is DiT (Document image Transformer) that is based on Google Bert and that is relying much more on computer vision than on OCR [9] [10].</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YASNKxLN35EZYJnkxobl4g.png" /><figcaption>Scheme 1: Scheme of the architecture of LayoutLM V2 by Microsoft team released in [6]</figcaption></figure><p>These technologies are state of the art as both paper of LayoutLM v3 and DiT have been released few weeks ago. <strong>While being very powerful, these algorithms still need to be tuned to our own documents and challenges.</strong> Indeed, as you can see on the Image 4 above, DiT has been trained to only classify visual token between four labels: “text”, “figure”, “table” and “title”. Similarly LayoutLM v2 has been trained to recognize the labels: “Question/Answer” as well as “title” or “other”. Thus to apply them to certain freight documents <strong>you would need to define the information you want to recover and label them in the boxes to fine-tune your transformer model</strong>.</p><h4>3. Grasping the meaning of words with embedding</h4><p>Once we have the text and labels for tokens, we might still need to understand what is the meaning of each token of word. Depending on the accuracy that you require from your labels, this can be an information to choose between two selected tokens of word.</p><p><strong>Due to the poor-context issue, we can’t rely on direct NLP outstanding technics</strong>, and this time we need to step back and consider a less state-of-the-art technic but still very efficient: word embedding. The idea behind it is simple: <strong>encode any word into a mathematical vector space</strong>, such that the difference of two vectors (typically cosine similarity) is defining the relationship between them [11]. <strong>This technic will, for example, help us find synonyms, which is useful when we intend to find a same information but present under another name</strong>.</p><p>To obtain such relationships between words, we need to train them on semantic sentences that should match our own topic of interest. However, once the training is done, we can apply word embedding even if we only have words and not sentence. Many python library exist to do this task such as Spacy, or NLTK with WordNet dictionary [12] [13].</p><h3>Our first use case: the reading of commercial invoices!</h3><p>Following this long study, we are very proud and happy to deliver our first feature with the reading of commercial invoices for stock vision. We have been able to develop a feature that is able to detect most of the products and their characteristics from a commercial invoice to have a better stock vision! As this is an R&amp;D topic, we will work on improving this first Beta feature. By testing new algorithms and architectures, we might crack even more the topic of poor-context document parsing !</p><p>If you are a customer interested in this new technology, please do reach out to <a href="/service/mailto:nader.zaim@ovrsea.com">our product team</a> to know more about this feature! And if you are a tech lover, who is craving for Data and want to be part of this terrific journey, apply to our amazing team! <a href="https://www.welcometothejungle.com/fr/companies/ovrsea">We are hiring!</a></p><p>#DataEngineer #DataAnalyst</p><h3>Bibliography</h3><ol><li>Manaadiar, H. Pitfalls of digitalisation in shipping and freight. <a href="https://www.shippingandfreightresource.com/pitfalls-of-digitalisation-in-shipping-and-freight/">https://www.shippingandfreightresource.com/pitfalls-of-digitalisation-in-shipping-and-freight/</a>.</li><li>The digitalization of the freight forwarding industry: essential for its survival. <a href="https://piernext.portdebarcelona.cat/en/logistics/the-digitalization-of-the-freight-forwarding-industry-essential-for-its-survival/">https://piernext.portdebarcelona.cat/en/logistics/the-digitalization-of-the-freight-forwarding-industry-essential-for-its-survival/</a>.</li><li>OpenAI API. <a href="https://openai.com/api/">https://openai.com/api/</a>.</li><li>Lutkevich, B. What is BERT (Language Model) and How Does It Work? <a href="https://www.techtarget.com/searchenterpriseai/definition/BERT-language-model">https://www.techtarget.com/searchenterpriseai/definition/BERT-language-model</a> (2020).</li><li>Vaswani, A. <em>et al.</em> Attention Is All You Need. (2017).</li><li>Xu, Y. <em>et al.</em> LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding.</li><li>Huang, Y., Lv, T., Cui, L., Lu, Y. &amp; Wei, F. LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking.</li><li>LayoutLMV2. <a href="https://huggingface.co/docs/transformers/main/en/model_doc/layoutlmv2#transformers.LayoutLMv2Model">https://huggingface.co/docs/transformers/main/en/model_doc/layoutlmv2#transformers.LayoutLMv2Model</a>.</li><li>Li, J. <em>et al.</em> Work in Progress DIT: Self-supervised Pre-training for Document Image Transformer. (2022).</li><li>Hugging Face. DiT documentation. <a href="https://huggingface.co/docs/transformers/master/en/model_doc/dit">https://huggingface.co/docs/transformers/master/en/model_doc/dit</a> (2022).</li><li>Bujokas, E. Creating Word Embeddings: Coding the Word2Vec Algorithm in Python using Deep Learning | by Eligijus Bujokas | Towards Data Science. <em>Towards Data Science</em> <a href="https://towardsdatascience.com/creating-word-embeddings-coding-the-word2vec-algorithm-in-python-using-deep-learning-b337d0ba17a8">https://towardsdatascience.com/creating-word-embeddings-coding-the-word2vec-algorithm-in-python-using-deep-learning-b337d0ba17a8</a> (2020).</li><li>Gubur, K. NLTK and Python WordNet: Find Synonyms and Antonyms with Python — Holistic SEO. <a href="https://www.holisticseo.digital/python-seo/nltk/wordnet">https://www.holisticseo.digital/python-seo/nltk/wordnet</a> (2021).</li><li>Revert, F. An Overview of Topics Extraction in Python with Latent Dirichlet Allocation — KDnuggets. <a href="https://www.kdnuggets.com/2019/09/overview-topics-extraction-python-latent-dirichlet-allocation.html">https://www.kdnuggets.com/2019/09/overview-topics-extraction-python-latent-dirichlet-allocation.html</a> (2018).</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a046faba1d57" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ovrsea/parsing-poor-context-documents-with-ai-a-major-challenge-for-the-freight-industry-a046faba1d57">Parsing poor-context documents with AI, a major challenge for the freight industry</a> was originally published in <a href="https://medium.com/ovrsea">OVRSEA</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup a Typescript project with ESLint, Prettier and VS Code]]></title>
            <link>https://medium.com/ovrsea/setup-a-typescript-project-with-eslint-prettier-and-vs-code-4ce4fcb76509?source=rss----91ee87b4533d---4</link>
            <guid isPermaLink="false">https://medium.com/p/4ce4fcb76509</guid>
            <category><![CDATA[prettier]]></category>
            <category><![CDATA[eslint]]></category>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Rémy van Dyk]]></dc:creator>
            <pubDate>Wed, 25 May 2022 09:30:33 GMT</pubDate>
            <atom:updated>2022-05-25T09:30:33.413Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_pOg2U--5Mo8ul6C.jpg" /><figcaption>Life feels better when developing in the right environment right ? (<a href="https://sgarq.com/wp-content/uploads/2017/06/confort-scaled.jpg">Source</a>)</figcaption></figure><p>Typescript is nowadays one of the <a href="https://insights.stackoverflow.com/survey/2021#technology-most-loved-dreaded-and-wanted">most loved languages</a> out there thanks to the power and robustness of its typing system.</p><p>Setting an appropriate environment to develop in Typescript is crucial to benefit most of all the features it has to offer.</p><p>In this article, I will show you how to setup a clean and simple development environment based on:</p><ul><li><a href="https://code.visualstudio.com/">VS Code</a>: a powerful editor that you can turn into an IDE to your liking.</li><li><a href="https://eslint.org/">ESLint</a>: a customizable linter to check your code for problems and bad practice.</li><li><a href="https://prettier.io/">Prettier</a>: a customizable code formatter to ensure you and your team follow the same code styling guidelines.</li><li><a href="https://yarnpkg.com/">Yarn</a>: a package manager, preferable to the default npm as it is way faster.</li></ul><h3>TL;DR</h3><p>You can find the final project setup in this <a href="https://github.com/Myreage/typescript-starter">GitHub repository</a> and clone it:</p><pre>git clone <a href="https://github.com/Myreage/typescript-starter">https://github.com/Myreage/typescript-starter</a><br>yarn install</pre><p>After that, simply install the <a href="https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint">ESLint VS Code extension</a> and the <a href="https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode">Prettier VS Code extension</a>.</p><p>You’re good to go !</p><h3>1. Project setup</h3><p>Initialize a new repository:</p><p>mkdir typescript-starter &amp;&amp; cd typescript-starter &amp;&amp; npm init</p><p>Install Typescript:</p><p>yarn add typescript --dev</p><p>Add a purposely badly formatted sample file in src/index.ts :</p><pre>type Book={<br>  title:string<br>  author: string</pre><pre>}</pre><pre>const book:Book ={<br>  title: &#39;Cool Book&#39;,<br>  author:&quot;John Doe&quot;</pre><pre>}</pre><h3>2. ESLint configuration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/0*UcKkHBbcJdQemHLE.png" /><figcaption>(<a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fwww.eficode.com%2Fblog%2Fhow-to-write-better-code-with-tools&amp;psig=AOvVaw0BslEgoxEjax6lUNdOhejQ&amp;ust=1653410491056000&amp;source=images&amp;cd=vfe&amp;ved=0CAwQjRxqFwoTCPCIhJyI9vcCFQAAAAAdAAAAABAP">Source</a>)</figcaption></figure><p>Install ESLint, its Typescript plugin and the related parser:</p><pre>yarn add eslint @typescript-eslint/parser @typescript-eslint/eslint-plugin --dev</pre><p>Add a basic ESLint for Typescript configuration in a new file .eslintrc.json :</p><pre>{<br>  “env”: {<br>    “browser”: true,<br>    “es2021”: true<br>  },<br>  “extends”: [<br>    “eslint:recommended”,<br>    “plugin:@typescript-eslint/recommended”<br>  ],<br>  “parser”: “@typescript-eslint/parser”,<br>  “parserOptions”: {<br>    “ecmaVersion”: “latest”,<br>    “sourceType”: “module”<br>  },<br>  “plugins”: [<br>    “@typescript-eslint”<br>  ]<br>}</pre><p>Install the <a href="https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint">ESLint VS Code extension</a>.</p><p>Finally, create a new .vscode folder at the root of your project. You can now edit a workspace settings configuration file .vscode/settings.json to have ESLint auto-fix your fixable problems on save:</p><pre>{<br>  &quot;editor.codeActionsOnSave&quot;: {<br>    &quot;source.fixAll.eslint&quot;: true<br>  },<br>}</pre><h3>3. Prettier configuration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*Nk89g6MFxztRmrAG.png" /><figcaption>(<a href="https://cdn.arstechnica.net/wp-content/uploads/2013/06/ars.png">Source</a>)</figcaption></figure><p>Install Prettier:</p><p>yarn add prettier eslint-config-prettier --dev</p><p>eslint-config-prettier ensures there will be no conflicts between ESLint and Prettier, by disabling ESLint rules that might be conflicting.</p><p>Edit your ESLint configuration so the &quot;extends” value looks like this:</p><pre>“extends”: [<br>  “eslint:recommended”,<br>  “plugin:@typescript-eslint/recommended”,<br>  “prettier”<br>],</pre><p>Install the <a href="https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode">Prettier VS Code extension</a> and edit your .vscode/settings.json to enable auto-formatting on save:</p><pre>“editor.defaultFormatter”: “esbenp.prettier-vscode”,<br>“editor.formatOnSave”: true</pre><p>Add a basic Prettier configuration of your choice in .prettierrc.json :</p><pre>{<br>  &quot;semi&quot;: true,<br>  &quot;trailingComma&quot;: &quot;all&quot;,<br>  &quot;singleQuote&quot;: true<br>}</pre><h3>4. Let the magic operate</h3><p>Go back to your index.ts file, and re-save it.</p><p>If everything went well, your file should have been formatted on save, and ESLint should have auto-fixed everything it could ! You’re left with an error on const book:</p><pre>‘book’ is assigned a value but never used.eslint<a href="https://typescript-eslint.io/rules/no-unused-vars">@typescript-eslint/no-unused-vars</a></pre><p>You’re ready to spend hours re-typing the world !</p><p>However, if you’re in the mood for some more magic, you could:</p><ul><li>Customize your ESLint/Prettier rules, or even create your own !</li><li><a href="https://marketplace.visualstudio.com/vscode">Look for some awesome plugins in the VS Code Marketplace</a></li><li><a href="https://code.visualstudio.com/docs/editor/userdefinedsnippets">Add custom VS Code snippets</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*H7aDZ4rxFZh_0SBg.gif" /><figcaption>In Your Face is a VS Code extension that shows you Doom ‘Ouch Faces’ that correlate to the number of errors in your code (<a href="https://marketplace.visualstudio.com/items?itemName=VirejDasani.in-your-face">Source</a>)</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ce4fcb76509" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ovrsea/setup-a-typescript-project-with-eslint-prettier-and-vs-code-4ce4fcb76509">Setup a Typescript project with ESLint, Prettier and VS Code</a> was originally published in <a href="https://medium.com/ovrsea">OVRSEA</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>