<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Victor Monginet on Medium]]></title>
        <description><![CDATA[Stories by Victor Monginet on Medium]]></description>
        <link>https://medium.com/@victor.monginet?source=rss-69021b3d54ad------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*a9OocMqgOw1mh5XOlVT60Q.jpeg</url>
            <title>Stories by Victor Monginet on Medium</title>
            <link>https://medium.com/@victor.monginet?source=rss-69021b3d54ad------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:43:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@victor.monginet/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deduplication of heterogeneous documents in near real-time]]></title>
            <link>https://medium.com/doctrine/deduplication-of-heterogeneous-documents-in-near-real-time-2bb4f31adedb?source=rss-69021b3d54ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bb4f31adedb</guid>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Victor Monginet]]></dc:creator>
            <pubDate>Fri, 13 May 2022 08:15:27 GMT</pubDate>
            <atom:updated>2022-05-13T08:15:27.707Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://www.doctrine.fr/">Doctrine</a> is a french <em>legal intelligence platform</em> helping thousands of lawyers achieve their work more efficiently. Dozens of millions of legal documents, from court orders to law texts, are centralized in a unique place.</h4><p>Each day we collect thousands of documents from many external and internal sources with various techniques.<strong> One of the difficulties we’re facing in processing this data is avoiding duplicates</strong>. Indeed, similar documents can be ingested in our data lake because they are consumed from mixed sources. The problem is difficult to solve because similar documents are not necessarily identical, some of their attributes and content can be different or missing. Furthermore, we want to process those documents continuously, and have them available in the application within minutes.</p><p><em>So how do we deduplicate those heterogeneous documents in near real-time at Doctrine?</em></p><figure><img alt="Which one are identical?" src="https://cdn-images-1.medium.com/max/1024/0*ZN3KI_f--_t2u2sO" /><figcaption>Which are identical? 🤯</figcaption></figure><p><strong><em>Currently, we are re-engineering the way we deduplicate court orders, this blog post describes the new process we are developing.</em></strong></p><h4>Objective</h4><p>When a new document (a court order) is collected, we want to find if similar documents exist in our system, and if so, merge them and update the document, all of this within minutes.</p><h4>Partitioning data to reduce the cardinality</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*eNr1yAN8My95DnDKbLreig.png" /></figure><p>Obviously, we should not compare a document with all the other ones for performance reasons (we have millions of court orders), even more in a near real-time fashion. So, we need to find the minimum subset on which to apply our deduplication algorithm. It means in order to find duplicates of a document, we can only compare it to a subset because we have the guarantee the others are not similar. In the case of court orders, we know for sure similar documents have always some identical attributes like the date, jurisdiction, and city if it exists. Consequently, <strong>similar documents can be found within this subset, and we will call it a partition.</strong></p><h4>A high-level overview of the pipeline</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I15TM4Eh4coGZrV12g43Uw.png" /><figcaption>Task processing of a document</figcaption></figure><p>Documents coming from various sources are sent in a <a href="https://kafka.apache.org/"><em>Kafka</em></a><em> </em>topic, which is basically a queue with messages to process. This topic is consumed by one or more processing tasks triggered when messages are available. When triggered, they will process batches of hundreds of those messages.</p><p>Documents need to be processed one by one in their respective partitions (to make sure we don’t miss a similar document). Parallelization is still achievable between partitions and inside them (we will cover it later).</p><p>A task operates on a batch of messages, the following steps are performed:</p><ul><li>Group every new document received into partitions</li><li>For each partition, load the existing documents from the data lake</li><li>For each partition, apply the deduplication algorithm</li><li>Merge all the similar documents</li><li>Save the results</li></ul><h4>Finding similar documents</h4><p>Depending on the document type and its source, the techniques to find similar documents can vary. In our case, similar documents are not necessarily identical, and many attributes can be missing.</p><p>For example, a court order has an identifier that may not only be missing but varies in its form, so we can’t totally rely on it. Also, the text content of a court order seems to be useful information to detect similarities with others, but the content can be missing or be partially different depending on the source (some sources add more content to the document, or if a text is processed with <a href="https://en.wikipedia.org/wiki/Optical_character_recognition">OCR</a> it can change from the digital version). To find similar documents, our deduplication algorithm operates on various attributes of the documents (like the identifier) and the text content if it exists.</p><p>In practice, a <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a> (a numerical statistic) is computed on all document texts within a partition. It gives us a vector representation of each text and allows us to calculate the <a href="https://en.wikipedia.org/wiki/Cosine_similarity">cosine similarity</a> between all texts, represented as a score between 0 and 1, where 1 indicates an identical content. Because most of the time the cardinality is low (&lt; 1000), the computation of the TF-IDF is extremely fast. Those scores are represented as a two-dimensional matrix with all combinations. We also compute matrices of combinations of all similar documents based on attribute equalities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*dsQ5HoVBwBDH88Mhc--QaA.png" /><figcaption>Each row or column represents a document compared to the others</figcaption></figure><p>Our algorithm takes numeral matrices in input (which are <a href="https://numpy.org/">NumPy</a> arrays) because this data representation allows fast operations and access to achieve good performance.</p><p>So, how does the algorithm actually execute?</p><blockquote>Given a partition, and the matrices containing the information on text similarity and attribute equality, a list of business rules is applied to each document to find similar documents within the partition.</blockquote><p>For example one of the rules can be:<em> if two documents have the same identifier and a text-based score similarity greater than 0.9, consider they are the same.</em></p><p>Every association found between two documents is then represented in a graph as an edge between two nodes (which are the two documents).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Jq_COB7Gnr4HjISWZJZ8w.png" /><figcaption>We know C and D are the same because they are linked together through A</figcaption></figure><p>By using a graph we can easily find groups of similar documents with transitive dependencies. And that’s it, we have all the groups of similar documents within a partition!</p><h4>Merging similar documents</h4><p>When a document is collected, if we found similar documents, we need to merge them. To do so various rules can be applied to each attribute to decide which one to keep or how to merge them. There is also a document that needs to act as a referent (the first seen) to keep an immutable primary identifier. Finally, the document can be updated or added to the data storage.</p><h4>Parallelizing</h4><p>Parallelization is achieved in two ways, at the task processing level, and the partition. We use the concept of <em>Topic partitions</em> specific to <em>Kafka</em>, which uses the same partition keys (jurisdiction, date, city) as our partitions defined previously, to handle the distribution between multiple tasks. It means a partition always belongs to one <em>Topic partition, </em>said differently<em>, </em>a partition can be processed by only one task at a time. So multiple tasks can run to handle the messages, and within a task, the deduplication is applied on every partition distinctly with parallelization.</p><p>Because our pipeline has good parallelization capability, we can reprocess a large quantity of data for internal needs with good overall performance.</p><p><em>Thank you for reading this blog post. This was an overview of how we can deduplicate documents at Doctrine. Feel free to ask any questions about this.</em></p><h4>And don’t hesitate to join us to work on ambitious projects like this, <a href="https://cooptation.hellotrusty.io/hytig1bm0q">we’re hiring</a> talented engineers! 🙂</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bb4f31adedb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doctrine/deduplication-of-heterogeneous-documents-in-near-real-time-2bb4f31adedb">Deduplication of heterogeneous documents in near real-time</a> was originally published in <a href="https://medium.com/doctrine">Inside Doctrine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>