<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Phil Chadenier on Medium]]></title>
        <description><![CDATA[Stories by Phil Chadenier on Medium]]></description>
        <link>https://medium.com/@Medno?source=rss-9a5f6b40b7f8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8CX2JeQ8anPd1I4PBss6wg.png</url>
            <title>Stories by Phil Chadenier on Medium</title>
            <link>https://medium.com/@Medno?source=rss-9a5f6b40b7f8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:19:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Medno/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Indexing and aggregating lawyers blogs posts at scale]]></title>
            <link>https://medium.com/doctrine/indexing-and-aggregating-lawyers-blogs-posts-at-scale-ad99ed19e7b4?source=rss-9a5f6b40b7f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad99ed19e7b4</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[legaltech]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Phil Chadenier]]></dc:creator>
            <pubDate>Wed, 09 Oct 2024 06:47:50 GMT</pubDate>
            <atom:updated>2024-10-09T06:47:50.981Z</atom:updated>
            <content:encoded><![CDATA[<h4>How we handle thousands of lawyers’ blogs using the News-Please library</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ERpCguXfJ3R2eVOc" /><figcaption>Photo by <a href="https://unsplash.com/@andrewtneel?utm_source=medium&amp;utm_medium=referral">Andrew Neel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Sitting in front of your computer, your files scattered like expedition maps, you’ve just discovered that a significant ruling has been made public. Its impact on the law is substantial, and you realize that this news deserves to be shared with the world. After hours of exploring the twists and turns of your ideas, you finally chart a clear and relevant article. Like an explorer discovering new territories, you mark the end of your expedition by clicking “Publish”. But days go by, and despite the importance of this discovery, your article remains a neglected corner of the map, lost in the vast expanse of the web, attracting only a few readers.</em></p><p>In the context of a newly established law firm, the website’s SEO may not be optimized, leading to low visibility of articles amid the plethora of online legal content.</p><p>As developers, we understand how frustrating it is to produce quality content that fails to reach its audience. In this context, automated analysis and optimization of article visibility become major technical challenges.</p><p>At Doctrine, our lawyer clients requested more visibility. We listened to their needs and now index thousands of lawyers’ blogs, enhancing the visibility of their articles and integrating them into our Legal Graph. To enable scalable and automated indexing of these sites, we have chosen to utilize <strong>News-Please</strong>. In 2019, the repository was forked into our codebase to add specific features, but over time, maintaining it became increasingly difficult. We then migrated to the open-source version, adding some of our features to make it accessible and beneficial for everyone.</p><p>In this article, we will introduce you to the library, explain the reasons behind the fork, discuss our migration back to the open-source version, outline our contributions to the public repository, and share the various lessons we’ve learned along the way.</p><h3>On the Hunt for Today’s News, Please!</h3><p>The open-source library <a href="https://github.com/fhamborg/news-please"><strong>News-Please</strong></a><strong> </strong>allows you to retrieve articles, blog posts, detailed topics from websites. It deeps dive into the website, analyzes the pages, and extracts only the articles along with their relevant information.</p><p>News-Please is a powerful exploration tool that maps the information on the web by relying on trusted tools like <strong>Scrapy</strong> and <strong>newspaper4k</strong>. Together, these instruments allow for efficient navigation through news sites, tracing a clear and detailed map of the latest news. But wait, what are those tools?</p><p><a href="https://github.com/scrapy/scrapy"><strong>Scrapy</strong></a> stands for the website explorer. It ventures into unknown territories (websites) and charts paths to bring back relevant information, much like a cartographer creating a new map of the web. Here, it is used to provide spiders and navigate various websites.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bun6uG4CskaPeufUoZygsA.png" /><figcaption><a href="https://docs.scrapy.org/en/latest/topics/architecture.html#data-flow">Scrapy — Data flow</a></figcaption></figure><p>The available spiders include:</p><ul><li>RssCrawler: Utilizes the URLs available in the RSS feed to perform crawling.</li><li>SitemapCrawler (inheriting from Scrapy&#39;s SitemapSpider): Analyzes sitemaps defined in the robots.txt file to extract URLs within.</li><li>RecursiveSitemapCrawler: Works like the SitemapCrawler but also recursively explores the URLs contained within the sitemaps.</li><li>RecursiveCrawler: Simply explores all the links available on the website.</li></ul><p>The library includes the integration of <strong>Downloader Middlewares</strong>, <strong>Spider Middlewares</strong>, and <strong>Pipelines</strong>. Several pipelines are available by default:</p><ul><li>ArticleMasterExtractor: This is the main pipeline that extracts data from the analyzed URLs.</li><li>The library supports various storage options, including Elasticsearch, Redis, PostgreSQL, MySQL, and local storage.</li></ul><p><a href="https://github.com/AndyTheFactory/newspaper4k"><strong>Newspaper4k</strong></a> is a library dedicated (within News-Please) to extracting information from news sites. It acts as our <strong>archivist</strong>. Once discoveries are made, it catalogs everything, carefully organizing the data to make it usable. It filters out superfluous elements and keeps only the essentials. It allows for the extraction of information such as article titles, authors, images, publication dates, language, content, and more.</p><h4>How to operate this super engine ?</h4><p>News-Please offers a significant advantage in crawling websites by combining the analytical power of Scrapy with the quality extraction provided by newspaper4k. Moreover, the library is flexible, allowing for easy customization of parameters such as how to crawl sites, data storage options, and what information to extract.</p><p>The library offers two modes of operation: through a command-line interface (CLI) or as a library. In this article, we focus on using the CLI.</p><p>Here’s an example of usage via the CLI:</p><pre>news-please --cfg-file-path ./my_configuration_folder</pre><p>The execution is orchestrated as follows:</p><ol><li>The desired number of threads are instanciated</li><li>For each thread, the appropriate crawler for the website is initialized.</li><li>The crawler analyzes every URL from the website it finds and determines, using certain heuristics, whether the webpage is an article.</li><li>If the page is identified as an article, the data is extracted from the HTML using Newspaper4k.</li><li>The extracted data is then passed to the pipelines.</li><li>And so on…</li></ol><p>The executable accepts a parameter for a configuration folder path. Two files are required to run News-Please: sitelist.hjson and config.cfg. You can find examples of these files <a href="https://github.com/fhamborg/news-please/tree/c83d0f87faf5f15423e7a9657256e56facf7d8c1/newsplease/config">here</a>.</p><h4>Where to Navigate?</h4><p>The sites you wish to crawl must be listed in the sitelist.hjson file. Here’s an example taken from the file available in the official repository. Currently, we maintain a simple configuration with only the url and crawler fields:</p><pre>{<br>  # Every URL has to be in an array-object in &quot;base_urls&quot;.<br>  # The same URL in combination with the same crawler may only appear once in this array.<br>  &quot;base_urls&quot;: [<br>    {<br>      # Start crawling from faz.net<br>      &quot;url&quot;: &quot;http://www.faz.net/&quot;,<br>      # Overwrite the default crawler and use the RecursiveCrawler instead<br>      &quot;crawler&quot;: &quot;RecursiveCrawler&quot;<br>    },<br>    ...<br>  ]<br>}</pre><p>The configuration is quite flexible and comprehensive. It is possible to add several additional parameters to customize acquisition by site:</p><ul><li>Each site can be associated with a heuristic that provides specific rules to determine if the content of a URL corresponds to an article.</li><li>It is also possible to specify that the acquisition of certain sites is done via a daemon by adding a daemonize field in the configuration.</li></ul><h4>Which directive to follow?</h4><p>The execution of News-Please depends not only on the website being analyzed but also on a configuration file : config.cfg. This file defines how to analyze the websites: whether to fetch only the latest articles or all the articles from the site, how many parallel runners to use, and how to configure Scrapy’s navigation.</p><p>An important and configurable feature in News-Please is the fallback system for crawlers. This allows you to define which crawler to use if the one you are testing is not supported by the site. For example, if you prefer to retrieve the latest articles via an RSS feed, but a site does not support it, you can easily configure the crawler to switch to sitemaps (if available, of course). Here’s an example of the fallback logic that we currently have in production.</p><pre>fallbacks = {<br>    &quot;RssCrawler&quot;: &quot;SitemapCrawler&quot;,<br>    &quot;RecursiveSitemapCrawler&quot;: &quot;RecursiveCrawler&quot;,<br>    &quot;SitemapCrawler&quot;: &quot;RecursiveSitemapCrawler&quot;,<br>    &quot;RecursiveCrawler&quot;: None,<br>    &quot;Download&quot;: None<br>}</pre><p><em>Note the “Download” element here. This is an acquisition method that retrieves only the content of a specified URL without exploring other links present on the page.</em></p><p>For those familiar with <strong>Scrapy</strong>, you will find this configuration comfortable as it adopts similar parameters.</p><p>The library also supports parallelism through <strong>multi-threading</strong>, which can be configured in the configuration file with the following parameter:</p><pre>number_of_parallel_crawlers = &lt;number of threads&gt;</pre><p>You can also configure the storage options you wish to use. The template supports options such as <strong>Elasticsearch</strong>, <strong>Redis</strong>, <strong>PostgreSQL</strong>, <strong>MySQL</strong>, and local storage.</p><p>Other parameters allow you to customize parallel crawling through daemons, or even define <strong>heuristics</strong> to filter articles based on specific criteria.</p><h3>From the side path to the main trail: Our return to the official News-Please path</h3><p>In 2019, we forked the code of News-Please to add specific processing features. Among these features were:</p><ul><li>Assigning the RecursiveCrawler if it’s the first time analyzing the site.</li><li>Adding logic during the fallback: if we haven’t retrieved any articles with a less exploratory crawler for a certain period, we automatically fall back to the next one.</li><li>Additional extractions via the RssCrawler: Extracting the publication date of articles from the RSS feed if present.</li><li>More sophisticated spiders (more details in the “Our Contributions” section).</li><li>Monitoring: To observe the evolution of crawlers used for each execution.</li></ul><p>Over time, our fork became like an old spyglass, next to a map left to gather dust in a forgotten corner of the archive, with some parts fading and becoming unreadable. We had once hoped to chart a course back to it, but eventually realized it was a burden from the past. The tools had become outdated and incompatible with newer versions of Python, making it as difficult to navigate this code — over 10,000 lines long — as it would be to explore uncharted lands with a worn-out map.</p><p>This year, we decided to return to the open-source version of News-Please. Several migration plans were considered:</p><ol><li>Completely move to another tool, using LLM (<a href="https://github.com/ScrapeGraphAI/Scrapegraph-ai">Scrapegraph-ai</a>, GPT 4o…)<br>This option was tested, but the results were mixed. The final quality was good, with an appropriate prompt. The main issue was the extraction time, which was significantly slower compared to our current tools, compounded by the cost per article that could increase rapidly.</li><li>Gradually replacing the fork with the official library by removing our code and modifying other Python imports.<br>However, this second option quickly proved impractical due to numerous outdated dependencies. For example, some imports had changed during the transition from <a href="https://github.com/codelucas/newspaper/">newspaper3k</a> (no longer maintained since 2020) to newspaper4k.</li><li>Duplicate our pipeline with an integration of the open-source tool that runs in parallel.<br>Thus, we opted for this option, creating a dedicated Docker image. Once ready for the transition, we simply redirected the input and output points to News-Please, and the migration went smoothly.</li></ol><p>The migration process took several weeks, as some of our specific features were scattered throughout our fork. However, this transition allowed us to benefit from recent open-source contributions, some of which even included features that we had developed.</p><h3>Charting Our Own Path</h3><p>Despite migrating to the open-source version of News-Please, some developments remain specific to our use case. For instance, since the number of blogs to analyze fluctuates daily, we had to implement dynamic generation of the sitelist.hjson file. Similarly, we configured the credentials for our storage systems dynamically.</p><p>When a site is indexed, the metadata of the analyzed URLs is stored in our PostgreSQL database, while the content is cached in Redis.</p><p>Optimization and scalability are major challenges for us, as we strive to ensure that the acquisition does not exceed one day. Although News-Please already supports multi-threading and increases the number of parallel requests, we found that some lawyers’ blogs contain a large volume of resources. Therefore, we needed to consider alternatives.</p><p>To generate our acquisition processes, we use Airflow and have integrated the principle of <a href="https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/dynamic-task-mapping.html#dynamic-task-mapping"><strong>Dynamic Task Mapping</strong></a> to parallelize tasks. Regarding News-Please, we are currently only exploiting a portion of it, as the number of tasks is static. In the future, we plan to distribute different sites dynamically, but for now, this static distribution suits our needs.</p><p>Additionally, we have added several middlewares, particularly to track precisely which crawler was used for each site. This enables us to detect any irregularities, especially if a fallback has been activated too quickly on certain crawlers.</p><h3>Expanding the explorer’s toolset</h3><p>To benefit the open-source community with the features we developed while facilitating our migration to the official repository, we have made several iterations on the News-Please project:</p><ul><li><strong>Sitemap analysis even in the absence of entries in </strong><strong>robots.txt</strong></li></ul><p>This helps to address issues with poorly configured robots.txt files.</p><ul><li><strong>Crawler fallback when </strong><strong>RssCrawler returns an empty list</strong></li></ul><p>The RSS feed can sometimes be misconfigured, and this functionality allows exploration to continue with another crawler.</p><ul><li><strong>Redis boilerplate</strong></li></ul><p>A smoother integration of Redis for caching storage.</p><ul><li><strong>New date extraction methods</strong></li></ul><p>Improvements for more accurately retrieving publication dates of articles.</p><ul><li><strong>Option to disable SSL certificate verification</strong></li></ul><p>A useful option for handling sites with improperly configured SSL certificates.</p><ul><li><strong>Multiple fixes</strong></li></ul><p>Several bugs have been corrected to improve the library’s stability.</p><p>These contributions allow for the retrieval of more articles with better quality, especially when certain robots.txt or sitemap files are misconfigured.</p><h4>End of the journey ?</h4><p>Between June and August 2024, the repository recorded about twenty contributions, whereas only six contributions had been made between mid-2021 and 2024. We have thus contributed to revitalizing the repository, adding new features with significant impact.</p><p>This experience also reminded us of the risks of maintaining a fork, which can significantly slow down the development cycle. The fork lacked the latest fixes and updates, leaving us unable to benefit from ongoing improvements. Additionally, we found ourselves stuck when trying to upgrade certain software versions.</p><p>The work is not over. We still have ideas for optimizations and new features to integrate. 🏃</p><p><em>Personal note: It was the first time I contributed to an open-source project. As a developer, it is an invaluable reward. Being able to revitalize an open-source project was truly a pleasure. I would also like to thank </em><a href="https://medium.com/u/8cf5ccf9d438"><em>Aurélien Didier</em></a><em> and </em><a href="https://medium.com/u/b0d7b3c26747"><em>Yves Lelievre</em></a><em> for their teamwork and valuable advice during the migration.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad99ed19e7b4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doctrine/indexing-and-aggregating-lawyers-blogs-posts-at-scale-ad99ed19e7b4">Indexing and aggregating lawyers blogs posts at scale</a> was originally published in <a href="https://medium.com/doctrine">Inside Doctrine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>