<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dmitrii Maksimov on Medium]]></title>
        <description><![CDATA[Stories by Dmitrii Maksimov on Medium]]></description>
        <link>https://medium.com/@maksimov.dmitry.m?source=rss-65387e04d923------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*O4b5OcxM9ez7jcBjynXGQA.jpeg</url>
            <title>Stories by Dmitrii Maksimov on Medium</title>
            <link>https://medium.com/@maksimov.dmitry.m?source=rss-65387e04d923------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:11:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@maksimov.dmitry.m/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Build a Better RAG System: Smart Hybrid Search for Tables]]></title>
            <link>https://medium.com/@maksimov.dmitry.m/how-to-build-a-better-rag-system-smart-hybrid-search-for-tables-7bbea69a31f2?source=rss-65387e04d923------2</link>
            <guid isPermaLink="false">https://medium.com/p/7bbea69a31f2</guid>
            <category><![CDATA[qdrant]]></category>
            <category><![CDATA[llamaindex]]></category>
            <category><![CDATA[hybrid-search]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[embedding]]></category>
            <dc:creator><![CDATA[Dmitrii Maksimov]]></dc:creator>
            <pubDate>Thu, 02 Jan 2025 15:23:41 GMT</pubDate>
            <atom:updated>2025-01-02T15:23:41.614Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8K85cPWodUjH0DSUpn_quA.jpeg" /><figcaption>photo credit: <a href="http://www.weka.io/">www.weka.io</a></figcaption></figure><h3>Introduction</h3><p>Retrieval-Augmented Generation (RAG) has emerged as a critical tool for extracting valuable insights from large datasets. At its core, RAG consists of two key components: retrieval and generation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D6jiLQUj4OgCtmoK2_Q2uQ.png" /></figure><p>While the generation step leverages large language models (LLMs) to produce responses, it faces inherent limitations due to context window constraints, computational costs and latency.</p><p>The retrieval step addresses these challenges by filtering the knowledge base to provide only the most relevant information to the generative model. Traditional approaches rely heavily on semantic search using text embeddings. However, even though these encoder models run efficiently, they often don’t capture the same depth of meaning that generative LLMs do. Because of this, retrieval often becomes the primary bottleneck in RAG pipelines: if the system retrieves the wrong information, the final answers will be incorrect — even if the generator is powerful.</p><h3>The Challenge: Working With Table Data</h3><p>Modern organizations work with both text-based sources (PDFs, documents, wiki pages) and table-based sources (spreadsheets, databases). The retrieval strategy for table data depends on specific use cases:</p><ol><li>Deterministic Query Patterns<em>.</em> If you already know exactly how you would retrieve information (e.g., “<em>for each request, I would write a specific SELECT query and execute the query to get the information</em>”), then a text-to-SQL pipeline is the best option. Why? Because it’s:<br>– Transparent<br>– Accurate<br>– Predictable</li><li>Complex Cases. Some scenarios make Deterministic Query approaches impractical:<br>– Excel files that resist clean database schema mapping<br>– Legacy data formats with inconsistent structures</li></ol><p>This article focuses on the second category, proposing a retrieval strategy for cases where direct SELECT query retrieval isn’t feasible.</p><h3>The Hybrid Solution: Summaries + Sparse/Dense Fusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*rV5eH19P4EAlpaQQTozaOw.png" /></figure><p>The solution combines two key components to maximize retrieval effectiveness for table data:</p><ol><li><strong>Dense Embeddings from Summaries<br></strong>– Generate natural language summaries of tables using LLM<br>– Create dense embeddings from summaries instead of raw tables</li><li><strong>Hybrid Search Strategy<br></strong>– Dense embeddings from summaries capture semantic relationships<br>– Sparse embeddings (keyword-based) from original tables maintain detailed information<br>– Score Fusion Methods:<br>a. Linear Combination: Experimenting with varying α parameter<br>(0 = keyword only, 1 = dense only)<br>b. Reciprocal Rank Fusion (RRF): As recommended by <a href="https://qdrant.tech/articles/hybrid-search/">Qdrant</a>, potentially offering better performance than linear combination</li></ol><p>This combined method is meant to fix the drawbacks of purely semantic search and still handle large tables effectively. By using summaries for dense embeddings, we can process tables that exceed typical embedding model context windows.</p><h3>Walking Through the Implementation</h3><h4>Data Overview</h4><p>Everything starts with the data. Let’s look at our tables and build our retrieval system step by step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wDeKQiZT76dMu2Fs6BPhCQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IYbWKyEhuLzr5IA68eny4g.png" /><figcaption>Source data examples</figcaption></figure><p>First, we need to convert our table data into plain text. Libraries like Markitdown, LlamaParse, and docling can help, but I recommend writing your own parser or at least integrating your understanding of the data into the parsing pipeline.</p><p>Here’s what our dataset looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0GoCUuCFPNTw0zq1KM3ICQ.png" /></figure><ul><li>Minimum length: 22 tokens</li><li>Maximum length: 169182 tokens</li><li>Median length: 463 tokens</li><li>90th percentile: 3430 tokens</li></ul><p>While our research is based on relatively small tables, the approach can be effectively scaled to larger datasets.</p><h4>Summary Generation</h4><p>Next, we use GPT-4o-mini to generate summaries for each table using this prompt:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f79efdb933f13e169cd598af0506b10a/href">https://medium.com/media/f79efdb933f13e169cd598af0506b10a/href</a></iframe><p>Let&#39;s have a look at the summaries distribution as well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*3w8ycsT4_NnsxcfT_JBjOg.png" /></figure><p>As we can see, the summaries are much shorter than original tables.</p><h4>Vector Store Setup</h4><p>Using text-embedding-3-large to create dense embeddings for summaries and BM25 to create sparse embeddings for original tables. Everything is stored in Qdrant:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/baaa2df6dc2c2ec03e551b3a5313b953/href">https://medium.com/media/baaa2df6dc2c2ec03e551b3a5313b953/href</a></iframe><h3>Experimental Setup</h3><p>To properly evaluate our approach, we needed a comprehensive test dataset. Here’s how it was built and tested:</p><p><strong>Test Dataset Creation</strong><br>GPT-4o-mini is used to generate 1000 table-like data samples. Each question was designed so that its answer could only come from one table. This gave us a clear way to check which retrievals were correct. Here’s the prompt we used for question generation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4cfe65d16b6cc1edc16bf5ad7c215b34/href">https://medium.com/media/4cfe65d16b6cc1edc16bf5ad7c215b34/href</a></iframe><p><strong>Testing Strategy<br></strong>We compared two different scenarios:</p><ul><li><strong>Baseline</strong>: Using dense embeddings from the original table text</li><li><strong>Our Approach</strong>: Using dense embeddings generated from table summaries</li></ul><p>For each scenario, we tested both linear fusion and RRF to combine dense and sparse search results.</p><p><strong>Evaluation Metrics<br></strong>Two complementary metrics provide a complete picture of retrieval performance:</p><ul><li><strong>HitRate@k</strong>: This practical metric shows if we can find the right table within the top k results.<br>Tested across multiple k values (1, 3, 5, 10, 20, 40)</li><li><strong>Mean Reciprocal Rank (MRR)</strong>: Measures retrieval precision<br>Higher scores indicate the correct table appears closer to the top<br>For example:<br>MRR = 0.5 → Average review of 2 tables needed<br>MRR = 0.25 → Average review of 4 tables needed</li></ul><h3>Results comparison</h3><p>Default Hybrid Search RAG (using original table, linear combination):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*84aYObbLBg_U9OAcr2Aocg.png" /><figcaption>Best MRR: 0.419, Best alpha based on MRR: 0.4</figcaption></figure><p>Summary-based Hybrid Search RAG (using summaries, linear combination):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uJ9mPZDB9bs754JMmbDDIw.png" /><figcaption>Best MRR: 0.410, Best alpha based on MRR: 0.4</figcaption></figure><p>Reciprocal Rank Fusion(Default and Summary-based):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nXoO7wR13trAU0d43QkM7g.png" /></figure><h4>Key findings</h4><ol><li><strong>Hybrid vs Dense vs Sparse Search</strong>:<br>At first glance, we noticed an interesting pattern: pure dense vector search (α=1) performs worse than pure keyword search (α=0) in both approaches.<br>– Summary-based: Dense (α=1): 0.246 → Hybrid (α=0.4): 0.410<br>(66% improvement)<br>– Original table: Dense (α=1): 0.287 → Hybrid (α=0.4): 0.419<br>(46% improvement)</li><li><strong>Summary vs Original Table</strong>:<br>– Hybrid (α=0.4): Summary (0.410) vs Original (0.419)<br>– Nearly identical performance<br>While the default approach shows marginally better results (0.419 vs 0.410), the test data mostly fits within model limits (8128 tokens). For large tables exceeding these limits, the summary approach remains viable while the default approach fails — making it a more scalable solution.</li><li><strong>Fusion Method Comparison(RFF vs linear combination)</strong>:<br> Surprisingly, linear combination fusion with optimized alpha outperforms Reciprocal Rank Fusion, contrary to initial expectations.</li></ol><h3>Key Takeaways</h3><p>The research reveals several key insights for building effective table retrieval systems:</p><ol><li><strong>Hybrid Search Superiority<br></strong>Combining dense and sparse embeddings significantly outperforms pure approaches: <br>– Summary-based: 66% improvement over pure dense search<br>– Original table: 46% improvement over pure dense search<br>– Optimal performance achieved at α=0.4 for both approaches</li><li><strong>Choosing the Right Strategy<br></strong>Two viable approaches emerged for hybrid search implementation:<br>–Default (using original tables): Slightly better performance (0.419 MRR)<br>–Summary-based: Nearly identical performance (0.410 MRR) with better scalability<br><strong>Production recommendation</strong>: Use the default approach for datasets where most tables are under 8128 tokens. Switch to the summary-based approach when dealing with larger tables or when scalability is crucial.</li><li><strong>Score Fusion Method<br></strong>Despite theoretical advantages of Reciprocal Rank Fusion, linear combination with optimized alpha provided better results, suggesting that simpler fusion methods can be effective when properly tuned.</li></ol><p>These findings demonstrate that hybrid search is a robust solution for table retrieval, with implementation choice depending primarily on your data characteristics and scalability requirements.</p><h3>P.S.</h3><p>While building your RAG system based on these findings, remember that data characteristics are crucial. While this research offers helpful insights, it’s not a one-size-fits-all fix. Your best approach may vary based on your own data and needs.</p><h3>Profile</h3><ol><li><a href="https://github.com/Maksimov-Dmitry/Excel-Rag/blob/main/6_dm_excels_embeddings.ipynb">Notebook with implementation</a></li><li><a href="https://github.com/Maksimov-Dmitry">GitHub</a></li><li><a href="https://www.linkedin.com/in/maksimov-dmitry/">LinkedIn</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7bbea69a31f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Baseline to SOTA: Enhancing Scene Text Recognition with the ParSeq Model]]></title>
            <link>https://medium.com/@maksimov.dmitry.m/baseline-to-sota-enhancing-scene-text-recognition-with-the-parseq-model-4b28936d7fbf?source=rss-65387e04d923------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b28936d7fbf</guid>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[scene-text-detection]]></category>
            <dc:creator><![CDATA[Dmitrii Maksimov]]></dc:creator>
            <pubDate>Fri, 28 Jul 2023 17:10:41 GMT</pubDate>
            <atom:updated>2023-07-28T17:45:45.040Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*YLMVFVfS4ZsYhXERs4fIEg.png" /></figure><h3>Introduction</h3><p>In the dynamic field of computer vision, Scene Text Recognition (STR) is gaining ever-increasing importance. In our journey through the world of STR, we’ll be taking a practical route — diving straight into a real-world application where we improve results using a state-of-the-art model, ParSeq, moving beyond the traditional baseline models.</p><p>In this article we’ll walk through a solution that I implemented for a Kaggle competition on Scene Text Recognition. This competition presented an exciting opportunity to put the ParSeq model into action and witness its advantages firsthand.</p><p>Rather than delving deep into the theoretical concepts, we’ll focus on this hands-on application, breaking down each step of my approach — from data preprocessing and model selection to performance evaluation and improvement.</p><h3>Data Exploration</h3><p>In this data exploration section, we’ll be diving deep into the details of our dataset for Scene Text Recognition. We’ll analyze various aspects like the volume of data, its structure, the diversity of the samples, and the balance of classes. Understanding these elements is crucial for preprocessing, model selection, and evaluation strategy.</p><p>But instead of just telling you about the dataset, I believe it’s more effective to show you. In the following section, I’ll present a series of plots and visualizations generated from our training dataset. Through these visuals, we’ll start to get a feel for our data, its characteristics, and the potential challenges that it might pose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*09WITCPWZtGnlFgYezTUiQ.png" /></figure><p>Looking at the initial samples from our dataset, some interesting characteristics immediately catch our eye. We observe a considerable diversity in our data, with variations in text length, image shape, and even the language of the text. These aspects add a layer of complexity to our Scene Text Recognition task but also make it more realistic and applicable to real-world scenarios.</p><p>Text length varies, reflecting the reality that in a natural scene, text snippets can range from a single character to a lengthy sentence. Image shapes also differ, reflecting the fact that text can appear in a variety of environments and perspectives. Finally, the inclusion of different languages increases the complexity, but also widens the potential applications of our model.</p><p>Let&#39;s start with plotting a histogram of text lengths in our dataset, enabling us to anticipate challenges and customize our modeling approach accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*J-FnyIbeRmvTgzm270-JoQ.png" /></figure><p>Upon analyzing the histogram of text lengths, the majority of the text instances in our dataset are fairly short, with a significant portion not exceeding a length of 20 characters.</p><p>The diversity of languages in our dataset makes our Scene Text Recognition task both challenging and exciting, broadening the applicability of our resulting model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*_SgyKdFRZp3yGAb9R9Mc1Q.png" /></figure><p>We observe that the ‘undefined’ language category contributes only a small fraction to our overall dataset. Given its size and the uncertainty it introduces, it would be reasonable to exclude these instances from our dataset. This would help us focus on clearly defined categories and ensure a more reliable and interpretable model performance.</p><p>Given the diverse nature of our dataset, we’re likely to encounter a mix of image shapes. To simplify this, we’ve categorized images into three primary classes — horizontal rectangles, vertical rectangles, and squares.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*NO9_NQ2suIpHTnFBpIp8eQ.png" /></figure><p>Traditional models often struggle with vertical rectangles orientations, primarily designed to handle horizontal text layout. This is due to the fact that reading systems, and thus, models, are typically built with the assumption of text lines being horizontal, following the pattern of most human languages.</p><p>However, the presence of vertical rectangles in our dataset underscores the necessity for a model that can handle this diversity in image shapes effectively.</p><p>Let&#39;s visualize the distribution of image shapes within each language category. This will not only provide insights into the most common image shapes for each language but also help us understand if certain shapes are more prevalent in one language than another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*jwXXD4BVqxF5m2OmtraWEQ.png" /></figure><h3>Baseline Model: RCNN</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TWunHgA4AUK668x5pMWqMw.png" /></figure><p>As we step into the realm of model building, it’s important to start with a baseline model. This serves as our initial benchmark and allows us to measure the impact of our improvements and adjustments.</p><p>For our task, we’ll use a Recurrent Convolutional Neural Network (RCNN) as the baseline. The RCNN model combines the power of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). CNNs excel at learning spatial hierarchies and are commonly used for image processing tasks, making them an ideal choice for handling the image component of our task. On the other hand, RNNs are capable of handling sequences and remembering past information, which is essential for our text recognitio n task. For a detailed understanding of the baseline implementation, I highly recommend referring to the original article¹.</p><p>From a birds eye view, the CNN component of the RCNN acts as a feature extractor. It takes in the raw input images and applies a series of convolutional and pooling layers to extract a hierarchy of spatial features from the images. The strength of CNNs lies in their ability to learn and capture complex patterns in image data, from simple edges and textures to more complex shapes and structures, which are essential for recognizing text within the images.</p><p>On the other hand, the RNN component acts as a sequence generator. It takes the high-level feature representations produced by the CNN and interprets them in a sequential manner. RNNs have the unique ability to process sequential data and remember past information, which is crucial for recognizing the sequence of characters in the text.</p><p>Here a code snippet which showcases how we’ve implemented our RCNN using ResNet-18 as a feature extractor and GRU as a sequence generator. This will provide a practical perspective on the concepts we’ve discussed.</p><p>CNN:</p><pre>class FeatureExtractor(Module):<br>    def __init__(self, input_size=(height_rescale, width_rescale), output_len=max_ctc_length):<br>        super(FeatureExtractor, self).__init__()<br><br>        h, w = input_size<br>        resnet = getattr(models, &#39;resnet18&#39;)(weights=&#39;ResNet18_Weights.DEFAULT&#39;)<br>        self.cnn = Sequential(*list(resnet.children())[:-2])<br>        with torch.no_grad():<br>            latent_space_width = self.cnn(torch.randn(size=(10, 3, height_rescale, width_rescale))).shape[3]<br><br>        self.pool = AdaptiveAvgPool2d((1, latent_space_width))<br>        self.proj = Conv2d(latent_space_width, output_len, kernel_size=1)<br><br>        self.num_output_features = self.cnn[-1][-1].bn2.num_features<br><br>    def apply_projection(self, x):<br>        &quot;&quot;&quot;Use convolution to increase width of a features.<br><br>        Args:<br>            - x: Tensor of features (shaped B x C x H x W).<br><br>        Returns:<br>            New tensor of features (shaped B x C x H x W&#39;).<br>        &quot;&quot;&quot;<br>        x = x.permute(0, 3, 2, 1).contiguous()<br>        x = self.proj(x)<br>        x = x.permute(0, 2, 3, 1).contiguous()<br><br>        return x<br><br>    def forward(self, x):<br>        # Apply conv layers<br>        features = self.cnn(x)<br><br>        # Pool to make height == 1<br>        features = self.pool(features)<br><br>        # Apply projection to increase width<br>        features = self.apply_projection(features)<br><br>        return features</pre><p>RNN:</p><pre>class SequencePredictor(Module):<br>    def __init__(self, input_size, hidden_size, num_layers, num_classes, dropout=0.3, bidirectional=False):<br>        super(SequencePredictor, self).__init__()<br><br>        self.num_classes = num_classes<br>        self.rnn = GRU(input_size=input_size,<br>                       hidden_size=hidden_size,<br>                       num_layers=num_layers,<br>                       dropout=dropout,<br>                       bidirectional=bidirectional)<br><br>        fc_in = hidden_size if not bidirectional else 2 * hidden_size<br>        self.fc = Linear(in_features=fc_in,<br>                         out_features=num_classes)<br><br>    def _init_hidden(self, batch_size):<br>        &quot;&quot;&quot;Initialize new tensor of zeroes for RNN hidden state.<br><br>        Args:<br>            - batch_size: Int size of batch<br><br>        Returns:<br>            Tensor of zeros shaped (num_layers * num_directions, batch, hidden_size).<br>        &quot;&quot;&quot;<br>        num_directions = 2 if self.rnn.bidirectional else 1<br><br>        h = torch.zeros(self.rnn.num_layers * num_directions, batch_size, self.rnn.hidden_size)<br><br>        return h<br><br>    def _reshape_features(self, x):<br>        &quot;&quot;&quot;Change dimensions of x to fit RNN expected input.<br><br>        Args:<br>            - x: Tensor x shaped (B x (C=1) x H x W).<br><br>        Returns:<br>            New tensor shaped (W x B x H).<br>        &quot;&quot;&quot;<br><br>        x = x.squeeze(1)<br>        x = x.permute(2, 0, 1)<br><br>        return x<br><br>    def forward(self, x):<br>        x = self._reshape_features(x)<br><br>        batch_size = x.size(1)<br>        h_0 = self._init_hidden(batch_size)<br>        h_0 = h_0.to(x.device)<br>        x, _ = self.rnn(x, h_0)<br><br>        x = self.fc(x)<br>        return x</pre><p>RCNN:</p><pre>class CRNN(Module):<br>    def __init__(self, num_classes=len(id2char),<br>                 cnn_input_size=(height_rescale, width_rescale), cnn_output_len=max_ctc_length,<br>                 rnn_hidden_size=128, rnn_num_layers=2, rnn_dropout=0.3, rnn_bidirectional=False):<br>        super(CRNN, self).__init__()<br>        self.features_extractor = FeatureExtractor(input_size=cnn_input_size, output_len=cnn_output_len)<br>        self.sequence_predictor = SequencePredictor(input_size=self.features_extractor.num_output_features,<br>                                                    hidden_size=rnn_hidden_size, num_layers=rnn_num_layers,<br>                                                    num_classes=num_classes, dropout=rnn_dropout,<br>                                                    bidirectional=rnn_bidirectional)<br><br>    def forward(self, x):<br>        features = self.features_extractor(x)<br>        sequence = self.sequence_predictor(features)<br>        return sequence</pre><p>We need a nuanced metric that can capture the specifics of our task — recognizing sequences of characters accurately. To this end, we’ve chosen the Mean Levenshtein Distance as our evaluation metric. The Levenshtein Distance, also known as the ‘edit distance’, quantifies how dissimilar two strings are by calculating the minimum number of single-character edits (insertions, deletions or substitutions) needed to change one word into the other.</p><p>After training our RCNN baseline model, we proceeded to evaluate its performance on our test dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*kLV7mJ4OcqUT7t8m8ngZhg.png" /></figure><p>Some test examples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U43Lb0TJ5CV9A8bXfF7T_A.png" /></figure><p>To delve deeper into the performance of our RCNN baseline model, let&#39;s break down the Mean Levenshtein Distance by the shape of the images — squares, horizontal rectangles, and vertical rectangles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*bOomVvuGIiOlJ6Xt94dBjQ.png" /></figure><p>As we expected, our RCNN baseline model faced challenges when dealing with vertical rectangles. The Mean Levenshtein Distance was notably higher for these images compared to squares and horizontal rectangles. This suggests that our model is finding it harder to recognize text correctly in vertical rectangles, requiring more single-character edits to match the actual text.</p><h3>SOTA Model: PARSeq</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rcv6f9n_GHLOlgy9GW5EnQ.png" /></figure><p>When embarking on a machine learning task, one of the crucial step should be to research the current state-of-the-art methods for similar tasks. Often, these are compiled and presented in benchmarks, which compare various models on the same datasets, providing a straightforward way to identify the most effective models.</p><p>Following this strategy can save significant time and effort. Instead of starting from scratch, you can leverage the latest advancements in your field, focusing your energy on optimizing and tailoring the state-of-the-art models to your specific task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GF4dqXxO_QrFjY4W5GmIYw.png" /></figure><p>In our quest to find the state-of-the-art model for our Scene Text Recognition task, we turned to the ICDAR2013 benchmark.</p><p>The theory and architecture of the PARSeq model are beyond the scope of this article, and for those who are interested in delving into those details, we recommend reading the original article on the model². The authors have done an excellent job explaining the mechanics of PARSeq, its architecture, and the philosophy behind its design.</p><p>Our focus here is on the practical application — we will explore how to effectively implement and train the PARSeq model on our dataset, and how to tweak and adjust it to handle our specific challenges.</p><p>Before we can start working with the PARSeq model, we first need to install some necessary libraries and download the pre-trained model.</p><pre>!pip install pytorch_lightning==1.6.5<br>!pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 torchtext==0.12.0 torchdata==0.3.0 --extra-index-url https://download.pytorch.org/whl/cu113<br><br>parseq = torch.hub.load(&#39;baudm/parseq&#39;, &#39;parseq&#39;, pretrained=True, decode_ar=True, refine_iters=6, max_label_length=max_target_length, batch_size=batch_size)</pre><p>Our pre-trained PARSeq model was trained on English text. However, our dataset contains text in different languages, including Russian. This means that we need to adapt our model to recognize these additional characters.</p><p>This adaptation involves two key steps:</p><ol><li>Adding New Tokens: The first step is to extend the model’s vocabulary with the new characters. We’ll add tokens for the new characters that aren’t covered by the original English-centric vocabulary. This will ensure our model can generate the correct text in all the languages in our dataset.</li><li>Customizing the Embedding Layer: Once we have the extended vocabulary, we need to update the model’s embedding layer. The embedding layer is responsible for converting our tokens into vector representations that the model can process. But, instead of creating a new embedding layer from scratch, we can leverage the pre-trained weights for the existing tokens. This allows our model to retain the knowledge it has learned from its pre-training. For the new tokens, we’ll initialize their embeddings randomly. Over time, during the training process, these embeddings will be updated and optimized to represent the new tokens effectively.</li></ol><p>Now, let’s look at how we can implement these steps in code:</p><pre>class PARSeq(pl.LightningModule):<br>    def __init__(self):<br>        super().__init__()<br><br>        self.model = torch.hub.load(&#39;baudm/parseq&#39;, &#39;parseq&#39;, pretrained=True, decode_ar=True, refine_iters=6, max_label_length=max_target_length, batch_size=batch_size)<br>        self._change_tokenizer()<br>        self.model.head = Linear(self.model.head.in_features, len(self.model.tokenizer) - 2)<br>        self._change_text_embed()<br><br>    def forward(self, images, max_length=None):<br>        return self.model.forward(images, max_length)<br><br>    def training_step(self, batch, batch_idx):<br>        return self.model.training_step(batch, batch_idx)<br><br>    def _eval_step(self, batch, stage):<br>        images, labels = batch<br><br>        metric = 0<br>        logits = self.forward(images)<br>        probs = logits.softmax(-1)<br>        preds, probs = self.model.tokenizer.decode(probs)<br>        for pred, gt in zip(preds, labels):<br>            metric += edit_distance(pred, gt)<br>        metric /= len(labels)<br>        self.log(f&#39;{stage}_metric&#39;, metric, on_epoch=True,  on_step=False, prog_bar=True)<br>        return None<br><br>    def validation_step(self, batch, batch_idx):<br>        return self._eval_step(batch, &#39;val&#39;)<br><br>    def test_step(self, batch, batch_idx):<br>        return self._eval_step(batch, &#39;test&#39;)<br><br>    def predict_step(self, batch, batch_idx, dataloader_idx=0):<br>        logits = self.forward(batch[0])<br>        probs = logits.softmax(-1)<br>        preds, probs = self.model.tokenizer.decode(probs)<br>        return preds<br><br>    def _change_tokenizer(self):<br>        self.model.tokenizer._itos = self.model.tokenizer._itos[:-2]<br>        last_id = len(self.model.tokenizer._itos) - 1<br>        for my_token in id2char:<br>            if my_token in (self.model.tokenizer.EOS, self.model.tokenizer.BOS, self.model.tokenizer.PAD):<br>                raise Exception<br>            if my_token not in self.model.tokenizer._stoi:<br>                self.model.tokenizer._itos += (my_token, )<br>                last_id += 1<br>                self.model.tokenizer._stoi[my_token] = last_id<br>        for token in (self.model.tokenizer.BOS, self.model.tokenizer.PAD):<br>            self.model.tokenizer._itos += (token, )<br>            last_id += 1<br>            self.model.tokenizer._stoi[token] = last_id<br><br>        self.model.tokenizer.bos_id = self.model.tokenizer._stoi[self.model.tokenizer.BOS]<br>        self.model.tokenizer.pad_id = self.model.tokenizer._stoi[self.model.tokenizer.PAD]<br><br>        self.model.bos_id = self.model.tokenizer.bos_id<br>        self.model.pad_id = self.model.tokenizer.pad_id<br><br>    def _change_text_embed(self):<br>        existing_weight = self.model.text_embed.embedding.weight<br>        new_weights = torch.empty(len(self.model.tokenizer), self.model.text_embed.embedding.embedding_dim)<br>        torch.nn.init.xavier_uniform_(new_weights)<br>        new_weights[:existing_weight.size(0) - 2] = existing_weight[:-2]<br>        new_weights[-2:] = existing_weight[-2:]<br><br>        self.model.text_embed.embedding = Embedding(num_embeddings=len(self.model.tokenizer), embedding_dim=self.model.text_embed.embedding.embedding_dim, _weight=new_weights)<br><br>    def configure_optimizers(self):<br>        agb = self.trainer.accumulate_grad_batches<br>        # Linear scaling so that the effective learning rate is constant regardless of the number of GPUs used with DDP.<br>        lr_scale = agb * np.sqrt(self.trainer.num_devices) * self.model.batch_size / 256.<br>        lr = lr_scale * self.model.lr<br>        optim = create_optimizer_v2(self.model, &#39;adamw&#39;, lr, self.model.weight_decay)<br>        sched = torch.optim.lr_scheduler.OneCycleLR(optim, lr, self.trainer.estimated_stepping_batches, pct_start=self.model.warmup_pct,<br>                           cycle_momentum=False)<br>        return {&#39;optimizer&#39;: optim, &#39;lr_scheduler&#39;: {&#39;scheduler&#39;: sched, &#39;interval&#39;: &#39;step&#39;}}<br><br>    def optimizer_zero_grad(self, *args, **kwargs):<br>        self.model.optimizer_zero_grad(*args, **kwargs)</pre><p>Let’s dive into the results!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*j-PnFQ0e_78X_UyEBcJOMg.png" /></figure><p>Some test examples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QBlJu034vmNpPzFHBsY-Gg.png" /></figure><p>As we can see, the results achieved with the PARSeq model demonstrate a substantial improvement over our initial baseline model. These results are a testament to the advanced capabilities of the PARSeq model, and the power of adapting and fine-tuning pre-trained models to our specific task.</p><h3>Summary</h3><p>In this article, we embarked on a journey to enhance Scene Text Recognition using the state-of-the-art PARSeq model. We started with a baseline RCNN model, explored our dataset’s unique characteristics, and gradually implemented and adapted the PARSeq model to improve our text recognition task performance significantly.</p><p>While we didn’t delve into some technical aspects, such as the explanation of the CTC loss used in the RCNN model, the specific augmentations used in our pipeline, and some other implementation details, these can be found in the corresponding Colab notebook linked at the end of this article. We encourage the interested reader to explore it for a deeper understanding and for reproducing our work.</p><h3>Profile</h3><ol><li><a href="https://colab.research.google.com/drive/1qRhqi4aWlKunKWwwFnCtS-RZW5JitrfX?usp=sharing">Colab notebook with full implementation</a></li><li><a href="https://github.com/Maksimov-Dmitry/MADE-DL/tree/main/computer_vision/ocr">Github</a></li><li><a href="http://linkedin.com/in/maksimov-dmitry">LinkedIn</a></li></ol><h3>References</h3><ol><li><a href="https://arxiv.org/abs/1507.05717">https://arxiv.org/abs/1507.05717</a></li><li><a href="https://arxiv.org/abs/2207.06966">https://arxiv.org/abs/2207.06966</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b28936d7fbf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>