<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Andrei Khobnia on Medium]]></title>
        <description><![CDATA[Stories by Andrei Khobnia on Medium]]></description>
        <link>https://medium.com/@aikho?source=rss-eeeb899dfa18------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4sJwq3nrN13uOm7ON3nhYA.png</url>
            <title>Stories by Andrei Khobnia on Medium</title>
            <link>https://medium.com/@aikho?source=rss-eeeb899dfa18------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:24:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aikho/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Explainability and Ad Hoc Fixing for Vector Search and RAG]]></title>
            <link>https://medium.com/@aikho/explainability-and-ad-hoc-fixing-for-vector-search-and-rag-7acd6835c399?source=rss-eeeb899dfa18------2</link>
            <guid isPermaLink="false">https://medium.com/p/7acd6835c399</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[information-retrieval]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Andrei Khobnia]]></dc:creator>
            <pubDate>Thu, 12 Feb 2026 13:21:01 GMT</pubDate>
            <atom:updated>2026-08-25T19:14:51.418Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jMOVMMfCCsf42KY6wEq8iQ.png" /><figcaption>Just illustration (AI-generated)</figcaption></figure><p>Vector search has become a critical component of modern agentic and RAG systems, serving as the bridge between vast knowledge bases and produced results. As these systems increasingly power customer support chatbots, code assistants, and autonomous agents, the quality of vector retrieval directly impacts their reliability and effectiveness. This is why it requires careful attention to search scores anomalies, systematic identification of issues in training data, and continuous quality improvements. Engineering teams need the ability to explain why certain documents or passages are retrieved, debug cases where relevant information is missed or irrelevant results surface, and iteratively refine their embedding models and indexing strategies. Without these diagnostic and optimization capabilities, even sophisticated agentic systems can fail silently, returning plausible but incorrect information or missing critical context that would lead to better responses. The ability to inspect, understand, and improve vector search quality is essential component of AI safety strategy and a foundation for building trustworthy AI systems that can operate reliably in production environments.</p><p>In this article we will use <strong>Retrivex</strong> toolkit to generate explanations of embedding model predictions and use them to make practical decisions about training data quality improvements. We will also go through a small demo example with ad hoc fix using a toy dataset to illustrate the idea.</p><h3>What is Retrivex?</h3><p><a href="https://github.com/aikho/retrivex">Retrivex</a> implements specialized explainability methods from academic research on retrieval models into a reliable, generalized toolkit. Embedding retrieval models used for vector search require specialized explainability methods because they operate fundamentally differently from standard classification or regression models. Rather than predicting discrete classes, they compute vectors and then continuous similarity scores between query-candidate pairs, making traditional tools like LIME inapplicable. Explanations must capture the interaction between two inputs simultaneously and account for how high-dimensional embeddings drive similarity. This necessitates tailored approaches such as gradient-based attribution in embedding space or concept-level alignment to provide meaningful insights.</p><h3>Explainability for Vector Similarity Scores</h3><p>Let’s look at a couple of examples. Start by installing dependencies:</p><pre>pip install torch &#39;transformers[torch]&#39; sentence-transformers <br>pip install retrivex</pre><p>We’ll use the all-MiniLM-L6-v2 model for vector search embeddings. While somewhat outdated, it effectively demonstrates explainability concepts for embedding models. Load the SentenceTransformer model:</p><pre>from sentence_transformers import SentenceTransformer<br>model = SentenceTransformer(&#39;sentence-transformers/all-MiniLM-L6-v2&#39;)</pre><p>Now let’s compute embedding similarity for these sentence pairs from the Quora similar questions dataset:</p><pre>sentences = [<br>    &quot;Which are some of the coolest VLSI startups in Silicon Valley?&quot;,   # query<br>    &quot;What are the VLSI startups in California?&quot;,                        # true positive<br>    &quot;What are the hottest startups in Silicon Valley?&quot;,                 # false positive<br>    &quot;What are the hottest startups to work at in Silicon Valley?&quot;       # false positive<br>]<br><br>embeddings = model.encode(sentences)<br><br>for i in range(1, 4):<br>    score = model.similarity(embeddings[0], embeddings[i])[0][0].float()<br>    print(f&quot;Similarity score for example #{i}: {score:.2f}&quot;)</pre><pre>Similarity score for example #1: 0.78<br>Similarity score for example #2: 0.81<br>Similarity score for example #3: 0.79</pre><p>Examples #2 and #3 (false positives) receive higher similarity scores than example #1 (ground truth). Example #1 should score highest because the question explicitly asks about VLSI startups. Examples #2 and #3 don’t mention VLSI at all.</p><p>Now let’s apply Retrivex explainability tool to understand these predictions. Import retrivex toolkit and load model to explainer:</p><pre>from retrivex.stransformers import IntJacobianExplainableTransformer<br><br>explainer = IntJacobianExplainableTransformer(&quot;sentence-transformers/all-MiniLM-L6-v2&quot;)</pre><p>Then generate explanation for example #1:</p><pre>attributions, tokens1, tokens2, _ = explainer.explain(sentences[0], sentences[1])</pre><p>Then visualize it:</p><pre>from retrivex.visualization import plot_connections<br>vis_params = {<br>    &quot;figsize&quot; :(4,4),<br>    &quot;query_label&quot;: &quot;text 1&quot;,<br>    &quot;candidate_label&quot;: &quot;text 2&quot;,<br>    &quot;token_fontsize&quot;: 9,<br>    &quot;connection_line_thick&quot;: 1.5,<br>    &quot;plot_pad&quot;: 0.05<br>}<br><br>plot_connections(attributions, tokens1, tokens2, **vis_params)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*04vJL8BI7fTvuxu3CvbqVA.png" /></figure><p>Then generate and visualize explanation for example #2:</p><pre>attributions, tokens1, tokens2, _ = explainer.explain(sentences[0], sentences[2])<br>plot_connections(attributions, tokens1, tokens2, **vis_params)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*8xaxU98VUS1Ot-JIaMsyzw.png" /></figure><p>These attribution plots reveal model biases. Notice the weak attribution for VLSI tokens and the model&#39;s failure to recognize that Silicon Valley and California are interchangeable in tech contexts. <strong>Adding examples with these variations</strong> to the training dataset will help <strong>mitigate these biases</strong>.</p><p>Let’s look at another example:</p><pre>sentences = [<br>    &quot;What is UX design?&quot;,                   # query<br>    &quot;What is user experience design?&quot;,      # true positive<br>    &quot;What is the work of a ux designer?&quot;,   # false positive<br>]<br><br>embeddings = model.encode(sentences)<br><br>for i in range(1, 3):<br>    score = model.similarity(embeddings[0], embeddings[i])[0][0].float()<br>    print(f&quot;Similarity score for example #{i}: {score:.2f}&quot;)</pre><pre>Similarity score for example #1: 0.66<br>Similarity score for example #2: 0.79</pre><p>Question #1 (true positive) is a <strong>direct rephrase</strong> of original question. But it is scored lower than question #2 which is a similar but different question (false positive). Let’s generate explanations and visualize them:</p><pre>attributions, tokens1, tokens2, _ = explainer.explain(sentences[0], sentences[1])<br>plot_connections(attributions, tokens1, tokens2, **vis_params)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*-7uNmtbKBIotq7aND9GRsw.png" /></figure><pre>attributions, tokens1, tokens2, _ = explainer.explain(sentences[0], sentences[2])<br>plot_connections(attributions, tokens1, tokens2, **vis_params)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*GC4QDvkWL2bEr8l4MMHz6g.png" /></figure><p>Weak attributions between tokens ##x and experience. Model is not robust to using them interchangeably in context of user experience. Ensuring that UX is a <strong>solid token</strong> in the vocabulary and <strong>adding examples where these two terms are used interchangeably</strong> to the training dataset will help mitigate this bias. Let&#39;s demonstrate this idea in the next section.</p><h3>Ad Hoc Fix</h3><p>Let’s demonstrate an ad hoc fix: we’ll add a new token to the model vocabulary and fine-tune for a few epochs using additional training samples. <strong>Important to node </strong>that this is a <strong>simplified proof-of-concept</strong> rather than a production-grade solution but it effectively shows the principle.</p><p>Adding new token to the vocabulary and embeddings:</p><pre>import torch<br>tokenizer = model.tokenizer<br><br>ux_tokens = tokenizer.tokenize(&quot;UX&quot;)<br>print(f&quot;Current tokenization of `UX`: {ux_tokens}&quot;)<br><br># Add UX as a special token if it&#39;s not already a single token<br>if len(ux_tokens) &gt; 1 or ux_tokens[0].lower() != &#39;ux&#39;:<br>    num_added = tokenizer.add_tokens([&#39;UX&#39;])<br>    print(f&quot;Added {num_added} new tokens to vocabulary&quot;)<br>    <br>    # Resize the model&#39;s token embeddings to place the new token<br>    model[0].auto_model.resize_token_embeddings(len(tokenizer), mean_resizing=False)<br>    <br>    # Initialize the new token&#39;s embedding as the mean of &quot;user&quot; and &quot;experience&quot; embeddings<br>    with torch.no_grad():<br>        user_id = tokenizer.convert_tokens_to_ids(&#39;user&#39;)<br>        experience_id = tokenizer.convert_tokens_to_ids(&#39;experience&#39;)<br>        ux_id = tokenizer.convert_tokens_to_ids(&#39;UX&#39;)<br>        <br>        embeddings = model[0].auto_model.embeddings.word_embeddings.weight<br>        if user_id &lt; len(embeddings) and experience_id &lt; len(embeddings):<br>            embeddings[ux_id] = (embeddings[user_id] + embeddings[experience_id]) / 2<br>            print(f&quot;Initialized &#39;UX&#39; embedding as mean of &#39;user&#39; and &#39;experience&#39;&quot;)<br>else:<br>    print(&quot;`UX` is already a single token in the vocabulary&quot;)</pre><pre>Current tokenization of `UX`: [&#39;u&#39;, &#39;##x&#39;]<br>Added 1 new tokens to vocabulary<br>Initialized &#39;UX&#39; embedding as mean of &#39;user&#39; and &#39;experience&#39;</pre><p><strong>Just to demonstrate the idea</strong>, let’s fine-tune the model for a couple of epochs using a toy dataset of approximately 50 samples where UX and user experience are used interchangeably, mixed with random similar question samples. All samples are triplets: original question (anchor), true positive, and hard negative (very similar but different question). <strong>Important note</strong>: this approach is not suitable for a real production system but is sufficient to demonstrate the idea.</p><pre>import csv<br>import random<br>from sentence_transformers import SentenceTransformer, losses, InputExample<br>from torch.utils.data import DataLoader<br><br>def load_triplets(csv_path: str) -&gt; list[InputExample]:<br>    &quot;&quot;&quot;Load anchor/positive/negative triplets from a CSV file.&quot;&quot;&quot;<br>    examples = []<br>    with open(csv_path, newline=&quot;&quot;, encoding=&quot;utf-8&quot;) as f:<br>        reader = csv.DictReader(f)<br>        for row in reader:<br>            examples.append(<br>                InputExample(texts=[row[&quot;anchor&quot;], row[&quot;positive&quot;], row[&quot;negative&quot;]])<br>            )<br>    return examples<br><br><br>ux_triplets = load_triplets(&quot;q_ux_triplets.csv&quot;)<br>general_triplets = load_triplets(&quot;q_general_triplets.csv&quot;)<br><br>all_triplets = ux_triplets + general_triplets<br>random.seed(42)<br>random.shuffle(all_triplets)<br><br>train_dataloader = DataLoader(all_triplets, shuffle=True, batch_size=16)<br>train_loss = losses.TripletLoss(model=model)</pre><p>Let’s run fine-tuning:</p><pre>from torch.optim import Adam<br><br>model.fit(<br>    train_objectives=[(train_dataloader, train_loss)],<br>    epochs=2,<br>    output_path=&quot;finetuned-ad-hoc-mixdataset&quot;,<br>    show_progress_bar=True,<br>    optimizer_class=Adam,<br>    optimizer_params={&quot;lr&quot;: 2e-5},<br>)</pre><p>Let’s check how texts are scored after this fine-tuning:</p><pre>sentences = [<br>    &quot;What is UX design?&quot;,                   # query<br>    &quot;What is user experience design?&quot;,      # true positive<br>    &quot;What is the work of a ux designer?&quot;,   # false positive<br>]<br><br>embeddings = model.encode(sentences)<br><br>for i in range(1, 3):<br>    score = model.similarity(embeddings[0], embeddings[i])[0][0].float()<br>    print(f&quot;Similarity score for example #{i}: {score:.2f}&quot;)</pre><pre>Similarity score with 1 example: 0.92<br>Similarity score with 2 example: 0.81</pre><p>Now ground truth is scored higher than false positive example.</p><p>All code in this article is available in <a href="https://github.com/aikho/ir-examples/blob/main/1_vector_search_explain_ad_hoc_fix/1_vector_search_explain_and_ad_hoc_fix.ipynb">this notebook</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7acd6835c399" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Learning in Information Retrieval. Part III: Ranking]]></title>
            <link>https://itnext.io/deep-learning-in-information-retrieval-part-iii-ranking-da511f2dc325?source=rss-eeeb899dfa18------2</link>
            <guid isPermaLink="false">https://medium.com/p/da511f2dc325</guid>
            <category><![CDATA[information-retrieval]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Andrei Khobnia]]></dc:creator>
            <pubDate>Fri, 13 Jan 2023 07:56:47 GMT</pubDate>
            <atom:updated>2023-01-16T08:31:50.002Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DLYC9Fk8EkvJfX5-F9Vu-g.png" /><figcaption>Example of ranking pipeline. Image by author</figcaption></figure><p>This is the third article in the series “Deep Learning in Information Retrieval”. This series is for those who know the basics of deep learning and would like to dive deep into the field of modern information retrieval. In previous parts, we have had a look at <a href="https://medium.com/itnext/deep-learning-in-information-retrieval-part-i-introduction-and-sparse-retrieval-12de0423a0b9">sparse</a> and <a href="https://medium.com/@aikho/deep-learning-in-information-retrieval-part-ii-dense-retrieval-1f9fecb47de9">dense</a> retrieval methods. Each family of methods has its own advantages and disadvantages. But as has already been mentioned in the previous article, dense retrieval and sparse retrieval methods are usually used as the first step of the pipeline of information retrieval systems. In this part, we will look at more powerful models that are used at further stages of information retrieval systems pipelines.</p><h4>Ranking and Re-ranking</h4><p>IR can be defined as the problem of ranking the collection of data entities by relevancy to a particular query. A straightforward approach is to design an algorithm or a model which sorts a list of all entities in accordance to relevancy to a search query. But it is impractical to apply some ranking algorithm to the whole set of entities in the index. After some scalable methods of sparse or dense retrieval is applied and a reasonable amount of candidates are retrieved from the index, it is possible to apply a more powerful ranking method to this list. In a complex information retrieval system, multi-stage ranking process can be applied. Usually, later stages of ranking have better ranking performance but worse speed than previous stages. Thus later stages of ranking take less amount of candidates from the previous stages. Also, later stages can use more signals to rank candidates. For example, creation date, metadata, author, or citation index can be taken into account in the ranking of documents. In some literature, ranking stages can be referred as <strong>pre-ranking</strong>, <strong>ranking</strong> and <strong>re-ranking</strong>. In this article, textual-based ranking is considered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mKzyTmX-ZczlugtGGAA5HA.png" /><figcaption>Ranking pipeline with several stages. Image by author</figcaption></figure><p>One of the popular and straightforward approaches to build a ranker is to design an algorithm or a model which predicts the relevancy score of a candidate for the query. Then candidates can be sorted by these scores. Other words, the idea of this approach is pair (query and candidate) scoring.</p><p>The concept of re-ranking by pair scoring was presented in papers about solving question answering. One work introduces answers re-ranking process for cloze-style reading comprehensions. The authors suggest refilling the candidate&#39;s answer into the blank of the query to double-check its appropriateness, fluency and grammar. They used several language models to score the resulting candidate sentences built by filling an answer into the blank of the query [<a href="#209f">1</a>].</p><p>It is reasonable to suggest that a similar approach of ranking by scoring pairs can be applied to ranking in information retrieval. Having a dataset of relevant pairs of queries and documents, it is possible to train a model which predicts the relevancy of a particular candidate to the query. Then these predicted values can be used as scores for re-ranking the list of candidates.</p><p>The next straightforward reasonable idea is applying pre-trained transformer-based language models for the task of pair scoring. Transformer architecture is based on<strong> self-attention mechanism</strong>. So if query and candidate are concatenated then it is possible for self-attention to model <strong>cross-interactions</strong> between tokens of query and tokens of candidate. These kinds of models are often referred as <strong>cross-encoders</strong>. Let’s look at how these models can be applied for ranking in the next section.</p><h4>Deep Learning and Transformers in Ranking</h4><p>One of the earliest works in applying large transformer-based language models for the re-ranking task has introduced a model called <strong>MonoBERT</strong>. This model is a simple re-implementation of BERT model for query-based passage re-ranking.</p><p>MonoBERT uses truncated query to have at most 64 tokens and truncated passage text such that the concatenation of query, passage, and separator tokens have a maximum length of 512 tokens. It is built based on BERT model as a binary classification model. [CLS] vector is used as input to a single layer neural network to obtain the probability of the passage being relevant. This probability is computed for each passage independently. The final list of passages is obtained by ranking them with respect to these probabilities. The model was trained from pre-trained BERT model using simple cross-entropy loss [<a href="#7966">2</a>].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1qF0XsSs-4A3FHa4sQSLYA.png" /><figcaption>MonoBERT ranking. Image by author</figcaption></figure><p>When MonoBERT was introduced, IR-pipeline based on this model had achieved the state of the art performance on the TREC-CAR dataset and the top entry in the leaderboard of the MS MARCO passage retrieval task, outperforming the previous state of the art by 27% (relative) in MRR@10.</p><p>In later work, the same authors suggested a more performant approach to re-ranking. The main idea of the suggested approach is using multi-stage ranking pipeline with two re-ranking models: <strong>MonoBERT</strong> and <strong>DuoBERT</strong>. MonoBERT is a BERT model used as a binary relevance classifier. The output from the MonBERT stage is used as input to the pairwise re-ranker called duoBERT. Within the framework of “learning to rank”, DuoBERT can be characterized as a “pairwise” approach, while MonoBERT can be characterized as a “pointwise” approach. In this pairwise approach, the re-ranker estimates the probability of the candidate dᵢ being more relevant than dⱼ. This re-ranker is also a BERT model that takes as input the query, candidate dᵢ and candidate dⱼ. The query, candidates dᵢ and dⱼ are truncated to 62, 223, and 223 tokens, respectively, so the entire sequence has at most 512 tokens when concatenated with the [CLS] token and the three separator tokens. Using the above truncation limits, in the datasets MS MARCO and TREC CAR none of the queries are truncated, and less than 1% of the documents are truncated [<a href="#b323">3</a>].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ELe8qN6dm_2aSSzfip82jg.png" /><figcaption>Multi-stage ranking pipeline with MonoBERT and DuoBERT. Image by author</figcaption></figure><p>On two large-scale datasets, MS MARCO and TREC CAR, experiments show that our model produces results that are either at or comparable to state of the art.</p><p>A lot of further works in this area are focused on using bigger language models or models trained differently. For example, in one work huge sequence-to-sequence model had been adopted for the task of document ranking. It was shown how a sequence-to-sequence model could be trained to generate relevance labels as “target words”, and how the underlying logits of these target words could be interpreted as relevance probabilities for ranking. In this work, re-ranking method is based on T5 model, which is a sequence-to-sequence model that uses a similar masked language modeling objective as BERT to pretrain its encoder–decoder architecture. T5 is pre-trained on a multi-task mixture of unsupervised and supervised tasks. Also, it is a much bigger model than BERT. BERT (in base version) has about 110 million parameters while T5 has 11 billion parameters. The model is fine-tuned to produce the words “true” or “false” depending on whether the document is relevant or not to the query. That is, “true” and “false” are the “target words” (i.e., ground truth predictions in the sequence-to-sequence transformation). At inference time, to compute probabilities for each query-document pair, softmax is applied only on the logits of the “true” and “false” tokens. Then the documents are re-ranked according to the probabilities assigned to the “true” token [<a href="#2ace">4</a>].</p><p>On the popular MS MARCO passage ranking task, experimental results show that our approach is at least on par with previous classification-based models and can surpass them with larger, more-recent models. On the test collection from the TREC 2004 Robust Track, zero-shot transfer-based approach outperforms previous state-of-the-art models requiring in-dataset cross-validation. Also thus approach significantly outperforms an encoder-only model in cases when only a few training examples are available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fL9xwO_uVs4udbdgu56yTg.png" /><figcaption>Multi-stage ranking pipeline with MonoT5 and DuoT5. Image by author</figcaption></figure><p>In later work, a re-ranking pipeline with pointwise and pairwise rankers based on T5 model was introduced. These rankers are referred as <strong>MonoT5</strong> and <strong>DuoT5 </strong>[<a href="#5d5d">5</a>].</p><p>In the approach (which was mentioned in <a href="https://medium.com/@aikho/deep-learning-in-information-retrieval-part-ii-dense-retrieval-1f9fecb47de9">previous article</a>), called <strong>RocketQA,</strong> model based on ERNIE 2.0 is used for re-ranking [<a href="#9733">6</a>]. <strong>ERNIE 2.0</strong> model uses a multi-layer Transformer as the basic encoder like other pre-training models such as GPT or BERT. But it introduces a continual pre-training framework on multiple pre-trained tasks. It uses several pre-training tasks to capture lexical, syntactic and semantic aspects of information in the training data [<a href="#0fee">7</a>]. RocketQA uses ranker based on ERNIE with pair of dense retrieval and uses ranker as teacher model for pseudo-labeling data to improve retriever [<a href="#9733">6</a>].</p><p>In the next version of this approach, called <strong>RocketQAv2</strong>, a joint training method for dense passage retrieval and re-ranking has been introduced. This approach has introduced the <strong>dynamic listwise distillation</strong>, where unified listwise training approach is used for both the retriever and the re-ranker. During the dynamic distillation, the retriever and the re-ranker can be adaptively improved according to each other’s relevance information. The main idea is to adaptively reduce the difference between the two distributions from the retriever and the re-ranker so as to mutually improve each other [<a href="#aaeb">8</a>]. To achieve adaptively mutual improvement, the KL-divergence between the two relevance distributions from the two modules is minimized. The KL-divergence loss and the supervised crossentropy loss are combined to the final loss function for this approach.</p><p>Recently, a new re-ranking approach has been introduced. It is called <strong>Hybrid List Aware Transformer Re-ranking (HLATR)</strong>. This approach uses multi-stage ranking. HLATR model is pipelined with the previous re-ranking model (for example, MonoBERT). HLATR model uses the multi-stage features fusion via coupling the <strong>retrieval ranking order</strong> in the retrieval stage and the final logit representation generated by the re-ranking model of each candidate. Retrieval ranking order is mapped into a learnable position embedding with dimension <em>d</em> and jointly optimized during the learning process. The position embedding is randomly initialized. For the re-ranking stage final encoded representation of query-document pair is converted to an embedding representation sharing the same dimension <em>d</em> as the position embedding. Finally, a linear projection layer is used to predict the relevance score between the input query and document. You can see the whole schema of the approach from the original paper below. In the schema: <em>q </em>is a query, <em>dᵢ</em> is a document with <em>i</em>-th order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r3YdB7fjYdJ5UYfWcjtEFA.png" /><figcaption>Architecture of HLATR approach. Image from [<a href="#f799">9</a>]</figcaption></figure><p>HLATR is optimized with a list-wise contrastive loss. From the perspective of the learning objective, HLATR can be classified as a re-ranking model, which takes a query and a document list as input and then produces the relevance score of each query-document pair among all candidate documents [<a href="#f799">9</a>]. But in the HLATR model, the relevance score of the entire input document list is calculated at one time. This <strong>list aware calculation</strong> method allows to better model the mutual relationship between candidate documents in the entire document list. Empirical experiments on MS-MARCO Document and MS-MARCO Passage datasets show that using HLATR approach can improve the ranking performance of existing multi-stage text retrieval methods.</p><h4>Conclusions</h4><p>In this article, we have considered the concepts of re-ranking ranking pipelines. Then we have had a look at several different approaches to build powerful rankers using deep transformer-based pre-trained language models. These kinds of models are often referred as cross-encoders because they use self-attention mechanism which is good at modeling cross-interaction between tokens. These methods can have very high-ranking performance. But it can be tricky to use them directly in interactive information retrieval systems like search engines due to their speed. These models can be applied directly in background information retrieval pipelines. For example, if the system retrieves recommended articles for user in advance, and the show stored recommendations in a user interface. But it can be very tricky to use these huge models in an interactive search when the response should be returned in sub-seconds. There are several approaches to this problem. We will look at most of them in the next part.</p><h4>References</h4><ol><li><a href="https://aclanthology.org/P17-1055/">Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu, Attention-over-Attention Neural Networks for Reading Comprehension, In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 593–602, Vancouver, Canada. Association for Computational Linguistics, 2017.</a></li><li><a href="https://arxiv.org/abs/1901.04085">Rodrigo Nogueira and Kyunghyun Cho, Passage re-ranking with BERT, arXiv preprint, 2019, arXiv:1901.04085.</a></li><li><a href="https://arxiv.org/abs/1910.14424">Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin, Multi-stage document ranking with BERT, arXiv preprint, 2019, arXiv:1910.14424.</a></li><li><a href="https://arxiv.org/abs/2003.06713">Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin, Document Ranking with a Pretrained Sequence-to-Sequence Model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 708–718, 2020, arXiv:2003.06713.</a></li><li><a href="https://arxiv.org/abs/2101.05667">Ronak Pradeep, Rodrigo Nogueira, Jimmy Lin, The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models, pp. 1–23, CoRR, 2021, arXiv:2101.05667</a>.</li><li><a href="https://arxiv.org/abs/2010.08191">Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835–5847, Online, 2021, arXiv:2010.08191</a>.</li><li><a href="https://ojs.aaai.org/index.php/AAAI/article/view/6428">Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. ERNIE 2.0: A continual pre-training framework for language understanding. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7–12, 2020, pages 8968–8975. AAAI Press.</a></li><li><a href="https://arxiv.org/abs/2110.07367">Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen, Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking, arXiv preprint, 2021, arXiv:2110.07367.</a></li><li><a href="https://arxiv.org/abs/2205.10569">Zhang, Yanzhao &amp; Long, Dingkun &amp; Xu, Guangwei &amp; Xie, Pengjun, HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Reranking, arXiv preprint, 2022, arXiv:2205.10569.</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da511f2dc325" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/deep-learning-in-information-retrieval-part-iii-ranking-da511f2dc325">Deep Learning in Information Retrieval. Part III: Ranking</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Learning in Information Retrieval. Part II: Dense Retrieval]]></title>
            <link>https://medium.com/@aikho/deep-learning-in-information-retrieval-part-ii-dense-retrieval-1f9fecb47de9?source=rss-eeeb899dfa18------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f9fecb47de9</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[information-retrieval]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Andrei Khobnia]]></dc:creator>
            <pubDate>Fri, 23 Dec 2022 22:50:29 GMT</pubDate>
            <atom:updated>2023-01-13T07:58:44.954Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9jujo5W8LKLuE8EeAXqtmg.png" /><figcaption>Dual-encoder architecture. Image by author</figcaption></figure><p>This is the second part of series of articles “Deep Learning in Information Retrieval”. This series is for those who knows basics of deep learning and would like to deep dive into the field of modern information retrieval. In <a href="https://medium.com/itnext/deep-learning-in-information-retrieval-part-i-introduction-and-sparse-retrieval-12de0423a0b9">part I</a> we have learned basic concepts of information retrieval, metrics used in IR, basics of IR-system design and one family of deep learning methods typically used at the first stage of IR-pipelines. In this part we will look at another family of methods used to obtain representations of searchable data entities. These methods are also typically used at the first stage of IR-system pipeline to retrieve initial candidates from an index at scale.</p><p>Sparse retrieval methods have some advantages like explainability, good fit for usage with classic algorithms like BM25, etc. But sparse representation of text based on tokens doesn’t reflect semantic of each term fully in a context of the whole text. This problem can’t be completely solved by just weighting importance of each term in a text or by document expansion. From another point of view a lot of methods to obtain contextual <strong>embeddings</strong> of tokens or whole text are available now. These methods allow to represent a text in a form of vector in some vector space with predefined dimension. Dimension of vectors does not depend on length of a text. This kind of vectors are usually referred as <strong>dense vectors</strong>. Semantically close texts are usually represented by vectors that are close in a vector space. Family of methods based on dense vectors is referred as <strong>dense retrieval</strong>. Obviously it is impossible to use classic retrieval algorithms like BM25 with dense vectors. So how does dense retrieval work? Let’s dive in.</p><h4>Dense Retrieval</h4><p>At first, let’s look at how dense representation models are trained. Usually some model is taken as a base model. It can be any kind of model which can produce vectors of fixed dimension as an output. Usually neural networks are taken as a base models. Particularly transformer-based models are used in modern approaches. Base model is usually referred as <strong>encoder</strong>. Then it is necessary to make base model produce close vectors for semantically close entities. It can be entities of the same type or different type like request and passage. In the second case two different encoders can be used. Usually <strong>siamese network</strong> architecture is used to train encoders to produce close vectors. This architecture is also referred as <strong>dual-encoder</strong> or <strong>bi-encoder</strong>. You can see schema of this architecture at the top of this article. The idea of this architecture is using two encoders during training. Each encoder gets one entity from a pair as an input. For exampe, first encoder gets query, second encoder gets document. Then distance between output vectors of each encoder is calculated. Usually <strong>cosine distance</strong> is used because magnitude of the vectors usually does not matter in textual similarity and it is simple to calculate using dot product of vectors. Then the whole bi-encoder is trained to minimize distance between matching pairs (<strong>positives</strong>) and maximize distance between non-matching pairs (<strong>negatives</strong>). Only positive pairs are often available in a dataset. In this case examples from other pairs in the same batch are used as negatives. Usually triplet loss is used to implement this kind of training. <strong>Triplet loss</strong> is a loss function for machine learning algorithms where a reference input (a query in case of dense retrieval) referred as <strong>anchor</strong> is compared to a matching input (positive) and a non-matching input (negative). In general, triplet loss can be defines the following way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*e9f18BTnpLKO3ecJqu_cBA.png" /></figure><p>where <strong><em>a</em></strong> is an anchor, <strong><em>p</em></strong> is a positive, <strong><em>n</em></strong> is a negative, <strong><em>d</em></strong> is a distance function and λ is a margin value to keep negatives far apart. This can be illustrated by the following picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*xNzu69MirNEO2rEvSdNPIg.png" /><figcaption>Triplet loss. Image by author</figcaption></figure><p>Different techniques are used to improve triplet loss. For example, it is possible to additionally penalize distance to the closest negative in the batch. Or it is possible to add external close negatives to the training. These techniques are referred as <strong>hard negatives mining</strong>.</p><p>At building of search index step trained encoder is used to obtain dense vectors for searchable entities (documents, passages). These vectors are stored in the index. At retrieval step trained encoder is used to produce dense vector of a query. Then this vector is used to calculate cosine distance between query vector and vectors of candidates from search index. Then some reasonable top of the closest candidates are retrieved from the index.</p><p>At scale it is impractical to calculate distances between query and each candidate from a huge index. But how to implement this in a scalable way? There are different solutions of this problem. One of the most common is to use LSH-index. <strong>LSH</strong> (<strong>locality-sensitive hashing)</strong> is an algorithm which hashes similar input entities into the same “buckets” with high probability [<a href="#b358">1</a>]. Using LSH allows to implement <strong>approximate nearest neighbor search</strong>. LSH-index uses set of LSH functions called <strong>LSH family</strong>. <strong>LSH function</strong> is a hash function which maps elements of the <a href="https://en.wikipedia.org/wiki/Metric_space">metric space</a> to buckets. A lot of hash tables are constructed with corresponding hash functions each of which is constructed by concatenating several randomly chosen hash functions from the same LSH family. At building of search index step all searchable entities are hashed into each hash table. At retrieval step algorithm iterates over hash tables and retrieves the entities that are hashed into the same bucket as query vector. As soon as an entity is found retrieval process is stopped. This approach allows to scale over tens and hundreds of millions of searchable entities. Libraries like <a href="https://faiss.ai/">FAISS</a>, <a href="https://github.com/google-research/google-research/tree/master/scann">ScaNN</a> and vector databases like <a href="https://milvus.io/">Milvus</a> and <a href="https://qdrant.tech/">Qdrant</a> use similar algorithms to implement approximate nearest neighbor search.</p><p>This is the high level description of how dense retrieval works. In the next section let’s look at how to apply deep learning models to obtain better dense representations.</p><h4>Deep Learning and Transformers in Dense Retrieval</h4><p>One of the earliest work which adopts large transformer-based models for dense retrieval is <strong>DPR </strong>(<strong>Dense Passage Retriever</strong>). In this work authors show that retrieval can be practically implemented using dense vectors, and qualitative embeddings of text passages can be learned from a small number of questions and passages by a bi-encoder type of model [<a href="#955d">2</a>].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H4AF36h9gH5WNO3ujrRp9A.png" /><figcaption>DPR bi-encoder architecture. Image by author</figcaption></figure><p>In Dense Passage Retriever two independent BERT networks are used as encoders for questions and passage. Special `[CLS]` token of BERT is used to take the representation at the token as the output. The whole bi-encoder model is trained by optimizing the loss function as the negative log likelihood of the positive passage. Dataset of questions and answers pairs is used for training. Usually negative examples are not available in a dataset and need to be sampled. As negatives DPR uses passages from the same mini-batch and one passage returned by BM25 which don’t contain the answer but match most question tokens . This strategy of mining negative examples can make the computation efficient while achieving great retrieval performance.</p><p>DPR was evaluated on a range of open-domain question answering datasets (Natural Questions, TriviaQA, WebQuestions, CuratedTREC) as a first stage retriever and outperformed a strong BM25-based IR-systems greatly by 9%-19% absolute in terms of top-20 passage retrieval accuracy.</p><p>Another model has been introduced in research paper dedicated to comparison of different methods of sparse, dense an hybrid retrieval. It’s called <strong>ME-BERT</strong>. It is a model in which every document is represented by exactly <strong><em>m</em></strong> vectors. Authors used <strong><em>m </em></strong>= 8 as a good compromise between cost and accuracy and found values of 3 to 4 for <strong><em>m</em></strong> more accurate on some datasets. A model with <strong><em>k</em></strong>-dimensional embeddings is referred to as <strong>ME-BERT-<em>k </em></strong>[<a href="#7fc7">3</a>]. In addition to using BERT `[CLS]` token as vector representation authors have implemented feed-forward layer with dimension 768 × k to obtain <strong><em>k</em></strong> vectors from single encoder. This model uses max over dot products of each document vector with query vector to get relevancy score for query-document pair.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*svemur5HS-WywjJLJh45Zg.png" /><figcaption>ME-BERT architecture. Image by author</figcaption></figure><p>Model is initialized from BERT-base, and all parameters are fine-tuned using a cross-entropy loss with 7 sampled negatives from a precomputed<br>200-document list and additional in-batch negatives (with a total number of 1024 candidates in a batch); the pre-computed candidates include 100<br>top neighbors from BM25 and 100 random samples. ME-BERT outperformed regular dual-encoders and sparse retrieval models in MS-MARCO-Document and MS-MARCO-Passage datasets.</p><p>Dense retrieval models are hard to train. It requires different training tricks like hard negative mining, etc. Later works in this field are dedicated to research of different enhancements in training process of such models.</p><p>For example, <strong>Approximate nearest neighbor Negative Contrastive Learning</strong> (<strong>ANCE</strong>) approach is a learning mechanism that selects hard training negatives globally from the entire corpus, using an asynchronously updated ANN index [<a href="#a504">4</a>].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TIHybFf3oZ4cil3g4-g1ag.png" /><figcaption>Approximate nearest neighbor Negative Contrastive Learning Process. Image from [<a href="#a504">4</a>]</figcaption></figure><p>ANCE consists of two components: Trainer and Inferencer. Both componets are used during model training process. The <strong>Trainer </strong>uses negatives from the ANN index to train a dense retrieval model. The <strong>Inferencer</strong> uses a recent checkpoint to update the representation of documents in the corpus then refreshes the ANN index with most up-to-date embeddings of documents. See schema from original paper above.</p><p>ANCE can be used to train any dense retrieval model. In original paper authors have used ANCE to train BERT-based bi-encoder with dot product<br>similarity scoring and negative log likelihood loss. The results have been evaluated on MSMARCO-Passage, TREC 2019 DL Passage and TREC 2019 DL Document. This approach outperformed all sparse retrieval methods at that moment and DPR in passage retrieval.</p><p>As it has already been mentioned, dense retrieval models are usually used at the first step of pipeline of information retrieval systems. The reason is that these models are designed to implement retrieval from a huge search-index at scale. This factor adds significant limitations to the model retrieval performance. There is different class of models that are trained to predict relevancy score for request-candidate (question-document) pair rather than to obtain good vector representations. These models allow to achieve much better retrieval perfomance in terms of MRR and NDCG. We will look at these models in the next part of this series of articles. But important point now is that this kind of models can be used to improve dense retrieval models using techniques called pseudo-labeling and knowledge distillation.</p><p><strong>Pseudo-labeling</strong> is a techniques of using one model to predict labels for unlabelled data and then adding these new data to the main labeled dataset for training of another model. <strong>Knowledge distillation</strong> is the process of transferring knowledge from a large model to a smaller one via training smaller model to match the logit to the logit of higher-performing large model. Later works in dense retrieval leverage these two techniques.</p><p>Approach called <strong>RocketQA</strong> have introduced three major technical improvements for dense retrieval model training: cross-batch negatives, denoised hard negatives and data augmentation [<a href="#67fa">5</a>]. <strong>Cross-batch negatives</strong> technique is used during training of model on multiple GPUs. First, passage embeddings are computed within each single GPU, and then these passage embeddings are shared among all the GPUs. Besides the in-batch negatives, dense vectors of all passages are collected from all other GPUs as the additional negatives for each question. See picture from original paper below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kfJFeeF_lpCSZ3e0yNIVMQ.png" /><figcaption>Cross-batch negatives<strong> </strong>technique. Image from [<a href="#67fa">5</a>]</figcaption></figure><p><strong>Denoised hard negatives </strong>technique allows to filter false negatives from sampled hard negatives. <strong>Data augmentation</strong> is a technique used to increase the amount of training data by adding newly created synthetic data from existing data. RocketQA implements both denoised hard negatives and data augmentation using pseudo-labeling. It uses more powerful large model (trained to predict relevancy score for a pair of question and passage) to label new unlabeled data and then uses them for augmentation of datatset. Also higher-performing large model is used to find false negatives during hard negatives mining. RocketQA have outperformed previous approaches on MSMARCO and Natural Questions dataset.</p><p>Later approach called <strong>PAIR</strong> (<strong>PA</strong>ssage-centric s<strong>I</strong>milarity <strong>R</strong>elations) was focused on passage-centric training and introduced three major technical contributions by introducing formal formulations of the two kinds of similarity relations, generating high-quality pseudo labeled data via knowledge distillation, and designing an effective two-stage training procedure that incorporates passage-centric similarity relation constraint. The aim of learning passage-centric similarity relation is to push negative passage farther from positive passage, and making the similarity between positive passage and query larger than the similarity between positive passage and negative passage [<a href="#e8ca">6</a>]. Authors of PAIR have combined query-centric and passage-centric loss to functions to implement this method. Also, they have used well-trained large model to obtain pseudo-labeled positives and hard negatives for unlabeled queries. In the pre-training stage of two-stage training process authors have trained the dual-encoder with pseudo-labeled data from unlabeled dataset by optimizing the combined loss function. In the fine-tuning stage they have fine-tuned the dual-encoder (pre-trained in the first stage) with both labeled and pseudo-labeled data by optimizing the query-centric loss only. All these techniques have allowed to achieve slightly better MRR and recall than for RocketQA approach on MS-MARCO and Natural Questions datasets.</p><p>Another approach called <strong>Adversarial Retriever-Ranker</strong> (<strong>AR2</strong>) has leveraged <strong>adversarial training</strong> technique to improve dense retrieval. AR2 consists of a dual-encoder retriever plus a highly-performant ranker. The two models are jointly optimized according to a minimax adversarial objective: the <strong>retriever</strong> learns to retrieve negative documents to cheat the ranker, while the <strong>ranker</strong> learns to rank a collection of candidates including both the ground-truth and the retrieved ones, as well as providing progressive direct feedback to the dual-encoder retriever. Through this adversarial game, the retriever gradually produces harder negative documents to train a better ranker, whereas the cross-encoder ranker provides progressive feedback to improve retriever [<a href="#e48e">7</a>]. AR2 has been evaluated on three benchmarks: Natural Questions, MS-MARCO and TriviaQA. It have shown slightly better results (+1.3–2.1% for recall) than previous state-of-the-art results.</p><p>Another technique to improve dense retrieval is <strong>domain matched pre-training</strong>. It has been demonstrated that pre-training large bi-encoder models on on a set of 65 million synthetically generated questions, and 200 million post-comment pairs from a preexisting dataset of Reddit conversations made available, allows to improve dense retrieval recall and MRR [<a href="#9dd3">8</a>]. It has been evaluated on a set of information retrieval and dialogue retrieval datasets: ConvAI2, Ubuntu v2, DSTC7.</p><h4>Conclusions</h4><p>In this article we have considered the concept of dense retrieval. Then we have had a look at several different approaches to improve performance of dense retrieval using deep learning: from DPR to relatively modern RocketQA, PAIR and AR2. You can see that most recent methods achieve better results on public information retrieval datasets like MS-MARCO or Natural Questions. But it doesn’t mean that the most recent approach should be used in building each production IR-system. Several factors should be taken into consideration: domain, availability of labeled data, amount of unlabeled data, quality of labeled data, type of retrieval task. For example, if your labeled data have a lot of unlabeled positives than your dense retriever training process will benefit from denoised hard negatives. If a lot of data from your problem domain are available and you have enough resources to train large highly-performant ranker then your dense retrieval model can benefit from pseudo-labeling technique. Also there is no obvious choice between dense and sparse retrieval methods. For example, in case your IR-system receive a lot of queries with relatively unique ids, names and abbreviations that target a small subsets of candidates from the index, then sparse retrieval can show better performance.</p><p>As it has already been mentioned, dense retrieval and sparse retrieval methods are usually used at the first step of pipeline of information retrieval systems. In the <a href="https://medium.com/@aikho/deep-learning-in-information-retrieval-part-iii-ranking-da511f2dc325">next part</a> of this series of articles we will look at more powerful models that are used at ranking stage of IR-pipelines.</p><h4>References</h4><ol><li><a href="http://www.mmds.org/">Rajaraman A, Ullman J, Mining of Massive Datasets, Ch. 3, Cambridge University Press, 2010.</a></li><li><a href="https://arxiv.org/abs/2004.04906">Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih, Dense Passage Retrieval for Open-Domain Question Answering, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16–20, 2020, pages 6769–6781, arXiv:2004.04906</a>.</li><li><a href="https://arxiv.org/abs/2005.00181">Yi Luan, Jacob Eisenstein, Kristina Toutanova, Michael Collins, Sparse, Dense, and Attentional Representations for Text Retrieval, CoRR, 2020, arXiv:2005.00181</a>.</li><li><a href="https://arxiv.org/abs/2007.00808">Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang,<br>Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold<br>Overwijk. 2020. Approximate nearest neighbor negative contrastive learning for dense text retrieval, CoRR, 2020, arXiv:2007.00808</a>.</li><li><a href="https://arxiv.org/abs/2010.08191">Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835–5847, Online, 2021, arXiv:2010.08191</a>.</li><li><a href="https://arxiv.org/abs/2108.06027">Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen, Pair: Leveraging passage-centric similarity relation for improving dense passage retrieval, 2021, arXiv preprint, arXiv:2108.06027.</a></li><li><a href="https://arxiv.org/abs/2110.03611">Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen, Adversarial retriever-ranker for dense text retrieval, 2021, arXiv preprint, arXiv:2110.03611</a>.</li><li><a href="https://arxiv.org/abs/2107.13602">Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Scott Yih, Sonal Gupta, and Yashar Mehdad, Domain-matched Pre-training Tasks for Dense Retrieval. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1524–1534, Seattle, United States. Association for Computational Linguistics, arXiv:2107.13602</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f9fecb47de9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Learning in Information Retrieval. Part I: Introduction and Sparse Retrieval]]></title>
            <link>https://itnext.io/deep-learning-in-information-retrieval-part-i-introduction-and-sparse-retrieval-12de0423a0b9?source=rss-eeeb899dfa18------2</link>
            <guid isPermaLink="false">https://medium.com/p/12de0423a0b9</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[information-retrieval]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Andrei Khobnia]]></dc:creator>
            <pubDate>Wed, 14 Dec 2022 04:09:44 GMT</pubDate>
            <atom:updated>2022-12-16T22:55:01.870Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rWbWwehwBwHQ1qlr0_OoNw.png" /></figure><p><strong>Information Retrieval (IR) </strong>is an important field of research in computer science. It is not only about building big <strong>search </strong>engines like Google or Bing. We can face IR systems when using an online store search, chatbot, or receiving an automatic email with recommended online courses. IR is a hot field in the industry. Big online marketplaces, streaming services, social media, job search services, and many other companies build their own information retrieval systems. The relevance of search or recommendation results has a crucial impact on business performance in these fields. Improving search quality by a few percent can lead to an increase in profits by millions of dollars. Some companies even create their own R&amp;D departments dedicated to research problems in information retrieval. A lot of problems are still unsolved in modern information retrieval. Researchers work to define and solve those kinds of problems. For example, <a href="https://www.kaggle.com/c/trec-covid-information-retrieval">TREC-COVID Information Retrieval Challenge</a> was held in 2020. Participants compete in building a pandemic document search system that helps to identify answers to some of the important questions for biomedical researchers, clinicians and policymakers. In 2022 more general <a href="https://fair-trec.github.io/index">TREC 2022 Fair Ranking Track</a>, along with other challenges, was held. This challenge was focused on fairly prioritizing Wikimedia articles for editing to provide fair exposure to articles from different groups.<br>One of the promising areas for building information retrieval systems is medical research. The vast majority of clinical trials fail to meet their patient recruitment goal. Efficient patient trial recruitment is a major barrier to medical research. An important solution to this problem is to build an information retrieval system that will utilize the vast amounts of patient data that is already available in the form of the electronic health record (EHR). In <a href="https://www.trec-cds.org/2022.html.">2022 Clinical Trials Track</a> participants were challenged to build an approach for retrieving clinical trials from <a href="http://clinicaltrials.gov/">ClinicalTrials.gov</a>, a required registry for clinical trials.</p><p>The field of IR has a relatively long history of research. The <a href="https://trec.nist.gov/">Text REtrieval Conference</a> (TREC) was founded in 1992. But recent advances in deep learning have changed this field dramatically. This is the first article of “Deep Learning in Information Retrieval” series. In this series of articles, we will look at how recent deep learning methods have been applied in modern information retrieval systems. We will focus mostly on textual information retrieval. Similarity search of images, sounds and other objects, and also multi-modal information retrieval are also important and noteworthy fields of research. But we will not dive deep into these topics in this series. In this article, we start with the problem definition of textual information retrieval, the general framework to solve IR, classical algorithms in IR, and ways to improve the performance of classical IR algorithms using deep learning.</p><h4><strong>Short Introduction to IR Systems</strong></h4><p>What is <strong>Information Retrieval (IR)</strong>? It is reasonable to define information retrieval as a ranking problem. We have a <strong>query</strong> (also called <strong>context</strong>) which is a formal statement of information needs. For example, the query can be a search string, a question from chat, the data of the user for whom we need to show recommendations, etc. And we have a set of <strong>candidates</strong> that are data entities available for retrieval to a particular <strong>query</strong>. For example, candidates can be paragraphs, documents, images, videos, audio files or other entities. In this series of articles, I will focus on textual data. In information retrieval, a query does not identify a unique object in the collection of candidates. Instead, several candidates may match the query with different degrees of <strong>relevancy</strong>. Some queries can have a strong match. The other can match only weak results with relatively low relevancy but still related to a particular query. This is the reason why most IR systems <strong>rank</strong> the results by some kind of relevancy score and return a list of top ranked results. Thus, IR can be defined as the problem of ranking the collection of data entities by relevancy to a particular query. This is an informal definition for better understanding of the IR problem.</p><p>There are different metrics to measure how good some IR algorithms or systems are at ranking tasks. One of the most important metrics is Recall@N and MRR. <strong>Recall</strong> is the fraction of relevant results that were retrieved. <strong>Recall@N</strong> means the fraction of relevant results that were retrieved in first <strong><em>N</em> </strong>top ranked results. The <strong>reciprocal rank</strong> of ranked results list for a query is the multiplicative inverse of the rank of the first correct result. The M<strong>ean Reciprocal Rank</strong> is the average of the reciprocal ranks of results for a dataset of queries:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*Z1QR52bBD1vTWwAKfWIDbw.png" /></figure><p>where <strong><em>Q</em></strong> is a sample of queries and <strong><em>rankᵢ</em></strong><em> </em>is a rank position of the <em>first</em> relevant candidate for the <em>i</em>-th query.</p><p>In the scientific world, different public benchmarks are available to measure the quality of IR systems. One of the largest and most popular for open domain IR is MS MARCO dataset. This dataset comprises of 1,010,916 anonymised questions sampled from Bing’s search query logs each with a human generated answer and 182,669 completely human rewritten generated answers. In addition, the dataset contains 8,841,823 passages extracted from 3,563,535 web documents retrieved by Bing that provide the information necessary for curating the natural language answers [<a href="#d52b">1</a>]. One of the tasks proposed by authors of MS MARCO is ranking a set of retrieved passages given a question. This task is a good benchmark for large scale open domain IR system. So this dataset is often used in scientific papers to benchmark new models and approaches in modern information retrieval. There is also TREC Complex Answer Retrieval (TREC CAR) dataset. It is smaller and based on ~7 million paragraphs from all Wikipedia pages [<a href="#efd3">2</a>]. TREC CAR dataset is also often used as a benchmark for modern IR approaches in scientific papers. Also, there are some other smaller and older public IR datasets, benchmarks for different variations of IR problem, datasets for different languages, etc. You can check <a href="https://github.com/allenai/ir_datasets/">ir_datasets</a> repo and Python library which provides a common interface to many IR ranking datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*lz3pwQx_7bCp7kjVRkqrjg.png" /><figcaption>Typical IR system pipeline</figcaption></figure><p>How to build an information retrieval system? A straightforward approach is to design an algorithm or a model which scores relevancy of candidate to query and to build a system that computes scores for all candidates and returns the ranked result list. However, this is impractical to rank the whole set of candidates for each query. This is the reason why most information retrieval systems usually use at least two stages pipeline (see picture above). First stage is usually referred as the <strong>candidates retrieval</strong> stage. Second and subsequent stages are referred as the <strong>ranking</strong> or <strong>re-ranking</strong> stages. At the candidates retrieval stage, some fast but not very precise algorithm is used to retrieve a reasonable subset of candidates which probably contains relevant results. Then at the ranking stages, more complex algorithms and machine learning models are used to rank retrieved subset and obtain top relevant results for end user. In subsequent sections of this article, we will dive into the candidates retrieval stage of IR pipeline and see how it works and how modern deep learning methods can be applied to improve results of this stage.</p><h4>Classic Algorithms (Sparse Retrieval)</h4><p>One of the central components of a classic information retrieval system is an <strong>inverted index</strong> data structure. <strong>Inverted index</strong> is an index storing a mapping from content, such as terms, words, numbers, etc., to its locations in a collection of data entities (documents, paragraphs, etc.) The purpose of an inverted index is to enable fast searches, at the cost of increased processing time when a new data entity is added to the system. Index design incorporates interdisciplinary concepts from linguistics, cognitive psychology, mathematics, informatics, and computer science.</p><p>One of the simplest approaches to text representation for building inverted index is <strong>Bag-of-Words (BoW)</strong> model. In BoW model, a data entity text <br>is represented as the bag (<a href="https://en.wikipedia.org/wiki/Multiset">multiset</a>) of its words (<strong>terms</strong> or <strong>tokens</strong>), disregarding grammar and word order. Let’s look at the following example. Here are two simple texts:</p><blockquote>This article is about information retrieval.</blockquote><blockquote>Information retrieval is the process of obtaining information resources that are relevant to an information need.</blockquote><p>Based on these two texts, a list of terms is constructed as follows for each text:</p><pre>&quot;this&quot;, &quot;article&quot;, &quot;is&quot;, &quot;about&quot;, &quot;information&quot;, &quot;retrieval&quot;</pre><pre>&quot;information&quot;, &quot;retrieval&quot;, &quot;is&quot;, &quot;the&quot;, &quot;process&quot;, &quot;of&quot;, &quot;obtaining&quot;, &quot;information&quot;, &quot;resources&quot;, &quot;that&quot;, &quot;are&quot;, &quot;relevant&quot;, &quot;to&quot;, &quot;an&quot;, &quot;information&quot;, &quot;need&quot;</pre><p>If we collect all tokens used in our two texts dataset to a single <strong>vocabulary</strong> and assign an order number to each token in the vocabulary then we will be able to present each bag of words as a vector using a number of occurrences of particular tokens in each bag. For example, let’s assume we define vocabulary the following way:</p><pre> 0: &quot;about&quot;,<br> 1: &quot;an&quot;, <br> 2: &quot;are&quot;, <br> 3: &quot;article&quot;, <br> 4: &quot;is&quot;,<br> 5: &quot;obtaining&quot;,<br> 6: &quot;of&quot;,<br> 7: &quot;need&quot;<br> 8: &quot;process&quot;,  <br> 9: &quot;information&quot;,<br>10: &quot;relevant&quot;,<br>11: &quot;retrieval&quot;<br>12: &quot;resources&quot;,<br>13: &quot;the&quot;, <br>14: &quot;this&quot;, <br>15: &quot;that&quot;, <br>16: &quot;to&quot;</pre><p>Then using number of occurrences of particular tokens bags for these two texts can be presented as the following two vectors:</p><pre>1: [1,0,0,1,1,0,0,0,0,1,0,1,0,0,1,0,0],<br>2: [0,1,1,0,1,1,1,1,1,3,1,1,1,1,0,1,1]</pre><p>Bag-of-Words is a simple vector space model. It does not take into account that terms can have different importance in a context of particular text. For example, common words like “a”, “the”, “this” are almost always have high frequency but carry very little quantities of information. One of the most simple ways to address this problem is using <strong>tf-idf </strong>or t<strong>erm frequency–inverse document frequency </strong>model. The tf–idf is calculated as a product of two metrics, <strong>term frequency</strong> and <strong>inverse document frequency</strong>. One data entity is referred to as a document. <strong>Term frequency</strong> is the relative frequency of term within the document and is usually calculated as the number of times that term occurs in the document divided by the total number of terms in the document. The <strong>inverse document frequency</strong> is correlated to<strong> </strong>how much information the term provides and usually calculated as logarithm of fraction of the total number of documents divided by the number of documents containing the term. Set of computed inverse document frequencies for vocabulary is usually referred as <strong>IDF table</strong>. For the two texts dataset from above IDF table computed using base 10 logarithm looks as the following:</p><pre>&quot;about&quot;: 0.30103,<br>&quot;an&quot;: 0.30103, <br>&quot;are&quot;: 0.30103, <br>&quot;article&quot;: 0.30103, <br>&quot;is&quot;: 0,<br>&quot;obtaining&quot;: 0.30103,<br>&quot;of&quot;: 0.30103,<br>&quot;need&quot;: 0.30103<br>&quot;process&quot;: 0.30103,  <br>&quot;information&quot;: 0,<br>&quot;relevant&quot;: 0.30103,<br>&quot;retrieval&quot;: 0,<br>&quot;resources&quot;: 0.30103,<br>&quot;the&quot;: 0.30103, <br>&quot;this&quot;: 0.30103, <br>&quot;that&quot;: 0.30103, <br>&quot;to&quot;: 0.30103</pre><p>Calculating term frequencies and then whole tf-idf metric for each term these two texts can be presented as the following two vectors:</p><pre>1: [0.05017,0.0    ,0.0    ,0.05017,0.0,0.0    ,0.0    ,0.0    ,0.0    ,0.0,0.0    ,0.0,0.0    ,0.0    ,0.05017,0.0    ,0.0    ],<br>2: [0.0    ,0.01881,0.01881,0.0    ,0.0,0.01881,0.01881,0.01881,0.01881,0.0,0.01881,0.0,0.01881,0.01881,0.0    ,0.01881,0.01881]</pre><p>Tf-ids anf Bag-of-Words are both simple vector space models. In real cases (with huge vocabulary of hundreds of thousands terms) they both produce vectors having a relatively small number of nonzero elements. Such vectors are called<strong> sparse vectors</strong>. And methods of information retrieval that use sparse vectors are usually referred as <strong>sparse retrieval</strong>.</p><p>One of the most known classic algorithms for sparse retrieval is Okapi BM25. <strong>Okapi</strong> <strong>BM25</strong> (BM — best match) is a family of scoring functions that ranks a set of documents based on the query terms appearing in each document. One of the popular variation of BM25 function is the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nxbO6uYgkDc1Xllhv_ZSMg.png" /></figure><p>where <strong><em>D </em></strong>is a document<em>, </em><strong><em>Q</em></strong> is a query containing terms <strong><em>q₁,…,qₙ</em></strong>, <strong><em>k₁</em></strong> and <strong><em>b </em></strong>are special constants, <strong><em>avgdl</em></strong> is the average number of terms in a document, <strong>|<em>D</em>|</strong> is number of terms in the document <strong><em>D</em></strong>, <strong><em>f(qᵢ,D)</em></strong> is the number of times that term <strong><em>qᵢ</em></strong> occurs in the document <strong><em>D</em></strong>, <strong><em>IDF(qᵢ)</em> </strong>is a variation of inverse-document frequency which is usually calculated as the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*-gg59JI4mtq68qxupue01A.png" /></figure><p>where <strong><em>n(qᵢ)</em></strong> is the number of documents containing the term <strong><em>qᵢ </em></strong>and <strong><em>N</em></strong> is the total number of documents.</p><p>In this article, we will deep into all theory behind BM25 algorithm. Just note that despite it was implemented in 1980s it is still used in a lot of information retrieval systems. Old classic algorithms for sparse retrieval have some advantages over recent machine learning based approaches:</p><ul><li>Indexing and retrieval speed. Classic algorithms are much faster than modern DL-based approaches at the indexing stage (calculating vectors) and in most cases at the retrieval stage.</li><li>Explainability. The meaning of sparse vector is obvious. We can easily check why particular entity was retrieved for particular query and what terms had the greatest impact. It is even possible to influence on results by simply deleting or updating particular terms or adding custom rules to calculate terms weights.</li></ul><p>So classic sparse retrieval approaches are still used in modern IR systems. And it is useful to know how they work. Modern deep learning methods can be used to improve even sparse retrieval. Let’s see how it works in the next section.</p><h4>Deep Learning in Sparse Retrieval</h4><p>Deep learning language models based on variations of Transformer architecture achieved state-of-the-art results in a vast majority of NLP tasks across different domains. One of the first and the most popular language model is BERT (Bidirectional Encoder Representations from Transformers) developed and published by Jacob Devlin and his colleagues from Google in 2018 [<a href="#0e9b">3</a>]. If you are not familiar with Transformer architecture or BERT then I recommend <a href="https://jalammar.github.io/illustrated-bert/">this</a> article. At this moment, a lot of more powerful variations of such models are available: RoBERTa, XLNet and others. But how to apply language models to information retrieval and particularly to sparse retrieval algorithms?</p><p>First, let’s look at an approach called <strong>DeepCT</strong>. As we see above, term frequency is a common method for identifying the importance of a term in a query or document. But it is a weak signal, especially when the frequency distribution is flat, such as in long queries or short documents where the text is of sentence/passage-length. Let’s look at the following example from the original paper:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CRvjRHpXHF7Gp9E1VW1T7Q.png" /></figure><p>Two passages that mention ‘stomach’ twice. But only the first passage is about the topic ‘stomach’. DeepCT paper proposes a <strong>Deep Contextualized Term Weighting</strong> framework that learns to map BERT’s contextualized text representations to context-aware term weights for texts. When applied to passages, DeepCT-Index produces term weights that can be stored in an ordinary inverted index for retrieval. When applied to query text, DeepCT-Query generates a weighted bag-of-words query. Both types of term weight can be used directly by typical first-stage retrieval algorithms [<a href="#df0e">4</a>].</p><p>DeepCT includes two main components: generating contextualized word embeddings through BERT, and predicting term weights through linear regression for each token. The contextualized word embedding is a feature vector that characterizes the word’s syntactic and semantic role in a given context. DeepCT linearly combines the features into a word importance score. Given the ground truth term weight for every word in text, DeepCT aims to minimize the mean square error. The DeepCT model, from BERT to the regression layer, is optimized end-to-end. The BERT component is initialized with a pretrained BERT model to reduce over-fitting. It is fine-tuned to align the contextualized word embeddings with the term-prediction task. The last regression layer is learned from scratch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sx9hCFs-1QrEvplLhjM2gg.png" /><figcaption>Predicting Term Weights Architecture</figcaption></figure><p>DeepCT is benchmarked using MSMARCO and TREC-CAR datasets. A BM25 retrieval on DeepCT-Index can be 25% more accurate than classic tf-based indexes, and are more accurate than some widely-used multi-stage retrieval systems. Variations of DeepCT approach are also called <strong>W-index retrieval</strong>.</p><p>Similar approaches can be used for long documents retrieval. For example, in work called<strong> HDCT</strong> (<strong>Context-aware Hierarchical Document Term weighting framework</strong>) a framework for document indexing and retrieval is presented [<a href="#1eee">5</a>]. HDCT first estimates the semantic importance of a term in the context of each passage. These fine-grained term weights are then aggregated into a document-level bag-of-words representation, which can be stored into a standard inverted index for efficient retrieval using classic sparse approaches like BM25. HDCT architecture from original paper:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UeQ9L5fABtgQ88cfsdUYvw.png" /><figcaption>The HDCT architecture</figcaption></figure><p>HDCT is trained to predict the importance weight got each token of a passage like in DeepCT approach. But the main difference is that authors of HDCT have proposed two weak supervision approaches for training HDCT instead of manually labeling the importance of every term in every passage. The first approach uses metadata fields of a document (such as title, keywords, and abstracts) to generate weak labels based on percentage of field instances that contain a token from document. The second approach uses dataset of queries and relevant documents to generate weak labels based on the percentage of relevant queries that mention term from document.</p><p>HDCT was evaluated on 4 document retrieval datasets: MS-MARCO Document Ranking dataset, ClueWeb09-B, ClueWeb09-C and ClueWeb12-C. It significantly improved sparse retrieval accuracy compared to typical term-frequency approach.</p><p>Improved term weighting does not help in cases when query and relevant text don’t match by terms directly. For example, if synonyms are used in query or query contains terms that are related indirectly to some text. For such cases another family of methods called <strong>document expansion</strong> are come into play. <strong>Document expansion</strong> is a way to improve information retrieval quality via expanding document by relevant terms or phrases. There are a lot of methods to do document expansion but here we consider a few deep learning based approaches.</p><p>First approach is called <strong>Doc2Query</strong>. It is a simple method that predicts which queries will be issued for a given document and then expands it with those predictions with a vanilla sequence-to-sequence neural network, trained using datasets consisting of pairs of query and relevant documents [<a href="#f459">6</a>]. The idea of the method is illustrated by the following picture from the original paper:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xjl86hN6krtnwYf1mkJQ_w.png" /><figcaption>Doc2Query Idea</figcaption></figure><p>Trained Doc2Query model is used to predict 10 queries using top-k random sampling and then generated queries are appended to each document in the index. Then BM25 algorithm is used for first stage retrieval. The next version of this methods called DocTTTTTQuery is trained the same way but uses modern transformer-based T5 model [<a href="#a696">7</a>].</p><p>Document expansion methods like Doc2Query can be used in combination with W-index retrieval. But Some modern deep learning methods do both document expansion and term weighting at the same time. Let’s look at some examples.</p><p>Framework called <strong>SparTerm</strong> directly learns sparse text representations in the full vocabulary space. SparTerm comprises an importance predictor to predict the importance for each term in the vocabulary, and a gating controller to control the term activation. These two modules cooperatively ensure the sparsity and flexibility of the final text representation, which unifies the term-weighting and expansion in the same framework [<a href="#a696">8</a>]. SparTerm architecture schema from original paper:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5mOZkUDLVPylRcTeOlGcZA.png" /><figcaption>SparTerm Architecture</figcaption></figure><p>The importance predictor outputs semantic importance of all the terms in the passage. The importance predictor is trained end-to-end by optimizing the ranking objective. But unlike of DeepCT it doesn’t directly fitted to the statistical term importance labels but views the importance as intermediate variables that can be learned by distant supervisory signal for passage ranking.</p><p>The gating controller generates a binary gating signal of which terms to activate to represent the passage. It learns to activate terms presented in the passage and some other terms related to the passage topic. This is how SparTerm does document expansion by activating additional terms that are not presented in the passage originally. Passage-query/summary parallel corpus is used in training of gating controller. Thus approach to document expansion is similar to Doc2Query. But the difference is that sparse representation of passages is learned directly and expansion and term weighting models are trained jointly. SparTerm outperforms DeepCT method on MSMARCO Passage Retrieval dataset.</p><p>More modern approach called <strong>SPLADE</strong> (<strong>SParse Lexical AnD Expansion</strong> model) adds slight, but essential changes to the SparTerm model that dramatically improve its performance [<a href="#e29b">9</a>]. First, it introduces a log-saturation effect to term weights which prevents some terms to dominate and naturally ensures sparsity in representations. Second, it introduces loss which uses hard negatives in the batch sampled by BM25. The in-batch negatives sampling strategy is widely used for training image retrieval models, and has shown to be effective in learning first-stage rankers. Third, it applyes some regularization techniques to the loss to to obtain a well-balanced index. These changes allows significantly improve SparTerm approach. In the next version of this method called <strong>SPLADEv2</strong> several significant improvements in terms of effectiveness and efficiency are introduced. More specifically, the pooling mechanism of the model is modified and models trained with distillation are introduced [<a href="#8367">10</a>].</p><p>There are also other deep learning approaches to sparse retrieval [<a href="#7a37">11</a>]. The general idea of applying deep learning to sparse retrieval is searching for a way to obtain sparse text representations that will work better for classic sparse retrieval algorithms like BM25.</p><h4>Conclusions</h4><p>In this part, we have considered basic concepts of information retrieval systems: inverted index, bag-of-words, TF-IDF, MRR and NDCG metrics, sparse retrieval and BM25 algorithm. Then we have a look at different approaches to improve performance of sparse retrieval using deep learning: W-index retrieval, document expansion models and hybrid approaches like SparTerm or SPLADE. Sparse retrieval methods have some advantages like explainability and existence of efficient implementations but other approaches can outperform sparse retrieval methods in some cases. In the next part we will look at alternative methods for first stage retrieval that can be used in IR systems.</p><h4><strong>References</strong></h4><ol><li><a href="https://arxiv.org/abs/1611.09268">Tri Nguyen et al., MS MARCO: A Human Generated MAchine Reading COmprehension Dataset, 2016, arXiv:1611.09268v3</a></li><li><a href="https://arxiv.org/abs/1705.04803">Federico Nanni, Bhaskar Mitra, Matt Magnusson, Laura Dietz, Benchmark for Complex Answer Retrieval, 2017, arXiv:1705.04803</a></li><li><a href="https://arxiv.org/abs/1810.04805">Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018, arXiv:1810.04805</a></li><li><a href="http://arxiv.org/abs/1910.10687">Zhuyun Dai and Jamie Callan, Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval, 2019, arXiv:1910.10687v2</a></li><li><a href="https://dl.acm.org/doi/10.1145/3366423.3380258">Context-Aware Document Term Weighting for Ad-Hoc Search, WWW ’20: Proceedings of The Web Conference 2020</a>.</li><li><a href="https://arxiv.org/abs/1904.08375">Rodrigo Nogueira et al., Document Expansion by Query Prediction, 2019, arXiv:1904.08375</a></li><li><a href="https://www.researchgate.net/publication/360890853_From_doc2query_to_docTTTTTquery">Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. From doc2query to doctttttquery, 2019</a></li><li><a href="https://arxiv.org/abs/2010.00768">Yang Bai et al., SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval, 2020, arXiv:2010.00768</a></li><li><a href="https://doi.org/10.1145/3404835.3463098">Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant, SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking, In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’21). Association for Computing Machinery, New York, NY, USA, 2288–2292, 2021, doi: 10.1145/3404835.3463098</a></li><li><a href="https://arxiv.org/abs/2109.10086">Thibault Formal, Carlos Lassance, Benjamin Piwowarski, Stéphane Clinchant, SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval, 2021, arXiv:2109.10086</a></li><li><a href="https://dl.acm.org/doi/10.1145/3511808.3557456">SpaDE: Improving Sparse Representations using a Dual Document Encoder for First-stage Retrieval, CIKM ’22: Proceedings of the 31st ACM International Conference on Information &amp; Knowledge Management, 2022, doi:10.1145/3511808.3557456</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12de0423a0b9" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/deep-learning-in-information-retrieval-part-i-introduction-and-sparse-retrieval-12de0423a0b9">Deep Learning in Information Retrieval. Part I: Introduction and Sparse Retrieval</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>