<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ritik Raturi on Medium]]></title>
        <description><![CDATA[Stories by Ritik Raturi on Medium]]></description>
        <link>https://medium.com/@ritikraturi?source=rss-026b4a2fc71e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MT4_yVVt9_wuacBpRFx7bA.jpeg</url>
            <title>Stories by Ritik Raturi on Medium</title>
            <link>https://medium.com/@ritikraturi?source=rss-026b4a2fc71e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 23:40:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ritikraturi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Evolution of Language Representation Techniques: A Journey from BoW to GPT]]></title>
            <link>https://medium.com/@ritikraturi/evolution-of-language-representation-techniques-a-journey-from-bow-to-gpt-dd0942c4310c?source=rss-026b4a2fc71e------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd0942c4310c</guid>
            <dc:creator><![CDATA[Ritik Raturi]]></dc:creator>
            <pubDate>Wed, 27 Nov 2024 17:09:20 GMT</pubDate>
            <atom:updated>2024-11-27T17:09:20.423Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZqpP99ZO92-VWlp-i9AQ-w.png" /><figcaption>credits: <a href="https://towardsdatascience.com/evolution-of-natural-language-processing-8e4532211cfe">https://towardsdatascience.com/evolution-of-natural-language-processing-8e4532211cfe</a></figcaption></figure><h3>Introduction</h3><p>Natural Language Processing (NLP) is an interdisciplinary field at the intersection of computer science, artificial intelligence, and linguistics.</p><blockquote>NLP is a study of processing, analyzing, understanding, generating text data.</blockquote><h4>Range of tasks in NLP:</h4><ol><li>Text Classification</li><li>Information Extraction (Named Entity Recognition &amp; Part of Speech Tagging)</li><li>Information Retrieval</li><li>Text Summarization</li><li>Machine Translation</li><li>Question &amp; Answering</li></ol><p>Text Classification, Information Extraction, Information Retrieval are being performed by <strong>Classical Machine Learning Algorithms</strong> from a long time.</p><p>Text Summarization, Machine Translation, Question &amp; Answering NLP tasks specifically employs <strong>Generative AI</strong>.</p><blockquote>Why is NLP hard?</blockquote><p>Because firstly, human language involves <strong>ambiguity</strong> (uncertainty of meaning)</p><p>For e.g. “The chicken is ready to eat”</p><p>this sentence is ambiguous because it’s not clear what this sentence means. It can have multiple meanings like:</p><ol><li>chicken can be eaten now as it is ready (chicken is now cooked perhaps)</li><li>chicken itself is ready to eat something else (maybe chicken was sleeping earlier, and now is woken up for eating)</li></ol><p>Apart from ambiguity, human language also involves metaphors, sarcasm, phrases, etc. all these are <strong>complex representations</strong> in human languages.</p><p>This article will now delve into language representation, embedding, some simple techniques like Bag of Words (BoW), TF-IDF, etc. and complex models like BERT &amp; GPT.</p><h3>What is a Language Representation and Vectorization Technique?</h3><p>In NLP, language representation and vectorization involve converting words, sentences, or documents into numerical vectors that machine learning models can comprehend. This ‘vectorization’ captures the semantic properties of the words, preserving the linguistic context in a mathematical format that can be processed by AI models.</p><p>Vectorization provides a bridge between the qualitative world of language and the quantitative world of machines.</p><h3>Different Types of Language Representations</h3><h4>1. Basic Vectorization Techniques</h4><p>Some Basic Vectorization Techniques are:</p><ul><li>Bag of Words</li><li>TF-IDF</li></ul><h4>1. a) Bag of Words</h4><p>This technique creates a vector representation based on the frequency of words in a sentence.</p><p>E.g. Let’s suppose there are following 3 sentences and using BoW, 3 vectors are created from them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X9TSAolfQxKsl9UOhkOrIQ.png" /></figure><p>Now, clearly each unique word in the corpus (set of all words) is assigned a position in each vector (representing a single sentence), and the value is word count in that sentence.</p><p>So, we achieved the objective of having a numerical representation for each text(sentence) but, there are some problems:</p><ul><li>BoW ignores word order (both vector 2 and vector 3 are same, even though sentence 2 &amp; sentence 3 are different).</li><li>High dimensionality for large vocabularies (as the number of sentences grow, corpus {number of unique words} grows too, which will increase the number of dimensions).</li><li>Semantic/meaning of word is lost (we obtained a numerical representation here, but it only captures frequency of words).</li><li>Out of vocabulary(corpus) words can’t be handled</li></ul><blockquote>To overcome problem of ignoring word order, Bag of n-grams is used.</blockquote><h4>Bag of n-grams</h4><p>n can have different values, for below e.g. we have taken n = 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TOJAT78KAnmNFxtEI_AFyg.png" /></figure><p>From above, we can see that the problem of word order (sequence information) is resolved as Sentence 2 &amp; 3’s numerical representations (i.e. Vector 2 &amp; Vector 3) are different now.</p><p>But the problem of High Dimensionality has worsened now. Why? because corpus (unique number of bigrams) in Bag of bigram(2-grams) method is greater than the corpus of words in BoW method (If we keep increasing n, the corpus increases).</p><blockquote>Also, note that we will get sparse matrix (most entries zero) as the number of sentences increases.</blockquote><p>One possible thing to reduce the huge number of dimensions is by removing Stop words and converting words to their root form.</p><h4>1. b) TF-IDF (Term Frequency-Inverse Document Frequency):</h4><p>TF-IDF improves on BoW by weighing words based on their frequency in a document and their rarity across all documents. High TF-IDF values indicate words that are important to a particular document but not commonly found in other documents. It does better differentiation of words’ importance than BoW and, effective for document similarity and information retrieval, but still lacks deep understanding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cmYyDjuRX6aeZErmpxs2oA.png" /><figcaption>credits: <a href="https://ted-mei.medium.com/demystify-tf-idf-in-indexing-and-ranking-5c3ae88c3fa0">https://ted-mei.medium.com/demystify-tf-idf-in-indexing-and-ranking-5c3ae88c3fa0</a></figcaption></figure><p>Some problems associated with TF-IDF:</p><ul><li>Ignores words order</li><li>High dimensionality for large vocabularies</li><li>Lacks Semantic/word understanding</li></ul><p>TF-IDF captures word importance whereas, BoW captures word frequency.</p><h4>2. Context Free Embedding</h4><p>Some Context Free Embeddings are:</p><ul><li>Word2Vec</li><li>GloVe</li><li>FastText</li></ul><h4>2. a) Word2Vec</h4><p>Word2Vec encodes words into a dense vector space, effectively capturing semantic relationships by positioning similar words in proximity to one another. It employs techniques such as Skip-gram and Continuous Bag of Words (CBOW) to facilitate the learning of these relationships. In the Skip-gram model, the primary word is used to predict its surrounding context words, while in the CBOW model, the context words are utilized to predict the target word at the center.</p><p>Word2Vec operates under unsupervised learning principles to generate word vectors. Shallow ANN algorithm is used by word2vec model to capture word semantics.</p><p>Word2vec, GloVe, FastText models don’t capture context.</p><h4>2. b) GloVe <strong>(Global Vectors for Word Representation)</strong></h4><p>GloVe (Global Vectors for Word Representation), leverages co-occurrence matrices to extract global statistical information from text, generating embeddings that emphasize semantic similarity among words. This unsupervised learning algorithm constructs word representations based on aggregated word-word co-occurrence statistics from a corpus, allowing it to capture meaningful relationships in a vector space. Its particularly effective for tasks like text classification and machine translation, enhancing the performance of various natural language processing applications.</p><h4>2. c) FastText</h4><p>FastText unlike traditional models (like Word2Vec) operates at the subword level by breaking words into subword units (character n-grams), allowing it to capture morphological variations and handle out-of-vocabulary words effectively. This feature is particularly beneficial for languages with rich morphology and rare words. It also employs techniques such as Hierarchical Softmax and Negative Sampling to optimize training speed and efficiency. Its versatility makes it suitable for various NLP tasks, including text classification and sentiment analysis.</p><h4>3. Context Based Embedding</h4><p>Some Context Based Embeddings are:</p><ul><li>ELMo</li><li>BERT</li><li>GPT</li></ul><p>Contextual embeddings, exemplified by models like BERT and ELMo, generates word vectors that account for surrounding words in a sentence. This context-dependent approach allows the same word to have different embeddings based on its usage, effectively capturing polysemy — where a single word can have multiple meanings depending on context. These embeddings are produced using transformer-based or deep learning architectures.</p><p><strong>ELMo</strong> generates context-dependent embeddings by capturing word meaning based on surrounding words in a sentence. This model is particularly valuable for understanding words with multiple meanings.</p><p>3. b) Whereas <strong>BERT</strong> generates context-dependent embeddings using transformer &amp; captures context bidirectionally, understanding each word in relation to all other words in a sentence. This makes it highly effective for complex NLP tasks requiring deep understanding of context.</p><p><strong>GPT</strong>, on the other hand, employs a unidirectional (left-to-right) transformer architecture. It generates text by predicting the next word in a sequence based solely on preceding words, which can limit its understanding of context compared to BERT.</p><p>Some Problems associated with them are:</p><ul><li>Computationally intensive and require substantial resources for training</li><li>Can be complex to interpret and deploy in real-time applications.</li><li>Training these models from scratch is difficult; they often rely on pre-trained versions to leverage their contextual understanding effectively.</li></ul><h3>What is an Embedding?</h3><p>An embedding is a dense, low-dimensional vector representation of text, such as a word, phrase, or entire document, where each element in the vector captures some meaning or context about the text. Embeddings capture the semantic meaning of words by placing similar words close to each other in the vector space.</p><h3>Difference between Word2Vec, BERT and GPT approaches</h3><p>BERT and GPT both utilize Transformers unlike Word2Vec.</p><h4>Transformers</h4><p>Transformers are sequence models that abandon the sequential structure of RNNs and LSTMs and adopt a fully attention-based approach.</p><p>The key differentiating feature of transformers from previous NLP models is the attention mechanism. This allows for:</p><ul><li>Parallelization, which means faster training and optimized performance.</li><li>Much larger contexts than recurrence, meaning transformers can craft more coherent, relevant, and complex outputs.</li></ul><p>Transformer has 2 parts:</p><ul><li>Encoder</li><li>Decoder</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yPBrQVKvhV_GjY67v1rwAw.png" /><figcaption>credits: <a href="https://www.comet.com/site/blog/explainable-ai-for-transformers/">https://www.comet.com/site/blog/explainable-ai-for-transformers/</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KgocM0fySJeDGfnU248eDQ.png" /></figure><h4>Word2Vec</h4><p><strong>Architecture :</strong> Employs basic architectures like Continuous Bag of Words (CBOW) and Skip-Gram models to represent words in a vector space.</p><p><strong>Methodology:</strong> Operates at the word level, focusing on individual word relationships without considering the sequence of words, which can lead to a limited understanding of context.</p><p><strong>Objective:</strong> The primary aim is to capture semantic similarity by understanding relationships between words based on their co-occurrence in a given corpus.</p><p><strong>Limitation:</strong> Its inability to capture the full context of sentences restricts its effectiveness in understanding nuanced meanings that depend on surrounding words.</p><h4>BERT (Bidirectional Encoder Representations from Transformers)</h4><p><strong>Architecture:</strong> Its built on a bidirectional Transformer architecture, which allows it to process text by considering both left and right contexts simultaneously. This architecture is designed for deep learning tasks and includes components such as tokenizers, embedding layers, and multiple transformer blocks.</p><p><strong>Methodology:</strong> Unlike traditional models that read text sequentially, BERT processes entire sequences at once. It uses a technique called Masked Language Modeling (MLM) to predict masked words in a sentence, thereby learning contextual relationships more effectively.</p><p><strong>Objective:</strong> It aims to understand the full context of language, making it particularly effective for tasks like question answering, sentiment analysis, and natural language inference. Its design allows it to achieve state-of-the-art results across various NLP benchmarks.</p><p><strong>Limitation:</strong> While BERT provides significant improvements in accuracy due to its bidirectional processing, it requires substantial computational resources for training and inference due to its complexity and size.</p><h4>GPT (Generative Pre-trained Transformer)</h4><p><strong>Architecture:</strong> GPT utilizes an autoregressive Transformer architecture that processes text in a unidirectional (left-to-right) manner. This approach focuses on generating text by predicting the next word based on previous words in the sequence.</p><p><strong>Methodology:</strong> GPT generates coherent text by leveraging its training on vast datasets. It predicts subsequent words in a sequence, which allows it to create fluid narratives or responses based on context established by preceding text.</p><p><strong>Objective:</strong> The primary goal of GPT is text generation, making it highly effective for applications like conversational agents, content creation, and any task requiring coherent language output.</p><p><strong>Limitation:</strong> One notable limitation of GPT is its tendency to produce unexpected or irrelevant outputs when it over-relies on preceding text without deeper contextual understanding. This can lead to inconsistencies or inaccuracies in generated content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Df1oqDI068GY6sfM2BRujA.png" /></figure><p>In above table,</p><ul><li>Dimensionality refers to the number of dimensions in which a word or token is represented as a vector.</li><li>Out of Vocabulary refers to whether the model can handle out of vocabulary words.</li></ul><h3>Conclusion</h3><p>The evolution of language representation techniques has significantly transformed the landscape of Natural Language Processing (NLP). From traditional methods like Bag of Words (BoW) and TF-IDF, which primarily focus on word frequency and importance, to advanced models such as Word2Vec, GloVe, and FastText that capture semantic relationships, the journey reflects a continuous quest for deeper understanding of language. The introduction of context-based embeddings, particularly through transformer architectures like BERT and GPT, marks a pivotal shift by enabling models to consider the surrounding context of words, thus addressing challenges like polysemy and ambiguity inherent in human language. Despite their advancements, these models face challenges including high computational demands and complexity in deployment. As NLP continues to evolve, ongoing research will be crucial in overcoming these hurdles while enhancing the capabilities of language representation techniques to better understand and generate human language.</p><p><em>This article is a part of my internship work at </em><a href="https://medium.com/u/e1a50e167688?source=post_page---user_mention--8dea2617412a--------------------------------"><em>Innomatics Research Labs</em></a><em>.</em></p><h3>Citations:</h3><ul><li><a href="https://www.scribbledata.io/blog/word-vectorization-101-the-journey-from-text-to-numbers/#Word_vectorization_in_LLMs">Word Vectorization 101: The Journey from Text to Numbers</a></li><li><a href="https://towardsdatascience.com/nlp-101-word2vec-skip-gram-and-cbow-93512ee24314">NLP 101: Word2Vec — Skip-gram and CBOW</a></li></ul><p><a href="https://www.comet.com/site/blog/explainable-ai-for-transformers/">https://www.comet.com/site/blog/explainable-ai-for-transformers/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd0942c4310c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hacking the System Design: How Search Engines Understand and Deliver Results]]></title>
            <link>https://medium.com/@ritikraturi/hacking-the-system-design-how-search-engines-understand-and-deliver-results-8dea2617412a?source=rss-026b4a2fc71e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8dea2617412a</guid>
            <dc:creator><![CDATA[Ritik Raturi]]></dc:creator>
            <pubDate>Fri, 25 Oct 2024 06:32:10 GMT</pubDate>
            <atom:updated>2024-10-25T07:19:30.835Z</atom:updated>
            <content:encoded><![CDATA[<p>The evolution of search engines has transformed information access. Understanding their functionality is crucial for developers, businesses, and users. This article explores the core components of search engines, how relevance is assessed using distance metrics, and the scalability challenges tackled by query rewriting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E5HegF1cvfEbxBSYo0Gseg.png" /></figure><h3>What Powers a Search Engine? Exploring the Core and Types</h3><p>Every search engine relies on a complex architecture designed to efficiently index, retrieve, and rank vast amounts of data.</p><p>The primary components include:</p><ol><li><strong><em>Crawlers:</em></strong> Automated bots that traverse the web to discover and index new content.</li><li><strong><em>Indexers:</em></strong> Systems that organize crawled data into a searchable format, often using an inverted index that maps keywords to document locations.</li><li><strong><em>Query Processors:</em></strong> These interpret user queries, retrieve relevant results from the index, and rank them using various algorithms.</li></ol><p>Search engines can be categorized into:</p><ol><li><strong><em>Traditional Search Engines:</em></strong> Such as Google and Bing, which index web pages based on relevance.</li><li><strong><em>Vertical Search Engines: </em></strong>Focused on specific sectors (e.g., job search engines like Indeed).</li><li><strong><em>Meta Search Engines:</em></strong> Aggregate results from multiple search engines for a comprehensive view.</li></ol><p>Each type utilizes core technologies like natural language processing (NLP) to better understand user intent and deliver accurate results.</p><h4>Web Crawling and Indexing</h4><p>Before a search query is submitted by user, an indexed database is already in place. Various types of indexed databases are utilized by different search engines to store information about web pages.</p><ol><li><strong><em>Relational Databases:</em></strong> Use structured data stored in tables with predefined schemas (e.g., MySQL, PostgreSQL). They are often used for complex queries in applications like e-commerce.</li><li><strong><em>NoSQL Databases:</em></strong> Designed for flexibility and scalability, handling unstructured or semi-structured data (e.g., MongoDB, Cassandra). Common in search engines indexing diverse data types quickly.</li><li><strong><em>Inverted Indexes:</em></strong> A specific indexing structure mapping keywords to their document locations, allowing for fast full-text searches (e.g., Google, Elasticsearch).</li><li><strong><em>Document Stores:</em></strong> Store data in document formats (like JSON) for flexible data structures (e.g., Elasticsearch, CouchDB). Ideal for quick retrieval based on various attributes.</li><li><strong><em>Key-Value Stores:</em></strong> Data stored as key-value pairs for fast access (e.g., Redis, DynamoDB). Useful for caching and session management.</li><li><strong><em>Graph Databases:</em></strong> Use graph structures to represent complex relationships (e.g., Neo4j, Amazon Neptune). Effective for applications like social networks.</li><li><strong><em>IndexedDB:</em></strong> A low-level API for client-side storage of structured data in web browsers, allowing persistent storage without server connection.</li><li><strong><em>Vector Databases:</em></strong> They allow search engines to understand user intent better by matching queries with semantically similar content, rather than relying solely on exact keyword matches. This capability is vital for applications like recommendation systems &amp; chatbots, enhancing user experience and accuracy in information retrieval. (e.g., ChromaDB)</li></ol><blockquote>The indexed database holds details about webpages, including URLs and metadata, which are updated by web crawlers.</blockquote><h4>Web Crawling</h4><p>A continuous operation where web crawlers systematically browse the internet to discover new content. Triggered by seed URLs, sitemaps, content updates, robots.txt directives, user behavior (indirectly), scheduled crawling, and machine learning insights.</p><blockquote>Every website includes a robots.txt file that informs web crawlers about which URLs should be crawled and which ones should not.</blockquote><h4>Indexing</h4><p>After crawling, webpages are analyzed and stored in a structured database. This process typically includes creating an inverted index mapping keywords to document locations while storing metadata about each page.</p><h3>Cracking the Relevance Code: How Distance Metrics Deliver Accurate Search Results</h3><blockquote>Search engines need to provide quick and accurate results; if not, users might switch to more effective options.</blockquote><p>For certain search engines focused on <strong>geographic relevance</strong> or <strong>vector similarity </strong>and having databases like<strong> vector database</strong>, distance metrics evaluate how closely search results align with user queries by quantifying similarity or dissimilarity between data points represented as vectors in multi-dimensional space.</p><p>Commonly used distance metrics include:</p><h4>Euclidean Distance:</h4><p>Measures straight-line distance between two points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*BsU1iavKnP101QJLHu4hSw.png" /></figure><h4>Manhattan Distance:</h4><p>Measures distance by moving along grid lines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*bSI2HWhIobLKC7XmSn6Qwg.png" /></figure><h4>Cosine Similarity:</h4><p>Measures the cosine of the angle between two vectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjvkIoG6fCIfKOUd3WrgWA.png" /></figure><h4>Hamming Distance:</h4><p>measures the number of positions at which two vectors differ. It is particularly useful for binary vectors.</p><p>d = min { d(x,y) :x, y∈C, x≠y }</p><p>For example:</p><p>A = [1, 2, 5, 8, 9, 0]<br>B = [1, 3, 5, 7, 9, 0]</p><p>Thus, Hamming distance is 2 for the above example since two values are different between the given vectors.</p><h4>Minkowski Distance:</h4><p>Generalizes both Euclidean and Manhattan distances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*Rsy4M80KptDdUzah" /></figure><p>At p = 1, it becomes Manhattan distance</p><p>At p = 2, it becomes Euclidean distance</p><h4>User Input and Query Processing</h4><p>When a query is typed into the search box, several processes are initiated:</p><p><strong><em>Tokenization: </em></strong>Breaking down the query into individual terms or tokens while removing common stop words.</p><p><strong><em>Normalization: </em></strong>Normalizing tokens through processes like stemming and lemmatization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*2HtWvYpI4NhFsY0l2yLW5w.png" /></figure><p>However, these processes are limited in their ability to expand the search scope for identifying relevant webpages.</p><h3>Scaling Search Engine Architecture: The Game-Changing Role of Query Rewriting</h3><p>As user demands grow, scaling search engine architectures becomes challenging. An innovative solution is query rewriting, which enhances efficiency and relevance.</p><blockquote><strong>What is Query Rewriting?</strong></blockquote><p>Query rewriting modifies user queries to improve match accuracy with indexed documents through spelling correction, expansion, or contraction.</p><blockquote><strong>Benefits of Query Rewriting</strong></blockquote><ol><li>Improved Performance: Optimizing queries reduces processing time and resource consumption.</li><li>Enhanced User Experience: Users receive more relevant results without constant query refinement.</li><li>Scalability: Efficient query handling allows systems to manage higher request volumes without performance degradation.</li></ol><p>For e.g. “top 5 ds instutus”</p><p>spelling correction, thus becomes “top 5 ds institutes”</p><p>query expansion, thus might become “top 5 data science institutes” or “top 5 data science institutions”</p><p>query contraction, thus might become “data science institutes”</p><p>Once processed, the search engine retrieves relevant webpages from its established index using distance metrics or other techniques.</p><blockquote><strong>Query Understanding</strong></blockquote><p>A critical process employed by search engines to interpret user queries effectively. Involves several steps &amp; methodologies aimed at deciphering the intent behind a user’s search input.</p><blockquote><strong>Components of Query Understanding:</strong></blockquote><ol><li>Keywords and Phrases: These are the specific words or phrases users enter into the search engine.</li><li>Context: Understanding the context surrounding a query is essential for accurate interpretation. This includes factors like the user’s location, previous interactions, and current situation, which can significantly influence the meaning of the query.</li><li>Intent: Involves discerning what the user hopes to achieve with their query, whether it’s seeking information, making a purchase, or finding a specific service.</li></ol><h4>Ranking Algorithms</h4><p>These retrieved relevant webpages must be ranked according to relevance using sophisticated algorithms like:</p><ol><li><strong><em>TF-IDF (Term Frequency-Inverse Document Frequency)</em></strong> measures word importance in a document relative to its frequency across all documents.</li><li><strong><em>PageRank</em></strong>, developed by Google, evaluates link quality and quantity to determine page authority.</li><li><strong><em>Machine Learning Models</em></strong> refine ranking accuracy based on user interactions over time.</li></ol><p>Contextual relevance also plays a role based on user history and account information (e.g., user’s sex, age).</p><h4>Result Retrieval</h4><p>After ranking, top results are retrieved based on their scores.</p><ol><li><strong><em>Result Formatting:</em></strong> Documents are formatted for user-friendly display with titles, snippets, and URLs.</li><li><strong><em>Pagination:</em></strong> Techniques manage large result sets efficiently without overwhelming users.</li></ol><h4>Delivery and User Interaction</h4><p>Finally, results are presented:</p><ol><li><strong><em>User Interface:</em></strong> Results are displayed with options for filtering or sorting based on criteria like date or relevance.</li><li><strong><em>Feedback Loop:</em></strong> User interactions provide valuable data for refining future searches and improving algorithms.</li></ol><h4>Evaluating Effectiveness of Ranking Models</h4><p><strong>Online Metrics:</strong></p><ol><li><strong><em>Click-Through Rate (CTR) </em></strong>indicates engagement levels with displayed results.</li><li><strong><em>Impressions</em></strong> measure visibility in search results regardless of clicks.</li><li><strong><em>Average Position</em></strong> reflects a webpage’s average rank for specific keywords over time.</li><li><strong><em>Organic Traffic</em></strong> tracks visitors arriving through non-paid search results.</li><li><strong><em>Bounce Rate</em></strong> measures visitors leaving after viewing one page; high rates may indicate unmet expectations.</li><li><strong><em>Daily Active Users</em></strong> count unique users interacting within 24 hours.</li><li><strong><em>Successful Session Rate = </em></strong>100 * number of successful sessions/total number of sessions</li></ol><p>where successful sessions are those in which dwelling time &gt; 10 sec.</p><blockquote>It is possible that a session can be considered success with dwelling time &lt; 10 sec., let’s say we search “Diwali date this year”, on the top of SERP itself we will find the answer &amp; we will close the tab afterwards. In this case even though we didn’t click on any webpage we still got our result. This scenario will be considered success considering several other things such as we don’t re-search similar query and ours’s past history of searching.</blockquote><p><strong>Offline Metrics:</strong></p><ol><li><strong><em>Keyword Rankings</em></strong>, tracking performance over time informs SEO strategies.</li><li><strong><em>Search Volume</em></strong>, understanding potential traffic informs content strategy.</li><li><strong><em>Competitor Analysis</em></strong>, evaluating rankings provides insights into market positioning.</li><li><strong><em>Historical Ranking Data,</em></strong> identifying trends helps inform SEO strategies.</li></ol><h4>Continuous Improvement</h4><p>Search engines operate in an iterative cycle of improvement:</p><ol><li>Data Collection monitors user behavior to identify effective results.</li><li>Algorithm Updates ensure adaptation to changing needs and content dynamics.</li><li>A/B Testing, presenting different result pages helps determine configurations yielding better engagement metrics.</li></ol><h4>Architectural Considerations</h4><p>Modern architectures must support dynamic scaling for read/write operations by separating indexing from query handling to minimize performance bottlenecks during peak loads. Techniques such as sharding (dividing data into smaller chunks) and replication (creating multiple copies) are essential for maintaining response times while accommodating increasing data volumes.</p><blockquote>This article is a part of my internship work at <a href="https://medium.com/u/e1a50e167688">Innomatics Research Labs</a>.</blockquote><h4>Citations:</h4><ol><li><a href="https://hughewilliams.com/2012/03/19/query-rewriting-in-search-engines/">https://hughewilliams.com/2012/03/19/query-rewriting-in-search-engines/</a></li><li><a href="https://highscalability.com/scaling-indexing-and-search-algolia-new-search-architecture/">https://highscalability.com/scaling-indexing-and-search-algolia-new-search-architecture/</a></li><li><a href="https://dev.to/memphis_dev/building-a-scalable-search-architecture-3jj0">https://dev.to/memphis_dev/building-a-scalable-search-architecture-3jj0</a></li><li><a href="https://www.confluent.io/blog/building-a-scalable-search-architecture/">https://www.confluent.io/blog/building-a-scalable-search-architecture/</a></li><li><a href="https://vinted.engineering/2024/09/05/goodbye-elasticsearch-hello-vespa/">https://vinted.engineering/2024/09/05/goodbye-elasticsearch-hello-vespa/</a></li><li><a href="https://dl.acm.org/doi/10.1145/3511808.3557068">https://dl.acm.org/doi/10.1145/3511808.3557068</a></li><li><a href="https://vertabelo.com/blog/database-index-types/">https://vertabelo.com/blog/database-index-types/</a></li><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Basic_Terminology">https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Basic_Terminology</a></li><li><a href="https://web.dev/articles/indexeddb">https://web.dev/articles/indexeddb</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8dea2617412a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>