<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Krishanu Saha on Medium]]></title>
        <description><![CDATA[Stories by Krishanu Saha on Medium]]></description>
        <link>https://medium.com/@krishanusaha_76915?source=rss-87a08126705------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*7Lc-Kjf_alxsTZJa.</url>
            <title>Stories by Krishanu Saha on Medium</title>
            <link>https://medium.com/@krishanusaha_76915?source=rss-87a08126705------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 15:47:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@krishanusaha_76915/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Quick semantic search model using open source libraries from scratch]]></title>
            <link>https://medium.com/@krishanusaha_76915/quick-semantic-search-model-using-open-source-libraries-from-scratch-a0435cfa746b?source=rss-87a08126705------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0435cfa746b</guid>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[spacy]]></category>
            <category><![CDATA[sentence-transformers]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Krishanu Saha]]></dc:creator>
            <pubDate>Wed, 10 Jan 2024 16:17:53 GMT</pubDate>
            <atom:updated>2024-01-10T16:17:53.101Z</atom:updated>
            <content:encoded><![CDATA[<p>Since the chatGPT was launched for the public in Nov ‘22, we are in the age of LLMs. LLM stands for <em>Large Language Model</em> which is an advanced Machine Learning model, usually based on the Transformer architecture, adept in the basics of language and understanding of how language works. It’s ability to achieve general-purpose language understanding and generation comes from the fact that it is usually been trained over gazillions of articles from the internet or other sources. The training of LLM is computationally intensive which is mostly an unsupervised process initially; however, later becomes supervised for human based feedback.</p><p><strong>Using SpaCy library cleaning of the text and basic NLP</strong></p><p>I have used the spacy library to manipulate the data, which had to be parsed into sentences. Since the data was in the form of paragraphs, we were not able to pass it to the model to create embeddings (as it was exceeding the token length). We could have chosen to use a splitter of certain chunk size and choose some overlap between different chunks, but we are going for a first-cut easier model.</p><p>The purpose of creating the embeddings (vector embeddings to be precise) is to be able to match it with the search string.</p><pre>import spacy<br>!python3 -m spacy download en_core_web_sm<br>nlp = spacy.load(&quot;en_core_web_sm&quot;)<br>from tqdm.auto import tqdm #To time the code execution<br><br># runs 30 seconds or so<br>sentences = []<br>for text in tqdm(df[&quot;text&quot;]):  #df is the dataframe with the &#39;text&#39; column which as all the textual data<br>    doc = nlp(text)<br>    for sentence in doc.sents:<br>        sentences.append(str(sentence).strip())<br><br>sentences</pre><p><strong>Using sentence_transformers library to create text embeddings</strong></p><p>After we have the different sentences of the text corpus, the next step would be to create the vector embeddings of the sentences. The vector embeddings is used to find out the semantic similarity between the search string and the corpus of text we have.</p><pre>!pip3 install sentence_transformers<br><br>from sentence_transformers import SentenceTransformer<br>model = SentenceTransformer(&#39;multi-qa-MiniLM-L6-cos-v1&#39;)<br><br>sembeddings = model.encode(sentences, show_progress_bar=True)<br><br><br>#Saving the embeddings in a .npy file<br>import numpy as np<br>with open(&quot;sentences.npy&quot;, &quot;wb&quot;) as f:<br>    np.save(f, sembeddings)</pre><p><strong>Finding the similarity between the search string and the data corpus</strong></p><p>The first step in this is to use the encoder to encode the search string. Next, we determine the cosine similarity between the search string and the corpus.</p><pre>def search(query, text, corpus_embeddings, model, top=5):<br>    # code query to restrict search space<br>    question_embedding = model.encode(query)<br>    <br>    # Determine similarity (vectors are normalized)<br>    sim = np.dot(corpus_embeddings, question_embedding)<br>    <br>    # Get most similar top_k by sorting<br>    hits = [ { &quot;text&quot;: text[i], &quot;score&quot;: sim[i] } <br>                     for i in sim.argsort()[::-1][0:top] ]<br>    <br>    # Return as dataframe<br>    return pd.DataFrame(hits)<br><br>search(&quot;Skin cream&quot;, sentences, sembeddings, model)</pre><p>I have detailed the steps for a quick semantic search engine, which can be deployed for searching through a database (maybe can be used by a e-commerce site in the search functionality).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0435cfa746b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>