<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mrityunjay Kumar on Medium]]></title>
        <description><![CDATA[Stories by Mrityunjay Kumar on Medium]]></description>
        <link>https://medium.com/@mjay.cse?source=rss-3c8f52f2eb16------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*t4_gLuoa98m4dPoVDe7EVg.jpeg</url>
            <title>Stories by Mrityunjay Kumar on Medium</title>
            <link>https://medium.com/@mjay.cse?source=rss-3c8f52f2eb16------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:24:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mjay.cse/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Handling Large text data in production]]></title>
            <link>https://medium.com/@mjay.cse/handling-large-text-data-in-production-30c2be4c7d91?source=rss-3c8f52f2eb16------2</link>
            <guid isPermaLink="false">https://medium.com/p/30c2be4c7d91</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Mrityunjay Kumar]]></dc:creator>
            <pubDate>Tue, 09 Oct 2018 19:30:54 GMT</pubDate>
            <atom:updated>2018-10-09T19:30:54.615Z</atom:updated>
            <content:encoded><![CDATA[<p>Question:</p><p>I have a 5GB a single JSON file which consists of millions of queries (questions) with its answers.</p><p>Can anyone please tell me,</p><p>1) How to handle such kind of large datasets?</p><p>2) The objective is to find out the top 5 similar query along with the answers which are similar to the provided a new test query.</p><p>Answer:</p><p>I can suggest one option which I implemented.</p><ol><li>Pyspark for data processing If you want to make this simple, you can use pyspark to load the dataset and perform operations accordingly. Because of the fact as soon as your data size will grow, it will eat more memory and a simple collective operation will cost you a lot. If I am not wrong, you might perform NLP techniques to analyze the structure underneath the question dataset.</li><li>After processing, you can store your pre-computed results in Graph database which will, in turn, help you in faster document matching and retrieval.</li></ol><p>Hope it helps.</p><p><em>Originally published at </em><a href="https://blog-mjay.firebaseapp.com/post/my-first-post/"><em>blog-mjay.firebaseapp.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30c2be4c7d91" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>