<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Khushboo Gupta on Medium]]></title>
        <description><![CDATA[Stories by Khushboo Gupta on Medium]]></description>
        <link>https://medium.com/@guptakhushi345?source=rss-4cdeefc94c5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_kYPImpZG236vSG6gtqKRg.jpeg</url>
            <title>Stories by Khushboo Gupta on Medium</title>
            <link>https://medium.com/@guptakhushi345?source=rss-4cdeefc94c5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 19:32:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@guptakhushi345/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My two cents on SWE Internships]]></title>
            <link>https://guptakhushi345.medium.com/my-two-cents-on-swe-internships-bdc378706534?source=rss-4cdeefc94c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/bdc378706534</guid>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[internships]]></category>
            <category><![CDATA[data-structures]]></category>
            <category><![CDATA[interview-preparation]]></category>
            <dc:creator><![CDATA[Khushboo Gupta]]></dc:creator>
            <pubDate>Thu, 22 Jun 2023 20:20:03 GMT</pubDate>
            <atom:updated>2023-06-22T20:21:36.960Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>My guide to tailoring one’s resume, applying to SWE internships, and preparing for them</blockquote><p>Hello everyone! I’m Khushboo Gupta, and I recently graduated from IIT Goa as a CSE Undergraduate. I have previously interned at Google and MathWorks, and I would like to share some tips I followed and learned from my preparation.</p><p>Preparing for SWE internships and getting your dream offer is truly an exhausting process. However, the process can become a tad easier with the right guidance and hard work. During my preparation days, I skimmed over many such articles to grab any useful information I could. Since the internship season is right around the corner, this is my way of giving back to the community by curating information that helped me during my preparation!</p><p>In this article, I will mainly focus on tech companies that follow the conventional DSA-based interview process and not on remote startups, which have a much different hiring process (often based on testing your knowledge on particular tech stacks).</p><h3>Getting that Interview call</h3><p>Before applying for an internship, the first and foremost step would be to build your resume. Your resume is that <em>“one”</em> document that will speak the most about you to get shortlisted, and hence, building a strong resume is of utmost importance.</p><ol><li><strong>Building your resume</strong></li></ol><p>For an individual applying to internships or for new grad roles, a resume should ideally be not more than one page. I prefer single-column resumes more; <a href="https://www.overleaf.com/latex/templates/jakes-resume-anonymous/cstpnrbkhndn">this</a> is one of my most preferred templates.</p><blockquote>Brownie tip: Always get your resume reviewed by at least 1–2 of your mentors or college seniors since they would be able to point out any errors or anything that you could have missed!</blockquote><p>Now comes the more interesting part. What would differentiate your resume from a pool of candidates while applying? This is where <strong><em>building meaningful projects, open-source contributions, working under professors, and participating in hackathons and activities conducted by your college’s coding club</em></strong> come into play. Moreover, getting <strong><em>appreciable ranks in contests on competitive programming sites like Codeforces, Atcoder, etc.,</em></strong> will help your resume stand out. Getting good ranks in Leetcode contests also helps a lot.</p><p><strong>2. Getting a referral</strong></p><p>Before applying for off-campus internships at big tech companies, it is always better to apply with a referral. It is important to note that in the case of a referral, the person who will be referring you will most probably share a link through which you will have to apply and not directly on the portal. That’s why asking for a referral before applying to the company is important. You can get a referral from your <strong><em>college seniors, alumni, and even strangers on LinkedIn</em></strong>. Also note that interns cannot refer you in most cases, so to be safer, messaging <strong><em>Full Time Software Developers</em></strong> would be the best way to go! It is important to be respectful and crisp while asking for referrals, so I usually go about using this template as follows:</p><blockquote><em>Hello! My name is [name] and I am currently a [year] year student majoring in [degree] from [college name]. My technical interests include [2–3 of your interests]. I recently saw an opening (</em><strong>remember to mention the Job ID</strong><em>) at [company name] and I was really interested in the opportunity as it aligns well with my career goals. But since [company name] does not come to my college, I would be very grateful to you if you could refer me for this role. I am also attaching my resume for further reference (</em><strong>add your resume link</strong><em>). Thank You!</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*POO2OHkZqYIB8-Ok" /><figcaption>Photo by <a href="https://unsplash.com/pt-br/@oskaryil?utm_source=medium&amp;utm_medium=referral">Oskar Yildiz</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Preparing for the OAs and Interviews:</h3><p>Congratulations, you are shortlisted for the interview process! Although some companies directly shortlist for interviews, some companies also have online assessment rounds. Solving questions within a particular time frame matters most here, and hence, the best way to practice would be to give contests. Continuously attempt <strong><em>Div. 2 contests on Codeforces and Leetcode contests</em></strong>. Upsolve questions you couldn’t solve during the contest, especially if they are medium-hard speaking in Leetcode language and C-D level when it comes to Codeforces.</p><p>Regarding interviews, this is the time to revise your Data Structures and Algorithms concepts and be thorough with things mentioned on your resume! Internship interviews mainly focus on DSA, a little about your projects or any past work experience that you might have done, and some CS fundamentals often taught in college. Let’s go over this step by step:</p><h4>Pointers to ace DSA:</h4><p>One of the most important concepts often overlooked during our preparation is <strong><em>time and space complexity</em></strong>. Getting a question’s time and space complexity wrong during an interview does not leave the best impression on the interviewer (<em>speaking from personal experience…</em>). To avoid such situations, it is better to compute the time and space complexity of every question you are solving (brownie tip: while solving questions on Leetcode, cross-check your computed complexity from any <em>of the most voted solutions available on the discuss tab</em>). Two resources that I would highly recommend when it comes to understanding and testing your knowledge of time and space complexity are:</p><ol><li><a href="http://englishonlineclub.com/pdf/Cracking%20the%20Coding%20Interview%20-%20189%20Programming%20Questions%20and%20Solutions%20(6th%20Edition)%20%5BEnglishOnlineClub.com%5D.pdf">Cracking the Coding Interview’s Big O section</a></li><li><a href="https://www.interviewbit.com/courses/programming/time-complexity">Interviewbit’s time complexity section</a></li></ol><p>Revise the time complexity of common operations for different data structures as well. For instance, insert and delete operations of an unordered map if you are using the STL library in C++. You can refer to <a href="https://www.bigocheatsheet.com/">this</a> cheatsheet for this.</p><p>Leetcode is hand down the best site for preparing for interviews. The site has one of the best communities, which actively posts about the most recent interview questions, experiences, detailed descriptions of the rounds, etc. One life-changing tip that I had gotten from my mentor was to go through the most voted answers in the discuss tab of every question. These solutions are very well written and consist of different concepts, patterns, and even several ways to approach the same problem. These key learnings often help in solving other questions as well and help in understanding the underlying patterns across DSA questions. I have listed certain important DSA concepts (<em>which, in my opinion, are a MUST DO</em>) below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*BFS1ggIojTGtd3ik.png" /><figcaption>Important DSA topics (although not exhaustive)</figcaption></figure><p>Covering all the topics in a breadth-wise manner and then jumping depth-wise helped me tremendously. For a particular topic, start with the easy questions (about 10–12), and once you have a little confidence, proceed to the medium (where the majority of the time should be spent) and, eventually, the harder ones (about 10–12). There are already several topic-wise lists available on Leetcode that you can clone. Other than that, solving one SDE sheet fully would make you much more prepared rather than juggling between several. These are a few sheets that thoroughly helped me during my preparation:</p><ol><li><a href="https://seanprashad.com/leetcode-patterns/">175 Leetcode Patterns list</a> (great resource!)</li><li><a href="https://leadcoding.in/dsa-sheet/">Fraz 250 DSA Shee</a>t</li><li><a href="https://leetcode.com/discuss/general-discussion/460599/blind-75-leetcode-questions">Blind 75 list</a> (for revision)</li><li><a href="https://takeuforward.org/interviews/strivers-sde-sheet-top-coding-interview-problems/">Striver 30-day sheet</a> (for revision)</li></ol><p>Regarding resources for certain topics, I strongly recommend <a href="https://www.youtube.com/watch?v=nqowUJzG-iM&amp;list=PL_z_8CaSLPWekqhdCPmFohncHwz8TY2Go">Aditya Verma’s Playlist</a> [Must do!] and <a href="https://www.youtube.com/watch?v=tyB0ztf0DNY&amp;list=PLgUwDviBIf0qUlt5H_kiKYaNSqJ81PMMY&amp;index=2">Striver’s Playlist</a> for Dynamic Programming, and <a href="https://www.youtube.com/watch?v=M3_pLsDdeuU&amp;list=PLgUwDviBIf0oE3gA41TKO2H5bHpPd7fzn">Striver’s Playlist</a> for Graphs.</p><h4>Preparing for CS fundamentals:</h4><p>Along with DSA, many companies often ask for core CS concepts like <strong><em>Object Oriented Programming, Operating Systems, Database Management Systems, and Computer Networks</em></strong>. In my opinion, OOPs concepts and Operating Systems are the most important ones (although this differs from company to company, so I recommend checking out their past interview experiences and preparing accordingly).</p><p>Many companies test OOPs’ knowledge by asking candidates to design systems like <a href="https://www.geeksforgeeks.org/hospital-management-system-in-c/">Hospital Management System</a>, <a href="https://www.geeksforgeeks.org/design-parking-lot-using-object-oriented-principles/">Parking Lot</a>, etc. I recommend going through GeeksForGeeks and Leetcode discuss sections to find such questions.</p><p><a href="https://www.youtube.com/@GateSmashers">GateSmashers</a> and <a href="https://www.youtube.com/@KNOWLEDGEGATE_kg">KnowledgeGate</a> playlists are the best resources to revise core CS concepts. These are also often taught in college if you are majoring in computer science, so course notes are another resource.</p><h4>Strategy for 1.5–2 months before your interviews:</h4><p>When you have only about 1.5–2 months before your internship coding rounds and interviews, it would be great to get started with InterviewBit. Compared to Leetcode, InterviewBit has a more comprehensive and selective question bank with top-notch quality questions. It is a great way to test your strengths and weaknesses regarding different topics in DSA for the upcoming interviews when you have limited time.</p><p>During this period, you cannot miss giving Leetcode bi-weekly and weekly contests (although I would recommend starting giving them much before!). Even if you miss the actual contest, always make sure to attempt the contest virtually. The level and topics of these questions are close to what is actually asked in interviews and will help you determine which areas require more practice. Make sure to upsolve the questions as well.</p><p>Reading company-wise interview experiences on Leetcode and GeeksForGeeks just before your scheduled interviews would help you know what you can expect in the interview. I would also recommend practicing on a Google doc since sometimes companies don’t use IDEs for their interviews. Ensuring the correct indentation and writing readable code becomes extremely important in this case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xRC6p6la_tozrD5A" /><figcaption>Photo by <a href="https://unsplash.com/@wocintechchat?utm_source=medium&amp;utm_medium=referral">Christina @ wocintechchat.com</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Going the extra mile:</h3><p>The first thing you are asked in the interviews is to introduce yourself. Preparing a crisp and strong introduction beforehand helps you. It should include your <strong><em>interests in the technical field, any notable achievements, and one line about your non-technical interests or hobbies</em></strong>.</p><p>Remember to remain calm during the interview and <strong><em>let your interviewer know what you are thinking at each step</em></strong>! They are genuinely there to help you out, and leaving long amounts of blank spaces would only cause unwanted stress. So let them know your thought process, walk them through different approaches, and discuss any time and space trade-offs.</p><p>While writing code, ensure that you write modular and readable code. Use <strong><em>meaningful variable names</em></strong> and <strong><em>write comments wherever necessary</em></strong>. Break your code into functions<strong><em> </em></strong>and<strong><em> reuse these functions if possible!</em></strong> Remember to <strong><em>mention your solution’s time and space complexity</em></strong>, and <strong><em>always dry-run your solution with an example!</em></strong> Mentioning the temporary variable values in comments in the code would help you explain your solution better and will also help you catch any errors beforehand.</p><p>Also, several companies have an HR round to assess whether you’ll fit the company culture well. I recommend creating a document beforehand with points regarding answers to common HR questions like your <strong><em>strengths, weaknesses, any project where you displayed leadership qualities, handled conflict, etc</em></strong>. You can easily find a list of questions online. This makes it easier to just open the document for any upcoming interviews rather than writing new answers each time.</p><p>Understanding the company values and what they stand for will also give you an edge and help you stand out. So I recommend reviewing company values and mottos before the interview and trying to frame your answers to highlight those values.</p><h3>Finally…</h3><p>Grabbing that dream internship offer is a lot of work and requires constant dedication and perseverance. Just remember to constantly put in your hard work and believe it will be worth it one day!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bdc378706534" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[WiMLDS: Mentorship Program (Week 3)]]></title>
            <link>https://guptakhushi345.medium.com/wimlds-mentorship-program-week-3-813bd7970833?source=rss-4cdeefc94c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/813bd7970833</guid>
            <category><![CDATA[mentorship-program]]></category>
            <category><![CDATA[coding-interviews]]></category>
            <category><![CDATA[resume]]></category>
            <category><![CDATA[women-who-code]]></category>
            <dc:creator><![CDATA[Khushboo Gupta]]></dc:creator>
            <pubDate>Sun, 18 Apr 2021 12:43:10 GMT</pubDate>
            <atom:updated>2021-04-18T12:43:10.877Z</atom:updated>
            <content:encoded><![CDATA[<p>The agenda for Week 3 under the Women in Machine Learning and Data Science Mentorship Program was focused on understanding your <strong><em>career paths, interview preparation (DSA/ML), and learning how to tailor one’s resume for different job profiles</em></strong>. My mentor briefed me on how practicing <strong><em>competitive programming</em></strong> is the key to cracking coding rounds and DSA based interviews.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SebCLXyjfgtEHnCfz4t4mQ.png" /></figure><p>For the resume, she gave me tips on how it should be of <strong><em>only one page</em></strong> and include all the necessary <strong><em>keywords</em></strong> pertaining to your skills and the role that you are applying. She also advised that your resume should have at least 3–4 good projects (this helps especially while applying off-campus). For off-campus opportunities, it is always best to <strong><em>apply on the career portal with a referral</em></strong> as it at least ensures that your resume is once seen.</p><p>As I was confused about whether one needs to know about advanced data structures like Segment trees, Fenwick trees, Binary Indexed trees, etc for internship interviews, my mentor told me that unless you have tailored your resume to show that you’re a great and active competitive programmer, questions from these topics are generally not asked. For behavioral interviews, it is always better to prepare your answers to a list of common questions that are asked <strong><em>keeping the company’s vision in mind</em></strong>. Week 3 had been really informative and cleared tons of my doubts regarding interviews. I am really looking forward to the final week of the mentorship program!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=813bd7970833" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[WiMLDS: Mentorship Program (Week 2)]]></title>
            <link>https://guptakhushi345.medium.com/wimlds-mentorship-program-week-2-47fbe7e967?source=rss-4cdeefc94c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/47fbe7e967</guid>
            <category><![CDATA[mentorship]]></category>
            <category><![CDATA[wimlds]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[mentorship-program]]></category>
            <category><![CDATA[women-in-tech]]></category>
            <dc:creator><![CDATA[Khushboo Gupta]]></dc:creator>
            <pubDate>Thu, 15 Apr 2021 14:22:55 GMT</pubDate>
            <atom:updated>2021-04-15T14:22:55.810Z</atom:updated>
            <content:encoded><![CDATA[<p>The second week’s agenda under the <strong>Women in Machine Learning and Data Science Mentorship Program </strong>was<strong> </strong>regarding how to get started with open source contributions and the different available opportunities. My mentor, Arushi Chauhan, briefed me that for getting started with open source, the <strong><em>knowledge of Git</em></strong> is essential. There are several open-source projects available (some tech giants like Microsoft, Google, etc, also have some open-source projects) on Github. One must start looking for them according to the tech stack that they are interested in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SebCLXyjfgtEHnCfz4t4mQ.png" /></figure><p>The world of open-source might be a bit intimidating at the start. But there are <strong><em>beginner-friendly</em></strong> or <strong><em>first-timer</em></strong> issues for beginners to get started with. My mentor briefed me regarding the advantages of contributing to open source as well, as it helps one gain some experience of <strong><em>how to write good code</em></strong> and <strong><em>learn proper practices for documentation</em></strong> as well (which also helps while working in a company). I could get my doubts regarding Google Summer of Code cleared as well. She advised me that managing GSoC along with an internship in the third year might be quite difficult and hence, the best time to apply for it is during second year. She also told me that sometimes very active and good open-source contributors often get <strong><em>internship/full time job opportunities</em></strong> from organisations like Mozilla, etc.</p><p>Considering my interest in Machine Learning and Natural Language Processing, my mentor advised me to start <strong><em>participating in Kaggle competitions</em></strong> as well as these competitions give you a chance to clean and visualise your data from scratch and create a model according to the problem statement. She also reminded me to continue <strong><em>participating in contests on Codeforces and Leetcode</em></strong> as it is extremely vital for clearing online rounds and technical interviews. The second week was extremely helpful as I could clear several of my doubts that I had regarding open source. I am really looking forward to what Week 3 has in store for us!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47fbe7e967" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[WiMLDS: Mentorship Program (Week 1)]]></title>
            <link>https://guptakhushi345.medium.com/wimlds-mentorship-program-week-1-2dd8799c1c08?source=rss-4cdeefc94c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dd8799c1c08</guid>
            <category><![CDATA[mentorship]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[mentorship-program]]></category>
            <category><![CDATA[wimlds]]></category>
            <dc:creator><![CDATA[Khushboo Gupta]]></dc:creator>
            <pubDate>Sun, 04 Apr 2021 15:16:18 GMT</pubDate>
            <atom:updated>2021-04-04T15:16:18.912Z</atom:updated>
            <content:encoded><![CDATA[<p>I got the opportunity to be selected as a mentee for the first-ever Mentorship Program organized by <strong>Women in Machine Learning and Data Science, Delhi</strong> circle, under my amazing and inspirational mentor, <strong>Arushi Chauhan</strong>! My mentor scheduled the first introductory meeting to introduce ourselves to our co-mentees and understand the different Computer Science fields that align with our interests, and we can further explore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wzv9ROf_aAqmqqODDOD-yQ.png" /></figure><p>I got to know about several new fields such as <strong><em>Semantic Web, Parallel Computing</em></strong>, etc., and various interdisciplinary fields that one can pursue and a Computer Science degree like <strong><em>Neurobiology, Computational finance, Design</em></strong>, etc. Our mentor discussed pursuing a tech career right after undergraduate, tradeoffs regarding pursuing higher studies and research in great detail. To get a Software Engineering role (SWE) in a good product-based company, the most important thing is to practice competitive programming continuously. If someone wishes to pursue higher studies, it is very much recommended to have a <strong><em>decent CGPA</em></strong> and<strong><em> at least one research paper</em></strong> published in a good conference as this increases the chances of getting accepted into a good university. We were advised to explore these different fields continuously, especially in our sophomore year, to figure out what aligns the best with our interests.</p><p>Next, to focus more on our individual interests, 1:1 sessions were organized for all the mentees. As I have not yet decided whether I want to pursue a master&#39;s degree right after my undergraduate, my mentor advised me to focus on competitive programming for cracking good product-based companies and looking for research projects under professors working in active research labs. This session helped clear my doubts regarding approaching professors for research projects and understanding how to manage both competitive programming and working on more research-focused projects. My mentor also advised me to set up <strong><em>weekly targets</em></strong> in competitive programming to lose touch with data structures and algorithms.</p><p>This week’s session was extremely informative, and I am really looking forward to the upcoming sessions!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dd8799c1c08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google India STEP Interview Experience (2021)]]></title>
            <link>https://guptakhushi345.medium.com/google-india-step-interview-experience-2021-d957ab19eb9?source=rss-4cdeefc94c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/d957ab19eb9</guid>
            <category><![CDATA[google]]></category>
            <category><![CDATA[google-step]]></category>
            <category><![CDATA[interview-experience]]></category>
            <category><![CDATA[google-india]]></category>
            <category><![CDATA[google-interview]]></category>
            <dc:creator><![CDATA[Khushboo Gupta]]></dc:creator>
            <pubDate>Fri, 26 Feb 2021 10:53:53 GMT</pubDate>
            <atom:updated>2021-04-30T04:41:19.391Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l2mUJFn2ngkbyF8fzkoNDA.jpeg" /></figure><p>Google STEP (<strong>Student Training Engineering Program</strong>) is a 10 or 12-week internship program for <strong>first and second-year underrepresented undergraduate students</strong> in circuital branches. I had the chance to give my first ever technical interview for the same during my sophomore year and even though I was not selected, I learned a lot from the experience. In this article, I would mainly be talking about how my interviews went and the parts where I could have improved, and not the resume shortlisting process. For that, you can refer to some amazing already existing articles like <a href="https://medium.com/@lavina16052/google-step-internship-5b5141e0cac6"><em>this one</em></a>.</p><h3><strong>Interview Structure</strong></h3><p>There are two 45 minute interviews scheduled usually on the same day. The interviews mainly test the candidate’s knowledge of <strong>Data Structures and Algorithms </strong>(very rarely<strong> Object Oriented Programming (OOPs concepts) </strong>related questions are asked) and questions related to CS fundamentals like DBMS, Computer Networks are not asked. One can code in the <strong>language of their choice</strong> which is filled by them in a form given by the recruiter. The interviews are conducted on <strong>Google Hangouts</strong> and one has to <strong>code in Google Docs</strong>. It is very helpful to have prior experience of coding in Google Docs because it is very much different from coding on an IDE as it does not have features like automatic indentation, auto-complete, etc.</p><blockquote>Tip: Practise <strong>easy-medium questions</strong> on interview preparation sites like <strong>Leetcode and Interviewbit</strong>. It is also very helpful to practise some of the <strong>famous hard level</strong> Leetcode problems if time permits as the interviews are usually scheduled just two weeks after the resume selection.</blockquote><h3>Interview Experience</h3><p>Two 45 minute back-to-back interviews were scheduled on the same day by my recruiter. My preferred language was C++.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lczOu6FfLFVuPfrkWVh4Ww.png" /></figure><h3><strong>First Interview</strong></h3><p>My first interview went really well as compared to my second one. The interviewer was a Software Engineer working at Google India. He asked for my brief introduction and then we straight away got to the question. I cannot reveal the exact questions that were asked because of a signed NDA but I was asked 2 <strong>Leetcode Medium </strong>level questions. The first question was relatively easy and it was very clear that it was to be solved using <strong>Hashmaps</strong>. One key piece of advice that I would give is to be very thorough with the<strong><em> time and space complexities of each and every function</em></strong> that you are using (even if it is from an inbuilt library, like STL for C++). I could answer this question quite easily and also give the correct time and space complexity.</p><p>The second question was a relatively open-ended question and I had to ask several <strong><em>clarifying questions</em></strong>. During the time of the interview, I thought that it was based on maths and greedy algorithms but recently, I saw a similar question and came to know that it could have also been solved using heaps as well.</p><blockquote><strong>Tip:</strong> Try to think of <strong>multiple approaches </strong>of solving a question by <strong>leveraging different data structures and algorithms</strong>. I was only thinking about trying to get into the depth of one approach whereas my friend tried answering each and every question with multiple approaches which really helped her in her selection.</blockquote><p>At the time of the interview, one thing which I did wrong was that without much thought I told the interviewer the first approach which came to my mind. After a small hint from the interviewer, I came to know that I was not thinking in the right direction. The interviewer then broke the question into two parts and I was able to explain my approach to the interviewer and code the most optimized solution. Then, after thinking for some time, I was able to come up with the brute force solution for the second part as well and coded the same. Towards the end, the interviewer told me that even though my solution worked, but it was not exactly what he was expecting from me and told me the best way to solve that problem.</p><blockquote><strong>Scope for improvement:</strong> I should have realised that when the interviewer broke the question into two parts, he was expecting me to <strong>make use of the first function that I had coded in order to solve the second problem</strong>. This would have made the code more<strong> modular and readable </strong>and would have prevented the extra work that my second function did.</blockquote><p>As the time was up now, he then asked me if I had any questions for him and I asked about the team which he works in and the work culture at Google.</p><h3><strong>Second Interview</strong></h3><p>My second interview was scheduled just after my first one. But unfortunately, after waiting for 15–20 minutes, I realized that due to some issues the second interviewer could not join and then contacted my recruiter. My recruiter was very helpful and immediately apologized for the same and asked me for my preferred date and timing and my second interview was rescheduled just after the weekend. My second interview was the most challenging one yet the one from which I learned the most.</p><p>The interviewer was a Software Engineer at Google India and was very friendly and kind. He started with his brief introduction and then we straight away got to the question. The question was based on the topic I fear the most,<strong> Dynamic Programming,</strong> and was of<strong> Leetcode Hard</strong> level. As I had not practiced dynamic programming much and was only familiar with the very basics, I knew this interview was not going to be easy for me.</p><blockquote><strong>Tip:</strong> Be thoroughly prepared with difficult topics like <strong>Dynamic Programming, Graph traversals and other graph algorithms and Tree traversals</strong>. Many a times, students including me think that they might not be interviewed on these topics as they are just in their third semester, but most of my friends were asked questions from these topics during their interviews.</blockquote><p>I was panicking because I was unsure how I could approach the problem. My interviewer was very helpful and helped me by first asking me for some <strong><em>edge cases</em></strong> that I can think of where a simple approach would have failed and then asked me for a <strong><em>recursive solution</em></strong> for the problem. I was able to explain my approach and then code the recursive algorithm. He then asked me about the time complexity for the code which I could get right in the second attempt with some hints from the interviewer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AS7BNgHypQTmjVWu" /></figure><blockquote><strong>Scope for Improvement: </strong>It is very unimpressive to get the time complexity wrong especially in an interview. After making this mistake, one thing I would strongly advice is to <strong>take some time to think about the time complexity of your solution and then get it right in the first attempt</strong>.</blockquote><p>I was then thinking hard about how can I optimize my code. One thing that I did wrong here was that I wasn’t very vocal with my interviewer about the thought process because I was panicking a lot as I was not comfortable with DP. Panicking is the worst thing that you can do to yourself in an interview as it blocks your thought process. The interviewers are there to help you and hence, <strong><em>try to be calm </em></strong>and at <strong><em>each and every point discuss your solution with your interviewer</em></strong>. After trying really hard with some hints from my interviewer, I was able to code a top-down DP solution for the same problem. While coding, I made some silly mistakes which I then corrected after they were brought to my notice by the interviewer.</p><blockquote>Tips: Always <strong>dry run</strong> your code yourself on some <strong>good edge cases</strong> and make sure that your <strong>code compiles without any errors</strong> before showing it to your interviewer.</blockquote><p>The interviewer then asked me about my experience and I asked him how it feels to work at Google. The interview concluded and I then received my rejection mail after almost a month. Through this article, I hope you get an in-depth idea about the Google STEP interview experience and understand the mistakes that I made and try to prevent them during your interviews. All the best!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d957ab19eb9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How many Topics: A detailed guide to Topic Modeling]]></title>
            <link>https://guptakhushi345.medium.com/how-many-topics-a-detailed-guide-to-topic-modeling-fa23eae385ef?source=rss-4cdeefc94c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa23eae385ef</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[girlscript-summer-of-code]]></category>
            <category><![CDATA[topic-models]]></category>
            <category><![CDATA[natural-language-processi]]></category>
            <category><![CDATA[topic-modeling]]></category>
            <dc:creator><![CDATA[Khushboo Gupta]]></dc:creator>
            <pubDate>Sun, 17 Jan 2021 19:43:19 GMT</pubDate>
            <atom:updated>2021-01-17T19:46:40.058Z</atom:updated>
            <content:encoded><![CDATA[<p>I had the opportunity to be a part of the Girlscript Summer of Code Extended’ 20 Program wherein I was one of the top 3 contributors of the open-source research project called ‘<strong><em>How Many Topics?’ </em></strong>based on<strong><em> Natural Language Processing </em></strong>and<strong><em> Topic Modeling</em></strong>.<strong><em> </em></strong>I am extremely grateful to my project mentors, Harini Suresh and Mansi for helping my team and I wherever required in successfully completing the project. I would also like to thank my teammates, Namya LG, Rashi Singh and Nagasuruthika for their constant support and enthusiasm towards the project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/0*rEejyernuFWwMeDf.PNG" /></figure><blockquote>Problem Statement</blockquote><p><strong><em>Topic modeling </em></strong>refers to the task of discovering the underlying thematic structure in a text corpus, where the output is commonly presented as a report of the top terms appearing in each topic. Despite the diversity of topic modeling algorithms that have been proposed, a common challenge in successfully applying these techniques is the selection of an appropriate number of topics for a given corpus.</p><p>We intend to work on a research paper where we propose a term-centric stability analysis strategy to address this issue, the idea being that a model with an appropriate number of topics will be more robust to perturbations in the data. The goal is to determine <strong><em>k, </em></strong>the number of topics that are relevant to a given piece of text.</p><p>The pipeline that was followed for the project was as follows:</p><ol><li>Dataset Collection and required pre-processing.</li><li>Studying and analysing several Topic modeling algorithms and running them.</li><li>Fine-tuning the parameters for topic modeling.</li><li>Classification</li></ol><p>The entire code for the project can be found <a href="https://github.com/khushboogupta13/How_Many_topics/tree/master/Covid_research"><strong><em>here</em></strong></a>.</p><blockquote>Introduction</blockquote><p><strong>Topic modeling</strong> is an unsupervised machine learning technique that is used to classify a set of documents, detecting words and phrase patterns within them, and automatically clustering word groups and similar expressions that best characterise a set of documents. To delve deep into the same, we researched Topic Modeling by using the COVID-19 Dataset which can be found <a href="https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*jKK4QFlMz8tFBeAS" /><figcaption>Word cloud showing the <strong>most prominent words</strong> in the COVID-19 title dataset.</figcaption></figure><h3>Description of the dataset</h3><p>In response to the COVID-19 pandemic, the White House and a coalition of leading research groups prepared the COVID-19 Open Research Dataset (CORD-19). CORD-19 is a resource of over 200,000 scholarly articles, including over 100,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses.</p><p>The research part of the project was implemented on the <strong><em>Title</em></strong> part of the COVID-19 Research paper dataset. To confirm our results, the best suited parameters and procedure was then run on the <strong><em>Abstracts</em></strong> part of the same dataset and several other datasets like <a href="https://www.kaggle.com/yelp-dataset/yelp-dataset"><strong><em>Yelp Reviews</em></strong></a>, <a href="https://github.com/tdhopper/topic-modeling-datasets/tree/master/data/raw/Nematode%20biology%20abstracts"><strong><em>Nematode Biology</em></strong></a>, <a href="http://qwone.com/~jason/20Newsgroups/"><strong><em>20 Newsgroups</em></strong></a> and <a href="https://www.kaggle.com/rtatman/blog-authorship-corpus"><strong><em>Blog Authorship</em></strong></a>.</p><h3>Data Pre-Processing</h3><p>From the entire COVID-19 Research Paper dataset, Topic Modeling is done in two parts i.e. on the <strong>Titles</strong> and the <strong>Abstract</strong> of the Research Papers. The following steps were performed to clean the data in order to prepare it for running several Topic Modeling algorithms:</p><ol><li><strong>Lowercasing, Punctuation, and Stopword Removal: </strong>All the texts were lowercase, and punctuations were removed with the help of the<strong><em> Regex library</em></strong> in Python. Then stopword removal was done and all the stopwords in the English language which are available in the <strong><em>gensim package</em></strong> of Python were removed.</li><li><strong>Tokenization: </strong>Word Tokenization for done so that these words can then further be stemmed and lemmatized. The <strong><em>RegexpTokenizer</em></strong> available in the <strong><em>nltk</em></strong> package for data preprocessing was used in this step.</li><li><strong>Stemming: </strong>Stemming was done for all the words in the dataset using <strong><em>Porter Stemmer</em></strong> in order to reduce the words to their root/base form. The tokenized words from the above step were the input for the stemmer.</li><li><strong>Lemmatization: </strong>Lemmatization was done so that words that are in the third person are changed to the first person and verbs in the past and/or future tense are changed to their present tense. <strong><em>WordNetLemmatizer</em></strong><em> </em>available in the<strong><em> nltk</em></strong> package was used for the same.</li></ol><blockquote>Observations</blockquote><h4><strong>1. Hyperparameter Tuning</strong></h4><p>For the<strong><em> COVID-19 Title dataset</em></strong>, the following were the questions which often serve as a standpoint for many who are getting their hands dirty with Topic Modeling. After several days of Research, the following were the observations made by us which worked well for our dataset.</p><ul><li><strong>Whether to use <em>Bag of Words</em> or <em>Term Frequence-Inverse Document Frequency</em>?</strong></li></ul><p>The BoW model captures the frequencies of the word occurrences in a text corpus. It is not concerned about the order in which words appear in the text; instead, it only cares about which words appear in the text. On the other hand, TF-IDF measures how important a particular word is with respect to a document and the entire corpus.</p><p>So as far as our algorithm goes, often while implementing the Latent Dirichlet Allocation algorithm (discussed later), TF-IDF model is only used because it has shown to provide better results as compared to when we only use a bag of words. However, the difference is not so significant. So as far as LDA is concerned, Bag of words or TF-IDF, both seemed to work fine for our dataset.</p><p>After trying both the models on several different values of k, the following observations were made by our team:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SqMoL4KKzDXWNRRi" /><figcaption><em>Method: Using Bag of Words, k=8, passes=2</em></figcaption></figure><p>Even though some of the topics were repetitive, overall the model gave satisfactory results for us to continue further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lmzPLzmc8A-CERnw" /><figcaption><em>Method: Using TF-IDF, k=8, passes=2 (keeping the other parameters same)</em></figcaption></figure><p>However, in our case, even more repetition among several topics was observed when the LDA model used TF-IDF. According to us, TF-IDF provided very general topics regarding COVID-19 which could have been common among all research papers whereas the Bag of Words model could give us a more clear picture of what was going on in the paper as more specific words came out. In order to check whether we were proceeding in the right direction, we did the <strong><em>performance evaluation</em></strong> of both models. Performance Evaluation is done to check whether our model can distinguish different topics by using the words in each topic and their corresponding weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LvnXFC6DF1k1UTn2" /><figcaption><em>Performance Evaluation according to the Bag of Words model</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZKja1XOizLoyeYeL" /><figcaption><em>Performance Evaluation according to TF-IDF model</em></figcaption></figure><p>Both the models performed more or less the same. For example, in the first case, Bag of Words performed better whereas in the second case TF-IDF model performed much better. So we went according to our previous observation and finally chose the Bag of Words Model as it gave us more detailed and specific topics.</p><ul><li><strong>Whether to use the <em>Latent Dirichlet allocation </em>(LDA) algorithm or the <em>Latent Semantic Analysis</em> (LSA) algorithm?</strong></li></ul><p>Latent Dirichlet Allocation (LDA) is the most popular topic modeling algorithm which is used to classify text in a document to a particular topic. It builds a topic per document model and words per topic model, modelled as Dirichlet distributions.</p><p><strong><em>Latent Semantic Analysis (LSA)</em> </strong>is another topic modeling algorithm that helps us to capture the context around hidden meaning/concepts in documents (topics). It mainly is a technique in the field of distributional semantics and analysis.</p><p>LSA is a technique that can help us figure out hidden concepts or topics behind the words. While implementing the LSA algorithm on the COVID-19 title dataset, because of limitations regarding computational time and resources, we restricted the number of features to 1000 while creating the document term matrix. This hyperparameter was also tuned to check if this affects the topic model in any way. However, by changing the number of max features, almost no change was observed in the results of our dataset. If enough computational power is available, it is advised to use all the terms while implementing the algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/0*fFVgFF5ZtoriTLNX" /><figcaption>LSA algorithm was implemented on the COVID-19 Title Dataset with k=8</figcaption></figure><p>The above has been repeated for varying values of k.</p><p>On implementing the same using LDA (as seen later in the document), our dataset gave much better results as compared to LSA. LDA gave a more variety of topics as compared to LSA which gave very redundant results in between the topics. One more reason why LSA has not given satisfactory results for our dataset is that the Latent Semantic Analysis algorithm as the name suggests tries to find the hidden concepts and meaning behind words that have the same spelling but mean different things in different situations. As most of the words in our dataset are technical and related to the COVID-19 field, they cannot be used in places where they have some other meaning (unlike words like a novel which can mean a book or original depending on the situation), this can be a reason that LDA performed better as compared to LSA for COVID-19 dataset. Hence, LDA is finally used for training.</p><ul><li><strong>Whether to use the implementation of LDA algorithm available in <em>Python’s Gensim</em> Package or <em>Scikit-learn Package</em></strong></li></ul><p>The Latent Dirichlet Allocation algorithm can be implemented using two libraries available in Python i.e. Gensim package and Scikit-learn Package. The more popular one used to implement LDA is the Gensim Package however, in order to check whether the results varied if another package was used, we tried to implement LDA using both the packages. A major drawback was that the Scikit-learn package was very slow and due to limited computational power, even on dividing our dataset into two parts, even one run of the algorithm could not be achieved. Hence, we decided to continue with the Gensim Package as it was much faster. If computational resources permit, then one should check the results by implementing LDA using both the packages.</p><h4>2. Number of Topics</h4><p>For the<strong><em> COVID-19 Title dataset</em></strong>, LDA algorithm was implemented twice, on the entire dataset as well as by dividing the dataset randomly into 2 parts to ease the implementation and reduce the running time. A little analysis was also done with regard to coherence, and we fixed the number of passes to one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/0*2_iKYSQjiiGVMm0P" /><figcaption>X-axis: Number of topics, Y-axis: Coherence Score</figcaption></figure><p>This is the output that was generated, though the max coherence increases with the number of topics, the topics are not relevant at all. Considering the local maxima, the coherence score for <strong><em>number of topics = 11 </em></strong>seems to<strong><em> </em></strong>work the best.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/0*zqxh-FlGosbln6G9" /><figcaption>X-axis: passes, Y-axis: Coherence score</figcaption></figure><p>The above is a graph between coherence and number of passes and number of topics as 8, it can be seen that the <strong><em>number of passes between 30–40</em></strong> is most favourable for the number of topics 8.</p><p>Earlier, the only parameters that were taken into account were as follows:</p><pre>lda_model = gensim.models.ldamodel.LdaModel(corpus,num_topics=8, id2word = dictionary, alpha=0.16, passes=i,chunksize=10000,per_word_topics=True)</pre><p>To delve deeper, the following have also been considered then:</p><pre>lda_model = gensim.models.ldamodel.LdaModel(corpus=corpus,id2word=dictionary,num_topics=8, random_state=100, update_every=1, chunksize=1000, passes=10,alpha=’auto’, per_word_topics=True)</pre><p>To date, the best results have been obtained for <strong><em>number of topics as 8</em></strong><em> </em>and <strong><em>number of passes as 40</em></strong><em> </em>and the <strong><em>coherence score calculated was 0.3253107958923857.</em></strong></p><h4>3. Coherence Value</h4><p><em>C_v</em> (Coherence value) measure is based on a sliding window, one-set segmentation of the top words, and an indirect confirmation measure that uses normalized pointwise mutual information (NPMI) and the cosine similarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5yWlW4Nqxof08stB" /><figcaption>The highest coherence value was obtained at <em>K=8 and K=38.</em></figcaption></figure><p>With the coherence score that seems to keep increasing with the number of topics, it may make better sense to pick the model that gave the highest CV before flattening out or a major drop. In this case, we picked k=8 as for a very huge value of k, the topics do not seem to add much value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/0*cyUK4s7rMVT4FZQq" /><figcaption>The Coherence value using LSA algorithm is 0.34 for k=9</figcaption></figure><p>The title dataset was then divided randomly into two equal halves in order to reduce the runtime and also to check whether that would give us better results as compared to feeding the entire dataset into the algorithm. As the best results were obtained for <strong><em>7&lt;=k&lt;=9</em></strong>, the two divided datasets were also checked in the same range only for easier comparison purposes.</p><blockquote><strong><em>First Half of the divided dataset</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nFu9Qt6Q0ovpRRl7" /><figcaption>Number of Topics = 7, passes = 40</figcaption></figure><p>The major observation was that dividing the dataset did help and gave much better results. From the above picture, we can see that newer words have come up (Topic 4 and Topic 5) which were not seen in earlier parts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6GFR-JxA-uaPaxPd" /><figcaption>Visualisation of Topic Model for k = 7 using pyLDAvis package</figcaption></figure><p>This dataset shows excellent results in terms of visualisation as the topics in the Intertopic Distance Map are quite far away. The best improvement that can possibly be done to our model to achieve excellent accuracy is to change the parameters such that (Topic 1 and Topic 5) and (Topic 2 and Topic 3) do not overlap for the best possible results. On observing this and the next visualisation, we thought that maybe a suitable number of topics might be 5 as the Intertopic Distance Map always is being divided into majorly 5 parts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pRtSHVo8_ORguD9N" /><figcaption>Number of Topics = 8, passes = 2</figcaption></figure><p>Almost the same results can be seen (this is also much better than the results obtained using the entire dataset) that were obtained by using k = 7. However, the one with more number passes gave a more in-depth insight into the topics. This also shows that the model which runs for a greater number of passes gives us better results (but this cannot always be tested because of limited computational power).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ip90NSH9R0Wj78Af" /><figcaption>Visualisation for k = 8</figcaption></figure><p>Topic 3 gave really good and unique results that did not come up before. This model is also excellent in terms of visualisation as all the topics are quite spread out. As we can again see in the above visualisation, some topics are a bit clustered together and there are 5 major categories/areas in which the topics fall into. So we decided to check whether k = 5 might be a good value to be chosen as the number of topics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R2Z_V03S9zNXV9gF" /><figcaption>Visualisation for Number of Topics = 5</figcaption></figure><p>From the previous observations, an idea that came up was to try for number of topics as 5 to get all separated topics. But still Topics 3 and Topics 5 were overlapping so this approach was not the correct one. Also, the quality of topics was maintained the same as that we could see with the Number of topics = 8. As a higher coherence value was obtained for k = 8, we decided to finally continue with that model only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yHhxISH_xKQZDLpB" /><figcaption>Visualisation for k = 8</figcaption></figure><p>This Intertopic Distance Distribution is the best one we have seen so far. All the topics are quite far from each other and again the dataset has been divided into 5 major sets. Some overlaps can be seen clearly in (Topic 6 and Topic3), (Topic 1 and Topic 2) and (Topic 4 and Topic 5). One thing which we observed that even with a lot of hyperparameter tuning, at least 2 topics were constantly overlapping.</p><blockquote><strong><em>Second Half of the divided dataset</em></strong></blockquote><p>The same procedure was now done in the second half of the dataset. As k = 8 gave the best results for the first half, we decided to check the second dataset with the same. But we increased the number of passes to 40 in order to achieve even better results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Let_bMKez08eZ2c4" /><figcaption>Number of Topics = 8, passes = 40, random_state = 42</figcaption></figure><p>Very satisfactory results were obtained (especially better than the one without splitting the dataset). New words have come up in this Topic Model too and comparatively there is less redundancy. One reason for new words coming up might be as the dataset is now divided, the dominant and most prominent words now change to ones that might have been not so dominant to the entire dataset.</p><blockquote>Implementation of our observations on the Abstracts part of the COVID-19 dataset</blockquote><p>The <strong><em>COVID-19 abstract</em></strong> was further divided into two halves randomly so that the operations performed can be easily implemented to understand the results.</p><blockquote><strong><em>First Half of the divided dataset</em></strong></blockquote><p>For the first half of the dataset, a similar procedure was followed as the one done for the title dataset. As the k value for our dataset (as found from the title dataset) ranges between <strong><em>8&lt;=k&lt;=10</em></strong>, these values were used to test whether they work the same for the abstract part too.</p><p>One observation was that values in the range<strong> k&lt;7 </strong>gave very unsatisfactory results as most of the words were redundant and not much relevant to our dataset. Fairly good results were coming up. So that’s when we knew that the optimum number of topics is going to be reached soon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*07JR7GRUCKXe00RU" /><figcaption>Number of topics=7 and passes=2</figcaption></figure><ul><li>For <strong>k=7</strong>: On increasing the number of passes, more general nouns came up as compared to more specific proper nouns (in passes = 2).</li><li>For <strong>k=8</strong>: Number of topics=8 gave a very good general idea of our dataset which could cover almost all the aspects of COVID-19 in general.</li></ul><p>The topics were not redundant which shows that k=7 and/or k=8 is a good number of topics that our dataset can be divided into. While k=7 gave more specific topics with proper nouns, k=8 gave a more general idea and hence, should be preferred over k=7.</p><p>As we decided to continue with k=8 for the first part of the abstract dataset, we made the <strong>Visualisation Model</strong> for the same using <strong><em>pyLDAvis package </em></strong>available in Python.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7T17NUN53sY584Ac" /></figure><blockquote>Striking Observation</blockquote><p>It is usually said that if the Intertopic Distance between the topics in a Topic Modelling Model is large, the model is said to be better. However, one striking observation that was observed while working with all the datasets was that if the Coherence Value for a particular number of topics was very good (i.e. high), the Intertopic distance for the same k was quite lower. This had caused a dilemma of whether to move forward with a better Coherence Value model or a better Visualisation Model. In the end, we decided to move forward with the Coherence Value Model because we thought that as our dataset was very much focused on topics related to COVID-19 only, there is a huge possibility that almost all the topics will have to be interrelated a lot more as compared to other datasets which have a very diverse range of topics.</p><p>An overall Visualisation of how the coherence score changes as compared to a number of topics (k) is shown below. From this graph, we can clearly see that the optimum range of k for our dataset is <strong><em>8&lt;=k&lt;=10</em></strong>, which is almost the same as our results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/0*iXmUQyPjH-wF6snL" /><figcaption>Coherence Value for k=8: 0.559898006654606 (A very good coherence score)</figcaption></figure><blockquote><strong><em>Second Half of the divided dataset</em></strong></blockquote><p>For the second half of the abstract dataset, we checked that what number of topics will suit the dataset best by plotting the coherence value graph and printing the topics’ names.</p><p>We used the coherence value graph method because as per our previous results we have seen that the coherence value is at the peak for the number of topics that suit the best for our dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wS_BQOYA3bM4vICK" /><figcaption>No. of topics = 4, Input size = complete dataset (second half of abstract dataset), passes=100</figcaption></figure><p>The words present in the topic model were not very much repetitive or redundant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DcNoVbv4A6gyoT1b" /><figcaption>No. of topics = 6</figcaption></figure><p>The words present in the topic were not very much repetitive or redundant except for topic 1, which consisted of only two-letter words that made no sense like <em>‘eg- de’, ‘la’, ‘en’,</em> etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H1KRbb99Nkadl_tU" /><figcaption>No. of topics = 8</figcaption></figure><p>The words present in the topic were not very much repetitive or redundant except for topic 6, which consisted of only one or two-letter words that made no sense.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4R32lh_I6OY4_LU2" /><figcaption>No. of topics = 4, Input size = 800, passes=100</figcaption></figure><p>Words like <em>‘covid19’</em> and <em>‘disease’</em> were seen to repeat in all the topics. The presence of irrelevant data was negligible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dPb3PrD8OcXNGboY" /><figcaption>No of topics = 11</figcaption></figure><p>Only one or two words were repeated but a lot of two or three-letter words were found which were irrelevant and did not add much value to our results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/0*tRwUOqowY6pEo231" /><figcaption>Coherence value graph for k = 9 (Best result)</figcaption></figure><blockquote>Visualisation</blockquote><p>For the <strong><em>COVID-Title Dataset</em></strong>, we tried visualising the results using word clouds, graphs, tables, etc.</p><ul><li><strong>Dominant topics and its percentage contribution</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y7f98tHXbm7YxYvY" /></figure><p>Repetition of words in various documents can be seen.</p><ul><li><strong>Frequency Distribution of Word Counts in Documents</strong></li></ul><p>The following graphs show the distribution of words present in various documents in the topics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3NaL_UIV64Ok28dl" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Yr_VVu5vCtUe46c1" /></figure><ul><li><strong>Word cloud with respect to different topic models:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/0*mIcQzc9BEHeE8lCv" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RVvyVBRnM4Hp8qzR" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OSMSxEk-9fgH72aD" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/0*rzjpjHyXDhh1zt0y" /></figure><blockquote><strong>Implementation of our observations on several other datasets</strong></blockquote><p>After successfully finding a particular flow to implement our topic modeling algorithm and finding the suitable number of topics along the way, our project mentors advised us to try the same on other datasets of different types to check whether we got the desired results. Each member took up some of the most famous datasets and worked on the same. I chose to work on the <strong><em>Blog Authorship dataset</em></strong>. This dataset contains text from 681,288 blogs which were written before August 2004 with each blog being the work of a single writer.</p><p>Because of the huge size of the dataset and limitation of computational power available, the dataset was split randomly into 2 halves to reduce the running time and for the ease of implementation of the algorithm. Data preprocessing was done which included the usual steps of lowercasing, punctuation and stopword removal, tokenization, stemming and lemmatization. Then the LDA model was run on the two divided datasets.</p><h4>Observations</h4><blockquote>First Half of the Blog Authorship Dataset</blockquote><p>As observed in the COVID-19 Research Paper dataset, <strong><em>8&lt;=k&lt;=10</em> </strong>was the best-suited value for the number of topics to be selected. Intuitively, we went for the same k to check if it also suited the Blog Authorship dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*snubVc2onBKXDmRS" /><figcaption>Number of Topics= 8</figcaption></figure><p>As can be seen from the above data, k=8 was a very bad value for a number of topics and did not give satisfactory results at all. Many words were repetitive and it did not give an idea about the overall topics at all. Hence, the value of k that worked for COVID-19 Research Paper dataset did not work for this. There were 2 reasons we could think of which might have caused this:</p><ol><li>The striking difference in the size of the datasets. The COVID-19 Research Paper dataset was much smaller (around 250,000 entries were there) as compared to the Blog Authorship dataset which contains 681,288 entries.</li><li>The Blog Authorship dataset can have a very large variety of topics as blogs can be about anything to everything whereas there is limited scope for topics in the COVID-19 Research Paper dataset.</li></ol><p>Because of the above-stated factors, there is a possibility that 8&lt;=k&lt;=10 might not have worked for <strong><em>huge datasets with very diverse topics</em></strong>.</p><p>A <strong><em>Coherence value graph</em></strong> was plotted for k ranging between 7 and 30 in order to find the local and global maxima of the Coherence Value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ttyBjwv_CytLEz14" /></figure><p>From the above graph, we can clearly see that there are local maximas occurring at<strong> <em>k = 12/13</em></strong> and<em> </em><strong><em>k = 17/18</em> </strong>and the best topic model could occur when <strong><em>k = 22/23</em></strong><em> </em>is selected as a higher coherence value corresponds to a better topic model. The coherence value for k&gt;23 then continues to decrease indicating that the highest suitable value for k cannot be greater than 23.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a3YNgl61DPhbwQs1" /><figcaption>Number of Topics= 12</figcaption></figure><p>k=12 also gave satisfactory results. This value for k can also be used as a topic model as coherence value gives a local maxima at k=12. However, as these topics are not much detailed, there is a very high possibility that many other major topics in the dataset are being missed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mlUVu0xQ25LIF4A_" /><figcaption>Visualisation model for number of topics=12</figcaption></figure><p>This model proved to be a good choice when it comes to visualisation of topic models as the topics were not so clustered together and had more intertopic distance as compared to other values of k (as seen later in the document). Similarly, as for the COVID-19 Research Paper Dataset, we can choose any one hyperparameter for tuning between Coherence Value and Visualisation. As Coherence Value has proven to be a better parameter, we continue with it only.</p><p>On checking for k=17 (as it was a point of local maxima when it came to Coherence Value), the LDA Model gave satisfactory results, but still a better model would be the one having k=22 because it achieved the maximum coherence Value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RmesX_HkxwvzzRSi" /><figcaption>Visualisation for number of topics=17</figcaption></figure><p>k = 17 gave the best model when considered from the Visualisation aspect. This is because the topics were comparatively very much spread out (i.e. the intertopic distance is quite large) as compared to other values of k and not so clustered together. This suggests that this value of k covers quite a large number of diverse topics which are dominant in the dataset.</p><p>Overall k=22 gave a very detailed explanation for each topic that is dominant in the first half of the Blog Authorship dataset, however there is a possibility that k=19/20/21 can also prove to be good as some of the topics did not make much sense as they included words like <em>‘like’</em> very frequently. As also seen from the Coherence Value graph that there is a global maximum at k=22, this value of k can finally be chosen as the optimum number of topics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oPP462GINSIMGBe6" /><figcaption>Visualisation model for number of topics = 22</figcaption></figure><p>When it comes to visualisation, it can again be observed that for a high coherence value, the intertopic distance is comparatively less as compared to lower values of k where a local maximum was achieved. But all the topics were not clustered together and as there is some appreciable intertopic distance, this factor can be ignored as Coherence Value outweighs the same.</p><blockquote>Top 15 words in the Dataset</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KPoknVJhtfLlD3oZ" /></figure><p>This histogram was plotted in order to check that the dominant topics that the LDA algorithm calculated were in accordance with the dominant words in the dataset. From the above graph we can clearly see that the most frequent words in the dataset also constitute topics which our topic model has calculated. This somewhat proves the authenticity of the procedure that we have followed for Topic Modeling.</p><blockquote>Second Half of the Blog Authorship Dataset</blockquote><p>Similar to the procedure followed for the first half of the dataset, a Coherence Value graph was plotted for<strong><em> 7&lt;k&lt;30</em></strong> in order to find the local and global maxima of the Coherence Value. One more reason for choosing <strong><em>k=22/23</em></strong> as compared to its lower values is that as this dataset is very huge with very diverse entries, in order to cover all the dominant topics, it is better that a larger number of topics is chosen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/0*WQCxj0o-pDE1rM-m" /></figure><pre>Number of Topics 7 has coherence value 0.417767</pre><pre>Number of Topics 12 has coherence value 0.441731</pre><pre>Number of Topics 17 has coherence value 0.444497</pre><pre>Number of Topics 22 has coherence value 0.458948</pre><pre>Number of Topics 27 has coherence value 0.452409</pre><p>A striking observation was that the Coherence Value Graph for both the first and second half of the Blog Authorship Dataset was exactly the same with local maxima at<strong> k = 12/13</strong> and <strong>k = 17/18 </strong>and the global maxima occurring for values of <strong>k = 22/23</strong>. As the Coherence Value decreases for k&gt;23, no other maxima occur after k=22/23.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hyYR48asPaVtheYI" /><figcaption>Number of Topics = 13</figcaption></figure><p>On running the algorithm for 13≤k≤15, the result observed was that all the topics made sense and gave a fairly good idea about what each topic was trying to say. Some of the words were redundant though which can be improved by choosing a greater value of k with a higher coherence score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FX3m2fMQTryYn5ZC" /><figcaption>Visualisation for number of topics = 15</figcaption></figure><p>On visualisation we can see that many topics are clustered together and it is a possibility that this model might not be a good one due to less intertopic distance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D1lzXZDdOCpAnrTo" /><figcaption>Number of Topics = 18</figcaption></figure><p>Even though k=17/18 had a higher coherence score as compared to k=12/13, the topics were not that good as many of them were very repetitive. It is observed that k = 12/13 would have sufficed rather than choosing a larger number such as 17/18.</p><p><em>The best topic model is achieved using k = 23</em>. This model gave a very detailed description of what each topic could have been and gives an overall idea of the different blogs that the dataset consists of. This model did not consist of redundant topics too and had the best coherence score as observed in the graph. Finally k = 22/23 is the final number of topics that can be chosen for the second half of the Blog Authorship dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DemJJU91-iSJrr_i" /><figcaption>Visualisation for number of topics = 23</figcaption></figure><p>Again it can be observed that for a very high coherence value, the visualisation model did not give very good results. Many topics were clustered together and the intertopic distance was also quite small. However, as Coherence Value outweighs the visualisation aspect, finally k = 22/23 is only chosen to train the model.</p><blockquote>Conclusion for the Blog Authorship dataset</blockquote><p>As both the first and second half of the dataset were pretty much in coordination and did not have any contradicting features at all, it can easily be said that <strong><em>k = 22/23 is the final number of topics chosen for the Blog Authorship dataset which gives a Coherence score of around 0.458948</em></strong>. The procedure and algorithm which was followed to find the topics for COVID-19 Research Paper dataset worked quite well for this dataset too. However, the value of k had to be increased because of the huge size and the diversity of topics of the dataset.</p><blockquote>Overall conclusions made for the COVID-19 datasets</blockquote><p>On the basis of our findings, the following are the conclusions made:</p><ul><li>The best-suited value for the <strong><em>number of topics i.e. k</em></strong> comes out to be in the range of <strong><em>8&lt;=k&lt;=10 </em></strong>for the<strong><em> COVID-19 Title </em></strong>and<strong><em> Abstract dataset</em></strong><em>.</em></li><li><strong><em>Latent Dirichlet Allocation (LDA)</em></strong> algorithm worked better for our dataset as compared to the Latent Semantic Analysis (LSA) algorithm.</li><li><strong><em>Coherence Value</em></strong><em> </em>was considered as the main feature used for the analysis of our topic model i.e. it was preferred over other factors like Perplexity score, Log-likelihood, Intertopic Distance Map, etc.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa23eae385ef" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>