<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nishant Sahoo on Medium]]></title>
        <description><![CDATA[Stories by Nishant Sahoo on Medium]]></description>
        <link>https://medium.com/@nishantsahoo?source=rss-c22f3d5bbc7b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-1nH3rDccZQmmGdfAWmDTA.jpeg</url>
            <title>Stories by Nishant Sahoo on Medium</title>
            <link>https://medium.com/@nishantsahoo?source=rss-c22f3d5bbc7b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:49:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nishantsahoo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Behavioral Finance]]></title>
            <link>https://medium.com/@nishantsahoo/behavioral-finance-6a9c11f592ff?source=rss-c22f3d5bbc7b------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a9c11f592ff</guid>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <dc:creator><![CDATA[Nishant Sahoo]]></dc:creator>
            <pubDate>Sat, 21 Jul 2018 02:58:37 GMT</pubDate>
            <atom:updated>2021-10-11T11:13:20.051Z</atom:updated>
            <content:encoded><![CDATA[<h4>What’s UP? Microsoft, or Amazon?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*ptjfmneeY5pEjU0J4I87nQ.jpeg" /><figcaption>Picture courtesy: https://www.thestreet.com</figcaption></figure><p>In this article, I’ll talk about how <a href="https://www.linkedin.com/in/harshini-n/"><strong>Harshini</strong></a> and I tried to predict the stock market behavior by performing Sentiment Analysis on Twitter tweets about NASDAQ 100 companies. We used <a href="https://textblob.readthedocs.io/en/dev/">TextBlob</a> (a Python library) to perform sentiment analysis, <a href="http://scikit-learn.org/">sklearn</a> to build the classifier model, and collected the Twitter tweets data set from <a href="http://followthehashtag.com/">followthehashtag</a> official website.</p><p>Many stock market investors try to predict the market behavior by analyzing previous trends, reading news articles, and understanding how well certain products of a company are performing in terms of revenue. While there are many other factors that influence market behavior, this article will focus on how the study of <a href="https://www.investopedia.com/terms/b/behavioralfinance.asp">behavioral finance</a> can help us gain insights into the stock market. Here are two famous incidents which show why the study of behavioral finance is important and how Twitter is a powerful platform that deeply influences the market -</p><ol><li>Snapchat shares dropped about 7% on 22nd Feb 2018 after Kylie Jenner tweeted the following, seen by her 25 million followers. Here’s an <a href="https://www.cnbc.com/2018/02/21/kylie-jenner-slams-snap.html">article</a> which talks about the same.</li></ol><h3></h3><p>sooo does anyone else not open Snapchat anymore? Or is it just me... ugh this is so sad.</p><p>2. Tesla shares rose 2.4% - which is an exceptional increase in the market — after Elon Musk replied to The Economist’s tweet</p><h3>The Economist on Twitter</h3><p>Tesla will need to raise $2.5bn to $3bn this year, according to Jefferies, a bank https://t.co/Uq1tlVIder</p><p>with the following tweet.</p><h3>Elon Musk on Twitter</h3><p>@TheEconomist The Economist used to be boring, but smart with a wicked dry wit. Now it&#39;s just boring (sigh). Tesla will be profitable &amp;amp; cash flow+ in Q3 &amp;amp; Q4, so obv no need to raise money.</p><p>These observations inspired me to work on a model which tries to predict the stock market behavior by performing sentiment analysis on Twitter tweets about a particular company.</p><h4>Sentiment analysis</h4><p>4 basic steps to perform sentiment analysis -<br>1. Data collection: We collected data for this model from this <a href="http://followthehashtag.com/datasets/nasdaq-100-companies-free-twitter-dataset/">data set</a>.<br>2. Text preparation/Text Preprocessing: We performed text preprocessing using the following code -</p><pre>import re</pre><pre>emoticons_str = r&quot;&quot;&quot;<br>    (?:<br>        [:=;] # Eyes<br>        [oO\-]? # Nose (optional)<br>        [D\)\]\(\]/\\OpP] # Mouth<br>    )&quot;&quot;&quot;<br> <br>regex_str = [<br>    emoticons_str,<br>    r&#39;&lt;[^&gt;]+&gt;&#39;, # HTML tags<br>    r&#39;(?:@[\w_]+)&#39;, # @-mentions<br>    r&quot;(?:\#+[\w_]+[\w\&#39;_\-]*[\w_]+)&quot;, # hash-tags<br>    r&#39;http[s]?://(?:[a-z]|[0-9]|[$-_@.&amp;amp;+]|[!*\(\),]|(?:%[0-9a-f][0-9a-f]))+&#39;, # URLs<br> <br>    r&#39;(?:(?:\d+,?)+(?:\.?\d+)?)&#39;, # numbers<br>    r&quot;(?:[a-z][a-z&#39;\-_]+[a-z])&quot;, # words with - and &#39;<br>    r&#39;(?:[\w_]+)&#39;, # other words<br>    r&#39;(?:\S)&#39; # anything else<br>]<br>    <br>tokens_re = re.compile(r&#39;(&#39;+&#39;|&#39;.join(regex_str)+&#39;)&#39;, re.VERBOSE | re.IGNORECASE)<br>emoticon_re = re.compile(r&#39;^&#39;+emoticons_str+&#39;$&#39;, re.VERBOSE | re.IGNORECASE)</pre><pre>def tokenize(s):<br>    return tokens_re.findall(s)</pre><pre>def preprocess(s, lowercase=False):<br>    tokens = tokenize(s)<br>    if lowercase:<br>        tokens = [token if emoticon_re.search(token) else token.lower() for token in tokens]<br>    return tokens</pre><p>By using the above code, one can remove emoticons, HTML tags, mentions of other Twitter accounts using “@”, hashtags, URLs, and numbers, leaving only a sequence of simple English tokens/words. This text can now be analyzed to calculate the sentiment score.</p><p>3. Sentiment calculation: Using TextBlob, we can find the sentiment score of a given text by using the following code -</p><pre><strong>from</strong> textblob <strong>import</strong> TextBlob<br><br>text_str = &#39;&#39;&#39;<br>Alexa is an amazing product by Amazon. Amazon is the best. Now I don&#39;t have to answer those stupid questions my kids ask me.<br>&#39;&#39;&#39;<br><br>blob = TextBlob<strong>(</strong>text_str<strong>)<br>sentiment_score = </strong>blob.sentiment.polarity<br>print(<strong>sentiment_score</strong>)<br># prints the sentiment score for the text above</pre><p>In our project, we’ve calculated the sentiment score for all tweets about a particular company of a given day. This sentiment score is further used for sentiment classification.</p><p>4. Sentiment classification: To predict whether the stock will increase or decrease, we’ve used a <a href="http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html">Decision Tree Classifier</a>. The end goal of our model is to classify whether the stock price for a given company on a given day will rise or fall. To do so, first, we calculate the public sentiment value associated with a company for a given day as shown in step 3. Then, we calculate the change in stock price for a given day, by subtracting the previous day’s closing stock price from the current day’s closing stock price. These two values, calculated for a given day, are used as training data inputs for our classification model.<br>A) Training: We train the classification model by using the following input values -</p><pre>[day_1 = {<br>    &#39;sentiment&#39;: sentiment_1,<br>    &#39;stock_change&#39;: change_value_1<br>},<br>day_2 = {<br>    &#39;sentiment&#39;: sentiment_2,<br>    &#39;stock_change&#39;: change_value_2<br>},<br>day_3 = {<br>    &#39;sentiment&#39;: sentiment_3,<br>    &#39;stock_change&#39;: change_value_3<br>}]</pre><p>If the change in stock price &gt; 0 (increase): stock_change is set to 1, and if the change in stock price ≤ 0 (decrease or no change): stock_change is set to 0.<br><br>B) Prediction: The sentiment score of all tweets about a company for a given day is used as input to the model, which is used to predict whether the stock price will increase or decrease. If the classification result is -<br>i) 1: Increase in stock price<br>ii) 0: Decrease in stock price</p><p>The following code can be used to predict the stock market behavior for day_x -</p><pre>clf = tree.DecisionTreeClassifier()<br>clf = clf.fit(X_Sentiment, Y_StockVal)<br>print(clf.predict[[sentiment_value_day_x]])</pre><p>The source code for the complete project can be found in this repository -<br><a href="https://github.com/nishantsahoo/Twitter-Stock-Analyzer">https://github.com/nishantsahoo/Twitter-Stock-Analyzer</a></p><p>I hope you found this article useful. :3<br>Give me a clap, or two, or forty (👏) if you want to read more such stuff from me.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a9c11f592ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Which movie should I watch?]]></title>
            <link>https://medium.com/@nishantsahoo/which-movie-should-i-watch-5c83a3c0f5b1?source=rss-c22f3d5bbc7b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c83a3c0f5b1</guid>
            <category><![CDATA[first-post]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Nishant Sahoo]]></dc:creator>
            <pubDate>Sun, 06 May 2018 02:37:00 GMT</pubDate>
            <atom:updated>2018-05-06T02:42:38.368Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_gFLhUX0FBv2IqZ380QEA.jpeg" /></figure><h4>Scraping Top 50 Movies on IMDb using BeautifulSoup, Python</h4><p>In this article, I will illustrate how to scrape a list of top 50 popular movies for each year from 1898–present as listed on IMDb, 2018.</p><h4>Web scraping is data scraping used for extracting data from websites. The legality of web scraping varies across the world. In general, web scraping may be against the <a href="https://en.wikipedia.org/wiki/Terms_of_use">terms of use</a> of some websites, but the enforce-ability of these terms is unclear. Always read the terms of use of the website you want to gather information from by data scraping.</h4><h4>Best Practices in Web Scraping -</h4><ol><li>Iterative: Always make sure, your code is as iterative as possible, keeping it dynamic, and not hard-coding any static values. This helps in cases where the website changes the number of items on their page keeping the structure same.</li><li>Compliant with Robots.txt and Terms &amp; Conditions: Don’t breach the implied contract, limits, permits, or prohibitions of web scraping that can be found in the terms and conditions and/or the robots.txt file.</li><li>Don’t Overburden the Website: Querying a website excessively will interfere with its normal processes, and slow down its performances. Make sure your queries aren’t excessive.</li><li>Use an API: If a site has the ability to download data via an API, obtain data that way, as opposed to scraping (even if there is a fee involved).</li></ol><h3>About <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">BeautifulSoup</a> -</h3><blockquote>Beautiful Soup is a Python package for parsing HTML and XML documents. It creates a parse tree for parsed pages that can be used to extract data from HTML, which is useful for web scraping. It is available for Python 2.6+ and Python 3.</blockquote><p>Let’s have a look at the basic structure of any website -</p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>  &lt;head&gt;<br>    &lt;title&gt;Page Title&lt;/title&gt;<br>  &lt;/head&gt;</pre><pre>&lt;body&gt;<br>    .<br>    .<br>    .<br>  &lt;/body&gt;<br>&lt;/html&gt;</pre><p>Every page starts and ends with an <em>html </em>tag. It has two components: <em>head,</em> and <em>body. Head </em>section contains the title of the page. <em>Body </em>section is where the content of the page lies.</p><h4>Basic usage of BeautifulSoup -</h4><pre>import urllib3<br>from bs4 import BeautifulSoup</pre><pre>url = &quot;enter_url_here&quot;<br>ourUrl = urllib3.PoolManager().request(&#39;GET&#39;, url).data<br>soup = BeautifulSoup(ourUrl, &quot;lxml&quot;)<br>print(soup.find(&#39;title&#39;).text)</pre><p>The above code prints the title of a website.</p><p>An example code to access a particular HTML tag with specific attributes in BeautifulSoup, say — “ &lt;div class=’upper’&gt; … &lt;/div&gt; “ using the <strong><em>find</em></strong> function -</p><pre>htmlEle = soup.find(&#39;div&#39;, attrs={&#39;class&#39;: &#39;upper&#39;})</pre><p>Explore findChildren, and findAll functions in the official documentation page of <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">BeautifulSoup</a>.<br><em>Bonus: </em><strong><em>findAll</em></strong> function returns a list of html elements of the same type.</p><p>I will be using the following URL template to access each page having top 50 movies for a particular <em>year</em>.</p><pre>url = “<a href="http://www.imdb.com/search/title?release_date=">http://www.imdb.com/search/title?release_date=</a>&quot; + year + “,” + year + “&amp;title_type=feature”</pre><p>Let’s have a look at the basic HTML structure of the list of movies for the above URL-<br>(can be viewed on the browser by opening the developer tools; ctrl+shift+i)</p><pre>&lt;div class = &quot;lister-list&quot;&gt;</pre><pre>  &lt;div class = &quot;lister-item mode-advanced&quot;&gt;<br>    .<br>    .<br>    &lt;div class = &quot;lister-item-content&quot;&gt;<br>      &lt;h3 class = &quot;lister-item-header&quot;&gt;<br>        ...<br>        &lt;a href = &quot;/title/...&quot;&gt;&lt;/a&gt; --&gt; Movie title<br>        ...<br>      &lt;/h3&gt;<br>    &lt;/div&gt;<br>    .<br>    .<br>  &lt;/div&gt;</pre><pre>  &lt;div class = &quot;lister-item mode-advanced&quot;&gt;<br>    .<br>    .<br>    .<br>  &lt;/div&gt;</pre><pre>  &lt;div class = &quot;lister-item mode-advanced&quot;&gt;<br>    .<br>    .<br>    .<br>  &lt;/div&gt;</pre><pre>  .<br>  .<br>  .</pre><pre>&lt;/div&gt;</pre><p>Code used to scrape the title of each movie for the above HTML structure -</p><pre>i = 1<br>movieList = soup.findAll(&#39;div&#39;, attrs={&#39;class&#39;: &#39;lister-item mode-advanced&#39;})</pre><pre>for div_item in tqdm(movieList):</pre><pre>    div = div_item.find(&#39;div&#39;,attrs={&#39;class&#39;:&#39;lister-item-content&#39;})<br>    print str(i) + &#39;.&#39;,</pre><pre>    header = div.findChildren(&#39;h3&#39;,attrs={&#39;class&#39;:&#39;lister-item-header&#39;})</pre><pre>    print &#39;Movie: &#39; + str((header[0].findChildren(&#39;a&#39;))[0].contents[0].encode(&#39;utf-8&#39;).decode(&#39;ascii&#39;, &#39;ignore&#39;))</pre><pre>    i += 1</pre><p>Example output as on 6th May, 2018 for Top 50 movies in 2018 -</p><pre>Most Popular Feature Films Released 2018-01-01 to 2018-12-31 : <br>1. Movie: Avengers: Infinity War<br>2. Movie: Venom<br>3. Movie: A Quiet Place<br>4. Movie: Black Panther<br>5. Movie: I Feel Pretty<br>6. Movie: Deadpool 2<br>7. Movie: Ready Player One<br>8. Movie: Super Troopers 2<br>9. Movie: Rampage<br>10. Movie: Den of Thieves<br>11. Movie: Truth or Dare<br>12. Movie: Red Sparrow<br>13. Movie: Pacific Rim: Uprising<br>14. Movie: Halloween<br>15. Movie: Jurassic World: Fallen Kingdom<br>16. Movie: Blockers<br>17. Movie: Traffik<br>18. Movie: Isle of Dogs<br>19. Movie: 12 Strong<br>20. Movie: Solo: A Star Wars Story<br>21. Movie: Peter Rabbit<br>22. Movie: Dude<br>23. Movie: Maze Runner: The Death Cure<br>24. Movie: Crazy Rich Asians<br>25. Movie: Deep Blue Sea 2<br>26. Movie: The Commuter<br>27. Movie: The Week Of<br>28. Movie: Tully<br>29. Movie: Bohemian Rhapsody<br>30. Movie: Annihilation<br>31. Movie: The Equalizer 2<br>32. Movie: Winchester<br>33. Movie: Love, Simon<br>34. Movie: The Predator<br>35. Movie: Overboard<br>36. Movie: Aquaman<br>37. Movie: Bharat Ane Nenu<br>38. Movie: The Meg<br>39. Movie: Don&#39;t Worry, He Won&#39;t Get Far on Foot<br>40. Movie: Ant-Man and the Wasp<br>41. Movie: Mile 22<br>42. Movie: The Guernsey Literary and Potato Peel Pie Society<br>43. Movie: Hereditary<br>44. Movie: I Can Only Imagine<br>45. Movie: The Titan<br>46. Movie: Incredibles 2<br>47. Movie: Forever My Girl<br>48. Movie: Wildling<br>49. Movie: Life of the Party<br>50. Movie: Ocean&#39;s 8</pre><p>To run this program yourself -<br>1. Clone this repo - <a href="https://github.com/nishantsahoo/IMDB_Top50_Scrape">https://github.com/nishantsahoo/IMDB_Top50_Scrape</a></p><p>2. Navigate to the project folder and run the following in the command line</p><pre>python IMDB_Top50_Scraper.py</pre><p>I hope you found this article useful. :3<br>Give me a clap, or two, or forty (👏) if you want to read more such stuff from me.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c83a3c0f5b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>