<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Saqlain Pasha on Medium]]></title>
        <description><![CDATA[Stories by Saqlain Pasha on Medium]]></description>
        <link>https://medium.com/@saqlainpasha1921?source=rss-1eab73e8f67e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*CBmDGo-gQrvjWQcY.jpg</url>
            <title>Stories by Saqlain Pasha on Medium</title>
            <link>https://medium.com/@saqlainpasha1921?source=rss-1eab73e8f67e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:51:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@saqlainpasha1921/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How i created a simple Stackoverflow related tags search webapp.]]></title>
            <link>https://medium.com/@saqlainpasha1921/how-i-created-a-simple-stackoverflow-related-tags-search-webapp-a4461998c0e?source=rss-1eab73e8f67e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4461998c0e</guid>
            <category><![CDATA[webapplicationdevelopment]]></category>
            <category><![CDATA[heroku]]></category>
            <category><![CDATA[stackoverflow]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[flask]]></category>
            <dc:creator><![CDATA[Saqlain Pasha]]></dc:creator>
            <pubDate>Sun, 02 Aug 2020 14:28:09 GMT</pubDate>
            <atom:updated>2020-08-02T15:44:45.106Z</atom:updated>
            <content:encoded><![CDATA[<p>I will be describing how i created a simple web app to find similar tags in stackoverflow. Stackexchange api along with requests module and beautifulsoup module is used to scrape data from stackoverflow. Networkx is used to create a weighted graph of these tags. Flask is used to create a webapp .This webapp is deployed to the web through Heroku platform.</p><p>This project is divided into 5 steps and each of these steps is discussed .<br> <em>1.Scraping stackoverflow for tags.<br> 2.Building a weighted graph using those tags.<br> 3.Getting related tags.<br> 4.Creating a webapp.<br> 5.Deploying it to the web.</em><br> The final product will look like <a href="https://related-tag-search.herokuapp.com/">this</a> (yeah i know my css is bad).<br> <br> Software requirements <br> Python 3.6+<br> Requests<br> Bs4<br> Flask<br> Networkx<br> Json<br> <br>Git and The Heroku CLI should also be installed.</p><h4>Step 1: Scraping stackoverflow for tags .</h4><p>A tag is a word or phrase that describes the topic of the question. Tags are a means of connecting experts with questions they will be able to answer by sorting questions into specific, well-defined categories.Tags can also be used to help you identify questions that are interesting or relevant to you.Each question in stackoverflow can contain no more than 5 tags.</p><p><a href="https://api.stackexchange.com/docs">Stackexchange api </a>is used for the url which would look like <em>‘</em><a href="https://api.stackexchange.com/2.2/questions?page=1&amp;pagesize=100&amp;order=desc&amp;sort=activity&amp;site=stackoverflow&#39;"><em>https://api.stackexchange.com/2.2/questions?page=1&amp;pagesize=100&amp;order=desc&amp;sort=activity&amp;site=stackoverflow&#39;</em></a><em>.</em> Here page is the page number in stackoverflow and pagesize is the maximum number of questions in the page (100 in each page).I will be scraping 100 pages which will get me tags of around 10,0000 questions. Once the raw data is scraped, tags are to be separated from it and cleaned .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*R6B0823nfoQKUqZtyUaiGQ.png" /><figcaption>Raw data of a single question</figcaption></figure><p>Tags of each question will be stored in a list and all of those lists will be stored in cleaned_tags_list.</p><pre>def scrape_tags(url, cleaned_tags_list):<br>    req = requests.get(url)  <br>    html_doc = req.text <br>    soup = BeautifulSoup(html_doc, &#39;lxml&#39;)<br>    paragraph = soup.find_all(&#39;p&#39;)       #finding the content within all of the paragraph <br>    string = str(paragraph)               #converting it to string to work with<br>    tag_split = string.split(&#39;tags&#39;)    <br>    tag_list = []                  #list to store the tags<br>    for i in range(1,len(tag_split)):<br>        tags = tag_split[i].split(&#39;owner&#39;)[0][2:-2]<br>        tag_list.append(tags)<br>    clean_tags(tag_list, cleaned_tags_list)<br>    <br>def clean_tags(tag_list, cleaned_tags_list):<br>    for tags in tag_list:<br>        tags = tags.split(&#39;,&#39;)<br>        temp_list = [] #a temporary list to store cleaned tags<br>        for tag in tags:<br>            temp_list.append(tag.replace(&#39;&quot;&#39;,&#39;&#39;).replace(&#39;[&#39;,&#39;&#39;).replace(&#39;]&#39;,&#39;&#39;))<br>        cleaned_tags_list.append(temp_list)</pre><pre>cleaned_tags_list = []  #list to stores tags of all the scraped questions<br>for page in range(1,101):<br>    url = &#39;<a href="https://api.stackexchange.com/2.2/questions?page=&#39;+str(page)+&#39;&amp;pagesize=100&amp;order=desc&amp;sort=activity&amp;site=stackoverflow&#39;">https://api.stackexchange.com/2.2/questions?page=&#39;+str(page)+&#39;&amp;pagesize=100&amp;order=desc&amp;sort=activity&amp;site=stackoverflow&#39;</a><br>    scrape_tags(url, cleaned_tags_list)<br>cleaned_tags_list[:5]</pre><p>cleaned_tags_list would look like</p><pre>[[&#39;javascript&#39;,&#39;node.js&#39;,&#39;express&#39;],[&#39;c++&#39;],[&#39;c++&#39;,&#39;visual-studio&#39;,&#39;include&#39;],[&#39;python-3.x&#39;,&#39;praw&#39;],[&#39;c#&#39;&#39;,&#39;asp.net-mvc&#39;,&#39;list&#39;]]</pre><h4>Step 2 : Building a weighted graph using those tags.</h4><p><a href="https://networkx.github.io/documentation/stable/tutorial.html">Networkx </a>,a python module for creating graphs is used . The tags will be the nodes and there will be a edge between two tags ,if they are from the same question (i.e from the same list). A weight of 1 will be initialized while creating a new edge between two tags. If an edge is already present ,the weight of the edge will be incremented.</p><pre>#creating a weighted graph using the tags     <br>tag_graph = nx.Graph()<br>for tag_list in cleaned_tags_list:<br>    #if there is only one tag in the list add it to the node if it is not already present<br>    if len(tag_list) == 1: <br>        if tag_list[0] not in list(tag_graph.nodes()):<br>                tag_graph.add_node(tag_list[0])<br>    else:    <br>        while(len(tag_list)&gt;1):<br>            tag = tag_list.pop(0) #taking a tag<br>            if tag not in list(tag_graph.nodes()): #checking if the tag is already present in the graph<br>                tag_graph.add_node(tag) #adding the tag as a node<br>            for temp_tag in tag_list:   #taking the remaining tags <br>                if (tag, temp_tag) not in tag_graph.edges([tag]): #checking if there is already a edge present between the two tags<br>                    tag_graph.add_edge(tag,temp_tag,weight=1) #adding an edge initialized with weight 1<br>                else: #if there is already an edge present<br>                    tag_graph[tag][temp_tag][&#39;weight&#39;] += 1 #increamenting the weight<br>        if tag_list[0] not in list(tag_graph.nodes()): #adding the last node in the current list to the graph if already not present<br>                tag_graph.add_node(tag_list[0])</pre><h4>Step 3 : Getting related tags.</h4><p>The graph is serialized into a json dictionary.Nodes and Edges are separated using that dictionary and stored in a nodes.txt and edges.txt files. These .txt files are used in the webapp instead of creating graph every time a request comes to the webapp. When a tag is given all of its edges using the edges.txt are found and sorted based on their weights. Top 10 tags with the highest weights are returned. If a new tag which is not present in the graph is given an error message is returned.</p><pre>#storing the graph in a dictonary to reuse the graph<br>serialized_graph = nx.readwrite.json_graph.node_link_data(tag_graph)<br>#nodes is a list of all the nodes in the graph<br>nodes = serialized_graph[&#39;nodes&#39;]<br>#edges is a list of all the edges in the graph<br>edges  = serialized_graph[&#39;links&#39;]<br>#storing the nodes in a text file<br>with open(&#39;nodes.txt&#39;, &#39;w&#39;) as outfile:<br>    json.dump(edges, outfile)<br>#storing the edges in a text file<br>with open(&#39;edges.txt&#39;, &#39;w&#39;) as outfile:<br>    json.dump(edges, outfile)<br>#function used in app.py to get related tags<br>def related_tags(edges_list, tag):<br>    max_related_tags = 10<br>    tag_weight = []<br>    for edge in edges_list:<br>        if edge[&#39;source&#39;] == tag:<br>            temp = [] #temporary list<br>            temp.append(edge[&#39;target&#39;])<br>            temp.append(edge[&#39;weight&#39;])<br>            tag_weight.append(temp)<br>        elif edge[&#39;target&#39;] == tag:<br>            temp = []<br>            temp.append(edge[&#39;source&#39;])<br>            temp.append(edge[&#39;weight&#39;])<br>            tag_weight.append(temp)    <br>    sorted_weight_tag = sorted(tag_weight , key = lambda x:x[1] , reverse = True) #sorting the weights in decreasing order<br>    top_related_tags_list = [x[0] for x in sorted_weight_tag] #getting the realted tags based on the weights<br>    if len(top_related_tags_list) &lt; max_related_tags: #if the total number of related tags is less than what we require <br>        return top_related_tags_list<br>    else:<br>        return top_related_tags_list[:max_related_tags]</pre><h4>Step 4 : Creating a webapp .</h4><p>Flask is used for creating the webapp. The two things which we need to begin with are :<br> 1)Python code to get user input from the web form , find related tags and return the results.<br> 2)The HTML templates that flask will render. This allows the user to input their own data and present the results.</p><p>The app will be structured like this:</p><pre>webapp/<br>    ├── templates/<br>    │   └──  main.html<br>    ├── app.py<br>    │   <br>    └── nodes.txt<br>    └── edges.txt</pre><p><strong>app.py</strong></p><p>This is the core of the webapp,which will run on the Heroku server ,send out webpages and process inputs from the user. This is how it looks like.</p><pre>import flask<br>import json</pre><pre>app = flask.Flask(__name__, template_folder=&#39;templates&#39;)</pre><pre>def related_tags(edges_list, tag):<br>    max_related_tags = 10<br>    tag_weight = []<br>    for edge in edges_list:<br>        if edge[&#39;source&#39;] == tag:<br>            temp = []<br>            temp.append(edge[&#39;target&#39;])<br>            temp.append(edge[&#39;weight&#39;])<br>            tag_weight.append(temp)<br>        elif edge[&#39;target&#39;] == tag:<br>            temp = []<br>            temp.append(edge[&#39;source&#39;])<br>            temp.append(edge[&#39;weight&#39;])<br>            tag_weight.append(temp)    <br>    sorted_weight_tag = sorted(tag_weight , key = lambda x:x[1] , reverse = True) #sorting the weights in decreasing order<br>    top_related_tags_list = [x[0] for x in sorted_weight_tag] #getting the realted tags based on the weights<br>    if len(top_related_tags_list) &lt; max_related_tags:<br>        return top_related_tags_list<br>    else:<br>        return top_related_tags_list[:max_related_tags]</pre><pre><a href="http://twitter.com/app">@app</a>.route(&#39;/&#39;, methods=[&#39;GET&#39;, &#39;POST&#39;])<br>def main():<br> if flask.request.method == &#39;GET&#39;:<br>  return(flask.render_template(&#39;main.html&#39;))<br> if flask.request.method == &#39;POST&#39;:<br>  tag_given = flask.request.form[&#39;Tag&#39;]<br>  with open(&#39;nodes.txt&#39;) as json_file:<br>   node_dict = json.load(json_file)<br>  node_list = []<br>  for node in node_dict:<br>   node_list.append(node[&#39;id&#39;])<br>  if tag_given not in node_list:<br>   return flask.render_template(&#39;main.html&#39;,<br>                                     original_input={&#39;Given tag&#39;:tag_given },<br>                                     result=[&#39;Error : Given tag not present&#39;],<br>                                     )</pre><pre>with open(&#39;edges.txt&#39;) as json_file:<br>   edge_list = json.load(json_file)</pre><pre>top_related_tags_list =  related_tags(edge_list,tag_given)<br>  return flask.render_template(&#39;main.html&#39;,<br>                                     original_input={&#39;Given tag&#39;:tag_given },<br>                                     result=top_related_tags_list,<br>                                     )<br>if __name__ == &#39;__main__&#39;:<br>    app.run(debug = True)</pre><p>Going to the base URL will trigger the main() function .The app will run in two modes. In one, it will be displaying the input form to the user. In the other, it will be retrieving that input. This uses two different HTTP methods: <em>GET </em>and <em>POST</em>. If a user loads the main URL for the app (e.g. <a href="http://myapp.herokuapp.com/">http://myapp.herokuapp.com/</a>), flask will receive a GET request and render <em>main.html</em>. If the user fills in the form on the page and clicks on a <em>Get related</em> <em>tags</em> button, flask receives a POST request, extracts the input, get related tags of the input and will finally render main.html with the results in place. The user can then view that result and submit more data for processing.<br>When the main function is triggered through POST method ,the nodes.txt is loaded and made sure that the given tag is present in the graph .If it is not present an error message is returned. If it is present edges.txt is loaded.Using this file all the edges of the given tag are extracted and sorted based on their weights in the <em>related_tags</em> function and the top most related tags are returned.</p><p><strong>main.html</strong></p><p>This has a form through which the user can provide input (tag).Minimal CSS is used as i dont know much CSS (you can improve the CSS part ). The action attribute of the form should have ‘<a href="http://localhost:5000&#39;">http://localhost:5000&#39;</a> if you want to run this in your local system . While running on the web ,it should contain the url of your webapp.(e.g <a href="https://related-tag-search.herokuapp.com/">https://related-tag-search.herokuapp.com/</a>).</p><pre>&lt;!doctype html&gt;<br>&lt;html&gt;<br>&lt;style&gt;<br><a href="http://twitter.com/import">@import</a> &quot;<a href="http://fonts.googleapis.com/css?family=Ubuntu">http://fonts.googleapis.com/css?family=Ubuntu</a>&quot;;<br>body{<br>margin:0;<br>padding:0;<br>font-family:&#39;Ubuntu&#39;,sans-serif<br>}<br>#first{<br>width:400px;<br>height:610px;<br>box-shadow:0 0 15px rgba(14,41,57,0.12),0 2px 5px rgba(14,41,57,0.44),inset 0 -1px 2px rgba(14,41,57,0.15);<br>float:left;<br>padding:10px 50px 0;<br>margin:2% 30%;<br>border-radius:5px<br>}<br>hr{<br>border-top:1px solid #ccc;<br>margin-bottom:30px<br>}<br>span{<br>float:left;<br>margin-top:8px;<br>font-size:17px<br>}<br>.one{<br>display:block;<br>margin-left:22px;<br>margin-top:10px<br>}<br>.submit{<br>width:100px;<br>margin-left:83px;<br>padding:10px;<br>margin-top:20px;<br>background:linear-gradient(#A4A4A4,#424242);<br>color:#fff;<br>font-weight:700<br>}<br>.submit:hover{<br>background:linear-gradient(#424242,#A4A4A4)<br>}<br>.result {<br>    margin: auto;<br>    width: 80%;<br>    border: 2px solid #ccc;<br>}<br>&lt;/style&gt;<br>&lt;head&gt;<br>    &lt;title&gt;Stackoverflow related tags search &lt;/title&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>&lt;div id = &quot;first&quot;&gt;<br>&lt;form action=&quot;{{url for &#39;main&#39;}}&quot; method=&quot;POST&quot;&gt;<br> &lt;h2 align=&quot;center&quot;&gt;Stackoverflow related tags search&lt;/h2&gt;<br> &lt;h4&gt;Please enter the tag&lt;/h4&gt;<br> &lt;label class = &quot;one&quot;&gt;<br>        &lt;span&gt;Tag :&lt;/span&gt;&amp;nbsp;<br>        &lt;input name=&quot;Tag&quot; type=&quot;text&quot; required&gt;<br>    &lt;/label&gt;<br>        &lt;br&gt;<br>        &lt;input type=&quot;submit&quot; value=&quot;Get related tags&quot;&gt;<br>    &lt;/form&gt;</pre><pre>&lt;div class=&quot;result&quot; &gt;<br>    {% if result %}<br>        {% for variable, value in original_input.items() %}<br>           &lt;b&gt;{{ variable }}&lt;/b&gt; : {{ value }}<br>        {% endfor %}<br>        &lt;p style=&quot;font-size:20px&quot; &gt;Related Tags :&lt;/p&gt;<br>           <br>         {% for variable in result %}<br>            &lt;p style=&quot;font-size:15px&quot; align=&quot;center&quot;&gt;{{ variable }}&lt;/p&gt;<br>         {% endfor %}<br>         {% endif %}    <br>&lt;/div&gt;<br>&lt;/div&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><h4>Step 5: Deploying it to the web.</h4><p>Now there is a functioning webapp on the local machine .I will deploy it to the web through Heroku . <br>Install git and Heroku CLI tool and signup for a free Heroku account.</p><p>The follwing steps will deploy the app to the web :<br>1.Inside the webapp folder, run the following command to create a git repositary for the webapp.</p><pre>git init</pre><p>2.Next log into your Heroku account using the CLI.</p><pre>heroku login</pre><p>3.Create a new Heroku app .Here related-tag-search is the name given to the webapp.</p><pre>heroku create related-tag-search</pre><p>4.Create the required Heroku files<br>Two files are required ,which are to be placed inside the webapp folder. i)<em>requirements.txt </em>— this tells Heroku which packages to install for your web app. It should look like this:</p><pre>flask<br>requests<br>networkx<br>gunicorn<br>bs4</pre><p>2)<em>Procfile </em>— this tells Heroku what kind of app you are running and how to serve it to users. It is a single line and should look like this:</p><pre>web: gunicorn app:app</pre><p>The webapp folder will finally look like this:</p><pre>       webapp/<br>            ├── templates/<br>            │   └── main.html<br>            ├── requirements.txt<br>            ├── Procfile<br>            └── app.py<br>            ├── nodes.txt<br>            └── edges.txt</pre><p>5.Add all the files to the git repositary</p><pre>git add .<br>git commit -m “First commit!”</pre><p>6.Set the remote destination for pushing from git to Heroku</p><pre>heroku git:remote -a related-tag-search</pre><p>7.Push the app to the web</p><pre>git push heroku master</pre><p>Heroku will upload your app files, install the packages it needs and start the app running.If everything goes as expected, you’ll see output showing things being installed and uploaded.</p><p>8.To open the deployed app run the following command.</p><pre>heroku open</pre><p>Alternatively going to <a href="https://related-tag-search.herokuapp.com/">https://related-tag-search.herokuapp.com/</a> will open the app.</p><p><strong>Further improvements</strong></p><ol><li>Using machine learning (specifically word2vec ) to find related tags instead of weighted graph.</li><li>Imroving the CSS part.</li></ol><p>The inspiration to this project is from <a href="https://blog.cambridgespark.com/deploying-a-machine-learning-model-to-the-web-725688b851c7">https://blog.cambridgespark.com/deploying-a-machine-learning-model-to-the-web-725688b851c7</a> .</p><p>The entire code for this project can be found here <a href="https://github.com/SaqlainAI/Stackoverflow-related-tags-search-webapp/">https://github.com/SaqlainAI/Stackoverflow-related-tags-search-webapp/</a> .</p><p>I would like to thank my mentor <em>Yavar Hussain</em> for guiding me in this project and Ureka Educational Group for giving me this internship opportunity ,where i was able to learn a lot of good stuff .</p><p>Connect with Ureka in linkedin <a href="https://www.linkedin.com/company/ureka-limited/">https://www.linkedin.com/company/ureka-limited/</a></p><p>Connect with me in linkedin <a href="https://www.linkedin.com/in/saqlain-pasha-388100158/">https://www.linkedin.com/in/saqlain-pasha-388100158/</a> .</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4461998c0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>