<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rifai Slamet on Medium]]></title>
        <description><![CDATA[Stories by Rifai Slamet on Medium]]></description>
        <link>https://medium.com/@rifaislamet1509?source=rss-bd872fe830e3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*ApOjCSG0kByMkHFgmyCpAg.jpeg</url>
            <title>Stories by Rifai Slamet on Medium</title>
            <link>https://medium.com/@rifaislamet1509?source=rss-bd872fe830e3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:07:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rifaislamet1509/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Bypass reCAPTCHA  to Scraping Australian Yellowpages using Python]]></title>
            <link>https://medium.com/@rifaislamet1509/bypass-recaptcha-to-scraping-australian-yellowpages-using-python-9330101e99a3?source=rss-bd872fe830e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9330101e99a3</guid>
            <category><![CDATA[captcha]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[tricks]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <dc:creator><![CDATA[Rifai Slamet]]></dc:creator>
            <pubDate>Sun, 26 Apr 2020 23:35:37 GMT</pubDate>
            <atom:updated>2020-04-26T23:35:37.326Z</atom:updated>
            <content:encoded><![CDATA[<h3>Bypass reCAPTCHA to Scraping Australian Yellowpages using Python</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gsTgiEDNKT35ylJ7q3Cf2g.png" /></figure><p>Dear readers, do you want to scrape a website but you can’t get the data you want because the web using reCAPTCHA security like <a href="http://yellowpages.com.au/"><strong>yellowpages.com.au</strong></a>? If you request on your python project, the output will be like this:</p><p>Don’t worry Dude, on this occasion I will share to you how to overcome it so you can scrape the website easily.</p><h3>What is reCAPTCHA?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fmy7ltvFeTMjktrD" /><figcaption>the image from <a href="https://id.wordpress.org/plugins/login-recaptcha/">https://id.wordpress.org/plugins/login-recaptcha/</a></figcaption></figure><p>reCAPTCHA is a free service that protects your website from spam and abuse. reCAPTCHA uses an advanced risk analysis engine and adaptive challenges to keep automated software from engaging in abusive activities on your site. It does this while letting your valid users pass through with ease. <strong><em>(Source: </em></strong><a href="https://www.google.com/recaptcha/intro/v3.html"><strong><em>https://www.google.com/recaptcha/intro/v3.html</em></strong></a><strong><em>)</em></strong></p><h4>Case study</h4><p>We will do web scraping from a website that uses <strong>reCAPTCHA</strong> so we must solve it first so we can enter the web. This time, I will give an example of one website that implements it, yellowpages.com.au. You can use other websites that implement the<strong> reCAPTCHA</strong> too.</p><h4>Preparation</h4><p>But Before starting, make sure you are ready with a new project, click <a href="https://medium.com/@rifaislamet1509/how-to-install-pycharm-ide-for-windows-and-setting-the-virtual-environment-7e2e0aa66541"><strong>here</strong></a> if you need a way to prepare a new project. Install the “<strong>bs4</strong>&quot; and “<strong>requests”</strong> libraries then import them to the project sheet.</p><pre><strong>from </strong>bs4 <strong>import </strong>BeautifulSoup<br><strong>import </strong>requests</pre><h3><strong>Let’s Get Started</strong></h3><h3>STEP 1: Get The Header Request</h3><p>Open the website and solve the<strong> reCAPTCHA</strong>. Then, enter the term location that you want to search. For example, I will look for designers in the Canberra area.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ubHA2524cqmCdEK2BAoOAA.png" /></figure><p>Inspect the web page then go to the network, then:</p><ol><li>Go to Network</li><li>Refresh the page</li><li>Search the target title on the search box.</li><li>Click the target request</li><li>Copy the request URL</li></ol><p>Yeah, we got the request header now… Next…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zpDNeVg3Ygb6y96EyIaoSw.png" /></figure><p>Then, open your project and create a new variable as the request URL:</p><pre>page = 1<br>url =<strong>f&quot;https://www.yellowpages.com.au/search/listings?clue=Designer&amp;eventType=pagination&amp;locationClue=Canberra&amp;pageNumber=</strong>{page}<strong>&amp;referredBy=www.yellowpages.com.au&quot;</strong></pre><p>Create a page variable to make it easier to access page numbers. If you want to retrieve data from many pages, just loop on that page variable</p><h3>STEP 2: Get the Header Request</h3><p>Just scroll down the step 1 result, then find the <strong>“Request Headers”</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KVHrvN6HFqNPHHBJWXaNcg.png" /></figure><p>Append all of the request headers to the <strong>“headers”</strong> variable.</p><pre>headers= {<strong>&#39;user-agent&#39;</strong>: <strong>&#39;&lt;enter your user agent&gt;&#39;</strong>,<br>          <strong>&#39;accept&#39;</strong>: <strong>&#39;text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9&#39;</strong>,<br><strong>&#39;accept-encoding&#39;</strong>: <strong>&#39;gzip, deflate, br&#39;</strong>,<br><strong>&#39;accept-language&#39;</strong>: <strong>&#39;en-US,en;q=0.9&#39;</strong>,<br><strong>&#39;cache-control&#39;</strong>: <strong>&#39;no-cache&#39;</strong>,<br><strong>&#39;cookie&#39;</strong>: <strong>&#39;_vwo_uuid_v2=DD8BABFDB0294210CDCAF95FD168FFF90|380874a2c78231991422698ff482667e; s_ecid=MCMID%7C54504064614746727123134366449041278759; _vwo_uuid=DD8BABFDB0294210CDCAF95FD168FFF90; _fbp=fb.2.1586234672325.905912023; _hjid=77c2aace-3aec-4200-a7b6-6f7222f0441f; _wingify_pc_uuid=830fa0b0345a40dd9171c21ec886428f; wingify_donot_track_actions=0; JSESSIONID=19B968432AC01A28817B26D0673DF141; yellow-guid=0f82ab9a-02a9-49fc-9cab-4acb666683f5; AMCVS_8412403D53AC3D7E0A490D4C%40AdobeOrg=1; AMCV_8412403D53AC3D7E0A490D4C%40AdobeOrg=1585540135%7CMCIDTS%7C18377%7CMCMID%7C54504064614746727123134366449041278759%7CMCAAMLH-1588294836%7C3%7CMCAAMB-1588294836%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1587697236s%7CNONE%7CMCAID%7CNONE%7CvVersion%7C4.4.0; _vis_opt_s=4%7C; _vis_opt_test_cookie=1; s_cc=true; _vis_opt_exp_76_combi=2; clue=Hotel; locationClue=brisbane; _vwo_ds=3%3Aa_0%2Ct_0%3A0%241586234651%3A48.22533329%3A%3A9_0%2C6_0%2C5_0%2C4_0%2C3_0%2C15_0%2C14_0%2C13_0%2C12_0%2C11_0%3A3_0%2C2_0%3A0; s_sq=%5B%5BB%5D%5D; _sdsat_Postcode=; _vwo_sn=1455318%3A4; _awl=3.1587690218.0.4-915a1a35-aac5594624c22b1ca9aebfd2425a235a-6763652d617369612d6561737431-5ea23aea-0&#39;</strong>,<br><strong>&#39;pragma&#39;</strong>: <strong>&#39;no-cache&#39;</strong>,<br><strong>&#39;sec-fetch-dest&#39;</strong>: <strong>&#39;document&#39;</strong>,<br><strong>&#39;sec-fetch-mode&#39;</strong>: <strong>&#39;navigate&#39;</strong>,<br><strong>&#39;sec-fetch-site&#39;</strong>: <strong>&#39;none&#39;</strong>,<br><strong>&#39;sec-fetch-user&#39;</strong>: <strong>&#39;?1&#39;</strong>,<strong>&#39;upgrade-insecure-requests&#39;</strong>: <strong>&#39;1&#39;</strong>}</pre><p>That is the trick to bypass the <strong>reCAPTCHA.</strong></p><h3>STEP 3: Check the Response</h3><p>Check the response text using the <strong>requests</strong> module and also check whether the response can be read by BS4 or not using the <strong>BeautifulSoup</strong> module.</p><p>if we don’t use the headers variable:</p><pre>res = requests.get(url)<br>print(res)</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u1yGw6U4BVP-MEYb2Skq9w.png" /></figure><p>Oh no, we were detected as robots and had to solve the reCAPTCHA. It will not be able to do a scraping to retrieve data from the page.</p><p>But what if we use the header variable? let’s execute it</p><pre>res = requests.get(url, headers=headers)<br>print(res)</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OAn2JXsuEgSnmuQdcoY7rA.png" /></figure><p>Perfect, the <strong>reCAPTCHA</strong> is not detected and we can enter the web page without being detected as a robot. COOL Right?</p><h3>STEP 4: Get the Target Area Element</h3><p>Inspect the webpage then looking for the element of the target area</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c5xhui_XGUgAhC0NQml-Ew.png" /></figure><p>Then enter the “items” variable, and loop the variable so that each item can be executed. In the loop area, enter any variable that represents the data field that we want to scrape, for example, I want to scrape the name of the term and don’t forget to print the variable so the python code is:</p><pre>items = soup.findAll(<strong>&#39;div&#39;</strong>,<strong>&#39;search-contact-card-table-div cag-groups-2 cag-items-4&#39;</strong>)<br><strong>for </strong>item <strong>in </strong>items:<br>    names = item.find(<strong>&#39;a&#39;</strong>,<strong>&#39;listing-name&#39;</strong>).text<br>    print(name)</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zBxLATTpsLC5oueuK19IZQ.png" /></figure><p>Perfect, we got the title of the term. You can develop the project to scrape another data field. And if you want to save data to the CSV file, you can see the step here. Good luck! ☺</p><p>Okay, this is the end of my tutorial. I hope we can take a good lesson from what I deliver today. I ask forgiveness for any words and behave which are not supposed to be. Thank you for your kind attention Guys. Stay tuned for my next articles!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9330101e99a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solved! Scraping All of Data from Google CSE Using Python]]></title>
            <link>https://medium.com/@rifaislamet1509/solved-scraping-all-of-data-from-google-cse-using-python-13d3e6f12413?source=rss-bd872fe830e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/13d3e6f12413</guid>
            <category><![CDATA[json]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Rifai Slamet]]></dc:creator>
            <pubDate>Sat, 25 Apr 2020 06:28:52 GMT</pubDate>
            <atom:updated>2020-04-26T23:10:26.592Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xUXePhZjwXsOGXmS2Bet1A.png" /></figure><p>Hi everyone, do you want to scrape any website? but you can’t get the HTML page source because the web uses requests from Google CSE like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E40rhtBAxJUooobIrQYVEQ.png" /></figure><p>The response can’t be parsed because the target is not found :(</p><p>Don’t worry, on this occasion I will share to you how to get the response so you can scrape the website easily.</p><h4>Case study</h4><p>We will do web scraping using Google CSE as a request from the web. this time I will give an example of one website that implements it, kompas.com. You can use other websites that implement Google CSE too.</p><h4>Preparation</h4><p>But Before starting, make sure you are ready with a new project, click <a href="https://medium.com/@rifaislamet1509/how-to-install-pycharm-ide-for-windows-and-setting-the-virtual-environment-7e2e0aa66541"><strong>here</strong></a> if you need a way to prepare a new project. If you are ready?</p><h3>Let’s get started!</h3><h3>STEP 1: Get The Search Engine ID</h3><p>Enter the topic you want to scrape in the search bar, then click the <strong>search icon.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dag9xmEXudLnLAqA5dvF6w.png" /></figure><p>Then, let’s inspect the webpage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qzCv6KDuixGDxj3wn-XNZA.png" /></figure><p>Go to the <strong>network </strong>and refresh the webpage, click the refresh icon, or press the “<strong>F5</strong>” button on your keyboard. Then, find the title element then click it.</p><p>Click Header, then looking for the cx value. This is the search engine id</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*66JwfiGak12R6b02FarvPw.png" /></figure><h3>STEP 2: Get the API Key</h3><p>Click <a href="https://developers.google.com/custom-search/v1/overview"><strong>here</strong></a> to get the API key. Then click the “<strong>Get Key”</strong> button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*28ztiXzb59R6M1PZcHFBHQ.png" /></figure><p>After that, add the new project then click the <strong>“OK”</strong> button, so we will see our API key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pq4b77B3HQCZ4-KNZV4ZDg.png" /></figure><h3>STEP 3: Get The Response from Google APIs</h3><p>We must ensure that the API key and Custom Search ID that we use can be read. To do that, enter at least three parameters (API key, search engine id, and query) in the Google API URL:</p><pre>url = &#39;<strong>https://www.googleapis.com/customsearch/v1?&#39;</strong></pre><p>Then add the three parameters. The name of the API key parameter is <strong>“key”</strong>, the search engine id parameter is “<strong>cx”</strong> and the query is “<strong>q”</strong>. So that the URL becomes like this:</p><pre>url = <a href="https://www.googleapis.com/customsearch/v1?">https://www.googleapis.com/customsearch/v1?</a>key=&lt;enter the API key here&gt;/cx=&lt;enter the search engine id here&gt;/q=&lt;enter the query here&gt;</pre><p>Enter the URL on the browser so the response like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6aQbioP83igl8KWEnBH5HA.png" /></figure><h3>STEP 4: Get the Response from the JSON Read</h3><p>Create a variable that defines it as URL</p><pre>API_KEY = ‘&lt;enter the API key here&gt;’<br>SEARCH_ENGINE_ID = ‘&lt;enter the search engine id here&gt;’<br>query = ‘coronavirus’<br>url = <strong>f&quot;https://www.googleapis.com/customsearch/v1?key={</strong>API_KEY<strong>}&amp;cx={</strong>SEARCH_ENGINE_ID<strong>}&amp;q={</strong>query<strong>}&quot;</strong></pre><p>Google CSE has 100 data and 10 pages (10 data per page). If we are going to scrape all of the data, we must provide additional parameters namely <strong>“num”</strong> and “<strong>start</strong>”. The <strong>“num”</strong> parameter gives the meaning of how many items will be taken at each page, while the <strong>“start” </strong>parameter means that each page starts from the number of the item.</p><pre>params = {<strong>&#39;num&#39;</strong>: 10, <strong>&#39;start&#39;</strong>: start}</pre><p>Then, we have to make a looping function with <strong>“for”</strong> so that the data for each page can be retrieved so the code is:</p><pre>starts = list(x <strong>for </strong>x <strong>in </strong>range(1,100,10))<br><strong>for </strong>start <strong>in </strong>starts :<em><br>    </em>API_KEY = <strong>&quot;AIzaSyD7wVwnpIsKgpAdvRzRsD8_sUEjBhwb5IU&quot;</strong><em><br>    </em>SEARCH_ENGINE_ID = <strong>&quot;018212539862037696382:-xa61bkyvao&quot;<br>    </strong>query = <strong>&quot;corona&quot;</strong><em><br>    </em>url = <strong>f&quot;https://www.googleapis.com/customsearch/v1?key={</strong>API_KEY<strong>}&amp;cx={</strong>SEARCH_ENGINE_ID<strong>}&amp;q={</strong>query<strong>}</strong></pre><p>Create also a variable that defines a request for the URL and the additional parameters. Then, read the response by JSON.</p><pre>res = requests.get(url, params=params).json()</pre><p>print the response:</p><pre>print(res)</pre><p>Output:</p><pre>{‘kind’: ‘customsearch#search’, ‘url’: {‘type’: ‘application/json’, ‘temp......</pre><h3>STEP 5: Get The Title Key</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k2B5b4_CAH_r-E6Buvivfw.png" /></figure><p>The title key is <strong>“title”</strong> from <strong>“item”</strong> key, then we can looping the title using this code:</p><pre>items = res[<strong>&#39;items&#39;</strong>]<br><strong>for </strong>item <strong>in </strong>items:<br>    title = item[<strong>&#39;title&#39;</strong>]<br>    print(title)</pre><p>Output :</p><pre>Liputan Khusus Wabah Virus Corona Halaman 1<br>Pengembangan Vaksin Perlu Pertimbangkan Dampak Mutasi Virus …<br>Manakah Skenario Ideal Pandemi Corona di Indonesia, Ahli Jelaskan<br>Komet Atlas, Lintang Kemukus yang Telah Betas di Tengah …<br>Update Corona di Bali, Tambah 10, Total 177 Kasus Positif<br>2 Anak di Solo Positif Corona, Diduga Tertular Orang Terdekat atau …<br>Update untuk Sumut, Pasien Positif Corona Mencapai 102 Orang …</pre><p>If you want to get more field such as “link” and “image source”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ffINzwmtQwyyuXnNlfe1SQ.png" /></figure><p>Yeah, the web scraping project is done! It’s your turn to try it … Good luck :)</p><p>This is the end of my tutorial. I hope we can take a good lesson from what I deliver today. I ask forgiveness for any words and behave which are not supposed to be. Thank you for your kind attention Guys. Stay tuned for my next articles!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13d3e6f12413" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Install Pycharm IDE for Windows and Setting The Virtual Environment]]></title>
            <link>https://medium.com/@rifaislamet1509/how-to-install-pycharm-ide-for-windows-and-setting-the-virtual-environment-7e2e0aa66541?source=rss-bd872fe830e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e2e0aa66541</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[installation]]></category>
            <category><![CDATA[pycharm]]></category>
            <dc:creator><![CDATA[Rifai Slamet]]></dc:creator>
            <pubDate>Sat, 25 Apr 2020 00:34:28 GMT</pubDate>
            <atom:updated>2020-04-25T00:40:29.629Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Install Pycharm IDE for Windows and Get the First Project</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*St0rISHbb1gekJoXQGN7Dg.png" /></figure><p><strong>PyCharm</strong> is an <a href="https://en.wikipedia.org/wiki/Integrated_development_environment">integrated development environment</a> (IDE) used in <a href="https://en.wikipedia.org/wiki/Computer_programming">computer programming</a>, specifically for the <a href="https://en.wikipedia.org/wiki/Python_(programming_language)">Python</a> language. It is developed by the Czech company <a href="https://en.wikipedia.org/wiki/JetBrains">JetBrains</a>.<a href="https://en.wikipedia.org/wiki/PyCharm#cite_note-6">[6]</a> It provides code analysis, a graphical debugger, an integrated unit tester, integration with <a href="https://en.wikipedia.org/wiki/Revision_control">version control systems</a> (VCSes), and supports web development with <a href="https://en.wikipedia.org/wiki/Django_(web_framework)">Django</a> as well as <a href="https://en.wikipedia.org/wiki/Data_science">Data Science</a> with <a href="https://en.wikipedia.org/wiki/Anaconda_(Python_distribution)">Anaconda</a>. <em>( Source: </em><a href="https://en.wikipedia.org/wiki/PyCharm"><em>Wikipedia</em></a><em>)</em>.</p><p>Let’s Get Started!</p><h3>STAGE 1: Installing Python</h3><p>Before installing the Pycharm IDE, we must first install Python.</p><p>Step 1: Download the Python, click <a href="https://www.python.org/downloads/"><strong>here</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nemPxunD0Ns1-hYTjiVfYA.png" /><figcaption>Click “Download Python 3.8.2”</figcaption></figure><p>Step 2: After downloading is complete, run the Python.exe. Then click on <strong>“Install Now”.</strong></p><p>Step 3: After the installing is complete, Now click on <strong>“Close”</strong>.</p><h3>STAGE 2: Installing Pycharm</h3><p>Step 1: We have to download Pycharm IDE first, click <a href="https://www.jetbrains.com/pycharm/download/"><strong>here</strong></a> to download it. Then click the <strong>“DOWNLOAD”</strong> link under the Community Section if you want the free version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S1ETKRN-q5d8hGsWLi1QdA.png" /></figure><p>Step 2: After the Pycharm was downloaded successfully, run the PyCharm.exe. On the setup wizard, click <strong>“Next” </strong>and so on until the install process runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B-eiOGHXx1kGykcmWu2ELQ.png" /></figure><p>Step 3: Once the installation finished, you should receive a message screen that PyCharm is installed. If you want to go ahead and run it, click the <strong>“Run PyCharm Community Edition”</strong> box first and click <strong>“Finish”</strong>.</p><h4>Congratulations! Pycharm IDE is ready to use.</h4><h3>STAGE 3: Create the project and set the virtual environment</h3><p>Step1: Click <strong>“Create New Project”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BwAZMZtepgeypVufWJ8BDA.png" /></figure><p>Step 2: Set the directory and name the project. Here, the project name is <strong>“First Project”</strong> in the <strong>“D:\Python” </strong>directory. Then click <strong>“Create”.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*BibX9KgrD2b0ljQ0MS-FAQ.png" /></figure><p>Step 3: Go to setting. Click <strong>“File”</strong>, then click <strong>“Setting”.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Op1cMw2ukwGm7yT77VWOVg.png" /></figure><p>Step 5: Set the virtual environment. Click <strong>“Project Interpreter”</strong>, then <strong>“setting icon” </strong>and then click <strong>“Add”.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BP9int70FlIdzr6COVXDsA.png" /></figure><p>Step 6: Set the directory and name the virtual environment. Here, the virtual environment name is <strong>“vnv_first”</strong> in the <strong>“D:\Python\vnv” </strong>directory. Then click <strong>“Ok”.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*9mWH_2iPFtRX2qLf_Z0leQ.png" /></figure><p>Step 7: Check the virtual environment is ready to use. Just go to <strong>Terminal.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hSeVWD696ZrVgMzc-z53wQ.png" /></figure><p>YEAH, Congratulations! Now, you can start to create a new project. Good luck!</p><p>This is the end of my tutorial. I hope we can take a good lesson from what I deliver today. I ask forgiveness for any words and behave which are not supposed to be. Thank you for your kind attention Guys. Stay tuned for my next articles!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e2e0aa66541" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Deploy a Flask App to Heroku]]></title>
            <link>https://medium.com/@rifaislamet1509/how-to-deploy-a-flask-app-to-heroku-d092ad2330d0?source=rss-bd872fe830e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d092ad2330d0</guid>
            <category><![CDATA[heroku]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Rifai Slamet]]></dc:creator>
            <pubDate>Fri, 17 Apr 2020 11:31:10 GMT</pubDate>
            <atom:updated>2020-04-17T11:49:06.045Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IuS7N4O7dNuMkMCXvhyJoA.png" /></figure><p>Hi Guys, see me again in this article. Now I try to make an article about how to deploy our flask application to Heroku. How to do that? Easy. Let’s read the article until it runs out. Before doing this tutorial, make sure you have a flask project ready to use.</p><h3>Let’s Get Started…</h3><h3>STEP 1: Create a New Repository on GitHub</h3><p>To create a new repository very easy, click on the profile picture then select “<strong>your repository</strong>”, then follow the following picture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*scRMtiDpz6bYLaMIzkzJuA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fmE4Ssf57supsdINaKUDtQ.png" /></figure><ol><li>Enter the repository name</li><li>Add the gitignore, search and select <strong>“python”</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*94-JdR7NrlrBU2-vgEntmQ.png" /></figure><h3>STEP 2: Create the Project on Pycharm</h3><p>Open Pycharm then select <strong>“Get from Version Control”, t</strong>hen paste the url in the cloned Github project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*fr86emCKqSDa01FhMAL5gw.png" /></figure><p>Wait for it to finish then select <strong>“yes”</strong> if asked to access the directory.</p><h3>STEP 3: Open File Manager</h3><p>Move or copy all the contents of the flask project folder to the project folder from the results of the cloned GitHub project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zgRtntvGpitVo4wRCTCy1A.png" /></figure><p>Then, open the PyCharm again, then we will be in our flask project. Then, use the virtual environment that has a flask installed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*It-pNXuWfLQPUuFO41lU9w.png" /></figure><h3>STEP 4: Create “requirements.txt” and Procfile File</h3><p>The way to create a “<strong>requirement.txt”</strong> file is very simple, type the following code in the terminal.</p><pre><strong>pip </strong>freeze &gt; requirements.txt</pre><p>After having the requirements.txt file, then we need to create a “<strong>Procfile</strong>” file so that Heroku can run our flask project. The way to do this is to press right-click on our project directory then select “<strong>New”</strong> then select “ <strong>File</strong>” and name it “<strong>Procfile”. </strong>After that<strong>, </strong>type this code in the <strong>“Procfile”</strong>.</p><pre>web: gunicorn application:app</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e420rmZ8ocgMlfqhVO068g.png" /></figure><h3>STEP 5: Commit and Push</h3><p>After we are ready with our code, then we commit and push to our GitHub project. Press “CTRL + K” to commit the code. After that, enter the author (your name and email), and commit message then click the commit button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*JnjIW43DTkLvCRZJBqdWuw.png" /></figure><p>After we successfully commit our code, then we push the code by pressing the <strong>“CTRL + SHIFT + K”</strong> button on your keyboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*5FYzv9idOTmymGZeCYvRUA.png" /></figure><p>Wait until the push process is complete.</p><h3>STEP 6: Log In to Heroku Account</h3><p>Next, we will go to the Heroku website using our browser and log in there. if you don’t have a Heroku account, please register first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4eCSqpDgoVYb5Rqnfb1Mgg.png" /></figure><p>After we press the “<strong>Log In</strong>” button, then we will be directed to our account homepage.</p><h3>STEP 7: Create a New App</h3><p>To create a new Heroku app, follow the following picture</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mAl-OMJ2eVkOVQjbrEXZDA.png" /></figure><p>Then, we give a name to our application. The name must start with a lowercase letter. Then, click “<strong>Create app</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kh1uQ6pjmmJhNHiFxJXYFw.png" /></figure><h3>STEP 8: Connect the GitHub Account</h3><p>After that, we chose the deployment method with GitHub, connect your GitHub account which has the created flask application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zYz1aMojga0c_UnCTMif0Q.png" /></figure><ol><li>The deployment method area</li><li>Connect to your GitHub account</li><li>Search the repository name</li><li>Click “Connect” to connecting the application to Heroku</li></ol><h3>STEP 9: Deploy the Flask Application</h3><p>After we connect Heroku with our GitHub account, scroll down then select “Enable Automatic Deploys” so that every time there is a code change in our project, automatically Heroku will also update the changes according to the latest code version of our project on GitHub. Then, click “<strong>Deploy Branch”.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oMdgOSz6Z3jxbZpz0roviA.png" /></figure><p>After we click “Deploy Branch”, Heroku will install the library that reads in the “<strong>requirements.txt</strong>” file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0roIFCfsCjv2F72Y8IiDnA.png" /></figure><p>wait until the installation process is complete. After the library is successfully installed, Heroku will display a <strong>“view”</strong> button to see the results of the deployment of our application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*C3fQt0XBIsCNtorG1xhfkQ.png" /></figure><p>The result of the deployment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-mZyAlNmZiSynRRNTbjhjQ.png" /></figure><p>Thank you for those of you who have read this article from beginning to end. Sorry, there are still many mistakes in writing this article. For this reason, the authors highly expect your criticisms and suggestions to become evaluations of learning for the next article. Once again, Thank you very much.</p><p><strong>So,</strong> what should you do after knowing how to do web scraping using python? <strong>Start coding, start practicing.</strong></p><h3>Good luck!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d092ad2330d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web Scraping Yellowpage Indonesia Menggunakan Python]]></title>
            <link>https://medium.com/@rifaislamet1509/web-scraping-yellowpage-indonesia-menggunakan-python-223c39d35736?source=rss-bd872fe830e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/223c39d35736</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-mining]]></category>
            <category><![CDATA[data-extraction]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Rifai Slamet]]></dc:creator>
            <pubDate>Fri, 17 Apr 2020 02:48:54 GMT</pubDate>
            <atom:updated>2020-04-26T08:29:35.293Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1wvAGRwHkyqzpCgenm062g.png" /></figure><p>Halo Guys, terimakasih telah klik judul artikel ini. Itu artiya kalian berniat untuk mempelajari python lebih dalam khususnya pada kasus web scraping. Betul? kalo iya, siapkan secangkir kopi/susu untuk menemanimu menyimak artikel ini. Srupuuuttt!!! Ahhh :D</p><h3>Web Scraping???</h3><p>Makanan apa itu web scraping? secara singkat web scraping adalah kegiatan merekam/mencatat/mengambil/mengikis/meeee meapalagi yaaa… meeeendapatkan data-data dari sebuah website. Apabila kita ingin mencantat data dari web yang banyaknya ribuan bahkan jutaan tentu kita akan kualahan jika dilakukan secara manual dengan cara menyalin dan menempel. Dengan web scraping, kegiatan tersebut dapat dengan cepat dilakukan secara otomatis dan cepat jadi mesin yang akan bekerja. Mesakne caahhh mesine… Oke Lanjuuutttt…</p><p>Eiitttss, bentar. Sebelum memulai tutorial ini, alangkah baiknya kita sudah siap dengan projek python baru kita yang sudah terinstall paket bs4, requests, dan csv pada virtual environmen yang kita gunakan. Jika belum, saya harap kalian sudah bisa menginstall library pada python. Jika masih belum juga, silakan kembali ke fundamental python pada bagian cara menginstall library/package pada Python. heheh :D</p><p>Tapi jangan khawatir, penulis tidak malas untuk membagikan cara menginstallnya kok, tetap akan penulis beritahu disini cara membuat virtual environment dan menginstall paket yang akan dipakai untuk projek kali ini menggunakan Pycharm. Bagi kalian yg sudah siap dengan projek baru bisa langsung loncat ke langkah 3 aja. Gassss!!!</p><h4>Studi Kasus :</h4><p>Kita akan melakukan scraping dari website yellowpages indonesia. Kita akan mengambil data dari hasil pencarian yaitu hotel yang terdapat di Jakarta. Berapa data yang bisa kita ambil? Terserah kita mau berapapun bisa asal masih di range halaman web tersebut. Mantap nggak? Wkwk</p><h3>Mari Kita Mulai</h3><p>Berdo’a, mulai….</p><p>Berdo’a selesai….</p><h4>Langkah 1: Membuat Virtual Environment</h4><p>Kita buka terlebih dahulu aplikasi Pycharm kemudian pilih buat projek baru <em>(create a new project). </em>Lalu kita namakan projek itu sesuka hati misalnya “<strong>YP</strong>” singkatan Yelow Page kemudian klik <strong>Creat</strong>. Maka kita akan diarahkan ke lembar projek kita. Untuk membuat virtual environment, pergi ke menu setting dengan menekan tombol “<strong>CTRL + ALT+ S</strong>”. Kemudian pilih <strong>“Project Interpreter</strong>” kemudian pilih <strong>“virtual environment”.</strong> Selanjutnya pada sebelah kanan kollom <strong>“New Environment</strong>” terdapat simbol gerigi. Klik saya jangan takut. Kemudian klik menu “<strong>Add</strong>” dan beri nama virtual environment sesuai keinginan. Coba lihat gambar berikut agar kata-kata penulis dapat lebih mudah dipahami.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*G2lxVlQj78oPmLWMMmTHog.png" /><figcaption>Pada menu setting, pilih “<strong>Add”</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*YBStFenahxc5YWG0odjhXw.png" /><figcaption>Yang saya blokadaah nama virtual environmennya, kemudian <strong>“ok”</strong></figcaption></figure><h3>Langkah 2: Menginstall Library</h3><p>Python Lybrary merupakan hal yang sangat penting untuk mengembangkan sebuah aplikasi python. Tanpa Lybrary, programmer akan merasa kesulitan untuk membuat sebuah aplikasi. Mereka akan kualahan, kerepotan, dan memakan banyak waktu jika membuat projek dengan library yang di buat sendiri. Maka dari itu para core of the corenya python sudah membuatkannya untuk kita sehingga kita lebih mudah dan praktis dalam membuat projek. Oke, bagaimana cara menginstallnya? muuuudaaahhh. Caranya pada langkah pertama diatas jangan ditutup dulu, terdapat lambang + di sebelah atas kanan dari menu virtual environment kemudian cari library yang akan diinstall, kemudian klik “<strong>Install Package</strong>”. Klik aja, jangan takut.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*dnbf3RWt2QFHSE24XELdaQ.png" /></figure><p>Atau bisa dengan mengetikkan seperti dibawah ini pada terminal kita</p><pre>pip install bs4 #tungggu hingga selesai<br>pip install requests #tunggu hingga selesai</pre><p>Untuk csv tidak usah diinstall yaa, karena beliau suda ada dari bawaan Python, jadi langsung import saja. Jika sudah seselai instal, kita lanjut ke langkah berikutnya</p><h4>Langkah 3: Import Modul dari Library</h4><p>Di dalam paket/library terdapat modul-modul yang dapat digunakan untuk projek kita. Kita ibaratkan library adalah kampus, modul adalah fakultas, dan class adalah prodi… paham ya? Hehe</p><p>Oke langsung saja kita import modul yang kita akan gunakan. Ketikkan saja di baris paling awal dari projek kita seperti dibawah ini</p><pre><strong>import </strong>requests<br><strong>from </strong>bs4 <strong>import </strong>BeautifulSoup <strong>as </strong>b<br><strong>import </strong>csv</pre><p>from itu untuk mengakses paketnya kemudian dari paket kita import modulnya dan enaknya lagi bisa kita beri inisial modul dengan huruf apapun biar lebih singkat dengan perintah “as” kemudian inisialnya mau kasih nama apa misal pada kasusdiatas yaitu “b”. Python memang mantap bisa seperti itu. Jika sudah maka modul siap digunakan.</p><h4>Langkah 4: Dapatkan Respon dari URL</h4><p>Kita pergi ke browser dan menuju ke situs yellowpage indonesia. Kemudian cari bisnis misal <strong>“hotel</strong>” yang ada di<strong> “jakarta”</strong>. Bisa kita ganti sesuka hati kita asalkan jangan ketik <strong>“dukun”</strong> di <strong>“Banyuwangi”</strong>. NGAAK ADAAA!!!</p><p>Apakah bisa pencarian tersebut diakukan pada lembar projek kita? BISA! dengan kekuatan fungsi input hal itu bisa dilakukan dengan mudah. Lihatlah dan bukalah mata hatimuuu, eh bukan. Lihatlah dan ikuti saja kode dibawah ini.</p><pre>nama_bisnis = input(<strong>&#39;Cari bisnis: &#39;</strong>)<br>tempat = input(<strong>&#39;Tempat : &#39;</strong>)<br>url = <strong>&#39;https://yellowpages.co.id/cari/bisnis/{0}/{1}&#39;</strong>.format(nama_bisnis, tempat)</pre><p>Angka <strong>“0”</strong> itu menunjukkan input yang pertama. Loh <strong>“0”</strong> kok pertama om? yaaa memang python itu menghitung angka dari <strong>0</strong> bukan <strong>1</strong>. Protes? Silakan mengadu ke pak “<strong>Guido van Rossum”. </strong>Siapa dia? Apa kehebatannya? Klik <a href="https://id.wikipedia.org/wiki/Guido_van_Rossum"><strong>disini</strong></a> untuk membaca biografinya. Jika sudah kita masukkan referernya yangterdapat pada request header dari halaman web tersebut.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eoT_RTutKyhBXyUMQoL0eA.png" /></figure><pre>headers = {<strong>&#39;Referer&#39;</strong>:<strong>&#39;https://yellowpages.co.id/cari/bisnis/{nama_bisnis}/{tempat}&#39;</strong>}</pre><p>Kalau sudah, request url tersebut dan buatlah variabel requestnya terlebih dahulu biar lebih mudah nantinya</p><pre>r = requests.get(url, headers=headers)</pre><p>Print variabel “r” tersebut untuk mengecek apakah respon dari url tersebut berhasil atau tidak. jika respon menunjukkan angka 200 maka request tersebut berhasil. Setelah itu kita ubah format html tersebut ke format BeautifulSoup agar dapat diparsing data dari url itu dengan bs4 pada variabel “soup” atau bisa yang lain terserah kita.</p><pre>soup= b(r.text, <strong>&#39;html.parser&#39;</strong>)</pre><h4>Langkah 5: Dapatkan Element dari Tiap Item</h4><p>Buka kemmbali browser dan inspeksi judul item pada halaman tersebut kemudian temukan element yang terdapat semua data yang akan kita ambil.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cDScnTGbWjDoMypN7GyL-w.png" /></figure><p>Pada gambar tersebut si item berada pada tag “<strong>div</strong>” dan kelasnya “<strong>‘home-list-pop list-spac</strong>” dan tampung di sebuah variabel “<strong>results</strong>”.</p><pre>results = soup.findAll(<strong>&#39;div&#39;</strong>, <strong>&#39;home-list-pop list-spac&#39;</strong>)</pre><p>Kita gunakan fungsi “<strong>FindAll</strong>” karena kita akan mengambil semua item ygberada pada tagdan kelas yang sama.</p><h4><strong>Langkah 6: Lakukan Looping dengan Fungsi “for”</strong></h4><p>Lihatkah kode dibawah ini, dan akan saya jelaskan dibawahnya.</p><pre><strong>for </strong>re <strong>in </strong>results:<br>    nama = re.find(<strong>&#39;h4&#39;</strong>).text.strip()<br>    p = re.findAll(<strong>&#39;p&#39;</strong>)<br>    kategori = p[0].text.strip()<br>    jalan = p[1].text.strip()<br>    kota = p[2].text.strip()<br>    kontak = re.find(<strong>&#39;div&#39;</strong>,<strong>&#39;col-md-4 home-list-pop-desc inn-list-pop-desc&#39;</strong>).findAll(<strong>&#39;a&#39;</strong>)</pre><p>Kita deklarasikan variabel “<strong>results</strong>” ke variabel baru misal “<strong>re</strong>” dan looping variabeltersebut dengan fungsi “<strong>for</strong>”. Selanjutnya kita akan buat variabel yangmenapungdatayang akan kita ambil.</p><p>Untuk nama terdapat pada tag “<strong>h4</strong>” sehingga kita tidak perlu menambahkan atributnya kemudian kita ambil teksnya saja dan kita strip agar karakter di depan dan dibelakang string teksnya dibersihkan. kemudian untuk infonya seperti “kategori”, “jalan”, “kota”, dan “kontak” terdapat pada tag yang sama yaitu “p”kita bisa “findAll” dan kita tinggal urutkan saja. Untuk data “kategori” pada rutan pertama maka kita panggil dengan “p[0]” dan seterusnya. untuk kontak terdapat 2 jenis yaitu email dan website yangberada pada tag yang sama sehingga dapat menggunakan <strong>“findAll(‘a’)”.</strong></p><h4>Langkah 7: Kendalikan Info Kontak</h4><p>Pada data kontak terdapat 2 data yaitu email dan website, namun tidak semua item punya web maupun email, ada juga yang hanya punya email. Untuk itu kita dapat kendaikan dengan fungsi “<strong>if elif else”</strong> atau bisa juga dengan fungsi <strong>“try except” </strong>terserah mau pakai yang mana.</p><p>Jika menggunakan fungsi <strong>“if elif else”</strong> kurang lebih akan seperti ini.</p><pre><strong>if </strong>len(kontak) &gt; 1:<br>    email = kontak[0][<strong>&#39;href&#39;</strong>].replace(<strong>&#39;mailto:&#39;</strong>,<strong>&#39;&#39;</strong>).strip()<br>    web = kontak[1][<strong>&#39;href&#39;</strong>].replace(<strong>&#39;https://www.&#39;</strong>,<strong>&#39;&#39;</strong>).replace(<strong>&#39;http://www.&#39;</strong>,<strong>&#39;&#39;</strong>)\<br>        .replace(<strong>&#39;https://&#39;</strong>,<strong>&#39;&#39;</strong>).replace(<strong>&#39;http://&#39;</strong>,<strong>&#39;&#39;</strong>).split(<strong>&#39;?&#39;</strong>)[0].strip()<br><strong>elif </strong>len(kontak) == 1:<br>    email, web = kontak[0][<strong>&#39;href&#39;</strong>].replace(<strong>&#39;mailto:&#39;</strong>, <strong>&#39;&#39;</strong>).strip(),<strong>&#39;Tidak Ada&#39;<br>else</strong>:<br>    email, web = <strong>&#39;Tidak Ditemukan&#39;</strong>, <strong>&#39;Tidak Diemukan&#39;</strong></pre><p>Jika mengunakan fungsi<strong> “try except”</strong> akan seprti ini :</p><pre><strong>try</strong>:<br>    email = kontak[0][<strong>&#39;href&#39;</strong>].replace(<strong>&#39;mailto:&#39;</strong>,<strong>&#39;&#39;</strong>).strip()<br><strong>except</strong>:<br>    email= <strong>&#39;Tidak ada&#39;<br>try</strong>:<br>    web = kontak[1][<strong>&#39;href&#39;</strong>].replace(<strong>&#39;https://www.&#39;</strong>, <strong>&#39;&#39;</strong>).replace(<strong>&#39;http://www.&#39;</strong>, <strong>&#39;&#39;</strong>) \<br>        .replace(<strong>&#39;https://&#39;</strong>, <strong>&#39;&#39;</strong>).replace(<strong>&#39;http://&#39;</strong>, <strong>&#39;&#39;</strong>).split(<strong>&#39;?&#39;</strong>)[0].strip()<br><strong>except</strong>:<br>    web = <strong>&#39;Tidak Ada&#39;</strong></pre><p>Metode yang bisa dipakai yaitu replace untuk mengubah karakter dalam string, split untuk memisahkan karakter, dan strip sudah sayjelaskan diatas. Kalian bisa otak-atik sendiri, jika sudah luus fundamental pasti lancar dan enjoy untuk kasus ini. Hehe. Lanjut…</p><p>Oh ya, jangan lupa untuk lakukan print setiap data dan jalankan untuk menguji apakah ada yang bermasalah atau tidak. Ok, Next yaa..</p><h4>Langkah 8: Panigation Web Scraping</h4><p>Tadi kita sudah ambil data dari 1 halaman saja, selanjutnya kita akan ambil banyak halaman sesuai yang kita inginkan. Caranya? Tinggal looping aja halamannya, beres… Eits tidak semudah itu Ferguso…</p><p>Pertama kita cari parameter halaman yang dipakai oleh web tersebut. Caranya adalah kita klik halaman lain misal 2. Nanti akan ketemu parameter halaman .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fSaSLREHfpJ-Uz07O_M2GA.png" /></figure><p><a href="https://yellowpages.co.id/cari/bisnis/hotel/jakarta?page=2&amp;sort=">“https://yellowpages.co.id/cari/bisnis/hotel/jakarta?<strong>page=2</strong>&amp;sort=</a>”</p><p>Ternyata web tersebut menggunakan kata “<strong>page</strong>” sebagai parameter halamannya. Kita bisa manfaatkan itu dengan looping urlnya dengan for sehingga code akan menjadi seperti dibawah ini:</p><pre><strong>for </strong>page <strong>in </strong>range(1,11): <em>#mentok 308<br>    </em>url = <strong>&#39;https://yellowpages.co.id/cari/bisnis/{0}/{1}?page={2}&#39;</strong>.format(nama_bisnis, tempat, page)<br>    headers = {<strong>&#39;Referer&#39;</strong>:<strong>f&#39;https://yellowpages.co.id/cari/bisnis/{</strong>nama_bisnis<strong>}/{</strong>tempat<strong>}&#39;</strong>}<br>    r = re...</pre><h4>Langkah 9: Append Data</h4><p>Jika berhasil mendapatkan data dari setiap halamannya, kita tampung semua data tersebut ke sebuah variabel list kosong dan kita beri nama misal <strong>“datas</strong>” dan letakkan itu di luar looping area.</p><pre>datas = []<br><strong>for </strong>page <strong>in </strong>range(1,3): <em>#mentok 308<br>    </em>url = <strong>&#39;https://yell...</strong></pre><p>Selanjutnya kita tambhkan data satu per satu data ke variabel datas dengan menggunakan metode append. Letakkan metote ini di akhir looping (masih dalam looping yaa).</p><pre>...<br>print(<strong>&#39;E-mail   : &#39;</strong>, email)<br>print(<strong>&#39;Website  : &#39;</strong>, web, <strong>&#39;\n&#39;</strong>)<br>datas.append([nama, kategori, jalan, kota, email, web])</pre><h4>Langkah Terakhir: Simpan ke Csv File</h4><p>Setelah kita berhasil mengambi/scrape data, kita dapat menyimpannya dalam format file CSV menggunakan metode “<strong>write</strong>”, untuk lebih detailnya metode apa saja yang ada pada library csv dapat kita klik <a href="https://docs.python.org/3/library/csv.html"><strong>di sini</strong></a>.</p><pre><strong>with </strong>open(<strong>&#39;hasil.csv&#39;</strong>, <strong>&#39;w&#39;</strong>, newline=<strong>&#39;&#39;</strong>) <strong>as </strong>file:<br>    writer = _csv.writer(file)<br>    headers = [&#39;<strong>Nama&#39;</strong>,<strong>&#39;Kategori&#39;</strong>,<strong>&#39;Jalan&#39;</strong>,<strong>&#39;Kota&#39;</strong>, <strong>&#39;Email&#39;</strong>,<strong>&#39;Website&#39;</strong>]<br><br>    writer.writerow(headers)<br>    <strong>for </strong>data <strong>in </strong>datas:<br>        writer.writerow(data)</pre><p>Tungu hingga proses scrapingnya selesai, kemudian cek file output dari projek kitayang berada pada folder yang sama dengan file aplikasi/projek scraping kita.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VAOuBR33wOt3v-CtkFnZEQ.png" /></figure><p>Saatnya bilang, SIIIPP…</p><p>Terimaksih bagi kalian yang sudah membaca artikel ini dari awal aku tak pernah, percaya kata-katamu, eh maaf wkwk, dari awal hingga akhir. Mohon maaf masih banyak kesalahan dalam penulisan artikel ini. Untuk itu penulis sangat mengharapkan kritik dan saran kalian agar menjadi evaluasi pembelajran untuk penulisan artikel selanjutnya… Tank youuu…</p><p><strong>So</strong>, apa yang harus kalian lakukan setelah mengetahui cara melakukan web scraping menggunakan python? <strong>Start coding, start practicing</strong>.</p><p><strong>Good luck</strong>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=223c39d35736" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Easy Peasy! Scraping CNN Web APIs to Flask Framework]]></title>
            <link>https://medium.com/@rifaislamet1509/easy-peasy-scraping-cnn-web-apis-to-flask-framework-457df79677ad?source=rss-bd872fe830e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/457df79677ad</guid>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Rifai Slamet]]></dc:creator>
            <pubDate>Thu, 16 Apr 2020 09:33:36 GMT</pubDate>
            <atom:updated>2020-04-16T09:33:36.089Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xRyYtgCVDM88UM6e6SeWJQ.png" /></figure><h3>Flask</h3><p><a href="http://flask.pocoo.org/">Flask</a> (<a href="https://github.com/pallets/flask">source code</a>) is a Python <a href="https://www.fullstackpython.com/web-frameworks.html">web framework</a> built with a <a href="http://flask.pocoo.org/docs/design/">small core and easy-to-extend philosophy</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/0*CLRYF-DIGJVdB0Ku.jpg" /></figure><h3>Why is Flask a good web framework choice?</h3><p>Flask is considered more <a href="http://blog.startifact.com/posts/older/what-is-pythonic.html">Pythonic</a> than the <a href="https://www.fullstackpython.com/django.html">Django</a> web framework because in common situations the equivalent Flask web application is more explicit. Flask is also easy to get started with as a beginner because there is little boilerplate code for getting a simple app up and running.</p><blockquote>souurce : <a href="https://www.fullstackpython.com/flask.html">https://www.fullstackpython.com/flask.htm</a>l</blockquote><p>After we know the meaning of Flask, we can expose our project to the framework so that it becomes a web application. Cool, right? Want to know the steps? The following are the steps so read carefully.</p><h3>STEP 1: Install the Packages</h3><p>this project only requires 2 packages, Flask and Requests. Here is how to install the packages.</p><h4>1. Flask</h4><p>Within the activated environment, use the following command to install Flask:</p><pre>$ pip install Flask</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*3tTpbeLzqSle_O9v.png" /></figure><p><strong>Flask is now installed:</strong> Check out the <a href="http://flask.pocoo.org/docs/1.0/quickstart/">Quickstart</a> or go to the <a href="http://flask.pocoo.org/docs/1.0/">Documentation</a>.</p><h4>2. Requests</h4><p>This part of the documentation covers the installation of Requests. The first step to using any software package is getting it properly installed.</p><h4>Distribute &amp; Pip</h4><p>Installing requests is simple with <a href="http://www.pip-installer.org/">pip</a>:</p><pre>$ pip install requests</pre><p>or, with <a href="http://pypi.python.org/pypi/setuptools">easy_install</a>:</p><pre>$ easy_install requests</pre><p>But, you really <a href="http://www.pip-installer.org/en/latest/other-tools.html#pip-compared-to-easy-install">shouldn’t do that</a>.</p><h4>Cheeseshop Mirror</h4><p>If the Cheeseshop is down, you can also install Requests from one of the mirrors. <a href="http://crate.io/">Crate.io</a> is one of them:</p><pre>$ pip install -i <a href="http://simple.crate.io/">http://simple.crate.io/</a> requests</pre><blockquote>source:<a href="https://2.python-requests.org/en/v1.1.0/user/install/"> https://2.python-requests.org/en/v1.1.0/user/install/</a></blockquote><h3>STEP 2: Get the Request URL</h3><p>To get it, open the CNN site on your browser then see the following picture along with a brief explanation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*efAN3JXPQ4Qpr32LwD4Z5A.png" /></figure><ol><li>Search the news about what you want to scrape. The example in the picture is about <strong>coronavirus,</strong></li><li>Inspect the webpage then copy the title,</li><li>Paste the title into the search box,</li><li>Click response from the search results,</li><li>Select the header tab,</li><li>Okay, we got the<strong> Request URL.</strong></li></ol><h3>STEP 3: Find the Key of Value</h3><p>In this step, find the key to the value that we will scrape. For example, I will take the title and source image of the news site. Then I have to know the key of the title and image source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hTmfzqE1DpPfMNGyHNH1Zg.png" /></figure><ol><li>Search the title,</li><li>This is key of the title</li><li>This is key to the image source, you can add other elements such as description, news upload date, etc.</li></ol><h3>STEP 4: Import the Flask and Requests Package</h3><p>In the first line, we have to initialize the module that we will use so that the project can run well. That matter can be done by importing the module.</p><pre><strong>import </strong>requests<br><strong>from </strong>flask <strong>import </strong>Flask, render_templateT</pre><h3>STEP 5: Let’s talk about this object.</h3><p>In <strong>todo/app.py</strong>, you&#39;ll create an <strong>app</strong> object, which is an instance of the <strong>Flask</strong> object. It&#39;ll act as the central configuration object for the entire application. It&#39;s used to set up pieces of the application required for extended functionality, e.g., a database connection and help with authentication.</p><p>It’s regularly used to set up the routes that will become the application’s points of interaction. To explain what this means, let’s look at the code it corresponds to.</p><pre>app = Flask(__name__)<br><br>@app.route(&#39;/&#39;)<br>def hello_world():<br>    &quot;&quot;&quot;Print &#39;Hello, world!&#39; as the response body.&quot;&quot;&quot;<br>    return &#39;Hello, world!&#39;</pre><blockquote>source : <a href="https://opensource.com/article/18/4/flask">https://opensource.com/article/18/4/flask</a></blockquote><p>Then we change the def function with the function that fits our project, for example, “def cnn”. Create a data variable that defines the results of the inspect on the web page.</p><pre><strong>def </strong>cnn():<br>    url = <strong>&#39;https://search.api.cnn.io/content?q=coronavirus&amp;page=6&#39;<br>    </strong>json_data = requests.get(url).json()<br>    data = json_data[<strong>&#39;result&#39;</strong>]<br>    <strong>return </strong>render_template(<strong>&#39;index.html&#39;</strong>, data= data)</pre><p>Do not forget to do debugging so that every time there is a change in the thesis automatically our application that runs on the flask framework will change too.</p><pre><strong>if </strong>__name__ == <strong>&quot;__main__&quot;</strong>:<br>    app.run(debug=<strong>True</strong>)</pre><h3>STEP 6: Create the HTML Tempates</h3><p>To create an HTML template we first create the “templates” directory in the same folder as our project file. then we create an HTML file called “index.html” in that directory. then in the HTML file write it as below. Here I use a template that is very basic and simple. You can create and develop it to make it look more attractive using Bootstrap or Angular, or something else</p><pre>&lt;!DOCTYPE <strong>html</strong>&gt;<br>&lt;<strong>html lang=&quot;en&quot;</strong>&gt;<br>&lt;<strong>head</strong>&gt;<br>    &lt;<strong>meta charset=&quot;UTF-8&quot;</strong>&gt;<br>    &lt;<strong>title</strong>&gt;Web Scraper Aplication&lt;/<strong>title</strong>&gt;<br>&lt;/<strong>head</strong>&gt;<br><br>&lt;<strong>body</strong>&gt;<br>{% for d in data %}<br>&lt;<strong>h1</strong>&gt;Name = {{d[&#39;headline&#39;]}} &lt;<strong>br</strong>/&gt;&lt;/<strong>h1</strong>&gt;<br>&lt;<strong>h2</strong>&gt;Image = {{d[&#39;thumbnail&#39;]}} &lt;<strong>br</strong>/&gt;&lt;/<strong>h2</strong>&gt;<br>{% endfor %}<br>&lt;/<strong>body</strong>&gt;<br>&lt;/<strong>html</strong>&gt;</pre><h3>STEP 6: Run The Python Project</h3><p>To run this project simply <strong>right-click</strong> then select <strong>“Run”</strong> on the project sheet or it can also be done by pressing the “<strong>CTRL + SHIFT + F10</strong>” button on the keyboard then it will be as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kS-pG3gEm1Kw5eJ6haI4eg.png" /></figure><p>After that, click on the localhost or you can do it manually with open your browser then enter the <strong>localhost:5000. </strong>Then we will be directed to the flask framework page through the chrome browser. there we will see the results of scraping data that we want. good luck :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k7AcsoRTMDoaA5NRCpbMiw.png" /></figure><p>I hope you can benefit from this article and develop it according to your own needs and imagination. And I ask forgiveness for any words and behave which are not supposed to be. Thank you for your kind attention Guys. Stay tuned for my next articles!…. :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=457df79677ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[API Web Scraping from FloatRates using Python]]></title>
            <link>https://medium.com/@rifaislamet1509/api-web-scraping-from-floatrates-using-python-1a53eed814e7?source=rss-bd872fe830e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a53eed814e7</guid>
            <dc:creator><![CDATA[Rifai Slamet]]></dc:creator>
            <pubDate>Wed, 15 Apr 2020 14:17:08 GMT</pubDate>
            <atom:updated>2020-04-15T14:17:08.381Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7H41xLrO6HoPmhw5MCa43w.png" /></figure><h3>WHAT IS API?</h3><p>An <strong>application programming interface</strong> (<strong>API</strong>) is a <a href="https://en.wikipedia.org/wiki/Interface_(computing)">computing interface</a> to a software component or a system, that defines how other components or systems can use it. It defines the kinds of calls or requests that can be made, how to make them, the data formats that should be used, the conventions to follow, etc. It can also provide extension mechanisms so that users can extend existing functionality in various ways and to varying degrees. An API can be entirely custom, specific to a component, or it can be designed based on an industry standard to ensure interoperability. Some APIs have to be documented, others are designed so that they can be “interrogated” to determine supported functionality. Since other components/systems rely only on the API, the system that provides the API can (ideally) change its internal details “behind” that API without affecting its users. (source: Wikipedia, click <a href="https://en.wikipedia.org/wiki/Application_programming_interface"><strong>here</strong></a> for more details).</p><h3>STEP 1: Preparation</h3><p>You can skip this step with a new project that is still empty and has installed bs, requests, and CSV packages.</p><h3>If you not ready for it</h3><p>I hope you have installed Pycharm or whatever Python IDE. If not, please click <a href="https://www.jetbrains.com/pycharm/download/#section=windows"><strong>here</strong></a> to download it, then select the community version because it’s free. If you have already installed it please do as you install other software. :)</p><p>Then create a virtual environment and install the requests and CSV packages. Click here to create a virtual environment and how to use it. then type below in the terminal to install packages.</p><pre><strong>pip install</strong> requests<br><strong>pip install</strong> csv</pre><h3>SEP 2: Import The Packages</h3><p>In the first line, we have to initialize the packages that we will use so that the project can run well. That matter can be done by importing the package.</p><pre><strong>import </strong>requests<br><strong>import </strong>csv</pre><h3><strong>STEP 3: Get The JSON Feeds from FloatRates APIs</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fAISoE4hagWOZND-_zreqQ.png" /><figcaption>Click “Feed” then “JSON Feeds”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k0VKVZqMMv1de2s7Lnai_Q.png" /></figure><p>After you click on one of these currencies, the FloatRates will show the JSON feeds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*649N7agGJMAsnXMQznRSSw.png" /></figure><h3>STEP 4: Request The JSON Feeds</h3><p>After we get the JSON feed, copy the URL, create a URL variable on your project, and paste it to the variable then we request the JSON feed on our project using the requests module.</p><pre>source = requests.get(url=<strong>&#39;http://www.floatrates.com/daily/usd.json&#39;</strong>)</pre><p>if we look at the JSON format as the dictionary format contained <strong>“keys”</strong> and <strong>“values”</strong>. Because we will take the values ​​part, we call that by adding <strong>“.values ​​()</strong>”</p><pre>json_data = source.json().values()</pre><h3>STEP 5: Determine The Keys and Values</h3><p>In these values ​​there are also “<strong>keys</strong>” and “<strong>values</strong>” again. from here we will call the values ​​by calling the key in square brackets “<strong>[“ &lt;key&gt; “]”</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4SSLXAl7jo2XflpJzXfJdg.png" /></figure><p>There is a lot of data we do the looping so that each item can be processed by declaring the “<strong>json_data</strong>” variable to the “<strong>data</strong>” variable.</p><pre><strong>for </strong>data <strong>in </strong>json_data:<br>    name = data[<strong>&#39;name&#39;</strong>]<br>    date =data[<strong>&#39;date&#39;</strong>]<br>    invers=data[<strong>&#39;inverseRate&#39;</strong>]</pre><p>Then, print the data serves to check whether the data we will take is in line with expectations.</p><pre>print(<strong>&quot;Name :&quot;</strong>,name)<br>print(<strong>&quot;Date :&quot;</strong>,date)<br>print(<strong>&quot;Invers Rate:&quot;</strong>,invers)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tTr-YoBdEbFrYt-1zNmcrQ.png" /></figure><h3>STEP 6: Append the Data</h3><p>Create a variable that contains an empty list, then enter the data into the empty list. for example, I named the variable “datas” which I placed before the loop.</p><pre>datas=[]<br><strong>for </strong>data <strong>in </strong>js...</pre><p>Then I will append the data that we have scrapped into the variable. Put the append function in the loop area.</p><pre>...</pre><pre>print(<strong>&quot;Invers Rate:&quot;</strong>,invers)<strong><br></strong>datas.append([name, date, invers])</pre><h3>STEP 7: Save to CSV File</h3><p>After we successfully scrape the data, we can save it in CSV file format using the “write” function, for more details click <a href="https://docs.python.org/3/library/csv.html"><strong>here</strong></a> to see how to complete the guide using the CSV library.</p><pre><strong>with </strong>open(<strong>&#39;FloatRates.csv&#39;</strong>, <strong>&#39;w&#39;</strong>, newline=<strong>&#39;&#39;</strong>) <strong>as </strong>file:<br>    writer = csv.writer(file)<br>    headers = [<strong>&#39;Name&#39;</strong>,<strong>&#39;Date&#39;</strong>,<strong>&#39;Invers Rates&#39;</strong>]<br>    writer.writerow(headers)<br>    <strong>for </strong>data <strong>in </strong>datas:<br>        writer.writerow(data</pre><p>Check the CSV file. Good luck :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6eXLNgn-2gMIfb6SjTKPUA.png" /></figure><p>I hope you can benefit from this article and develop it according to your own needs and imagination. And I ask forgiveness for any words and behave which are not supposed to be. Thank you for your kind attention Guys. Stay tuned for my next articles!…. :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a53eed814e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Powerful, Scraping Thousands of Record from Futbin Database to CSV file using Python]]></title>
            <link>https://medium.com/@rifaislamet1509/powerful-scraping-thousands-of-record-from-futbin-database-to-csv-file-using-python-207abda35bba?source=rss-bd872fe830e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/207abda35bba</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Rifai Slamet]]></dc:creator>
            <pubDate>Wed, 15 Apr 2020 01:22:31 GMT</pubDate>
            <atom:updated>2020-04-15T01:22:31.888Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EGxk5UxGhNXrBlxkg9ts-Q.png" /></figure><p>Are you a data entry or a data report? if you are going to take a lot of data from the internet whose numbers are thousands, even millions, of course, it is very heavy if done manually or copy-paste. first, waste of your energy. secondly wastes internet quota because there will be lots of clicks and loads of images. In Python, you can retrieve any data you want quickly. Can even be done while lying down. It’s cool, right? If so, pay close attention to this article.</p><h3>STEP 1: Preparation</h3><p>You can skip this step with a new project that is still empty and has installed bs, requests, and CSV packages.</p><h4>If you not ready for it</h4><p>I hope you have installed Pycharm or whatever Python IDE. If not, please click <a href="https://www.jetbrains.com/pycharm/download/#section=windows"><strong>here</strong></a> to download it, then select the community version because it’s free. If you have already installed it please do as you install other software. :)</p><p>Then create a virtual environment and install three main packages: bs4, requests, and CSV. Click here to create a virtual environment and how to use it. then type below in the terminal to install packages.</p><pre><strong>pip install</strong> bs4<br><strong>pip install</strong> requests<br><strong>pip install</strong> csv</pre><h3>SEP 2: Import The Packages</h3><p>In the first line, we have to initialize the packages that we will use so that the project can run well. That matter can be done by importing the package.</p><pre><strong>import </strong>bs4<br><strong>import </strong>requests<br><strong>import </strong>csv</pre><h3>STEP 3: Get the Response Text</h3><p>To get the response text, we must know the URL that we are targeting to do the scraping.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Id7sagAbFZeE9f4Yw4E2mg.png" /></figure><p>After that, enter the URL into the requests function with the get method.</p><pre>response = requests.get(<strong>&#39;https://www.futbin.com/players?page=1</strong>))</pre><p>Then, print the response by adding <strong>“.text”</strong> behind the response word and run the project.</p><pre>print(response.text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tpMhm1ofJtKSStU4Ik6rew.png" /></figure><p>After that, change the response format to bs4 format.</p><pre>soup = bs4.BeautifulSoup(response.text,<strong>&#39;html.parser&#39;</strong>)</pre><h3>STEP 4: Get the Player’s Element</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0K7H2GS-vI0Es1KmkKBZ8g.png" /></figure><p>In the picture, each product is in the <strong>“tr”</strong> tag and the <strong>“player_tr_1</strong>” and “<strong>player_tr_2</strong>” class. Because there are two different classes, we loop it so we can get all the data players. After that, we can use the “findAll” function in the beautifulsoup module to retrieve all information contained in the product.</p><pre>numbers = [1,2]<strong><br>for </strong>number <strong>in </strong>numbers:<br>    players = soup.findAll(<strong>&#39;tr&#39;</strong>,<strong>&#39;player_tr_&#39;</strong>+str(number))</pre><h3>STEP 5: Lopping the Players Variable with “for” Function</h3><p>Then we will parse the data contained in the product class by repeating the ‘for’ function. The data that I took in this article include name, club, nation, and league. The name, nation, and league are in the same tag and class so we apply it with the <strong>‘FindAll’</strong> function so that it becomes a list of data, then we will call the data in what order. You can develop your own by adding data variables that you will take and enter into the “find (‘ &lt;tag&gt; ‘, {‘&lt;attribute name&gt;’:’attribute value’})” function. Use the string strip method to remove the characters at the front and back of the string.</p><pre><strong>for </strong>p <strong>in </strong>players:<br>    tds = p.findAll(<strong>&#39;td&#39;</strong>)<br>    name = tds[0].text.strip()<br>    clubs = p.find(<strong>&#39;span&#39;</strong>, <strong>&#39;players_club_nation&#39;</strong>).findAll(<strong>&#39;a&#39;</strong>)<br>    club = clubs[0][<strong>&#39;data-original-title&#39;</strong>].replace(<strong>&#39;Icons&#39;</strong>, <strong>&#39;unknown&#39;</strong>).strip()<br>    nation = clubs[1][<strong>&#39;data-original-title&#39;</strong>].replace(<strong>&#39;Icons&#39;</strong>, <strong>&#39;unknown&#39;</strong>).strip()<br>    league = clubs[2][<strong>&#39;data-original-title&#39;</strong>].replace(<strong>&#39;Icons&#39;</strong>, <strong>&#39;unknown&#39;</strong>).strip()</pre><p>Then, print the data serves to check whether the data we will take is in line with expectations.</p><pre>print(<strong>&#39;Name     : &#39;</strong>,name)<br>print(<strong>&#39;Club     : &#39;</strong>,club)<br>print(<strong>&#39;Nation   : &#39;</strong>,nation)<br>print(<strong>&#39;League   : &#39;</strong>,league,<strong>&#39;\n&#39;</strong>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i0prLpFfAnDxxgIu6kEv0A.png" /></figure><p>Congratulations you have the data you want on one page. If you are going to scrape data from many pages and get thousands of data, let’s look at the next step.</p><h3>STEP 6: Pagination</h3><p>To do pagination, we do a page lopping with a certain range, for example from pages 1 to 5, then we loop the range with the “for” function that loops all the variable data that we scrape so that later the data from each page can be read properly. On the futbin site there are 379 pages and there are 30 players per one page. if you will do scraping all player data, so you can loop for the page in range 1 to 380 because python will read the loop from 1 to numbers before 380, it is 379.</p><pre><strong>for </strong>page <strong>in </strong>range(1, 380):<br>    req =  requests.get(<strong>&#39;https://www.futbin.com/players?page=&#39;</strong>+str(page))<br>    soup = bs4.Beautiful...</pre><p>You will get 739 x 30 records = 22,170 records.</p><h3>It’s Amazing!!!</h3><h3>STEP 7: Append the Data</h3><p>Create a variable that contains an empty list, then enter the data into the empty list. for example, I named the variable “datas” which I placed before the loop.</p><pre>datas =[]<br><strong>for </strong>page <strong>in </strong>range(1, 380):<br>    req =  requests.get(<strong>&#39;https://w...</strong></pre><p>Then I will append the data that we have scrapped into the variable. put the append function in a loop.</p><pre>...<br>print(<strong>&#39;League   : &#39;</strong>,league,<strong>&#39;\n&#39;</strong>)<br>datas.append([name, club, nation, league])</pre><h3>STEP 8: Save to CSV File</h3><p>After we successfully scrape the data, we can save it in CSV file format using the “write” function, for more details click <a href="https://docs.python.org/3/library/csv.html"><strong>here</strong></a> to see how to complete the guide using the CSV library.</p><pre><strong>with </strong>open(<strong>&#39;futbin.csv&#39;</strong>, <strong>&#39;w&#39;</strong>, newline=<strong>&#39;&#39;</strong>) <strong>as </strong>file:<br>    writer = csv.writer(file)<br>    headers = [<strong>&#39;Name&#39;</strong>,<strong>&#39;Address&#39;</strong>,<strong>&#39;Post Code&#39;</strong>, <strong>&#39;Telp&#39;</strong>,<strong>&#39;Website&#39;</strong>]<br>    writer.writerow(headers)<br>    <strong>for </strong>data <strong>in </strong>datas:<br>        writer.writerow(data</pre><p>Check scraped CSV file. Finish, good luck :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CuefhUs9KFrl-tR2ddlwdw.png" /></figure><p>I hope you can benefit from this article and develop it according to your own needs and imagination. And I ask forgiveness for any words and behave which are not supposed to be. Thank you for your kind attention Guys. Stay tuned for my next articles!…. :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=207abda35bba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web Scraping from HCAD Site to CSV file with Post Method Using Python]]></title>
            <link>https://medium.com/@rifaislamet1509/web-scraping-from-hcad-site-to-csv-file-with-post-method-using-python-51cf46457496?source=rss-bd872fe830e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/51cf46457496</guid>
            <category><![CDATA[csv]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Rifai Slamet]]></dc:creator>
            <pubDate>Tue, 14 Apr 2020 08:35:39 GMT</pubDate>
            <atom:updated>2020-04-14T10:39:27.460Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BdLY_p14PS0fGIUkG3FB6A.png" /></figure><p>Hey everyone, thanks for clicking on the title of this article. What will I write here? yes, that’s right. I will share how to easily scrape the hcad site. And as usual, I will not beat around the bush for this article, I will explain it briefly and clearly and easily understood, Oke? Next...</p><p>Before started, I hope you are on a new project and have the “<strong>requests</strong>”, “<strong>bs4</strong>” and “<strong>CSV</strong>” libraries installed on the virtual environment you use.</p><p>Are you ready? YEAH :D</p><h3>Let’s Get Started!</h3><h4>STEP 1: Import BeautifulSoup and Requests Module</h4><p>In the first line, we have to initialize the module that we will use so that the project can run well. That matter can be done by importing the module.</p><pre><strong>import </strong>bs4<br><strong>import </strong>requests<br><strong>import </strong>csv</pre><h4>STEP 2: Create URL, Headers and Form Data Variable</h4><p>Enter the user agent and referer site into the headers variable and enter the data in the form of a dictionary (key and value) according to the website.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*blO-QlMdEAk1tVZ_F0Bwog.png" /></figure><ol><li>Enter the name you want to search,</li><li>Click network settings to inspect the page. To inspect the page, press <strong>right click</strong> on the mouse then select “<strong>inspect</strong>” or press the “<strong>CTRL + SHIFT + I</strong>” button on the keyboard,</li><li>Enter the name again to see the URL request from the data that we will take,</li><li>Click on the name that appears,</li><li>Click Header to see the URL request and form data,</li><li>Data forms have been found there.</li></ol><pre>url = <strong>&quot;https://public.hcad.org/records/QuickRecord.asp&quot;</strong><br>hearders = {<strong>&#39;User-Agent&#39;</strong>: <strong>&#39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36&#39;</strong>,<br><strong>&#39;Referer&#39;</strong>: <strong>&#39;https://public.hcad.org/records/QuickRecord.asp&#39;</strong>}<br><br>form_data = {<br><strong>&#39;searchtype&#39;</strong>: <strong>&#39;name&#39;</strong>,<br><strong>&#39;searchval&#39;</strong>: <strong>&#39;Michael&#39;</strong>,<br><strong>&#39;taxyear&#39;</strong>: <strong>&#39;2020&#39;<br></strong>}</pre><p>Note: user-agent can be found on google by typing “<strong>whats my user-agent</strong>”. or you can click <a href="https://www.google.com/search?q=whats+my+user+agent&amp;oq=whats&amp;aqs=chrome.2.69i57j0j69i59j0l3j69i60l2.4426j0j7&amp;sourceid=chrome&amp;ie=UTF-8"><strong>here</strong></a><strong> </strong>to get it.</p><h4>STEP 3: Create the Peoples Variable</h4><p>Before we doing that, we request by the post method first and change the text format of the response to the bs4 format.</p><pre>response = requests.post(url, data=data, headers=hearders)<br>soup = BeautifulSoup(response.text, <strong>&#39;html.parser&#39;</strong>)</pre><p>Then, inspect the elements first in the browser that we use to find the class of the target that we are going to scrape.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZyI_28dNIN8JcC9PVF4izw.png" /></figure><p>In the picture, each people is in the <strong>“tr”</strong> tag and the <strong>“</strong>#ffffff” bgcolor. For that, we can use the “findAll” function in the beautifulsoup module to retrieve all information contained in the peoples.</p><pre>peoples = soup.find_all(<strong>&#39;tr&#39;</strong>,{<strong>&#39;bgcolor&#39;</strong>:<strong>&quot;#ffffff&quot;</strong>})</pre><h4>STEP 4: Lopping the Product Variable with “for” Function</h4><p>Then we will parse the data contained in the product class by repeating the ‘<strong>for’</strong> function. the data that I took in this article include <strong>name, account number, zip code,</strong> and<strong> address</strong>. You can develop your own by adding data variables that you will take and enter into the <strong>“find (‘ &lt;tag&gt;’ , {‘&lt;attribute name&gt;’:’attribute value’})”</strong> function.</p><p>Because many tags and attributes are the same, we can take all the tags and attributes into a list. To find the people&#39;s name, we just need to call the names in which order. <strong>Remember! python sorts data calls in a list from the number 0.</strong></p><pre><strong>for </strong>people <strong>in </strong>peoples:<br>    name = people.findAll(<strong>&#39;td&#39;</strong>,{<strong>&#39;valign&#39;</strong>:<strong>&quot;center&quot;</strong>})[1].text.strip()<br>    id = people.findAll(<strong>&#39;td&#39;</strong>,{<strong>&#39;valign&#39;</strong>:<strong>&quot;center&quot;</strong>})[0].text.strip()<br>    address = people.findAll(<strong>&#39;td&#39;</strong>,{<strong>&#39;valign&#39;</strong>:<strong>&quot;center&quot;</strong>})[2].text.strip()<br>    zip = people.findAll(<strong>&#39;td&#39;</strong>,{<strong>&#39;valign&#39;</strong>:<strong>&quot;center&quot;</strong>})[3].text.strip()</pre><p>Then, print the data serves to check whether the data we will take is in line with expectations.</p><pre>print(name)<br>print(id)<br>print(zip)<br>print(address, <strong>&#39;\n&#39;</strong>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SLJKyj6lYztGpSBZXlAR0w.png" /></figure><p>Congratulations you got the data. Then let’s go to the next step to store it in CSV file.</p><h4>STEP 5: Append the Data</h4><p>Create a variable that contains an empty list, then enter the data into the empty list. for example, I named the variable “datas” which I placed before the loop.</p><pre>datas=[]<br><strong>for </strong>people <strong>in </strong>peoples:<br>    name = peop...</pre><p>Then I will append the data that we have scrapped into the variable. put the append function in a loop.</p><pre>...<br>zip = row.findAll(<strong>&#39;td&#39;</strong>,{<strong>&#39;valign&#39;</strong>:<strong>&quot;center&quot;</strong>})[3].text.strip()<em><br></em>datas.append([name,id,address,zip])</pre><h3>7. Save to CVS File</h3><p>After we successfully scrape the data, we can save it in CSV file format using the “write” function, for more details click <a href="https://docs.python.org/3/library/csv.html"><strong>here</strong></a> to see how to complete the guide using the CSV library.</p><pre><strong>with </strong>open(<strong>&#39;hcad.csv&#39;</strong>, <strong>&#39;w&#39;</strong>, newline=<strong>&#39;&#39;</strong>) <strong>as </strong>file:<br>    writer = csv.writer(file)<br>    headers = [<strong>&#39;Name&#39;</strong>,<strong>&#39;Address&#39;</strong>,<strong>&#39;Post Code&#39;</strong>, <strong>&#39;Telp&#39;</strong>,<strong>&#39;Website&#39;</strong>]<br>    writer.writerow(headers)<br>    <strong>for </strong>data <strong>in </strong>datas:<br>        writer.writerow(data</pre><p>Check scraped CSV file. Finish, good luck :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gen051bMSMbUV1HqRVQWcA.png" /></figure><p>I hope you can benefit from this article and develop it according to your own needs and imagination. And I ask forgiveness for any words and behave which are not supposed to be. Thank you for your kind attention Guys. Stay tuned for my next articles!…. :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51cf46457496" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to do Web Scraping with Pagination from yell.com to CSV file Using Python]]></title>
            <link>https://medium.com/@rifaislamet1509/how-to-do-web-scraping-with-pagination-from-yell-com-to-csv-file-using-python-960e9ab38f06?source=rss-bd872fe830e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/960e9ab38f06</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[csv]]></category>
            <dc:creator><![CDATA[Rifai Slamet]]></dc:creator>
            <pubDate>Tue, 14 Apr 2020 07:01:00 GMT</pubDate>
            <atom:updated>2020-04-26T07:57:51.999Z</atom:updated>
            <content:encoded><![CDATA[<h3>Easily Web Scraping with Pagination from yell.com to CSV file using Python</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0E_jtdp2DTKU21nMDQvhdA.png" /></figure><p>Dear reader, would you like to scrap directory sites like yell.com? and do you want to save the results in CSV file format? If true, then consider this article to the end while enjoying a cup of your coffee. Enjoy :)</p><p>I will share how I do web scraping quickly on directory websites such as in this case study, yell.com. I will make this article brief but clear so that you do not get bored with long-winded words.</p><p>Before doing so make sure you are on a new project in your python idle. I use Pycharm as python idle because it’s very easy and you have installed bs4 and requests. I did not explain how to install it here because I will only focus on how to do the scraping. please visit my previous article, there are ways on how to install the package and use the virtual environment. Thank you.</p><h3>Let’s get Started</h3><h4>1. Import BeautifulSoup and Requests Modul</h4><p>In the first line, we have to initialize the module that we will use so that the project can run well. That matter can be done by importing the module.</p><pre><strong>import </strong>bs4<br><strong>import </strong>requests<br><strong>import </strong>csv</pre><h4>2. Create URL and header variables</h4><p>In the URL variable, we can create an input function that functions as the keyword that we are looking for. The header variable is filled with user-agent.</p><p>the URL variable like as follows:</p><pre>url = <strong>&#39;https://www.yell.com/ucs/UcsSearchAction.do?keywords={0}&amp;location={1}&amp;scrambleSeed=509559697&amp;pageNum=1’</strong>.format(input(<strong>’Enter term : &#39;</strong>),input(<strong>’Enter location :’</strong>))<br></pre><p>the headers variable like as follows:</p><pre>header = {<strong>&#39;User-Agent&#39;</strong>: <strong>&#39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36&#39;</strong>}</pre><p>Note: user-agent can be found on google by typing “<strong>whats my user-agent</strong>”. or you can click <a href="https://www.google.com/search?q=whats+my+user+agent&amp;oq=whats&amp;aqs=chrome.2.69i57j0j69i59j0l3j69i60l2.4426j0j7&amp;sourceid=chrome&amp;ie=UTF-8"><strong>here</strong></a><strong> </strong>to get it.</p><h4>3. Create Product Variable</h4><p>Before we do, we request by the method of getting it first and change the text format of the request to the bs4 format.</p><pre>req = requests.get(url, headers= headers)<br>soup = bs4.BeautifulSoup(.text, &#39;html.parser’)</pre><p>Then, inspect the elements first in the browser that we use to find the class of the target that we are going to scrape.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FEhbVjVWoFFCGfFFwm4REA.png" /></figure><p>In the picture each product is in the <strong>“div”</strong> tag and the <strong>“row businessCapsule — mainRow</strong>” class . For that we can use the “findAll” function in the beautifulsoup module to retrieve all information contained in the product.</p><pre>product = soup.findAll(&#39;div&#39;, &#39;row businessCapsule--mainRow&#39;)</pre><h4>4. Lopping the Product Variable with “for” Function</h4><p>Then we will parse the data contained in the product class by repeating the ‘for’ function. the data that I took in this article include name, address, telephone number and website. then you can develop your own by adding data variables that you will take and enter into the “find (‘ &lt;tag&gt; ‘, {‘&lt;attribute name&gt;’:’attribute value’})” function.</p><pre>for x in product:<br>   name = x.find(&#39;span&#39;,&#39;businessCapsule--name&#39;).text<br>   address = x.find(&#39;span&#39;, {&#39;itemprop&#39;:&#39;streetAddress&#39;}).text<br>   post_code = x.find(&#39;span&#39;, {&#39;itemprop&#39;:&#39;postalCode&#39;}).text<br>   telp = x.find(&#39;span&#39;, &#39;business--telephoneNumber&#39;).text<br>   web = x.find(&#39;a&#39;, {&#39;rel&#39;: &#39;nofollow noopener&#39;})[&#39;href&#39;].split(&#39;?&#39;)[0].replace(&#39;<a href="https://&/#39;,&#39;&#39;).replace(&#39;http://&#39;,&#39;&#39;).replace(&#39;http:&#39;,&#39;&#39;).replace(&#39;www.&#39;,&#39;&#39;)">https://&#39;,&#39;&#39;).replace(&#39;http://&#39;,&#39;&#39;).replace(&#39;http:&#39;,&#39;&#39;).replace(&#39;www.&#39;,&#39;&#39;)</a><br>specifically for the web we control with the function &quot;try except&quot; because not all products have a website</pre><p>Then, print the data serves to check whether the data we will take is in line with expectations.</p><pre>print(&#39;Name      :&#39;,name)<br>print(&#39;Addres    :&#39;,address)<br>print(&#39;Post Code :&#39;,post)<br>print(&#39;Tel       :&#39;,telp)<br>print(&#39;Website   :&#39;,web,&#39;\n&#39;)</pre><p>Specifically for the web data that we control with the “try except” function because not all products have a website. if you find the same thing in the data you want to scrape, then just add the data variable to the function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WIl6hMKXL3lvivdXemomJw.png" /></figure><p>Congratulations you have the data you want on one page. If you are going to scrape data from many pages, let’s look at the next step.</p><h4>5. Pagination</h4><p>To do pagination, we do a page lopping with a certain range, for example from pages 1 to 5, then we loop the range with the “for” function that loops all the variable data that we scrape so that later the data from each page can be read properly.</p><p>For example, I will be scraping data from pages 1 to 100, then the numbers in the range function are 1 and 101. why 101? because python will read the loop from 1 to numbers before 101, it is 100.</p><pre><strong>for </strong>page <strong>in </strong>range(1,101):<br>    req = requests.get(url+str(page), headers= headers)<br>    soup = bs4.BeautifulSoup(req.text, <strong>&#39;html.parser&#39;</strong>)<br>    product = soup.find....<br></pre><h4>6. Append the Data</h4><p>Create a variable that contains an empty list, then enter the data into the empty list. for example, I named the variable “datas” which I placed before the loop.</p><pre>datas=[]<br><strong>for </strong>page <strong>in </strong>range(1,101):<br>    req = requests.get...</pre><p>Then I will append the data that we have scrapped into the variable. put the append function in a loop.</p><pre>...<br><strong>except</strong>:<br>    web = <strong>&#39;Nothing&#39;<br></strong>datas.append([num, name, street, local, post, telp, web])</pre><h4>7. Save to CSV File</h4><p>After we successfully scrape the data, we can save it in CSV file format using the “write” function, for more details click <a href="https://docs.python.org/3/library/csv.html"><strong>here</strong></a> to see how to complete the guide using the CSV library.</p><pre><strong>with </strong>open(<strong>&#39;yell.csv&#39;</strong>, <strong>&#39;w&#39;</strong>, newline=<strong>&#39;&#39;</strong>) <strong>as </strong>file:<br>    writer = csv.writer(file)<br>    headers = [<strong>&#39;Name&#39;</strong>,<strong>&#39;Address&#39;</strong>,<strong>&#39;Post Code&#39;</strong>, <strong>&#39;Telp&#39;</strong>,<strong>&#39;Website&#39;</strong>]<br>    writer.writerow(headers)<br>    <strong>for </strong>data <strong>in </strong>datas:<br>        writer.writerow(data</pre><p>Check scraped CSV file. Finish, good luck :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dttqneAurDNhrTfz4LDaBg.png" /></figure><p>I hope you can benefit from this article and develop it according to your own needs and imagination. And I ask forgiveness for any words and behave which are not supposed to be. Thank you for your kind attention Guys. Stay tuned for my next articles!…. :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=960e9ab38f06" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>