<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Data engineering on Medium]]></title>
        <description><![CDATA[Stories by Data engineering on Medium]]></description>
        <link>https://medium.com/@dataengineering3?source=rss-5dc47af8639e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*4gWg_ghEkbEbFcgC8dv_Rw.jpeg</url>
            <title>Stories by Data engineering on Medium</title>
            <link>https://medium.com/@dataengineering3?source=rss-5dc47af8639e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:03:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dataengineering3/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[IMDB: Rating of Movies]]></title>
            <link>https://medium.com/@dataengineering3/imdb-rating-of-movies-1f80af64cddd?source=rss-5dc47af8639e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f80af64cddd</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Data engineering]]></dc:creator>
            <pubDate>Mon, 02 Dec 2019 15:32:49 GMT</pubDate>
            <atom:updated>2019-12-02T15:32:49.302Z</atom:updated>
            <content:encoded><![CDATA[<p>Web scraping is the process of extracting data from websites. All the job is carried out by a piece of code which is called a “scraper”. First, it sends a “GET” query to a specific website. Then, it parses an HTML document based on the received result. After it’s done, the scraper searches for the data you need within the document, and, finally, converts it into the specified forma</p><p>IMDB is an Internet Movie Database that contain information related to films, television programs, home videos, video games, and streaming content online. IMDB is also including cast, production crew and personal biographies, plot summaries, trivia, fan and critical reviews, and ratings.</p><p><strong>EXTRACT :</strong></p><p>During the process of web scraping, we have to make sure that all the data we pick is legal to scrape due to the content copy protection, such as multi-level layout, using JavaScript for content rendering and user-agent validations.</p><p>We also found out that if the data in the websites that we are going to scrape is quite small, it is better to write our own scraper and customize it according to each specific website. The quality of the output content should be 100%. If the number of websites to scrape goes beyond “small”, we suggest using a complex approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/0*Z4ZvXT91pWZE4RLL" /></figure><p>To scrape the data, we will right click on the selected web page to get the inspect of elements. Firstly we will count the number of movies that contain in one page. Then, we will start to scrape the first movie’s elements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/0*xQP0nA16liQiTjjL" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/0*baUPX9O2_JgJ74hq" /></figure><p>From the elements, we can start to identify the data attributes that contain in each of the movie. The attributes that we have chosen is movie’s name, movie’s poster, movie’s MPAA rating, movie’s year, movie’s runtime, movie’s rating, movie’s metascore, movie’s votes and movie’s genre.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/0*Q-lEFfbTdM8JIL-t" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/0*XBkUicrV8f_XQSUt" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/0*486eR5WIC2uy-AnZ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/0*onuN_6HfjafPrUnH" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/0*2X2gHPDYRkWP0NJM" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/0*5mlvAEY5pzHtptH2" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*Nl0-SVvPkZvNCuyx" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/0*0wNw0SOJpte8TOBV" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/0*O-2WwVOmDgDYr5NM" /></figure><p>Below is show the overall code for extracting data:</p><p><strong>(2) TRANSFORM:</strong></p><p><strong><em>Data Processing</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/0*4_YVkhn4sHwTJoRL" /></figure><p>Since the result of year will display I or II for different series such as (I)(2017), we need to remove the “(I)(“ and “)” in year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/0*xWMEBrnym8S7jO2j" /></figure><p>To calculate the average runtime for all the movies on the web page that we want to scrape, we need to remove “min” in runtime for an easier calculation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t0j_v11Jg9QsfGD_" /></figure><p>Since there is an irrelevant data in the gross attribute, we need to drop it to prevent the irrelevant data disturb our scraped data. For example, ‘Bird Box’ movie holds a ‘PG-13’ as it’s gross attribute, which should the value should be the amount of value that earned through this movie.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/0*vUeh0npqu0X-GUEW" /></figure><p>We found that there have incorrect data types assigned to some features. We corrected the data types and converted them into correct data type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/0*ejlzDUl1YBr8YW3S" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*dHigSXzWG6_h7Mwz" /></figure><p>We also removed the unwanted white space in genre.</p><p><strong><em>Data Visualization</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8t802pnqU2YoBbI_" /></figure><p>We also carry out the data visualization to sum up the total number of certificates in each year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ojGwdb67sQ9ydo0G" /></figure><p>Three histogram has been drawn for data visualization.</p><p><strong>(3)LOAD:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/0*Hk7SdFHqtCTxYR1S" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f80af64cddd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>