<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Angela/Qiuyun Shen on Medium]]></title>
        <description><![CDATA[Stories by Angela/Qiuyun Shen on Medium]]></description>
        <link>https://medium.com/@shenqiuyun228?source=rss-80ecc4d52399------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*AJ4zKnJ0gdtF1ytb.</url>
            <title>Stories by Angela/Qiuyun Shen on Medium</title>
            <link>https://medium.com/@shenqiuyun228?source=rss-80ecc4d52399------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 08:55:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shenqiuyun228/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Movie Grosses Prediction]]></title>
            <link>https://medium.com/@shenqiuyun228/movie-grosses-prediction-89c9a3f0447c?source=rss-80ecc4d52399------2</link>
            <guid isPermaLink="false">https://medium.com/p/89c9a3f0447c</guid>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[polynomial-regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Angela/Qiuyun Shen]]></dc:creator>
            <pubDate>Tue, 06 Feb 2018 00:25:53 GMT</pubDate>
            <atom:updated>2018-02-06T00:25:53.754Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Movie Grosses Prediction</strong></p><p>This project is for movie future investment firm. Data was scraped from the web, linear regressions and polynomial regression were used for worldwide grosses prediction and all models and features were selected based on Cross-Validation test. The models were then regularized and optimized to help movie investment.</p><p><strong>1.</strong> <strong>Data Collection</strong></p><p>Data was collected from boxofficemojo.com from 1989 to 2017 with 7373 movies and then stored in csv files. For each movie, the information collected includes Rank, Link, Title, Worldwide_Grosses, Domestic_Grosses, Domestic%, Overseas_Grosses, Overseas%, Distributor, Genre, Runtime, MPAA_Rating, Theaters, Release_Date, Release_Days, Production_Budget, Director, Writer, Actor_1, Actor_2, Now_Opening. The target of this project is Worldwide_Grosses.</p><p><strong>2.</strong> <strong>Data Cleaning</strong></p><p>a. Read csv files into Pandas DataFrame, and delete movies lack of Production Budget information;</p><p>b. Delete movies with other NULL data;</p><p>c. The distribution of worldwide grosses is right skewed, then delete movies with worldwide grosses than 800 million dollars;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*n61iXp8jtB7hos549xxL8g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*cTQ4dppypbdXlQhTNZFD0g.png" /><figcaption>Grosses distribution before and after deleted outliers</figcaption></figure><p>d. Split release_date into month and year, treat the month as categorical, and year as numeric.</p><p>7097 movies were narrowed down to 1663.</p><p><strong>3.</strong> <strong>OLS Modeling</strong></p><p><strong>Analysis</strong>: Select all the numeric columns and model with statsmodels OLS. Check coefficient of numeric features with worldwide grosses. Production budget shows the highest coefficient with worldwide grosses (0.725923), and release year has the lowest coefficient with worldwide grosses. The R2 of OLS model with all numeric features is 0.627, after deleting the release year feature, the R2 reduced to 0.622. The shape of predicted <em>vs</em> residual plot is funnel, which may be caused by having a distribution with a very wide range of values, or having features that lack the full information needed to predict the problem.</p><p><strong>Conclusion</strong>: Keep all the numeric features for future modeling, and add more categorical features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*7jfj46wirCs7PV1-vrR6rw.png" /><figcaption>Residuals of OLS modeli</figcaption></figure><p><strong>4.</strong> <strong>Models and features selection</strong></p><p><strong>Analysis: </strong>Different combinations of categorical features were added with numeric features for four models regression. The models tried were linear regression, ridge regression, lasso regression and degree 2 polynomial regression, and then selected through cross-validation test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9f_tXYa_hVqmxp5m7-wQdA.png" /></figure><p><strong>Conclusion:</strong></p><p>The models and features selected:</p><p>Lasso Regression R2=0.641: Release Month + Genre + Numeric<br> Degree 2 Polynomial R2=0.676: Genre + Numeric</p><p><strong>5.</strong> <strong>Lasso Regularization for linear regression:</strong></p><p>The linear regression was then regularized using Lasso Regularization. 4 columns with 0 coefficient according to Ridge regularization were deleted. The Cross-Validation test R2 was increased to 0.642.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*cq2oRk4j9gkMoy8EvpCssw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*pMZsgDSD5JrTK4gg9HuhiA.png" /></figure><p><strong>6.</strong> <strong>Degree 2 Polynomial regression:</strong></p><p>The Cross-Validation test R2 of degree-2 polynomial was 0.676.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*dfEP85QV7Y9I1dYfO4KofA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*eakXMNjSwe0OjhwzrzW75A.png" /></figure><p><strong>7.</strong> <strong>Conclusion:</strong></p><p>The best model for worldwide grosses prediction selected in this project is degree 2 polynomial regression, and the features for prediction were runtime, theaters, release days, production budget, release year and genre.</p><p>8. <strong>Next Steps:</strong></p><p>a. Try More feature, such as writers, directors, actors, etc..</p><p>b. Try more regressions, such as logistic regression.</p><p>c. Get more budget data for most of the movies, collect from other resource or predict based on other features.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89c9a3f0447c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Billboard Placement in NY Subway Station]]></title>
            <link>https://medium.com/@shenqiuyun228/billboard-placement-in-ny-subway-station-375fc29e7e72?source=rss-80ecc4d52399------2</link>
            <guid isPermaLink="false">https://medium.com/p/375fc29e7e72</guid>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[ny-subway-station]]></category>
            <dc:creator><![CDATA[Angela/Qiuyun Shen]]></dc:creator>
            <pubDate>Thu, 25 Jan 2018 00:46:51 GMT</pubDate>
            <atom:updated>2018-01-25T00:48:10.258Z</atom:updated>
            <content:encoded><![CDATA[<p>Last week was my first week in Metis 2018 Winter Data Science Bootcamp. I have learned a lot about downloading data, reading the data into pandas dataframe, exploring the data, and cleaning the data with deleting the anomalous records. And then have learned how to get desired information from the data using pandas functions and visualize the results using matplotlib library and pandas.plot.</p><p>During learning the pandas data analysis library, we got our hands into the Benson project. The data was obtained from turnstile data files published by the New York City M for a period of two months (2017.08–2017.09). Working as a team of four, we tried to find out the top 10 subway stations in New York City with the most ridership for the billboard display of a company named Green Napkin.</p><p>NYC MTA data contained a unique identifier for each turnstile (control area, remote unit for a station, subunit channel position, station name and train lines can be boarded at the station), date and time, and the cumulative entry and exit counts for each turnstile. The data needed extensive cleaning and preprocessing, including dropping duplicate rows, sorting train line names (some train line names for one station have different orders), removing anomalous turnstiles (the cumulative counts are significant large or small). Some of the devices were counting down which was corrected by taking the absolute count.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*ZMQpAq910RZHeGLw0WVk-w.png" /></figure><p>After data cleaning and analysis, we obtained the top 10 subway stations with the most ridership in New York, and the first one is GRD CNTRL-42 ST.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NYLc0ZGOk56xwXQm2U4qnA.png" /></figure><p>We then located the top 10 stations on the map. The first nine were in Manhattan, and the last one was in Brooklyn. The Average HouseHold Income (AHI) data was obtained from Zillow based on the ZIP code. The AHI of New York city is $88,437, and that of Manhattan is $138,758.</p><p>All 10 stations are located in an area with higher than average AHI, and the AHI’s of the top 8 areas are higher than that of Manhattan in general. This indicates that all selected stations are located in areas with medium and medium-high household incomes. There is no data for the 9th station (47th–50th Streets–Rockefeller Center), which suggests that this station is located in an office or commercial area without residential spaces.</p><p>The above work is just the tip of the iceberg. We can analyze years of data to get more accurate results and track trends to make data-driven predictions<em>. </em>A more accurate analysis would focus on excluding tourist areas, and we can recalculate the results for stations with lower mean ridership but higher ad retention rates. We can also analyze other cities and even countries!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=375fc29e7e72" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>