<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lauren Faulds on Medium]]></title>
        <description><![CDATA[Stories by Lauren Faulds on Medium]]></description>
        <link>https://medium.com/@lfaulds?source=rss-53441b27c02d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*T9uotUxjePyOREF6zkp8cg.jpeg</url>
            <title>Stories by Lauren Faulds on Medium</title>
            <link>https://medium.com/@lfaulds?source=rss-53441b27c02d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:24:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lfaulds/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Sliding Window Price Predictions]]></title>
            <link>https://medium.com/analytics-vidhya/sliding-window-price-predictions-20fea38bae6d?source=rss-53441b27c02d------2</link>
            <guid isPermaLink="false">https://medium.com/p/20fea38bae6d</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[timeseries-forecasting]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Lauren Faulds]]></dc:creator>
            <pubDate>Tue, 21 Jul 2020 19:35:32 GMT</pubDate>
            <atom:updated>2020-08-02T15:58:12.061Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The code used here is available in its original </em><a href="https://github.com/manyshapes/Produce-Soup/blob/master/Web%20scraping%20fruit%20index.ipynb"><em>repository</em></a><em> in .ipynb format. You can download it &amp; fiddle with it in Jupyter Notebook on your own device.</em></p><h3>Intro</h3><p>Today we’ll be seeing how we can use historic produce prices to make predictions over a twenty year period. This will be done in <strong>Python</strong> using a simple linear regression model. <strong>Beautiful Soup 4</strong> helps with parsing the observations from an online source. This data will then be accessed &amp; manipulated from a <strong>Pandas</strong> dataframe. Training will be done on a sliding window; this and model fitting will be conducted using<strong> Sci Kit Learn</strong>.</p><p><em>Our package imports:</em></p><pre>from bs4 import BeautifulSoup<br>import requests<br>import numpy as np<br>import pandas as pd<br>from pandas import concat<br>from sklearn.model_selection import TimeSeriesSplit<br>from sklearn.linear_model import LinearRegression<br>from sklearn.metrics import mean_squared_error<br>from math import sqrt</pre><p>This post covers the core functions for linear regression using a sliding window on time series data. To see more on visualizing the results of our linear regression with <strong>Yellowbrick </strong>as well as other findings, please see original repository.</p><h4><em>✧ Outline of this Post:</em></h4><ol><li><em>Preparation</em> — Getting model ready data with acquisition, cleaning &amp; feature engineering</li><li><em>Model Fitting </em>— Linearly regressing time series splits</li><li><em>About the Data </em>— More on produce</li></ol><figure><img alt="Produce in basket" src="https://cdn-images-1.medium.com/max/800/1*UydXpQ9495G__e5uDws2Gw.jpeg" /></figure><h3>Acquisition</h3><p>The function created for acquiring Produce data is very much tailored for the unique website: <a href="http://www.producepriceindex.com/">Produce Price Index</a>. Here’s a look at the sort of information we’ll be extracting.</p><figure><img alt="Commodity, Record Date, Farm Price, Atlanta Retail, Chicago Retail, Los Angeles Retail, New York Retail, Average Spread" src="https://cdn-images-1.medium.com/max/1024/1*Vp7oLh4cm9FMzPa-cmwlGA.png" /></figure><p>We’ll need to parse the HTML to save each row’s unique values. For 638 pages worth, there are about 15,950 rows. Luckily with the package <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/"><strong>Beautiful Soup</strong></a> (v4) it’ll be easy enough to distill what we need from this resource (see link for installation process). Values are held in specific HTML classes, such as Strawberries in “fix-width-16”. These classes’ contents will be emptied then zipped together as tuples.</p><pre>def getproducedata(inpages):<br>    &#39;&#39;&#39;<br>    The number argument inpages will be the page the function finishes on. Each row&#39;s values will be stored in a tuple &amp; added to returned list tuple_data.<br>    &#39;&#39;&#39;</pre><pre>    tuple_data = []<br>    url = &quot;<a href="http://www.producepriceindex.com/produce-price-index?field_ppi_commodity_target_id=All&amp;field_ppi_date_value%5Bmin%5D&amp;field_ppi_date_value%5Bmax%5D&amp;page={">http://www.producepriceindex.com/produce-price-index?field_ppi_commodity_target_id=All&amp;field_ppi_date_value%5Bmin%5D&amp;field_ppi_date_value%5Bmax%5D&amp;page={</a>}&quot;<br>    page_nums = list(range(inpages))<br>    <br>    for page_num in page_nums:<br>        page_url = url.format(page_num)<br>        rawhtml = requests.get(url.format(page_num)).text<br>        body = BeautifulSoup(rawhtml, &quot;lxml&quot;).tbody<br>        <br>        producelist = []<br>        for produceitem in body.find_all(&quot;td&quot;, {&quot;class&quot;: &quot;fix-width-16&quot;}):<br>            producelist.append(produceitem.text.strip())<br>        <br>        pricedatelist = []<br>        for pricedate in body.find_all(&quot;td&quot;, {&quot;class&quot;: &quot;fix-width-12&quot;}):<br>            pricedatelist.append(pricedate.text.strip())<br>        # The list is broken up into pieces, for 6 values in each row<br>        pricedate_list = [pricedatelist[x:x+6] for x in range(0, len(pricedatelist),6)]<br>        <br>        #Spaces are removed before the spread value is added<br>        spreadlist = []<br>        for each in body.find_all(&quot;td&quot;, {&quot;class&quot;:&quot;views-field views-field-expression-4&quot;}):<br>            spreadlist.append(each.text.strip())<br>        <br>        #combine spread with the rest of the pricedate list<br>        pricedate_list = [x + [y] for x, y in zip(pricedate_list, spreadlist)]<br>        <br>        #combine produce names &amp; pricedates<br>        producetuples = list(zip(producelist, pricedate_list))<br>        tuple_data += producetuples<br>    <br>    return tuple_data</pre><p>We’ve successfully retrieved data by formatting the URL &amp; caching the relevant information from each page. Next we’ll work it into a usable data frame.</p><h3>Cleaning</h3><figure><img alt="Photo from http://www.personalcreations.com/" src="https://cdn-images-1.medium.com/max/1024/1*DgEjkK_e--0vOrsiTjbeFw.jpeg" /></figure><p>We’ll use <strong>Pandas</strong> to pop our tuples into a dataframe and get cleaned up. We’ll be fixing our columns, so each column carries one value. Additional columns will also be created that carry information like averaged price &amp; date attributes.</p><pre>def cleanproduce(dataframe):<br>    &#39;&#39;&#39;<br>    Values will be divided into appropriate columns <br>    Errors will be removed in the process <br>    The Date will become the index for time series analysis</pre><pre>    &#39;&#39;&#39;</pre><pre>    #The 2nd column&#39;s data is broken up, &amp; the 2nd column removed<br>    dataframe[[&#39;Date&#39;, &#39;Farm_Price&#39;, &#39;Atlanta_Retail&#39;, &#39;Chicago_Retail&#39;, &#39;LA_Retail&#39;, &#39;NY_Retail&#39;, &#39;AVGspread&#39;]] = pd.DataFrame(produceframe[&quot;Attris&quot;].to_list(), columns=[&#39;Date&#39;, &#39;Farm_Price&#39;, &#39;Atlanta_Retail&#39;, &#39;Chicago_Retail&#39;, &#39;LA_Retail&#39;, &#39;NY_Retail&#39;, &#39;AVGspread&#39;])<br>    <br>    # Columns made numeric with nonnumber characters removed<br>    for each in [&#39;Farm_Price&#39;, &#39;Atlanta_Retail&#39;, &#39;Chicago_Retail&#39;, &#39;LA_Retail&#39;, &#39;NY_Retail&#39;]:<br>        dataframe[each] = pd.to_numeric(dataframe[each].str[1:])<br>    dataframe[&#39;AVGspread&#39;] = dataframe[&#39;AVGspread&#39;].replace({&#39;\%&#39;: &#39;&#39;, &#39;,&#39;: &#39;&#39;}, regex=True).astype(float)<br>    produceframe[&#39;AVGspread&#39;] = produceframe[&#39;AVGspread&#39;] / 100<br>    dataframe[&#39;AVGprice&#39;] = dataframe[[&#39;Atlanta_Retail&#39;, &#39;Chicago_Retail&#39;, &#39;NY_Retail&#39;, &#39;LA_Retail&#39;]].mean(axis=1)<br>    <br>    # Removing data intake errors<br>    dataframe.dropna(inplace=True)<br>    dataframe = dataframe[dataframe.AVGprice &lt; 9]<br>    <br>    #Order the data chronologically<br>    dataframe = dataframe.iloc[::-1]<br>    # The frame can be made into a time series, a numeric index is preserved<br>    dataframe[&#39;Date&#39;] = pd.to_datetime(dataframe.Date)<br>    dataframe[&#39;Year&#39;] = dataframe.Date.dt.year<br>    dataframe[&#39;Month&#39;] = dataframe.Date.dt.month<br>    dataframe[&quot;Num_Index&quot;] = range(1, (len(dataframe)+1))<br>    dataframe = dataframe.set_index(&#39;Date&#39;)<br>    <br>    # Dummie variables are created for produce category<br>    dataframe[&quot;Produce_&quot;] = dataframe[&quot;Produce&quot;]<br>    dataframe = pd.get_dummies(dataframe, columns=[&quot;Produce_&quot;], prefix=&#39;Produce&#39;, drop_first=True)</pre><pre>    return dataframe</pre><p>In dummy variables the produce column is now a binary feature and usable for linear regression. The original categorical variable is still preserved, this significantly enables groupby functions, which assist in our feature engineering.</p><h3>Creating Features</h3><p>The time series features we can add now will be simple but effective. Each produce price observation will include the previous logged price &amp; an average of last 3. This will create a small amount of empty values, where no previous or no 3 previous is available &amp; will be removed afterwards.</p><pre>def timeseriesfeatures(dataframe):<br>    # Last entry as well as average of last 3 prices saved<br>    previousprice = dataframe.groupby(&#39;Produce&#39;,  as_index=False)[&#39;AVGprice&#39;].shift(1)<br>    avg3prev = previousprice.rolling(3).mean()<br>    <br>    # Dataframe of these values created &amp; indiced <br>    prevsavgs = concat([avg3prev, previousprice], axis =1)<br>    prevsavgs.columns = [&#39;Window_AVG&#39;, &#39;Prev_AVG&#39;]<br>    <br>    # Original dataframe will aquire new features &amp; drop resulting na<br>    dataframe = concat([dataframe, prevsavgs], axis = 1)<br>    dataframe = dataframe.dropna()<br>    <br>    return dataframe</pre><p>Our model uses these previous observations to estimate future price.</p><h3>Modeling</h3><figure><img alt="Child looking out window" src="https://cdn-images-1.medium.com/max/910/1*AeADsIE4UKV60zgDSaNvWA.jpeg" /><figcaption>Looking only at what’s present</figcaption></figure><p>When we’re doing time series predictions we want machine learning where only the information available at a given time is used. Data will be divided into folds for training on information &amp; testing novel predictions.</p><p>We might be given one training fold to test one testing fold — this would be a sliding window. Or we might progressively accumulate our folds to test with more data— this would be an expanding window. Sci Kit Learn offers a time series split object that easily provides an expanding window.</p><p>You may be wondering <em>why pick sliding window</em>?</p><p>Our data set stretches a long period of time &amp; experiences produce price volatility. Here, previous patterns may hinder novel training. I found smaller test errors occurring from a sliding window.The linear regression model will only have a selected frame before the event &amp; can learn on the most relevant trends.</p><p>So we will make a sliding window from our expanding window object to make predictions as well as analyze our mistakes.</p><pre>def timeseriesregression(X):<br>    &#39;&#39;&#39;<br>    Builds tscv expanding window into sliding window<br>    Fitting to train test splits &amp; storing test scores &amp;<br>    indices in lists<br>    &#39;&#39;&#39;<br>    trained_on = []<br>    r_scores = []<br>    rmse_scores = []<br>    prev_train = 0<br>    for train_index, test_index in tscv.split(X):</pre><pre>#   An array of indices are created that starts<br>    #   at the finish of the previous training set<br>    #   &amp; ends on the start of the current test set</pre><pre>        last_step = test_index[0]<br>        a_train_index = np.arange(prev_train, last_step)</pre><pre>#   Train &amp; Test portions are then allocated   <br>        X_train, X_test = X.iloc[a_train_index], X.iloc[test_index]<br>        y_train, y_test = y.iloc[a_train_index], y.iloc[test_index]</pre><pre>#   Linear Regression model is fit to the current training set  <br>        lrm = LinearRegression()<br>        lrm.fit(X_train, y_train)</pre><pre>        print(&#39;Test For Rows:&#39;, test_index)<br>        print(&#39;For Years:&#39;, X_test[&#39;Year&#39;].unique())</pre><pre>#   Validation scores are saved with their indices for<br>    #   later analysis<br>        r_score = lrm.score(X_test, y_test)<br>        r_scores.append(r_score)<br>        rmse_score = sqrt(mean_squared_error(y_test, (lrm.predict(X_test))))    <br>        rmse_scores.append(rmse_score)<br>        print(r_score, rmse_score, &#39;\n&#39;)</pre><pre>        trained_on.append(a_train_index)</pre><pre>#   The final row is saved for the next iteration  <br>        prev_train = train_index[-1]</pre><p>Looking at the results, this model gives us an average R-Score of 0.91 over 10 iterations. Performance is best on the first couple of iterations as well as the last couple of iterations. Further analysis can be done to see what may need to be optimized for the middle of the dataset where performance is lower.</p><h3>About the Dataset</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*qExDPGsS3DDdLIkzC4lmUQ.png" /><figcaption><a href="http://www.producepriceindex.com/"><strong>Produce Price Index</strong></a></figcaption></figure><p>Starting in 1999 Western Growers took on documenting prices of produce at farm and retail in cities Atlanta, Chicago, Los Angeles &amp; New York. Two decades later in 2019 Western Growers concluded their expedition in price tracking — at least for the time being. They are advocates for producers in the agricultural sector and more on their work can be <a href="https://www.wga.com/">found here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20fea38bae6d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/sliding-window-price-predictions-20fea38bae6d">Sliding Window Price Predictions</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>