<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Michael Salmon on Medium]]></title>
        <description><![CDATA[Stories by Michael Salmon on Medium]]></description>
        <link>https://medium.com/@msalmon00?source=rss-8e7466898cd5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3t1UsGDbh_3cvk9ML8d5RQ.jpeg</url>
            <title>Stories by Michael Salmon on Medium</title>
            <link>https://medium.com/@msalmon00?source=rss-8e7466898cd5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:44:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@msalmon00/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Helpful Python Code Snippets for Data Exploration in Pandas]]></title>
            <link>https://medium.com/@msalmon00/helpful-python-code-snippets-for-data-exploration-in-pandas-b7c5aed5ecb9?source=rss-8e7466898cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7c5aed5ecb9</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[code-samples]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Michael Salmon]]></dc:creator>
            <pubDate>Thu, 06 Jul 2017 18:54:12 GMT</pubDate>
            <atom:updated>2017-07-06T18:54:42.789Z</atom:updated>
            <content:encoded><![CDATA[<p>For anyone new to data exploration, cleaning, or analysis using Python, <a href="http://pandas.pydata.org/pandas-docs/stable/">Pandas</a> will quickly become one of your most frequently used and reliable tools. It is extremely versatile in its ability to work with a wide variety of existing data files (including csv, excel, json, html, and sql, among others), and can easily assemble data from lists or dictionaries into standard “data frames” that effectively display data in tabular form for easy manipulation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSVjTlE_-qCkD63F9Csy3g.jpeg" /></figure><p>Despite (and perhaps because of) pandas’ versatility in exploring and manipulating data, it can be easy for programmers to find themselves reusing or adapting code used in previous work to perform similar operations for new projects. Rather than revisiting old project files for the appropriate code, it can be handy to maintain a reference file that can house useful code snippets in a singular location.</p><p>As such, I wanted to provide some of the most helpful, shorthand pandas code snippets I have utilized in the past. This shouldn’t be taken as a definitive list of pandas code snippets; however, these should help in addressing many of the initial data exploration and manipulation tasks one might perform on data of interest. I have accumulated these from a variety of sources, though I want to give special credit to <a href="https://medium.com/@josephofiowa">Joseph Nelson</a> for providing much of the initial structure for the outlay below. Other great resources for troubleshooting pythonic coding problems include <a href="https://chrisalbon.com/">Chris Albon’s blog</a>, and the ever faithful <a href="https://stackoverflow.com/">Stack Overflow</a>.</p><p>Without further ado, I hope you find some of the following code snippets useful. If nothing else, I know that I can come back here to reference these for myself if need be. Note, that I’ve abstracted most of these snippets to refer to a data frame as “df”, and various columns as “column_x” and so on.</p><pre>#Code snippets for Pandas<br>import pandas as pd</pre><pre>‘’’<br>Reading Files, Selecting Columns, and Summarizing<br>‘’’</pre><pre># reading in a file from local computer or directly from a URL<br># various file formats that can be read in out wrote out<br>‘’’<br>Format Type     Data Description      Reader           Writer<br>text                  CSV            read_csv          to_csv<br>text                 JSON            read_json         to_json<br>text                 HTML            read_html         to_html<br>text             Local clipboard  read_clipboard     to_clipboard<br>binary             MS Excel          read_excel        to_excel<br>binary            HDF5 Format        read_hdf           to_hdf<br>binary           Feather Format     read_feather      to_feather<br>binary              Msgpack         read_msgpack      to_msgpack<br>binary               Stata           read_stata        to_stata<br>binary                SAS             read_sas <br>binary        Python Pickle Format   read_pickle       to_pickle<br>SQL                   SQL             read_sql          to_sql<br>SQL             Google Big Query      read_gbq          to_gbq<br>‘’’<br>#to read about different types of files, and further functionality of reading in files, visit: <a href="http://pandas.pydata.org/pandas-docs/version/0.20/io.html">http://pandas.pydata.org/pandas-docs/version/0.20/io.html</a></pre><pre>df = pd.read_csv(‘local_path/file.csv’)<br>df = pd.read_csv(‘<a href="https://file_path/file.csv&#39;">https://file_path/file.csv&#39;</a>)</pre><pre># when reading in tables, can specify separators, and note a column to be used as index separators can include tabs (“\t”), commas(“,”), pipes (“|”), etc.</pre><pre>df = pd.read_table(‘<a href="https://file_path/file&#39;">https://file_path/file&#39;</a>, sep=’|’, index_col=’column_x’)</pre><pre># examine the df data<br>df           # print the first 30 and last 30 rows<br>type(df)     # DataFrame<br>df.head()    # print the first 5 rows<br>df.head(10)  # print the first 10 rows<br>df.tail()    # print the last 5 rows<br>df.index     # “the index” (aka “the labels”)<br>df.columns   # column names (which is “an index”)<br>df.dtypes    # data types of each column<br>df.shape     # number of rows and columns<br>df.values    # underlying numpy array — df are stored as numpy arrays for effeciencies.</pre><pre># select a column<br>df[‘column_y’]         # select one column<br>type(df[‘column_y’])   # determine datatype of column (e.g., Series)<br>df.column_y            # select one column using the DataFrame attribute — not effective if column names have spaces</pre><pre># summarize (describe) the DataFrame<br>df.describe()          # describe all numeric columns<br>df.describe(include=[‘object’]) # describe all object columns<br>df.describe(include=’all’)      # describe all columns</pre><pre># summarize a Series<br>df.column_y.describe()   # describe a single column<br>df.column_z.mean()       # only calculate the mean<br>df[“column_z”].mean()    # alternate method for calculating mean<br> <br># count the number of occurrences of each value<br>df.column_y.value_counts()   # most useful for categorical variables, but can also be used with numeric variables</pre><pre>#filter df by one column, and print out values of another column<br>#when using numeric values, no quotations<br>df[df.column_y == “string_value”].column_z<br>df[df.column_y == 20 ].column_z    <br> <br># display only the number of rows of the ‘df’ DataFrame<br>df.shape[0]</pre><pre># display the 3 most frequent occurances of column in ‘df’<br>df.column_y.value_counts()[0:3]</pre><pre>‘’’<br>Filtering and Sorting<br>‘’’</pre><pre># boolean filtering: only show df with column_z &lt; 20<br>filter_bool = df.column_z &lt; 20    # create a Series of booleans…<br>df[filter_bool]                # …and use that Series to filter rows<br>df[filter_bool].describe()     # describes a data frame filtered by filter_bool<br>df[df.column_z &lt; 20]           # or, combine into a single step<br>df[df.column_z &lt; 20].column_x  # select one column from the filtered results<br>df[df[“column_z”] &lt; 20].column_x     # alternate method <br>df[df.column_z &lt; 20].column_x.value_counts()   # value_counts of resulting Series, can also use .mean(), etc. instead of .value_counts()</pre><pre># boolean filtering with multiple conditions; indexes are in square brackets, conditions are in parens</pre><pre>df[(df.column_z &lt; 20) &amp; (df.column_y==’string’)] # ampersand for AND condition <br>df[(df.column_z &lt; 20) | (df.column_z &gt; 60)] # pipe for OR condition</pre><pre># sorting<br>df.column_z.order()          # sort a column<br>df.sort_values(‘column_z’)   # sort a DataFrame by a single column<br>df.sort_values(‘column_z’, ascending=False)     # use descending order instead</pre><pre># Sort dataframe by multiple columns<br>df = df.sort([‘col1’,’col2&#39;,’col3&#39;],ascending=[1,1,0]) <br> <br># can also filter ‘df’ using pandas.Series.isin <br>df[df.column_x.isin([“string_1”, “string_2”])]</pre><pre>‘’’<br>Renaming, Adding, and Removing Columns<br>‘’’</pre><pre># rename one or more columns<br>df.rename(columns={‘original_column_1’:’column_x’, ‘original_column_2’:’column_y’}, inplace=True) #saves changes <br> <br># replace all column names (in place)<br>new_cols = [‘column_x’, ‘column_y’, ‘column_z’]<br>df.columns = new_cols</pre><pre># replace all column names when reading the file<br>df = pd.read_csv(‘df.csv’, header=0, names=new_cols)</pre><pre># add a new column as a function of existing columns<br>df[‘new_column_1’] = df.column_x + df.column_y<br>df[‘new_column_2’] = df.column_x * 1000   #can create new columns without for loops</pre><pre># removing columns<br>df.drop(‘column_x’, axis=1)   # axis=0 for rows, 1 for columns — does not drop in place<br>df.drop([‘column_x’, ‘column_y’], axis=1, inplace=True) # drop multiple columns</pre><pre># Lower-case all DataFrame column names<br>df.columns = map(str.lower, df.columns)</pre><pre># Even more fancy DataFrame column re-naming<br># lower-case all DataFrame column names (for example)<br>df.rename(columns=lambda x: x.split(‘.’)[-1], inplace=True)<br> <br> <br>‘’’<br>Handling Missing Values<br>‘’’</pre><pre># missing values are usually excluded by default<br>df.column_x.value_counts()             # excludes missing values<br>df.column_x.value_counts(dropna=False) # includes missing values</pre><pre># find missing values in a Series<br>df.column_x.isnull()  # True if missing<br>df.column_x.notnull() # True if not missing</pre><pre># use a boolean Series to filter DataFrame rows<br>df[df.column_x.isnull()]  # only show rows where column_x is missing<br>df[df.column_x.notnull()] # only show rows where column_x is not missing</pre><pre># understanding axes<br>df.sum()       # sums “down” the 0 axis (rows)<br>df.sum(axis=0) # equivalent (since axis=0 is the default)<br>df.sum(axis=1) # sums “across” the 1 axis (columns)</pre><pre># adding booleans<br>pd.Series([True, False, True])       # create a boolean Series<br>pd.Series([True, False, True]).sum() # converts False to 0 and True to 1</pre><pre># find missing values in a DataFrame<br>df.isnull() # DataFrame of booleans<br>df.isnull().sum() # count the missing values in each column</pre><pre># drop missing values<br>df.dropna(inplace=True)   # drop a row if ANY values are missing, defaults to rows, but can be applied to columns with axis=1</pre><pre>df.dropna(how=’all’, inplace=True)  # drop a row only if ALL values are missing</pre><pre># fill in missing values<br>df.column_x.fillna(value=’NA’, inplace=True) <br># fill in missing values with ‘NA’<br># value does not have to equal a string — can be set as some calculated value like df.column_x.mode(), or just a number like 0<br> <br> <br># turn off the missing value filter<br>df = pd.read_csv(‘df.csv’, header=0, names=new_cols, na_filter=False)</pre><pre>‘’’<br>Split-Apply-Combine<br>Diagram: <a href="http://i.imgur.com/yjNkiwL.png">http://i.imgur.com/yjNkiwL.png</a><br>‘’’</pre><pre># for each value in column_x, calculate the mean column_y <br>df.groupby(‘column_x’).column_y.mean()</pre><pre># for each value in column_x, count the number of occurrences<br>df.column_x.value_counts()</pre><pre># for each value in column_x, describe column_y<br>df.groupby(‘column_x’).column_y.describe()</pre><pre># similar, but outputs a DataFrame and can be customized<br>df.groupby(‘column_x’).column_y.agg([‘count’, ‘mean’, ‘min’, ‘max’])</pre><pre>df.groupby(‘column_x’).column_y.agg([‘count’, ‘mean’, ‘min’, ‘max’]).sort_values(‘mean’)</pre><pre># if you don’t specify a column to which the aggregation function should be applied, it will be applied to all numeric columns</pre><pre>df.groupby(‘column_x’).mean()<br>df.groupby(‘column_x’).describe()</pre><pre># can also groupby a list of columns, i.e., for each combination of column_x and column_y, calculate the mean column_z<br>df.groupby([“column_x”,”column_y”]).column_z.mean()</pre><pre>#to take groupby results out of hierarchical index format (e.g., present as table), use .unstack() method<br>df.groupby(“column_x”).column_y.value_counts().unstack()</pre><pre>#conversely, if you want to transform a table into a hierarchical index, use the .stack() method<br>df.stack()</pre><pre>‘’’<br>Selecting Multiple Columns and Filtering Rows<br>‘’’</pre><pre># select multiple columns<br>my_cols = [‘column_x’, ‘column_y’]  # create a list of column names…<br>df[my_cols]                   # …and use that list to select columns<br>df[[‘column_x’, ‘column_y’]]  # or, combine into a single step — double brackets due to indexing a list.</pre><pre># use loc to select columns by name<br>df.loc[:, ‘column_x’]    # colon means “all rows”, then select one column<br>df.loc[:, [‘column_x’, ‘column_y’]]  # select two columns<br>df.loc[:, ‘column_x’:’column_y’]     # select a range of columns (i.e., selects all columns including first through last specified)</pre><pre># loc can also filter rows by “name” (the index)<br>df.loc[0, :]       # row 0, all columns<br>df.loc[0:2, :]     # rows 0/1/2, all columns<br>df.loc[0:2, ‘column_x’:’column_y’] # rows 0/1/2, range of columns</pre><pre># use iloc to filter rows and select columns by integer position<br>df.iloc[:, [0, 3]]     # all rows, columns in position 0/3<br>df.iloc[:, 0:4]        # all rows, columns in position 0/1/2/3<br>df.iloc[0:3, :]        # rows in position 0/1/2, all columns</pre><pre>#filtering out and dropping rows based on condition (e.g., where column_x values are null)<br>drop_rows = df[df[“column_x”].isnull()]<br>new_df = df[~df.isin(drop_rows)].dropna(how=’all’)<br> <br> <br> <br>‘’’<br>Merging and Concatenating Dataframes<br>‘’’ <br>#concatenating two dfs together (just smooshes them together, does not pair them in any meaningful way) - axis=1 concats df2 to right side of df1; axis=0 concats df2 to bottom of df1<br>new_df = pd.concat([df1, df2], axis=1)</pre><pre>#merging dfs based on paired columns; columns do not need to have same name, but should match values; left_on column comes from df1, right_on column comes from df2<br>new_df = pd.merge(df1, df2, left_on=’column_x’, right_on=’column_y’)</pre><pre>#can also merge slices of dfs together, though slices need to include columns used for merging<br>new_df = pd.merge(df1[[‘column_x1’, ‘column_x2’]], df2, left_on=’column_x2&#39;, right_on=’column_y’)</pre><pre>#merging two dataframes based on shared index values (left is df1, right is df2)<br>new_df = pd.merge(df1, df2, left_index=True, right_index=True)<br> <br> <br>‘’’<br>Other Frequently Used Features<br>‘’’</pre><pre># map existing values to a different set of values<br>df[‘column_x’] = df.column_y.map({‘F’:0, ‘M’:1})</pre><pre># encode strings as integer values (automatically starts at 0)<br>df[‘column_x_num’] = df.column_x.factorize()[0]</pre><pre># determine unique values in a column<br>df.column_x.nunique()   # count the number of unique values<br>df.column_x.unique()    # return the unique values</pre><pre># replace all instances of a value in a column (must match entire value)<br>df.column_y.replace(‘old_string’, ‘new_string’, inplace=True)</pre><pre>#alter values in one column based on values in another column (changes occur in place)<br>#can use either .loc or .ix methods</pre><pre>df.loc[df[“column_x”] == 5, “column_y”] = 1<br> <br>df.ix[df.column_x == “string_value”, “column_y”] = “new_string_value”</pre><pre>#transpose data frame (i.e. rows become columns, columns become rows)<br>df.T</pre><pre># string methods are accessed via ‘str’<br>df.column_y.str.upper() # converts to uppercase<br>df.column_y.str.contains(‘value’, na=’False’) # checks for a substring, returns boolean series</pre><pre># convert a string to the datetime_column format<br>df[‘time_column’] = pd.to_datetime_column(df.time_column)<br>df.time_column.dt.hour   # datetime_column format exposes convenient attributes<br>(df.time_column.max() — df.time_column.min()).days   # also allows you to do datetime_column “math”<br>df[df.time_column &gt; pd.datetime_column(2014, 1, 1)]   # boolean filtering with datetime_column format</pre><pre># setting and then removing an index, resetting index can help remove hierarchical indexes while preserving the table in its basic structure<br>df.set_index(‘time_column’, inplace=True)<br>df.reset_index(inplace=True)</pre><pre># sort a column by its index<br>df.column_y.value_counts().sort_index()</pre><pre># change the data type of a column<br>df[‘column_x’] = df.column_x.astype(‘float’)</pre><pre># change the data type of a column when reading in a file<br>pd.read_csv(‘df.csv’, dtype={‘column_x’:float})</pre><pre># create dummy variables for ‘column_x’ and exclude first dummy column<br>column_x_dummies = pd.get_dummies(df.column_x).iloc[:, 1:]</pre><pre># concatenate two DataFrames (axis=0 for rows, axis=1 for columns)<br>df = pd.concat([df, column_x_dummies], axis=1)</pre><pre>‘’’<br>Less Frequently Used Features<br>‘’’</pre><pre># create a DataFrame from a dictionary<br>pd.DataFrame({‘column_x’:[‘value_x1’, ‘value_x2’, ‘value_x3’], ‘column_y’:[‘value_y1’, ‘value_y2’, ‘value_y3’]})</pre><pre># create a DataFrame from a list of lists<br>pd.DataFrame([[‘value_x1’, ‘value_y1’], [‘value_x2’, ‘value_y2’], [‘value_x3’, ‘value_y3’]], columns=[‘column_x’, ‘column_y’])</pre><pre># detecting duplicate rows<br>df.duplicated()       # True if a row is identical to a previous row<br>df.duplicated().sum() # count of duplicates<br>df[df.duplicated()]   # only show duplicates<br>df.drop_duplicates()  # drop duplicate rows<br>df.column_z.duplicated()   # check a single column for duplicates<br>df.duplicated([‘column_x’, ‘column_y’, ‘column_z’]).sum()  # specify columns for finding duplicates</pre><pre># Clean up missing values in multiple DataFrame columns<br>df = df.fillna({<br> ‘col1’: ‘missing’,<br> ‘col2’: ‘99.999’,<br> ‘col3’: ‘999’,<br> ‘col4’: ‘missing’,<br> ‘col5’: ‘missing’,<br> ‘col6’: ‘99’<br>})</pre><pre># Concatenate two DataFrame columns into a new, single column - (useful when dealing with composite keys, for example)<br>df[‘newcol’] = df[‘col1’].map(str) + df[‘col2’].map(str)</pre><pre># Doing calculations with DataFrame columns that have missing values<br># In example below, swap in 0 for df[‘col1’] cells that contain null<br>df[‘new_col’] = np.where(pd.isnull(df[‘col1’]),0,df[‘col1’]) + df[‘col2’]<br> <br># display a cross-tabulation of two Series<br>pd.crosstab(df.column_x, df.column_y)</pre><pre># alternative syntax for boolean filtering (noted as “experimental” in the documentation)<br>df.query(‘column_z &lt; 20’) # df[df.column_z &lt; 20]<br>df.query(“column_z &lt; 20 and column_y==’string’”)  # df[(df.column_z &lt; 20) &amp; (df.column_y==’string’)]<br>df.query(‘column_z &lt; 20 or column_z &gt; 60’)        # df[(df.column_z &lt; 20) | (df.column_z &gt; 60)]</pre><pre># Loop through rows in a DataFrame<br>for index, row in df.iterrows():<br> print index, row[‘column_x’]</pre><pre># Much faster way to loop through DataFrame rows if you can work with tuples<br>for row in df.itertuples():<br> print(row)</pre><pre># Get rid of non-numeric values throughout a DataFrame:<br>for col in df.columns.values:<br> df[col] = df[col].replace(‘[⁰-9]+.-’, ‘’, regex=True)</pre><pre># Change all NaNs to None (useful before loading to a db)<br>df = df.where((pd.notnull(df)), None)</pre><pre># Split delimited values in a DataFrame column into two new columns<br>df[‘new_col1’], df[‘new_col2’] = zip(*df[‘original_col’].apply(lambda x: x.split(‘: ‘, 1)))</pre><pre># Collapse hierarchical column indexes<br>df.columns = df.columns.get_level_values(0)</pre><pre># display the memory usage of a DataFrame<br>df.info()         # total usage<br>df.memory_usage() # usage by column</pre><pre># change a Series to the ‘category’ data type (reduces memory usage and increases performance)<br>df[‘column_y’] = df.column_y.astype(‘category’)</pre><pre># temporarily define a new column as a function of existing columns<br>df.assign(new_column = df.column_x + df.spirit + df.column_y)</pre><pre># limit which rows are read when reading in a file<br>pd.read_csv(‘df.csv’, nrows=10)        # only read first 10 rows<br>pd.read_csv(‘df.csv’, skiprows=[1, 2]) # skip the first two rows of data</pre><pre># randomly sample a DataFrame<br>train = df.sample(frac=0.75, random_column_y=1) # will contain 75% of the rows<br>test = df[~df.index.isin(train.index)] # will contain the other 25%</pre><pre># change the maximum number of rows and columns printed (‘None’ means unlimited)<br>pd.set_option(‘max_rows’, None) # default is 60 rows<br>pd.set_option(‘max_columns’, None) # default is 20 columns<br>print df</pre><pre># reset options to defaults<br>pd.reset_option(‘max_rows’)<br>pd.reset_option(‘max_columns’)</pre><pre># change the options temporarily (settings are restored when you exit the ‘with’ block)<br>with pd.option_context(‘max_rows’, None, ‘max_columns’, None):<br> print df</pre><p>Thanks, all!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7c5aed5ecb9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analysis of Web Scraped Job Data to Predict Relative Salaries]]></title>
            <link>https://medium.com/@msalmon00/analysis-of-web-scraped-job-data-to-predict-relative-salaries-c7237954184a?source=rss-8e7466898cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7237954184a</guid>
            <category><![CDATA[predictive-modeling]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Michael Salmon]]></dc:creator>
            <pubDate>Sat, 17 Jun 2017 18:44:54 GMT</pubDate>
            <atom:updated>2017-06-17T18:44:54.254Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello friends, today I bring you the long-awaited (I presume) conclusion to my work with web scraped job data taken from the job-posting site <a href="https://www.indeed.com/">Indeed.com</a>. For the earlier segments of this story, you can read up on <a href="https://medium.com/@msalmon00/web-scraping-job-postings-from-indeed-96bd588dcb4b">how I initially scraped this data</a>, as well as the <a href="https://medium.com/towards-data-science/data-cleaning-web-scraped-job-data-6c2a2d963cd">initial cleaning of the data</a> that I performed leading into these analyses.</p><p>When we last left off, I had conducted initial cleaning of my scraped data such that my dataframe (scrape_data) contained seven columns, noting several pieces of information for each posting:</p><ol><li>the city entered when scraping;</li><li>the job title;</li><li>the company name;</li><li>the location associated with the job (sometimes the associated city, sometimes a suburb of said city);</li><li>a brief snippet of the job summary;</li><li>the estimated annual salary for each position, where available; and</li><li>the original salary period specified for each posting, where available (e.g., if salary was originally posted as hourly, this figure was converted to an annual estimate in the salary column, but the original salary period was retained here).</li></ol><p>Below is a quick look at the tail of our dataframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EtWdOxC1smOa7op43TMGYA.png" /></figure><p>While not visible in the output image above, the vast majority of job postings scraped did not contain any salary information whatsoever. Of 6,399 unique job postings scraped, only 449 of them (7%) contained salary information. Ideally, we could analyze information about those postings that do contain salary information in order to build a model that can inform us of the relative salary prospects of those jobs containing no salary information at all.</p><p>While we could go about this in a number of ways, for simplicity’s sake, here we’ll build a model that attempts to predict whether a given job posting has a salary above or below the median salary value of all jobs scraped. This way, when approaching future job applications, we could focus our efforts on those jobs that are likely to pay a desirable wage.</p><p>So, from here, we have several objectives when analyzing our scraped data:</p><ol><li>Determine the median salary value of those jobs containing existing salary information.</li><li>Explore job postings containing salary data to determine those characteristics most associated with above- vs. below-median salaries. (We’ll predominantly use <strong>natural language processing</strong> during this step.)</li><li>Build a predictive model evaluating jobs’ salary prospects, and apply this model to those job postings containing no salary information.</li></ol><h4>Initial Data Exploration and Prep</h4><p>To begin, we’ll import some basic python libraries to assist both in the analysis and visualization of the data. We’ll do most of our initial work in Pandas, utilizing NumPy. We’ll also import Seaborn and Matplotlib to assist with visualizations.</p><pre>import seaborn as sns<br>import matplotlib.pyplot as plt<br>import pandas as pd<br>import numpy as np</pre><pre># display plots in the notebook<br>%matplotlib inline</pre><pre># increase default figure and font sizes for easier viewing<br>plt.rcParams[‘figure.figsize’] = (8, 6)<br>plt.rcParams[‘font.size’] = 17</pre><p>The ultimate goal of this analysis is to predict a binary variable — whether the salary is above or below the median value. So, we’ll first need to compute the median salary and then create a new binary variable that equals 1.0 when the salary is high (above the median), and 0.0 when it is low (below the median).</p><pre>#what is the median salary? (calculate mean salary, too, for fun)</pre><pre>print(&quot;salary median: &quot; + str(scrape_data[&quot;salary&quot;].median()))<br>print(&quot;salary mean: &quot; + str(scrape_data[&quot;salary&quot;].mean()))</pre><pre>#creating new target variable &quot;above_med_sal&quot; where 0 == below or equal to the median, and 1 == above the median; any postings without salary information will have NaN values for this variable</pre><pre>scrape_data[&quot;above_med_sal&quot;] = np.nan</pre><pre>scrape_data.ix[scrape_data[&quot;salary&quot;] &gt; scrape_data[&quot;salary&quot;].median(), &quot;above_med_sal&quot;] = 1</pre><pre>scrape_data.ix[scrape_data[&quot;salary&quot;] &lt;= scrape_data[&quot;salary&quot;].median(), &quot;above_med_sal&quot;] = 0</pre><pre>scrape_data.tail()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CBh1Y-Al30a6LNqmL6ir4A.png" /></figure><p>We see that our median salary value is $87,915 and that our mean salary value is $98,754. We can tell from these figures that our data is positively skewed, but a plot of the salaried data can give us a better approximation of what this looks like. Let’s plot the distribution of available job salaries, and add some vertical lines indicating the median and mean salary values.</p><pre>#plotting salary distribution, with vertical lines to represent the mean and median salary</pre><pre>sal_plot = scrape_data[scrape_data[&quot;salary&quot;].notnull()]</pre><pre>ax = sns.distplot(sal_plot[&quot;salary&quot;])<br>ax.axvline(sal_plot[&quot;salary&quot;].median(), lw=2.5, ls=&#39;dashed&#39;, color=&#39;black&#39;)<br>ax.axvline(sal_plot[&quot;salary&quot;].mean(), lw=2.5, ls=&#39;dashed&#39;, color=&#39;red&#39;)<br>sns.plt.title(&#39;Indeed Salary Distribution with Median (black) and Mean (red)&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IqthKqDWIXeK-ifRq6iAsA.png" /></figure><p>Looking at the total range of salaries, our available jobs provide anywhere from $22,100 to $270,400 on an annual basis, with the most frequently occurring salaries in the realm of $50,000.</p><p>Now that we have a sense of how our data is distributed, we can begin analyzing the data in greater depth to examine those characteristics that are most indicative of a job’s estimated annual salary.</p><p>When approaching the model-building process, we will only be working with job postings containing salary information. We do this because a job’s salary being above or below the median value is our target of interest (i.e., the means by which we determine the effectiveness of our model). For now, we’ll hold those postings with no salary information in reserve for our final predictions.</p><pre>#We&#39;ll only be training our model on jobs that have salary data, so separate out salary jobs</pre><pre>salary_data = scrape_data[scrape_data.above_med_sal.notnull()]</pre><pre>#splitting out our predictor variables from the salary data</pre><pre>sal_X = salary_data.iloc[:,0:len(salary_data.columns)-1]</pre><pre>#sal_y will be above_med_sal, creating as a dataframe</pre><pre>sal_y = pd.DataFrame(salary_data[&quot;above_med_sal&quot;])</pre><p>Now, another best practice when building models is to conduct a train-test split. This effectively segments our data into separate groups, one for building our model (the train set) and the other for testing and validating our model (the test set).</p><p>The test set helps to validate our model by serving as a source of data that was not used at all in the building of our model. If the model performs well on data it has never seen before, we can feel reasonably confident in our model’s ability to perform well on other similar data we might collect in the future.</p><p>We’ll import the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html">train-test module from scikit-learn</a>, and apply it to our salaried data. We’ll do a 80:20 split here, so that 80% of our salaried data is used to train the model and the remaining 20% is used for testing and validation. We want to stratify our data on the y variable (above_med_sal) to make sure the ratio of above- vs. below-median salary is preserved across the train and test sets. We’ll set a random state here to ensure that our work can be replicated if we rerun our analysis in the future. Here, I used the google random number generator to choose a random_state of 74.</p><p>I reset the index of each of the generated sets for ease of use.</p><pre>from sklearn.model_selection import train_test_split</pre><pre>X_train, X_test, y_train, y_test = train_test_split(sal_X, sal_y, test_size=0.2, stratify=sal_y, random_state=74)</pre><pre>X_train.reset_index(drop=True, inplace=True)<br>X_test.reset_index(drop=True, inplace=True)<br>y_train.reset_index(drop=True, inplace=True)<br>y_test.reset_index(drop=True, inplace=True)</pre><h4>Exploring Job Characteristics with Natural Language Processing</h4><p>As you no doubt noticed from the dataframe screenshots above, almost all of the information we have on our job postings is qualitative in nature — that is, it’s narrative. To best examine this information, we’ll use natural language processing (NLP) to examine frequencies of words and phrases (also known as n-grams) in each of our different variables. From there, we’ll determine those n-grams that are strongly associated with above- vs. below-median salary job postings.</p><p>Presumably, n-grams that are strongly associated with above-or below-median salary postings should be fairly predictive of the median salary status of jobs we have held out of our training set. For example, if our training set shows a term like “data scientist” appears frequently in job postings with above-median salaries, and rarely in job postings with below-median salaries, we can assume job postings in our test set containing the term “data scientist” will be more likely to have an above-median salary. As such, we could then engineer some new features based on the findings of our NLP for use in our predictive model.</p><p>For each of the features in X_train, I utilized the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html">count vectorizer from scikit-learn</a> to examine the frequencies of n-grams associated with above-median and below-median salary jobs. I then generated ratios examining the relative appearance of each n-gram in above- vs. below-median salary positions. The higher the ratio above 1.0, the more strongly associated the n-gram is with above-median salaries; the lower the ratio is below 1.0, the more strongly associated the n-gram is with below-median salaries. Ratios closer to 1.0 indicate that n-grams appeared relatively as frequently in above-median jobs as they did in below-median jobs, indicating little predictive power.</p><p>I will review this process step-by-step, using analysis of a single feature as an example.</p><p>First, let’s initialize our count vectorizer.</p><p>For this analysis, I set my n-gram range from 1–3, meaning our analysis will examine consecutive strings of between one and three words. I was also only interested in words that appeared in at least 5% of postings, to cut out sparsely represented words that wouldn’t have helpful discriminating power. Finally, I set my vectorizer to ignore common english words like “the,” “is,” “a,” etc. via stop_words=‘english’.</p><pre>#initializing count vectorizer to examine text-based data for logistic regression. </pre><pre>from sklearn.feature_extraction.text import CountVectorizer</pre><pre>cvec = CountVectorizer(stop_words=&#39;english&#39;,ngram_range=(1, 3), min_df = .05)</pre><p>Now, I’ll demonstrate the process of examining job titles from the train set. Remember, though, that this process was applied to each of the variables within the training set.</p><pre>#fitting count vectorizer to job title</pre><pre>cvec.fit(X_train[&quot;job_title&quot;])</pre><pre>#transforming job_title data into job_train</pre><pre>job_train = pd.DataFrame(cvec.transform(X_train[&quot;job_title&quot;]).todense(),<br>                       columns=cvec.get_feature_names())</pre><p>To examine how n-grams from job titles are associated with median salary status, we’ll concatenate our salary data onto the job_train dataframe.</p><pre>word_observe = pd.concat([job_train, y_train], axis=1)</pre><pre>word_observe.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RUsEwjUwwvGhQPj-luMAsg.png" /></figure><p>Now, we’ll take a quick look at the sheer frequencies of n-grams associated with below-median salary jobs (above_med_sal == 0) and above-median salary jobs (above_med_sal == 1).</p><pre>#Observing most common job title n-grams for both above and below median salaries</pre><pre>for i in xrange(2):<br>    word_count = word_observe[word_observe[&quot;above_med_sal&quot;]==i].sum(axis=0)<br>    print i, &quot;most common words&quot;<br>    cw = word_count.sort_values(ascending = False).head(20)<br>    print cw<br>    print</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yyvYswxJqkMrEUw7F5cpHw.png" /></figure><p>While we can get a ballpark view of some terms (e.g., “research” appears pretty frequently in below-median jobs, and pretty rarely in above-median jobs), being able to see the ratios of these terms’ appearance will make our job a lot easier in discerning the relative discriminatory power of our n-grams.</p><pre>#creating separate lists of the top appearing words so that these can be combined and compared to create ratios of appearances in above-median vs. below-median salaries</pre><pre>word_count_below = word_observe[word_observe[&quot;above_med_sal&quot;]==0].sum(axis=0)</pre><pre>word_count_above = word_observe[word_observe[&quot;above_med_sal&quot;]==1].sum(axis=0)</pre><pre>#creating as dataframe, using method &quot;.T&quot; to transpose columns with index.</pre><pre>word_count_compare = pd.DataFrame([word_count_below, word_count_above]).T</pre><pre>#creating a &quot;ratio&quot; column to determine frequency of words associated with above median jobs vs. below median jobs</pre><pre>word_count_compare[&quot;above_below_ratio&quot;] = word_count_compare[1]/word_count_compare[0]</pre><pre>word_count_compare.above_below_ratio.sort_values(ascending=False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NQfCMKtViOkZYo-WVTLJZg.png" /></figure><p>Voila! Looking at these ratios, we can see that job title terms like “data scientist,” combinations of “machine,” “learning,” “engineer,” and “data” all have powerful predictive power for being above the median salary. Comparatively, job title terms like “research,” “analyst,” and “associate” are all associated with below-median salaries.</p><p>We can utilize these insights when engineering features for our predictive model to determine their appearance in job postings and predict jobs’ median salary status accordingly. For instance, we can use terms with ratios above 3.0 to generate a “good job title” feature, and terms with ratios below 0.5 to generate a “bad job title” feature.</p><p>After performing similar analysis on the other features of the scraped job postings, I determined several exploratory findings:</p><p><strong>City</strong> — cities most associated with above-median salaries were Chicago, Washington DC, and San Francisco.</p><p><strong>Job Title</strong> — job title terms like “data scientist”, combinations of “machine,” “learning,” “engineer,” and “data” all have powerful predictive power for being above the median salary; job title terms like “research,” “analyst,” and “associate” are all associated with below-median salaries.</p><p><strong>Company Name</strong> — company names were somewhat difficult to analyze, as company names could include a lot of unique words or acronyms that didn’t necessarily translate to the type of work being done. Working for (<em>presumably)</em> consulting firms (i.e., companies with “associates” in the title) is associated with above-median salaries, while working for a “university” is associated with below-median salaries.</p><p><strong>Location </strong>— Due to the variety of locations, and some overlap with the City feature above, I decided look at whether or not a job’s location matched the associated city name. That is, was the job posted for the city itself or a suburb thereof? This was meant to examine whether a job being in the city it’s associated with (or outside of it) has any impact.</p><p><strong>Job Summary</strong> — job summary terms including “machine learning,” “machine,” “learning,” “data scientist,” “data science,” “senior,” “lead,” “big data,” “analytics,” and “years” (presumably looking for at least so many years of experience) are all associated with above-median salaries; job summary terms including “research,” “analyze,” and “analysis” are all associated with below-median salaries</p><h4>Feature Engineering and Model Building</h4><p>Now that we have analyzed our job posting data for relevant information connected to median salary status, we can engineer features relevant to our predictive model building. From the insights derived above, I decided to build a model with eight binary features derived from each job posting to predict whether a job would offer an above-median or below-median salary:</p><ol><li>Was the job located in Chicago, Washington DC, or San Francisco?</li><li>Did the job title contain “good” terms including “data scientist,” “machine,” “learning,” or “engineer”?</li><li>Did the job title contain “bad” terms including “research,” “analyst,” or “associate”?</li><li>Did the company name include the “good” term “associates”?</li><li>Did the company name include the “bad” term “university”?</li><li>Was the job posting located in the city of its posting, or in a nearby area (e.g., being in Seattle vs. Redmond, Washington)?</li><li>Did the job summary contain “good” terms including “machine learning,” “data scientist,” “data science,” “senior,” “lead,” “big data,” “analytics,” or “years”?</li><li>Did the job summary contain “bad” terms including “research,” “analyze,” and “analysis”?</li></ol><p>Once these features were engineered for each posting, I utilized a logistic regression model to predict whether each job posting would provide an above-median or below-median salary.</p><p>The data preparation was a bit extensive, but you can see below how I engineered each of these features for my data set below, generally by examining whether variable strings contained n-grams or word stems of interest.</p><pre>#Creating eight binary variables utilizing insights above to prepare data for logistic regression</pre><pre>#splitting out our predictor variables from the salary data</pre><pre>sal_X = salary_data[[&quot;city&quot;, &quot;job_title&quot;, &quot;company_name&quot;, &quot;location&quot;, &quot;summary&quot;, &quot;og_salary_period&quot;]]</pre><pre>sal_X.reset_index(drop=True, inplace=True)</pre><pre>#sal_y will be above_med_sal, creating as a dataframe, then converting to a 1-D array to allow for easier use below with k-folds, etc.</pre><pre>sal_y = pd.DataFrame(salary_data[&quot;above_med_sal&quot;])</pre><pre>sal_y = pd.DataFrame.as_matrix(sal_y).ravel()</pre><pre>#Creating variable for if city is chi, DC, or sf</pre><pre>sal_X[&quot;is_chi_dc_sf&quot;] = 0</pre><pre>sal_X.ix[(sal_X[&quot;city&quot;].str.contains(&quot;chicago&quot;) | <br>          sal_X[&quot;city&quot;].str.contains(&quot;washington dc&quot;) | <br>          sal_X[&quot;city&quot;].str.contains(&quot;san francisco&quot;)), &quot;is_chi_dc_sf&quot;] = 1</pre><pre>#Creating variable for if job title includes &quot;data scientist&quot;, combinations of &quot;machine&quot;, &quot;learning&quot;, &quot;engineer&quot;, and &quot;data&quot;</pre><pre>sal_X[&quot;good_job_title&quot;] = 0</pre><pre>sal_X.ix[(sal_X[&quot;job_title&quot;].str.contains(&quot;data scientist&quot;) | <br>            sal_X[&quot;job_title&quot;].str.contains(&quot;machine&quot;) | <br>            sal_X[&quot;job_title&quot;].str.contains(&quot;learning&quot;) |<br>            sal_X[&quot;job_title&quot;].str.contains(&quot;engineer&quot;) |<br>            sal_X[&quot;job_title&quot;].str.contains(&quot;data&quot;)), &quot;good_job_title&quot;] = 1</pre><pre># job_title terms like &quot;research&quot;, &quot;analyst&quot; &quot;associate&quot; are all associated with below median income</pre><pre>sal_X[&quot;bad_job_title&quot;] = 0</pre><pre>sal_X.ix[(sal_X[&quot;job_title&quot;].str.contains(&quot;research&quot;) | <br>            sal_X[&quot;job_title&quot;].str.contains(&quot;analyst&quot;) | <br>            sal_X[&quot;job_title&quot;].str.contains(&quot;associate&quot;)), &quot;bad_job_title&quot;] = 1</pre><pre># working for *presumably* consulting firms e.g., &quot;associates&quot; is associated with above median salaries, while working for &quot;university&quot; is associated with below median salaries.</pre><pre>sal_X[&quot;good_company&quot;] = 0</pre><pre>sal_X.ix[(sal_X[&quot;company_name&quot;].str.contains(&quot;associates&quot;) ), &quot;good_company&quot;] = 1</pre><pre>sal_X[&quot;bad_company&quot;] = 0</pre><pre>sal_X.ix[(sal_X[&quot;company_name&quot;].str.contains(&quot;university&quot;) ), &quot;bad_company&quot;] = 1</pre><pre># For location data - I will look at whether or not location matches city name to examine whether a job being in the city it&#39;s associated with (or outside of it) has any impact.</pre><pre>sal_X[&quot;in_city&quot;] = 0</pre><pre>sal_X.ix[(sal_X[&quot;location&quot;].str.contains(&quot;new york&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;seattle&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;san francisco&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;houston&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;denver&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;austin&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;washington dc&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;miami&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;phoenix&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;pittsburgh&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;portland&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;philadelphia&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;boulder&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;dallas&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;los angeles&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;atlanta&quot;) |<br>            sal_X[&quot;location&quot;].str.contains(&quot;chicago&quot;)), &quot;in_city&quot;] = 1</pre><pre># job summary terms including &quot;machine learning&quot;, &quot;machine&quot;, &quot;learning&quot;, &quot;data scientist&quot;, &quot;data science&quot;, &quot;senior&quot;, &quot;lead&quot;, &quot;big data&quot;, &quot;analytics&quot;, &quot;years&quot; (presumably looking for at least so many years) are all associated with above median salaries</pre><pre>sal_X[&quot;good_summary&quot;] = 0</pre><pre>sal_X.ix[(sal_X[&quot;summary&quot;].str.contains(&quot;machine&quot;) | <br>            sal_X[&quot;summary&quot;].str.contains(&quot;learning&quot;) | <br>            sal_X[&quot;summary&quot;].str.contains(&quot;data scien&quot;) |<br>            sal_X[&quot;summary&quot;].str.contains(&quot;senior&quot;) |<br>            sal_X[&quot;summary&quot;].str.contains(&quot;lead&quot;) |<br>            sal_X[&quot;summary&quot;].str.contains(&quot;big data&quot;) |         <br>            sal_X[&quot;summary&quot;].str.contains(&quot;analytics&quot;) |<br>            sal_X[&quot;summary&quot;].str.contains(&quot;years&quot;)), &quot;good_summary&quot;] = 1</pre><pre># &quot;research&quot;, &quot;analyze&quot; and &quot;analysis&quot; are all associated with below median salaries</pre><pre>sal_X[&quot;bad_summary&quot;] = 0</pre><pre>sal_X.ix[(sal_X[&quot;summary&quot;].str.contains(&quot;research&quot;) | <br>            sal_X[&quot;summary&quot;].str.contains(&quot;analyze&quot;) | <br>            sal_X[&quot;summary&quot;].str.contains(&quot;analysis&quot;)), &quot;bad_summary&quot;] = 1</pre><pre>#dropping original data columns from sal_X to make analysis easier.</pre><pre>sal_X.drop([&quot;city&quot;, &quot;job_title&quot;, &quot;company_name&quot;, &quot;location&quot;, &quot;summary&quot;, &quot;og_salary_period&quot;], axis=1, inplace=True)</pre><pre>#converting 1 and 0 values in matrix to float</pre><pre>sal_X = sal_X.astype(&quot;float&quot;)</pre><pre>sal_X.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GGrcXS31wRaN-H4DsX3e8g.png" /></figure><p>We are now left with eight binary variables describing information about each job posting. Let’s do a quick correlation matrix via Seaborn to examine how interrelated these variables are to one another.</p><pre># Perform Pearson correlation coefficients using NumPy and Seaborn</pre><pre>feature_correlation_matrix = sal_X.corr()</pre><pre>#Creating heatmap correlation matrix of predictor coefficients.</pre><pre>sns.heatmap(feature_correlation_matrix,<br>           annot = True,<br>           linewidths = 0.5)</pre><pre>sns.plt.title(&#39;Correlation Heatmap of Eight Binary Variables for Predicting Job Salary&#39;)<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZjEJfLeH7zCx2ombE1bb0Q.png" /></figure><p>All things considered, there is pretty good separation within our eight variables. Good_job_title and good_summary are the most highly correlated of the variables, with a positive correlation of 0.51. While moderately positively related, this correlation isn’t particularly worrisome for our current purposes.</p><p>Now that we’re about ready to begin modeling our data, we’ll redo our train-test split in order to fit our model to the train set and validate it on our test set.</p><pre>X_train, X_test, y_train, y_test = train_test_split(sal_X, sal_y, test_size=0.2, stratify=sal_y, random_state=74)</pre><p>Now, when fitting our logistic regression to the train set, we’ll perform cross-validation to get a sense of how consistently our model performs with different slices of the train set. We’ll be importing <a href="http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html">cross validation</a> tools and the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html">logistic regression</a> model from scikit-learn to conduct our model building.</p><pre>#now score a logistic regression on X_train and y_train using cross_val_scores</pre><pre>from sklearn.model_selection import cross_val_score<br>from sklearn import metrics<br>from sklearn.linear_model import LogisticRegression</pre><pre>logreg = LogisticRegression()<br>logreg.fit(X_train, y_train)<br>print &quot;Score:&quot;, logreg.score(X_train, y_train)     </pre><pre>scores = cross_val_score(logreg, X_train, y_train, cv=6)<br>print &quot;Cross-validated scores:&quot;, scores</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7VYYumuEXH_-gdt2OFf7lg.png" /></figure><p>Looking at the initial score for our train set, we can see that our logistic regression model was approximately 83% accurate in assessing our training set, with a range between 76% and 93% accuracy when conducting cross-validation. All things considered, this is pretty good.</p><p>Now, let’s utilize our model to predict the median salary classifications within our test set. We’ll import and utilize a <a href="http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html">confusion matrix</a> and <a href="http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html">classification report</a> from scikit-learn to evaluate the predictive performance of our model.</p><pre># Predicting salaries in X_test given predictors developed with X_train</pre><pre>y_probs = logreg.predict(X_test)   <br>#generates predicted values of Y_test from X_test based off of training set.</pre><pre>print &quot;Score:&quot;, logreg.score(X_test, y_test)</pre><pre>#Create classification and confusion matrix</pre><pre>from sklearn.metrics import classification_report, confusion_matrix, accuracy_score</pre><pre># printing confusion matrix </pre><pre>print(confusion_matrix(y_test, y_probs,))</pre><pre>#printing classification report</pre><pre>target_names = [&#39;below_med&#39;, &#39;above_med&#39;]</pre><pre>print(classification_report(y_test, y_probs, target_names=target_names))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NEgTI0JFqY5MWD1QQl64wQ.png" /></figure><p>From this, we can see the model scored quite well on the test set (83% accurate).</p><p>For the Confusion Matrix, note that the rows reference actual values (e.g., actual below-median salaries; actual above-median salaries), while columns represent predictions (e.g., predicted below-median salaries; above-median salaries).</p><p>We can see from this that five below-median salary jobs (11%) in the test set were incorrectly predicted as above-median salaries, while 10 above-median salary jobs (22%) were incorrectly predicted as below-median jobs.</p><p>For the classification report, a brief recap of terminology:</p><blockquote>precision scores = true positives / (true positives + false positives) — of all predicted, the % correct</blockquote><blockquote>recall = true positives / (true positives + false negatives) — of all actual, the % correct</blockquote><blockquote>f1-score = weighted harmonic mean of precision and recall — f1-score reaches its best value at 1 and worst at 0.</blockquote><blockquote>support is the number of actual observations for each class</blockquote><p>We see the model performs fairly well all-around, but is most effective at predicting above-median salaries, and discerning actual below-median salaries.</p><h4>Applying logistic regression to predict above- vs. below-median salaries of job postings with no salary information</h4><p>Finally, now that we have built and validated our predictive model on the known salary jobs, we can apply our model to predict the median salary status of those jobs that contained no salary information when initially scraped.</p><p>We’ll first separate out the non-salaried postings from our scrape_data dataframe into a new predict_data dataframe. We know that the above_med_sal column will be null for each of the job postings we want.</p><pre>predict_data = scrape_data[scrape_data.above_med_sal.isnull()]</pre><p>Next, we’ll process the predict_data it in the same way we processed our salaried data above to create our eight predictor variables. Once this is complete, we’re left with a new predict_X dataframe that looks like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mMTDfiDdfCTbaa2Uz-Xbag.png" /></figure><p>We can now utilize the logistic regression’s predict method to generate median salary predictions for each of our job postings, and append these predictions as a new column (“salary_prediction”) to the predict_data dataframe as follows:</p><pre>#generating median salary predictions for unsalaried job postings<br>predict_y = logreg.predict(predict_X)   </pre><pre>predict_data[&quot;salary_prediction&quot;] = predict_y</pre><pre>predict_data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EXq0bC7CBJyQxboO422rVw.png" /></figure><p>When applying the predictive model to the remaining 5,950 job postings without salary information listed, 3,128 jobs (52%) were predicted to have salaries below the median, while 2,822 (48%) were predicted to be above the median. Under ideal circumstances, we would expect approximately 50% of postings to be below the median and approximately 50% to be above the median. As such, the predictive results are encouraging with respect to representative classification.</p><p>Let’s take a brief look at how our model predicted above-median salaries vs. below-median salaries for each of the cities we specified when scraping our initial data.</p><pre>city_predictions = pd.DataFrame(predict_data[[&quot;city&quot;, &quot;salary_prediction&quot;]])</pre><pre>city_predictions[&#39;median_salary_prediction&#39;] = city_predictions.salary_prediction.map({0:&quot;below_median&quot;, 1:&quot;above_median&quot;})</pre><pre>sns.countplot(x=&quot;city&quot;, hue=&quot;median_salary_prediction&quot;, data=city_predictions)<br>plt.ylabel(&quot;Number of Job Postings&quot;)<br>plt.xticks(rotation=45)<br>sns.plt.title(&#39;Plot of Salary Predictions by City (8 variable log regression)&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8WmotFNiah4TEMJrEMbcpw.png" /></figure><p>Among the predicted results, San Francisco is predicted to be the biggest hub of above-median salary jobs, and had the highest predicted ratio of above-median jobs to below-median jobs (2.87 above-median jobs for every below-median job). Chicago also had a predicted ratio of 2.06 above-median jobs for every below-median job.</p><p>The least appealing cities for data science jobs included Miami (8.63 below-median jobs for every above-median job), Philadelphia (4.16 below-median jobs for every above-median job), and Houston (2.79 below-median jobs for every above-median job).</p><h4>Conclusions</h4><p>While this evaluation has produced valuable information on the expected salaries of data science jobs in various regions of the United States, there are several caveats worth raising regarding these findings.</p><p>First, the predictive ability of this model is somewhat restricted to job postings from cities that were initially surveyed. If job postings would be added from cities not in the initial set, the predictive ability of the model would not be able to account for locations that were not assessed during the model’s creation.</p><p>Second, the classification model inherently assumes that the distribution of salaries for jobs with listed salaries is approximately the same as without listed salaries. Employers listing higher paying jobs may intentionally not post salary information with their postings, which could lead jobs with unposted salaries following a different salary distribution than those jobs with posted salaries.</p><p>Finally, while the classification model utilized binary variables (that is, indicated yes or no for a given item), they may be adjusted in future evaluations to capture a gradient of information, which could improve the predictive ability of the model. That is, certain “good” terms like “data scientist” or “machine learning” could be more powerfully predictive of above-median salary than other “good” terms like “engineer”. Capturing the relative strength or weakness of these terms could help to further distinguish the likelihood of above- or below-median salaries depending on their relative presence in any given job posting.</p><p>At any rate, I hope you enjoyed this review of my analysis of web scraped job data. The web is brimming with valuable and informative data, and the sky is effectively the limit in terms of how this information might be collected and analyzed for personal benefit or broader impacts.</p><p>Thanks for taking the time to read!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7237954184a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Trends in Substance Use through Data Visualization]]></title>
            <link>https://medium.com/@msalmon00/exploring-trends-in-substance-use-through-data-visualization-428019133b4b?source=rss-8e7466898cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/428019133b4b</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[drugs]]></category>
            <category><![CDATA[policy]]></category>
            <dc:creator><![CDATA[Michael Salmon]]></dc:creator>
            <pubDate>Thu, 25 May 2017 14:45:29 GMT</pubDate>
            <atom:updated>2017-06-07T14:56:52.318Z</atom:updated>
            <content:encoded><![CDATA[<p>The <a href="http://datafiles.samhsa.gov/study-series/national-survey-drug-use-and-health-nsduh-nid13517">National Survey on Drug Use and Health (NSDUH)</a> is an annual survey conducted by the Substance Abuse and Mental Health Services Administration (SAMHSA). This survey employs rigorous statistical sampling to capture current and former substance use statistics of nearly 60,000 civilian, non-institutionalized US citizens over the age of 12.</p><p>It is a veritable treasure trove of information, surveying respondents on their experiences with a wide battery of legal and illicit substances. The survey also includes many other measures gauging aspects of each individual’s personal life, including their self-reported physical and mental health, social environment, demographics, education, employment, household composition, health insurance status, and income. While this brief summary grossly simplifies the richness of the available data, as the <a href="http://datafiles.samhsa.gov/study-dataset/national-survey-drug-use-and-health-2015-nsduh-2015-ds0001-nid16894">final 2015 NSDUH data set</a> includes some 2,666 separate variables, it hopefully gives you a broad understanding of the sort of information captured by the survey.</p><p>Recently, I took it upon myself to explore the 2015 NSDUH data set, in order to examine some of the underlying statistics about substance use, abuse, and treatment in America today.</p><p>In a future post, I’ll discuss some of the inferential modeling I did to examine factors most associated with obtaining substance abuse treatment. For today, however, I’ll examine some of the initial exploratory analysis and data visualization I did while getting a handle of the broader data set.</p><h4>Lifetime Reported Use of Substances</h4><p>Some of the initial questions asked of respondents relate to lifetime use of different substances — that is, has the respondent used “Substance X” ever in their lives? The gated logic of the survey is structured to provide respondents further questions about each substance if answered with a “yes,” or else move on to the next series of questions if answered with a “no.”</p><p>The various substances included in the “ever used” questions can include both discrete substances (e.g., LSD) as well as families of substances (e.g., hallucinogens). I initially examined the proportion of total respondents reporting lifetime use of various substances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5O_NLcD4M56i6bryDPl5xg.png" /><figcaption>In order, substances include: cigarettes, alcohol, marijuana, cocaine, crack, heroin, LSD, PCP, peyote, mescaline, psilocybin, ecstasy, ketamine, DMT/AMT/FOXY, salvia, hallucinogens, inhalants, methamphetamine, pain relievers (off-prescription), tranquilizers (off-prescription), stimulants (off-prescription), and sedatives (off-prescription)</figcaption></figure><p>From the results, we can see that the three most used substances include alcohol (over 70%), cigarettes (over 50%), and marijuana (over 40%). Notably, over 10% of respondents also report lifetime use of hallucinogens, cocaine, pain relievers (off-prescription), and inhalants.</p><h4>First Use and Most Recent Use of Substances (Respondents 50 years and Older)</h4><p>In addition to lifetime use of substances, I wanted to examine when individuals typically first use substances as well as their most recent use of substances across their lifetimes. The idea behind this is predominantly wondering which substances tend to be used first, but also which substances tend to be used continually over time once started.</p><p>Because I wanted to look at the use of substances over long periods of time, I restricted this analysis to examine only those respondents who are 50 years and older. For younger individuals, first use and most recent use are likely much closer together, and so could otherwise obfuscate broader trends over a lifetime.</p><p>I used violin plots for these analyses, which essentially draw a vertical distribution plot for each substance. Wider blobs indicate a larger proportion of the total measured sample.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YWb7B40O_QLy-0DiUyFiHw.png" /><figcaption>Substances are, in order: cigarettes, alcohol, marijuana, cocaine, crack, heroin, LSD, PCP, ecstasy, hallucinogens, inhalants, and methamphetamine.</figcaption></figure><p>It’s clear from this first plot that individuals’ late teens and early 20s reflect the “experimental” phases often attributed with these ages. Interestingly, crack and ecstasy have the highest median ages of first use (as indicated by the white dots in the black interquartile bars). Considering that respondents over 50 in this sample would have been born in 1965 or earlier, perhaps the later adoption of crack and ecstasy is partly related to the timing of when these substances first became broadly available?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jZRgqJX18NRsYwcMAON4yw.png" /><figcaption>Substances are, in order: cigarettes, alcohol, marijuana, cocaine, crack, heroin, LSD, PCP, ecstasy, hallucinogens, inhalants, and methamphetamine.</figcaption></figure><p>Examining the timing of last use, it appears that most respondents left hallucinogens and inhalants largely behind in their late teens and 20s. This perhaps implies these substances being used while going through an experimental phase, or may speak to a general difficulty in obtaining these substances once one moves into a more mature life stage.</p><p>Alcohol is clearly the most persistent substance in overall lifetime use, with the largest portion of the distribution congregating in individuals’ 50s, 60s, and 70s.</p><p>It is interesting to see the more uniform distributions of substances like crack and heroin. Perhaps this implies that use of these substances is less of a “phase”?</p><h4>Substance Use over Prior 30 Period</h4><p>For individuals reporting use of any given substance within the past month, I also looked at the general distributions of how frequently each substance was used in the 30 days prior to respondents completing their surveys. For example, a respondent reporting use of cigarettes in 30 out of the last 30 days implies daily use, and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ykwn_gTuihCRVRrEI0724A.png" /><figcaption>Substances include, in order: cigarettes, alcohol, marijuana, cocaine, crack, heroin, hallucinogens, inhalants, methamphetamine, pain relievers (off-prescription), tranquilizers (off-prescription), stimulants (off-prescription), and sedatives (off-prescription)</figcaption></figure><p>Some findings immediately stand out. Cigarette smokers report predominantly daily use, with cigarettes being by far the most frequently used of all substances on average.</p><p>Squatter distributions, such as that for users of hallucinogens, may imply a more “recreational” nature of certain substances, something that is used less frequently for a particular kind of experience.</p><p>Marijuana appears to have something of a bimodal distribution, with noticeable swells of use at both the frequent and infrequent sides of the distribution.</p><p>Other, “harder” drugs such as heroin and methamphetamine have thinner, more uniform distributions. Perhaps this indicates something about use based on availability? Further research may reveal more.</p><h4>Substance Dependence/Abuse over Prior 12 Month Period</h4><p>Finally, of all respondents reporting use of a substance within the last 12 months, how many qualified as either developing a dependence on that substance, or otherwise abusing that substance?</p><p>The NSDUH codebook defines dependence and abuse of substances based on respondents’ answers to a series of questions about each substance they have used.</p><p>Dependence was determined as a positive answer to at least three of the following:</p><ol><li>Spent a great deal of time over a period of a month getting, using, or getting over the effects of the substance.</li><li>Unable to keep set limits on substance use or used more often than intended.</li><li>Needed to use substance more than before to get desired effects or noticed that using the same amount had less effect than before.</li><li>Unable to cut down or stop using the substance every time he or she tried or wanted to.</li><li>Continued to use substance even though it was causing problems with emotions, nerves, mental health, or physical problems.</li><li>Reduced or gave up participation in important activities due to substance use.</li><li>(For select substances) experiencing withdrawal symptoms at one time that lasted for longer than a day after they cut back or stopped using</li></ol><p>To qualify as abusing a substance, respondents needed to answer in the affirmative for one of the following, and not being determined as dependent on the substance (Note: the survey codebook indicates individuals cannot be classified as “abusing” tobacco products):</p><ol><li>Respondent reported having serious problems due to substance use at home, work or school.</li><li>Respondent reported using substance regularly and then did something where substance use might have put them in physical danger.</li><li>Respondent reported substance use causing actions that repeatedly got them in trouble with the law.</li><li>Respondent reported having problems caused by substance use with family or friends and continued to use substance even though it was thought to be causing problems with family and friends.</li></ol><p>Again, the following graphic displays the proportions of respondents reporting use of a substance in the last 12 months who were classified as <em>either</em> dependent upon or abusing the substance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*KEzptLAYMbIWkPqFf2S7VA.png" /><figcaption>Substances are, in order: nicotine products, alcohol, marijuana, cocaine/crack, heroin, hallucinogens, inhalants, methamphetamine, pain relievers (off-prescription), tranquilizers (off-prescription), stimulants (off-prescription), sedatives (off-prescription), any psychotropics (off-prescription), any illicit substance, any illicit substance except marijuana, and any illicit substance plus alcohol</figcaption></figure><p>Again, some findings immediately stand out.</p><p>Over 60% of respondents using heroin in the last 12 months qualified as being dependent on or abusing the substance. In addition, nearly 50% of respondents using methamphetamine qualified as dependent or abusing, and nearly 30% of nicotine product users qualified as dependent.</p><p>Finally, approximately 18% of respondents using any illicit drug (which includes marijuana, cocaine, heroin, methamphetamine, inhalants, hallucinogens, and off-prescription psychotropics) reported dependence or abuse of at least one of these substances.</p><p>While this has been a relatively brisk look through only a sliver of the data contained within the 2015 NSDUH, I look forward to further discussing some of the deeper analysis I did in an upcoming post.</p><p>I also acknowledge that I am not an expert in the use or trends of use in substances across America. I appreciate there is a vast literature that could better frame understanding of these findings. If you happen to know of any additional resources, please feel free to share!</p><p>Thanks, all!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=428019133b4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trump vs. Drumpf: How well can Natural Language Processing distinguish the President’s tweets from…]]></title>
            <link>https://medium.com/@msalmon00/trump-vs-drumpf-how-well-can-natural-language-processing-distinguish-the-presidents-tweets-from-cfea8411d110?source=rss-8e7466898cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfea8411d110</guid>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Michael Salmon]]></dc:creator>
            <pubDate>Mon, 15 May 2017 04:50:14 GMT</pubDate>
            <atom:updated>2017-05-15T13:08:59.386Z</atom:updated>
            <content:encoded><![CDATA[<h3>Trump vs. Drumpf: How well can Natural Language Processing distinguish the President’s tweets from parody?</h3><p>I’ve recently begun using the Twitter API (application programming interface) to examine the content of users’ tweets via natural language processing (NLP). While the set-up of a tweet miner could be its own separate post, this week I wanted to focus on a fun, little side project I conducted to examine the contents of President Trump’s voluminous tweeting habits against those of a competing parody twitter handle that mimics the mannerisms and word stylings of the president to humorous effect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*1nf6h8EuKftzYZuNZPSifA.png" /><figcaption>Image retrieved from the Twitter stream of <a href="https://twitter.com/RealDonalDrumpf">@RealDonalDrumpf</a>—<a href="https://twitter.com/RealDonalDrumpf/status/862142415460814849">https://twitter.com/RealDonalDrumpf/status/862142415460814849</a></figcaption></figure><p>While there are several quality Trump parody handles to choose from, I decided to contrast the tweets of <a href="https://twitter.com/realDonaldTrump">@realDonaldTrump</a> with those of <a href="https://twitter.com/RealDonalDrumpf">@RealDonalDrumpf</a>. Some quick examples of their respective postings:</p><h3>Donald J. Trump on Twitter</h3><p>I am watching the Democrats trying to defend the &quot;you can keep you doctor, you can keep your plan &amp;amp; premiums will go down&quot; ObamaCare lie.</p><p>Compared to:</p><h3>Donald J. Drumpf on Twitter</h3><p>On this #nationaldayofprayer, I am proud to repeal Obamacare so that poors can have the same healthcare Jesus used: miracles.</p><p>Now, it is clear that one of these tweets is a clear parody of the other (can you guess which?), but I thought it would be fun to draw out several tweets from both of these handles, perform natural language processing on their contents, and then build a classification model to predict which tweets came from each handle.</p><p>The burning question is, therefore, how similar is President Trump to a parody of himself?</p><h3>The Data — cleaning and preparation</h3><p>For this analysis, I drew 150 tweets from both the Trump and Drumpf handles from approximately a two week period (~April 23 — May 5, 2017). For examining and manipulating the data, I utilized the pandas library in python.</p><p>After filtering out all retweets (as these would not reflect original content from the respective authors), I was left with 136 tweets from Trump and 146 from Drumpf. I called my data frame “all_tweets”. Here’s a quick look at the structure of its contents:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WwK97mpQ6WfTj8j7aLU8Hw.png" /></figure><p>Before submitting the contents of the tweets to NLP, I utilized a function to remove special characters, to improve the ability of the NLP to focus on words regardless of attached punctuation or capitalization. As such, I specified a “cleaning list” of characters to remove and lowercased all characters.</p><pre>def data_clean(df, column):<br>  cleaning_list = [“+”, “:”,”#”,”$”,”/”,”,”,”?”,”.”,”;”,”-”,”@”,”!”,”&amp;”,”%”,”^”,”*”,”)”,”(“, “\n”, “http”]<br>  for item in cleaning_list:<br>  df[column] = df[column].str.replace(item, “”)<br>  df[column] = map(str.lower, df[column])</pre><p>I then applied the function to the “text” column of the all_tweets data frame.</p><pre>data_clean(all_tweets, “text”)</pre><p>Next, I dropped all remaining columns except for the text and twitter handle for purposes of implementing NLP more easily:</p><pre>all_tweets.drop([“created_at”, “mined_at”, “retweet_count”, “tweet_id”], axis=1, inplace=True)</pre><pre>all_tweets.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TFkkHJa965Ax6RmapSLkFg.png" /></figure><p>Now, in building any decent model, we typically want to do what’s known as a “train-test” split, which essentially divides our data into separate sets — one for building (or “training”) our model, and another set for “testing” the efficacy of the model on data not used in the training process. A typical split might utilize 70% of your data for training the model, while leaving 30% for testing purposes.</p><p>In this exercise, I conducted a train-test split of 75 tweets/25 tweets, and used the remaining tweets (36 for Trump, 46 for Drumpf) as an additional “hold-out” set to be used for further validation of the classification model. Here’s the code I used to set this up:</p><pre>#create a holdout set of 36 Trump tweets and 46 Drumpf tweets <br>#(taking everything after the 100th tweet for each)</pre><pre>hold_trump = all_tweets[all_tweets[“handle”]==”Donald J. Trump”].iloc[101:,:]</pre><pre>hold_drumpf = all_tweets[all_tweets[“handle”]==”Donald J. Drumpf”].iloc[101:,:]</pre><pre>#merging two holdout dfs together<br>hold_total = pd.concat([hold_trump, hold_drumpf])</pre><pre>#removing holdout_total from all_tweets<br>all_tweets = all_tweets[~all_tweets.isin(hold_total)].dropna(how=’all’)</pre><pre>#importing train-test split, applying to all_tweets, <br>#stratifying on handle:</pre><pre>from sklearn.model_selection import train_test_split</pre><pre>X_train, X_test, y_train, y_test = train_test_split(all_tweets[“text”], all_tweets[“handle”], test_size=0.25, random_state=42, stratify=all_tweets[“handle”])</pre><p>Now I could begin conducting NLP on the data itself. For this exercise, I utilized the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html">tf-idf vectorizer</a>. Tf-idf (or term frequency / inverse document frequency) can be useful for distinguishing between authors who may use a lot of the same words or phrases. It effectively prioritizes those words or phrases that are largely unique to one source or another to help distinguish between them.</p><p>To hone in on the words and phrases most pertinent for our analysis, I utilized a common “stop word” list, which helps prevent words like “a”, “the” “and”, etc. from being considered during NLP. I had my vectorizer consider phrases between 1–2 words in length (also called n-grams).</p><p>I restricted my analysis to those words appearing in at least 2.5% of tweets (to restrict one-off words or unique URL strings) and appearing in no more than 95% of tweets (because I’m not interested in words that are always present).</p><p>After importing the tf-idf vectorizer, I “fit” the vectorizer to my training data, and then transformed my train and test sets into vectorized matrices. These matrices are essentially data tables of each n-gram extracted from the train set, with weighted coefficients representing the frequency of an n-gram in each tweet compared to its relative frequency among all tweets.</p><pre>from sklearn.feature_extraction.text import TfidfVectorizer</pre><pre>tvec = TfidfVectorizer(stop_words=’english’,ngram_range=(1, 2), min_df = .025, max_df=.95)</pre><pre>tvec.fit(X_train)</pre><pre>#transforming X_train, X_test and hold_out into dataframes</pre><pre>tweet_train = pd.DataFrame(tvec.transform(X_train).todense(),<br> columns=tvec.get_feature_names())</pre><pre>tweet_test = pd.DataFrame(tvec.transform(X_test).todense(),<br> columns=tvec.get_feature_names())</pre><pre>tweet_holdout = pd.DataFrame(cvec.transform(hold_total[&quot;text&quot;]).todense(),<br>                       columns=cvec.get_feature_names())</pre><p>Here’s a snapshot of the tweet_test data frame, with different n-gram columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_KSKcrra1bHcal80xL-V9A.png" /></figure><h3>Conducting the Analysis</h3><p>With our train and test data effectively processed, we can now develop our model and test its initial efficacy. For this analysis, I utilized a <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html">Logistic Regression</a>, which classifies observations based on the probability that an observation belongs to one class or another (typically splitting at 0.5 on a scale of 0.0 to 1.0).</p><p>To examine the output of the analysis, I used both a confusion matrix and a classification report. These two outputs relate the effectiveness of the model’s predictions in classifying tweets to their appropriate authors.</p><pre>from sklearn.metrics import classification_report, confusion_matrix, accuracy_score</pre><pre>from sklearn.linear_model import LogisticRegression</pre><pre># Setup logistic regression and score train set<br>logreg = LogisticRegression()</pre><pre>logreg.fit(tweet_train, y_train)</pre><pre>print “Train score: “, logreg.score(tweet_train, y_train)<br>print</pre><pre>#using logreg on test set<br>y_probs = logreg.predict(tweet_test) #predict y values for X_test</pre><pre>print “Output for Tested Model:”<br>print “Confusion Matrix of Predictions: “<br>print<br>print(confusion_matrix(y_test, y_probs))<br>print</pre><pre>#printing classification report<br>print “Classification Matrix: “<br>print<br>print(classification_report(y_test, y_probs, target_names=[“Drumpf”, “Trump”]))<br>print &quot;Test score: &quot;, logreg.score(tweet_test,y_test)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Aj8BntdS8FqAJq5mPfE_Lw.png" /></figure><p>From this, we can see the model scored pretty well on the training set (87% accurate), and still did pretty well on the test set (73% accurate).</p><p>For the Confusion Matrix, note that the rows reference actual values (e.g., actual Drumpf Tweets; actual Trump Tweets), while columns represent predictions (e.g., predicted Drumpf Tweets; predicted Trump Tweets).</p><p>We can see from this that four Drumpf tweets in the test set were incorrectly predicted as Trump tweets, while 10 Trump tweets were incorrectly predicted as Drumpf tweets.</p><p>This means that 2 in 5 Trump tweets (40%) were mistaken for parody (Drumpf) tweets, compared to 15% the other way around.</p><p>For the classification report:</p><p>precision scores = true positives / (true positives + false positives) — of all predicted, the % correct</p><p>recall = true positives / (true positives + false negatives) — of all actual, the % correct</p><p>f1-score = weighted harmonic mean of precision and recall — f1-score reaches its best value at 1 and worst at 0.</p><p>support is the number of actual observations for each class</p><p>Now, let’s analyze the additional hold-out set, and see how our scores change:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*puubudFZld0-iAZg9MdUjA.png" /></figure><p>Here again, the model performs quite well in accurately classifying the Drumpf tweets (11% misclassified as Trump tweets), but does quite poorly in predicting the tweets of the president himself (49% misclassified as Drumpf tweets).</p><h3>Common Words and Phrases</h3><p>Before wrapping things up, let’s take a quick look at the words and phrases most commonly used by the President vs. the Parody. To do this, I used a separate count vectorizing process to reexamine the content of each handle’s tweets in order to provide a raw count of their appearances (I utilized the same stop words, n-gram sizes, and minimum and maximum frequencies I set with my tf-idf above).</p><p>Here were the most frequent words employed by President Trump:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xwgJbKte-UCiHfifkv8TKg.png" /></figure><p>And now, the most frequent words employed by Mr. Drumpf:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t5k0-zdfqIQjdub5n6pavw.png" /></figure><p>Interestingly, both handles actually use many of the same words, often just in different frequencies:</p><p>Great, big, obamacare, healthcare, democrats, today, lol, and president are all used relatively frequently by both handles. In a sense, it’s almost more impressive that the classification model was able to perform as well as it<br>did considering how similar the lists of the most frequent words were between each handle.</p><h3>Take-aways and Conclusions</h3><p>While this was meant purely as a fun distraction, and is not meant to be taken with any real seriousness, I did find it very interesting that a classification model taking a first pass at processing the text of President Trump’s tweets often could not tell the President’s real words from parody.</p><p>These first several months of President Trump’s administration have been rife with both confusion and excitement, often spurred by the President’s infamous tweeting habits. However, it might bring us some pause to truly reflect on the outsized nature of the President’s personality and the sentiments conveyed in the messages he transmits to his 29+ million Twitter followers. Indeed, many media figures can be left guessing at the seriousness of the President’s often inflammatory or outright baffling remarks, wondering whether the President is testing or openly “trolling” the American media and the broader American public.</p><p>While this classification model is not perfect by any stretch of the imagination (it could benefit from larger tweet repositories, further text cleaning, and tuning of parameters), it does seem to reflect the fact that neither man nor machine often has a good sense of the seriousness of this President’s messages to the world.</p><p>Still, one might be left with a lingering notion upon reviewing future tweets from The Donald himself: is this Presidential or Parody?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfea8411d110" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bayes Rule and Cookies — A Primer in Code]]></title>
            <link>https://medium.com/@msalmon00/bayes-rule-and-cookies-a-primer-in-code-3ee15b6dc755?source=rss-8e7466898cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ee15b6dc755</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[cookies]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <dc:creator><![CDATA[Michael Salmon]]></dc:creator>
            <pubDate>Sun, 07 May 2017 04:05:11 GMT</pubDate>
            <atom:updated>2017-05-07T04:05:11.676Z</atom:updated>
            <content:encoded><![CDATA[<p>Bayes’ Rule (and Bayesian statistics in general) is an increasingly prominent force within the modern scientific community. When examining any given event, Bayes’ Rule allows us to update our beliefs about the probability of an event based on data we observe in the world around us.</p><p>This can be seen in <a href="https://medium.com/@msalmon00/replicating-the-monty-hall-problem-through-code-a2a93bcab734">my earlier discussion of the Monty Hall Problem</a>. As Monty Hall opens one door to reveal a “Zonk” prize, we are able to update our assumptions on the likelihood of our original door pick concealing the grand prize compared to the likelihood of the prize being behind the last remaining unopened door.</p><p>Today, I want to briefly explore the different components of Bayes’ Rule, and provide a short demonstration of Bayes’ Rule in action with a bit of coding in Python. To make this more fun, our demonstration will involve… cookies!!</p><h3>A Case of Two Cookie Jars</h3><p>Let’s say you have two cookie jars in front of you. Both cookie jars contain vanilla and chocolate cookies, but in different proportions. The first jar contains 30 vanilla cookies and 10 chocolate cookies. The second jar contains 20 vanilla cookies and 20 chocolate cookies.</p><p>You’re blindfolded and told to pick a cookie at random from one of the two jars. After picking, you’re told that you selected a vanilla cookie. What is the probability that you picked the cookie from either jar?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ekYXDhNwGvfvP-KkELqXQ.jpeg" /><figcaption>Maybe you picked more than one cookie — maybe you picked all of them?</figcaption></figure><p>While this is a relatively simple example, let’s examine how to solve this problem using Bayes Rule.</p><h4>Bayes in Brief</h4><p>For this problem, we want to solve the following question: what is the probability that I selected from a particular jar given that my selected cookie is vanilla?</p><p>Let’s first examine the notation of Bayes’ Rule, and then express this problem in Bayes’ notation.</p><p>Here is one of the ways Bayes’ Rule is commonly expressed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*9YuCNcICo5PW5qqQug6Yqw.png" /></figure><p>While appearing somewhat intimidating at first glance, there are four basic components to this rule. Let’s take a quick look at each.</p><p>P(A) — this is the probability of event A given no supplemental information. This is also known as the “prior probability.” In this case, P(A) is the probability of selecting from a particular jar given no information about what is selected.</p><p>P(B|A) — this is the likelihood of seeing evidence (data) B assuming that event A occurs. This is also simply known as the “likelihood.” In this case, P(B|A) is the probability of selecting a vanilla cookie given that we selected from a particular jar.</p><p>P(B) — this is probability of event B given no supplemental information. This is also known as the “marginal likelihood,” and is a way of scaling our probabilities to ensure we’re only looking at A within the context of B occurring. In this case, P(B) is the likelihood of selecting a vanilla cookie from the first jar plus the likelihood of selecting a vanilla cookie from the second jar. (Don’t worry if this is confusing — marginal likelihood is the most difficult aspect of this to pick up intuitively.)</p><p>P(A|B) — this is the probability of event A given that we have seen evidence B. This is known as the “posterior probability,” and specifies the probability of selecting from a given jar given that the cookie is vanilla.</p><h4>From Letters of Numbers</h4><p>Now that our eyes have sufficiently glazed over from looking at the statistical notation above, let’s plug some numbers into the formula to see how this problem resolves itself.</p><p>Because there are two different jars we could choose from, we’ll solve the above formula for both jars. We’ll call them jar_1 and jar_2.</p><p>From the problem statement above, we know that there are two jars. Without any further information, we can assume an equal probability of selecting from either of the two jars.</p><p>As such, we can assume that: P(A1) = P(jar_1) = 0.5 and P(A2) = P(jar_2) = 0.5</p><p>Step 1 complete!</p><p>Now, we’re told that jar_1 has 40 cookies inside — 30 vanilla and 10 chocolate. If we select from jar_1, we can assume the probability of selecting a vanilla cookie is 30/40 = 3/4 = 0.75. As such, P(B-vanilla|A1) — the probability of selecting a vanilla cookie, given we are selecting from jar_1 = 0.75</p><p>We also know that jar_2 has 40 cookies — 20 vanilla and 20 chocolate. If we select from jar_2, we can assume the probability of selecting a vanilla cookie is 20/40 = 1/2 = 0.5. As such, P(B-vanilla|A2) — the probability of selecting a vanilla cookie, given we are selecting from jar_2 = 0.5</p><p>Step 2 complete!</p><p>Marginal likelihood — our P(B) — is the most troublesome aspect to calculate. However, since we have solved P(A) and P(B|A) for both jars above, this will make solving P(B) much simpler. P(B) is a normalizing term for the likelihood of selecting a vanilla cookie from either jar, and so will be the same for both of our equations.</p><p>To solve for this, we sum the collective probabilities of selecting a vanilla cookie from jar_1 and selecting a vanilla cookie from jar_2. This relates to something called “The Law of Total Probability,” which is the sum of all probabilities of B for all possible As.</p><p>From Step 2 above, we know several things:</p><ul><li>the probability of selecting a vanilla cookie from jar_1 is 0.75</li><li>the probability of selecting from jar_1 is 0.5</li><li>the probability of selecting a vanilla cookie from jar_2 is 0.5</li><li>the probability of selecting from jar_2 is 0.5</li></ul><p>As such, we can determine the marginal likelihood of selecting a vanilla cookie from any jar to be P(B) = ((0.75 * 0.5) + (0.5 * 0.5)) = 0.625</p><p>Step 3 complete!</p><p>Now that we have found P(A), P(B|A) and P(B) for both jar_1 and jar_2, we can finally solve our equations.</p><p>The posterior probability of selecting from jar_1 given that we selected a vanilla cookie is:</p><p>P(A1|B-vanilla) = (0.75 * 0.5)/0.625 = 0.6 = 60%</p><p>The posterior probability of selecting from jar_2 given that we selected a vanilla cookie is:</p><p>P(A2|B-vanilla) = (0.5 * 0.5)/0.625 = 0.4 = 40%</p><p>Pretty cool, huh?</p><p>If this is not immediately intuitive, do not fret. These are not easy concepts to pick up immediately, so try to be patient and see if reviewing the steps again helps in understanding how we reached our answers for both jars.</p><p>Now that the pen and paper math is complete, let’s build this problem-solving method into python and have computers do the hard work for us!</p><h4>Cookies in Code</h4><p>Let’s up the ante a bit when coding this problem in python.</p><p>We’ll still assume that there are two jars, each containing different amounts of vanilla and chocolate cookies. However, each jar can now contain completely different ratios and numbers of these cookies AND we can set different probabilities of picking from either jar (i.e., we no longer have a 50/50 chance of picking from either jar).</p><p>Given these wide-ranging parameters, we still want to know the probability of picking from either jar, assuming that we select one vanilla cookie at random from either jar.</p><p>We’ll set up our function to take three arguments:</p><ul><li>the first argument will be a list of total vanilla and chocolate cookies in jar_1 (e.g. [30, 10] equates to 30 vanilla cookies and 10 chocolate)</li><li>the second argument will be a list of total vanilla and chocolate cookies in jar_2</li><li>the third argument will be a list of probabilities for selecting from either jar_1 or jar_2 (we’ll set this to a default 50/50 chance for both jars)</li></ul><p>We’ll set this up so that, even if the entered probabilities don’t sum to 1.0, our function will normalize the probabilities to allow the math to work properly. Here’s our code:</p><pre>def cookie_monster(jar1, jar2, jar_probs=[.5, .5]):<br>  new_probs = [] <br>  new_probs.append(float(jar_probs[0])/float(sum(jar_probs)))<br>  new_probs.append(float(jar_probs[1])/float(sum(jar_probs)))<br>  new_jar_1 = [float(i) for i in jar1] <br>  new_jar_2 = [float(i) for i in jar2]<br>  jar1_prob = ((new_jar_1[0]/sum(new_jar_1)) * new_probs[0]) <br>  jar2_prob = ((new_jar_2[0]/sum(new_jar_2)) * new_probs[1])<br>  norm_jar_1 = jar1_prob/(jar1_prob + jar2_prob)<br>  norm_jar_2 = jar2_prob/(jar1_prob + jar2_prob)<br>  return norm_jar_1, norm_jar_2</pre><p>Let’s examine the code a little closer:</p><pre>  new_probs = [] <br>  new_probs.append(float(jar_probs[0])/float(sum(jar_probs)))<br>  new_probs.append(float(jar_probs[1])/float(sum(jar_probs)))</pre><p>This creates a new list for the jar_probs to ensure they sum to 1.0. As you can see, we generate an empty list, and then append two items — the relative probability of selecting from jar_1, and the relative probability of selecting from jar_2. We ensure that these values are understood to be float values.</p><pre>  new_jar_1 = [float(i) for i in jar1] <br>  new_jar_2 = [float(i) for i in jar2]</pre><p>Next, we ensure that the cookie lists passed for both jars are converted to float values. Python can be really stubborn in not converting integer values to floats, so we add this step to ensure that all cookie totals are decimal values.</p><pre>  jar1_prob = ((new_jar_1[0]/sum(new_jar_1)) * new_probs[0]) <br>  jar2_prob = ((new_jar_2[0]/sum(new_jar_2)) * new_probs[1])</pre><p>This is effectively the P(B|A)*P(A) portion of each formula we calculated above. This specifies the probability of selecting a vanilla cookie from a given jar multiplied by the probability of selecting from that jar.</p><pre>  norm_jar_1 = jar1_prob/(jar1_prob + jar2_prob)<br>  norm_jar_2 = jar2_prob/(jar1_prob + jar2_prob)</pre><p>Finally, we normalize the probabilities for both jars by dividing each probability by the marginal likelihood. From above, we know the marginal likelihood is the sum of the probability of selecting a vanilla cookie from jar_1 and the probability of selecting a vanilla cookie from jar_2.</p><pre>  return norm_jar_1, norm_jar_2</pre><p>Finally, our function returns the probability of selecting a vanilla cookie from jar_1 along with the probability of selecting a vanilla cookie from jar_2.</p><p>Voila!</p><p>Now that our code is in place, let’s try a few examples.</p><p>First, we’ll state the problem as originally framed — 30 vanilla and 10 chocolate in jar_1, 20 vanilla and 20 chocolate in jar_2, and an even 50/50 chance of selecting from either jar (remember jar_probs defaults to 50/50):</p><pre>cookie_monster([30, 10], [20, 20])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*w1XsnoUu5-QVZHFX5d0SnQ.png" /></figure><p>Excellent! Confirmation that our math and code result in the same answer — this is promising!</p><p>Now, let’s change up the probability of selecting from either jar. For this example, we’ll keep the cookie counts the same, but boost our probability of selecting from jar_2 to 80%</p><pre>cookie_monster([30, 10], [20, 20], [.2, .8])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*okdnrA2Q3600GHKd81go3Q.png" /></figure><p>We see how much our chances of selecting from jar_2, given a vanilla cookie, have improved.</p><p>Next, we’ll set impossible probabilities to ensure our probability transformation is working properly.</p><pre>cookie_monster([30, 10], [20, 20], [1, 1])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*in38JEW9wPdwk8o6zzp1wg.png" /></figure><p>Yep! All good!</p><p>Finally, what if we keep the cookie counts the same for both jars, but alter the jar probabilities?</p><pre>cookie_monster([20, 20], [20, 20], [.333, .667])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*AgTzBfLBSGAFHbfhVNuBlQ.png" /></figure><p>As we might expect, if the counts and ratios of vanilla and chocolate cookies are the same in both jars, than the probability of selecting a vanilla cookie from either jar depends solely on the probability of selecting from either jar.</p><p>Hopefully, this has been an interesting and insightful look into Bayes’ Rule, and how it can be explored in a simple example through code in python.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ee15b6dc755" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Cleaning Web Scraped Job Data]]></title>
            <link>https://medium.com/@msalmon00/data-cleaning-web-scraped-job-data-6c2a2d963cd?source=rss-8e7466898cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c2a2d963cd</guid>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Michael Salmon]]></dc:creator>
            <pubDate>Mon, 01 May 2017 01:15:09 GMT</pubDate>
            <atom:updated>2017-05-02T00:50:36.643Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi all,</p><p>In a <a href="https://medium.com/@msalmon00/web-scraping-job-postings-from-indeed-96bd588dcb4b">previous post</a>, I reviewed a process for scraping job posting data from the website indeed.com. I did this with the intention of collecting a variety of job postings from different cities, and then building a model to predict their relative salaries.</p><p>Before I could build my model, however, I needed to perform some initial analysis and cleaning of the scraped data. This post will briefly review some of the techniques I utilized in my cleaning process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IgjxRBJPX47zLC7gQzBrdQ.jpeg" /></figure><p>For this task, I utilized python and its assorted libraries, including pandas and numpy.</p><p>When we last left off, I had successfully scraped and saved a multitude of job postings from different US cities into a single pandas data frame, which looks something like the following (you’ll recall, the majority of job postings did not include salary information):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SaW9voYFEGNPMChs_Om8yg.png" /></figure><p>In approaching the task of cleaning my data, I had a few objectives:</p><ul><li>eliminate all duplicate job postings from my data</li><li>format all available salary data so that salaries were presented in terms of a single figure detailing expected annual compensation</li><li>conduct any initial cleaning of string data that will make future natural language processing easier when building my predictive models</li></ul><h4>Eliminating Duplicate Job Postings</h4><p>I began by reading in my scraped data from my saved csv, and checking its shape. I then eliminated all duplicate rows, and assessed how many truly unique postings I gathered in my scraping.</p><pre>#reading csv with index_col = 0, otherwise I get an additional<br>#unnamed column of separate index values</pre><pre>scrape_data = pd.read_csv(“scrape_data.csv”, index_col=0)<br>scrape_data.shape</pre><pre>#dropping any duplicate rows:<br>scrape_data = scrape_data.drop_duplicates()<br>scrape_data.reset_index(drop=True, inplace=True)<br>scrape_data.shape</pre><p>From this process alone, my data frame was reduced from 128,289 rows down to 6,399. While not complex in terms of coding, I only wanted to use unique job postings when conducting my future analysis.</p><h4>Formatting Salary Data</h4><p>To prepare the salary data, I first wanted to look at the sorts of values I was dealing with:</p><pre>scrape_data[“salary”].value_counts()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CCtEDsg22tQuH2a0wHZ9rw.png" /></figure><p>As you can see, there are several aspects of this data that could make further calculations difficult. Python doesn’t know how to handle characters like commas and dollar-signs when conducting mathematical operations, so we need to get rid of these and “\n” strings before we can go much further.</p><p>On top of this, you’ll notice that salaries can be presented as single figures, or ranges, and that they can appear on different scales of payment — per year, per month, or per hour. I needed to take care of all of these items in the course of preparing my salary data for further analysis.</p><p>I began by stripping out some of the extraneous characters that I knew I needed to eliminate:</p><pre>#some targeted cleaning of salary information to make parsing easier #—&gt; remove “\n”, “$”, and “,”</pre><pre>scrape_data[“salary”] = scrape_data[“salary”].str.replace(“\n”, “”)<br>scrape_data[“salary”] = scrape_data[“salary”].str.replace(“,”, “”)<br>scrape_data[“salary”] = scrape_data[“salary”].str.replace(“$”, “”)</pre><p>While I knew I would need to remove string data on the scale of pay (e.g., “a year,” “a month”) from the salary data before I could work with it, I did want to preserve information about these original pay scales for possible future use. As such, I created a new column of data to capture this. I called this new column “og_salary_period”:</p><pre>scrape_data[“og_salary_period”] = np.nan</pre><pre>#if the salary contains information on time period, save that time<br>#period string in the og_salary_period column</pre><pre>scrape_data.ix[scrape_data[“salary”].str.contains(“year”), “og_salary_period”] = “year”</pre><pre>scrape_data.ix[scrape_data[“salary”].str.contains(“month”), “og_salary_period”] = “month”</pre><pre>scrape_data.ix[scrape_data[“salary”].str.contains(“week”), “og_salary_period”] = “week”</pre><pre>scrape_data.ix[scrape_data[“salary”].str.contains(“day”), “og_salary_period”] = “day”</pre><pre>scrape_data.ix[scrape_data[“salary”].str.contains(“hour”), “og_salary_period”] = “hour”</pre><p>I then isolated all jobs with salary information into a separate data frame that I could work with to scale them all accordingly. Notice, I also removed these salaried rows entirely from the original scale_data table. I did this knowing that I would re-append the salaried rows once I had effectively standardized their appearance.</p><pre>#filtering out the results with salary data != “Nothing_found”</pre><pre>salary_data = scrape_data[scrape_data[“salary”] != “Nothing_found”]</pre><pre>#removing all rows in salary data from scrape data, and converting #all “Nothing_found” values to NaN, so that float salary values can #be easily reintegrated later</pre><pre>scrape_data = scrape_data[~scrape_data.isin(salary_data)].dropna(how=’all’)</pre><pre>scrape_data[“salary”].replace(“Nothing_found”,np.nan, inplace=True)<br>scrape_data[“salary”].astype(‘float’)</pre><p>Here’s a quick look at the salary data frame. You’ll notice that the og_salary_period column captures information about the original scale of the salary information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yMD7KXi-XLQJwEUco1gzxA.png" /></figure><p>From here, I separated job postings with salary information according to the scale in which original salary data was presented. I also removed the string data associated with the pay scale.</p><pre>#separating out data based on whether contains hour, day, week, #month, year</pre><pre>year_salaries = salary_data[salary_data[“salary”].str.contains(“year”)]</pre><pre>month_salaries = salary_data[salary_data[“salary”].str.contains(“month”)]</pre><pre>week_salaries = salary_data[salary_data[“salary”].str.contains(“week”)]</pre><pre>day_salaries = salary_data[salary_data[“salary”].str.contains(“day”)]</pre><pre>hour_salaries = salary_data[salary_data[“salary”].str.contains(“hour”)]</pre><pre># removing string values(“ a year”, “ a week”, etc. from salary dfs)</pre><pre>year_salaries[“salary”] = year_salaries[“salary”].str.replace(“ a year”, “”)</pre><pre>month_salaries[“salary”] = month_salaries[“salary”].str.replace(“ a month”, “”)</pre><pre>week_salaries[“salary”] = week_salaries[“salary”].str.replace(“ a week”, “”)</pre><pre>day_salaries[“salary”] = day_salaries[“salary”].str.replace(“ a day”, “”)</pre><pre>hour_salaries[“salary”] = hour_salaries[“salary”].str.replace(“ an hour”, “”)</pre><p>Next, I defined a function that would detect when salary information was provided in a range (by looking for a hyphen in the data), and return an average of the two values. If there is no hyphen, it simply returns the single value as a float.</p><pre>def split_sal(i):<br> try:<br>   splt = i.split(‘ — ‘,1)<br>   first = float(splt[0])<br>   second = float(splt[1])<br>   return (first + second)/2<br> except:<br>   return float(i)</pre><p>With my function in place, I could then clean the salary data and transform any salary data that wasn’t on an annual scale to approximate yearly earnings.</p><p>(While there may be discussion to be had regarding whether to keep jobs with hourly and weekly salaries, I decided to preserve these jobs for the sake of having more data available, and because I felt that the pay grade could still approximate the true value of the work on offer.)</p><pre>year_salaries[“salary”] = year_salaries[“salary”].apply(split_sal)</pre><pre>month_salaries[“salary”] = month_salaries[“salary”].apply(split_sal)<br>month_salaries[“salary”] = month_salaries[“salary”] * 12</pre><pre>week_salaries[“salary”] = week_salaries[“salary”].apply(split_sal)<br>week_salaries[“salary”] = week_salaries[“salary”] * 52</pre><pre>day_salaries[“salary”] = day_salaries[“salary”].apply(split_sal)<br>day_salaries[“salary”] = day_salaries[“salary”] * 260</pre><pre>hour_salaries[“salary”] = hour_salaries[“salary”].apply(split_sal)<br>hour_salaries[“salary”] = hour_salaries[“salary”] * 2080</pre><p>Finally, I appended the various salaried data frames together, and rejoined them to the original scrape_data. While I did this in two steps, it could be consolidated into a single step.</p><pre>#rejoining salary data into main scrape_data df</pre><pre>combined_salaries = pd.concat([year_salaries, month_salaries, week_salaries, day_salaries, hour_salaries], axis=0)</pre><pre>scrape_data = pd.concat([scrape_data, combined_salaries], axis=0)</pre><p>Huzzah! Salary data ready!</p><h4>Additional Assorted Data Cleaning</h4><p>Before I felt ready to begin my modeling, I wanted to accomplish a few more cleaning tasks to prepare data for natural language processing.</p><p>Before removing special characters from all data, I recognized that there were several instances of “r&amp;d” within the data. To prevent simply stripping the ampersand and leaving two separate strings of “r” and “d”, I wanted to targetedly change this particular string before doing a further blanket removal of special characters:</p><pre>#targeted replace of “r&amp;d” in job titles and summaries with #“research development”, as I want to remove “&amp;” <br>#from listings as part of general clean-up</pre><pre>scrape_data[“job_title”] = scrape_data[“job_title”].str.replace(“R&amp;D”, “research development”)</pre><pre>scrape_data[“summary”] = scrape_data[“summary”].str.replace(“R&amp;D”, “research development”)</pre><p>Next, I defined a function to scan a column and remove any of a list of special characters. I then applied this function to all columns except for “salary” (a column of float data) and “og_salary_period” (as I specifically created this column to not contain any of these characters.</p><pre>#creating data character cleaning function, and applying to all #columns, also lowercasing all string data for ease of later nlp</pre><pre>def data_clean(df, column):<br>  cleaning_list = [“+”, “$”,”/”,”,”,”?”,”.”,”;”,”-”,”@”,”!”,”&amp;”,”%”,”^”,”*”,”)”,”(“, “\n”]<br>  for item in cleaning_list:<br>    df[column] = df[column].str.replace(item, “ “)<br>    df[column] = map(str.lower, df[column])</pre><pre>#can’t clean the salary column due to float values, and don’t need #to clean og_salary, so keeping out of the for loop</pre><pre>for column in scrape_data.columns[0:len(scrape_data.columns)-2]:<br>  data_clean(scrape_data, column)</pre><p>Finally, I cleaned up the location data a bit. I noticed that certain job postings contained location data that simply said “united states”. Because this was completely unhelpful, where this was the case, I inputed these values with the city names I collected in my scrape.</p><p>Also, to simplify location data, I kept only the city and state names from each posting. This meant splitting off location data at the appearance of zip codes. I realized I was losing some information with this, but I felt it would make examining groups of locations easier, with fewer unique representations of the same places that won’t add much to natural language processing analysis.</p><pre>#where location == “united states” I filled this in with the city #location name, otherwise it’ll be useless if more than one city has #location data of “united states”</pre><pre>#using numpy is faster than using .replace</pre><pre>scrape_data[‘location’] = np.where(scrape_data[‘location’] == “united states”, scrape_data[“city”], scrape_data[‘location’])</pre><pre>#I’m also preemptively simplfying location data to include only city #and state, cutting data off at zip code - note: “ \d” splits #information on the first numeric digit in the string.</pre><pre>scrape_data[“location”] = scrape_data[“location”].str.split(‘ \d’, expand=True, n=1)</pre><pre>scrape_data.tail()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UfLoZCtfC84qHMTdrQ558Q.png" /></figure><p>And that’s it! The final step was to save my data as a cleaned csv to make it easier to load and begin my modeling.</p><pre>scrape_data.to_csv(“scraped_clean.csv”)</pre><p>Thanks for reading, stay tuned for more!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c2a2d963cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Probability in a Weighted Coin-flip Game using Python and Numpy]]></title>
            <link>https://medium.com/data-science/probability-in-a-weighted-coin-flip-game-using-python-and-numpy-bc1686c49a35?source=rss-8e7466898cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc1686c49a35</guid>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Michael Salmon]]></dc:creator>
            <pubDate>Sat, 22 Apr 2017 22:17:38 GMT</pubDate>
            <atom:updated>2021-04-28T15:19:35.044Z</atom:updated>
            <content:encoded><![CDATA[<p>A short, fun one for this week:</p><p>Two players are playing a game where they flip a <em>not necessarily</em> fair coin, starting with Player 1. The first person to flip heads wins. The probability that a coin flipped lands on heads is <em>p</em>. What is the probability that Player 1 will win the game?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YUB4NGZ7zjszmbwR" /><figcaption>Photo by <a href="https://unsplash.com/@yirage?utm_source=medium&amp;utm_medium=referral">Andriyko Podilnyk</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We can explore this problem with a simple function in python. Let’s write a function that takes in two arguments: 1.) the number of games to be played, and 2.) the probability that a coin flip will result in heads (set to a default of 0.5 or 50%).</p><p>Let’s call this function “P1_win_prob_weighted_coin_game”</p><p>We can simulate this game with the following code:</p><pre>import numpy as np</pre><pre>def P1_win_prob_weighted_coin_game(num_games, prob_heads=.5): <br>  player_one_wins = 0 <br>  for n in range(0,num_games): <br>    num_flips = 0 <br>    win = 0<br>    while win == 0: <br>      turn = np.random.uniform(0,1) <br>      num_flips += 1 <br>      if turn &lt;= prob_heads: <br>        if num_flips % 2 != 0: <br>          player_one_wins += 1 <br>        win += 1 <br>  return float(player_one_wins)/float(num_games)</pre><p>Now, let’s walk through this briefly step-by-step.</p><pre>import numpy as np</pre><pre>def P1_win_prob_weighted_coin_game(num_games, prob_heads = 0.5): <br>  player_one_wins = 0</pre><p>We begin by importing numpy, as we can utilize its random choice functionality to simulate the coin-flipping mechanism for this game.</p><p>We define the name of our function, and specify our two arguments. We’ll set a default probability of heads to 0.5, in case we simply want to specify a number of games using a fair coin to see what results. Obviously, we can enter custom probabilities here as we like.</p><p>We also set a global counter for the number of games won by Player 1, to begin at 0 for each round of simulations.</p><pre>  for n in range(0,num_games): <br>      num_flips = 0 <br>      win = 0<br>      while win == 0: <br>        turn = np.random.uniform(0,1) <br>        num_flips += 1</pre><p>Now, for every game in the specified number of simulations (e.g., range(0, num_games), we will count the number of coin flips starting from zero. This both keeps track of how the long the game has gone on, and will help us to determine who the winner is once a heads has been flipped.</p><p>We also specify a win condition “win” that will help us know when a game has been completed. While win remains equal to zero, the players continue to flip the coin. Each “turn” constitutes a flip of the coin, which is a randomly selected decimal value between 0 and 1. Each time a turn is taken, we add one to our num_flips counter.</p><pre>        if turn &lt;= prob_heads: <br>          if num_flips % 2 != 0: <br>            player_one_wins += 1 <br>          win += 1</pre><p>If the randomly chosen “turn” value is less than or equal to our set “prob_heads” value, we consider this a winning condition.</p><p>We can understand this in the following way: if the probability of flipping a heads is 0.6, than 60% of the values between 0 and 1 could be interpreted as a flip of heads (e.g., all of the values between 0.0 and 0.6).</p><p>Once the winning condition is met, we check how many times the coin has been flipped. Because player 1 always has the first flip, we know that an odd number of flips means that player 1 was the last player to go, and would therefore be the winner. If player 1 won the game, we add one to our player_one_wins counter at the top of the function. We also change the value of “win” to 1.0, which will break off the current game, and begin the next game in the preset number of simulations.</p><pre>  return float(player_one_wins)/float(num_games)</pre><p>Once all simulations have been run, the function returns the number of games won by player 1 divided by the total number of games played. This will be expressed as a decimal win percentage.</p><p>Okay! Now that we have written our function, let’s play 50,000 games with a fair coin and see what results:</p><pre>P1_win_prob_weighted_coin_game(50000)</pre><pre>0.66848</pre><p>Over 50,000 games, we see that player 1 has a distinct advantage by going first. In fact, player 1 has about a 2/3 chance of winning the game as a result of flipping first, even when using a fair coin.</p><p>What if we adjust the probability of the coin turning up heads? What if the coin has a 75% chance of coming up heads?</p><pre>P1_win_prob_weighted_coin_game(50000, .75)</pre><pre>0.80066</pre><p>With a weighted coin coming up heads 75% of flips, player 1 would be expected to win about 80% of the time.</p><p>What if we really scale back the likelihood of a head appearing? Say, if heads only came up 1% of the time?</p><pre>P1_win_prob_weighted_coin_game(50000, .01)</pre><pre>0.50498</pre><p>It’s only when we drastically cut back the likelihood of a heads turning up that both players have approximately the same chances of winning.</p><p>Fun stuff! Play around with the function as you like to see how different numbers of simulations and probabilities for heads effect the likelihood of player 1 winning.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc1686c49a35" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/probability-in-a-weighted-coin-flip-game-using-python-and-numpy-bc1686c49a35">Probability in a Weighted Coin-flip Game using Python and Numpy</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web Scraping Job Postings from Indeed]]></title>
            <link>https://medium.com/@msalmon00/web-scraping-job-postings-from-indeed-96bd588dcb4b?source=rss-8e7466898cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/96bd588dcb4b</guid>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[job-hunting]]></category>
            <dc:creator><![CDATA[Michael Salmon]]></dc:creator>
            <pubDate>Sun, 16 Apr 2017 23:36:31 GMT</pubDate>
            <atom:updated>2017-04-16T23:36:31.877Z</atom:updated>
            <content:encoded><![CDATA[<p>So you’re in the job market, and you want to work smarter rather than harder at finding new and interesting jobs for yourself? Why not build a web scraper to collect and parse out job posting information for you? Set it, forget it, and come back to your data riches in a nice tidy format! How, you ask? Let’s take a look together!</p><p>[A disclaimer before beginning, many websites can restrict or outright bar scraping of data from their pages. Users may be subject to legal ramifications depending on where and how they attempt to scrape information. Many sites have a devoted page to noting restrictions on data scraping at www.[site].com/robots.txt. Be extremely careful if looking at sites that house user data — places like facebook, linkedin, even craigslist, do not take kindly to data being scraped from their pages. Scrape carefully, friends]</p><p>For this project, I wanted to explore data science-related jobs posted to a variety of cities on indeed.com, a job aggregator that updates multiple times daily. I conducted my scraping using the “requests” and “BeautifulSoup” libraries in python to gather and parse information from indeed’s pages, before using the “pandas” library to assemble my data into a dataframe for further cleaning and analysis.</p><h3>Examining the URL and Page structure</h3><p>First, let’s look at a sample page from <a href="https://www.indeed.com/jobs?q=data+scientist+%2420%2C000&amp;l=New+York&amp;start=10">indeed</a>.</p><p>Notice a few things about the way the URL is structured:</p><ul><li>note “q=” begins the string for the “what” field on the page, separating search terms with “+” (i.e. searching for “data+scientist” jobs)</li><li>when specifying salary, it will parse by commas in the salary figure, so the beginning of the salary will be preceded by %24 and then the number before the first comma, it will then be broken by %2C and continue with the rest of the number (i.e. %2420%2C000 = $20,000)</li><li>note “&amp;l=” begins the string for city of interest, separating search terms with “+” if city is more than one word (i.e. “New+York”</li><li>note “&amp;start=” notes the search result where you want to begin (i.e., start by looking at the 10th result)</li></ul><p>The URL structure will come in handy as we build a scraper to look at and gather data from a series of pages. Keep this in mind for later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TramcWKus8K18aq7ADspIg.png" /></figure><p>Each page of job results will have 15 job posts. Five of these are “sponsored” jobs, which are specially displayed by indeed outside of the normal order of results. The remaining 10 results are specific to the page being viewed.</p><p>All of the information on this page is coded with HTML tags. HTML (HyperText Markup Language), is the coding that tells your internet browser how to display a given page’s contents upon accessing it. This includes its basic structure and order. HTML tags also have attributes that are a helpful way of keeping track of what information can be found where within the structure of the page.</p><p>Chrome users can examine the HTML structure of a page by right-clicking on a page and choosing “Inspect” from the menu that appears. A menu will appear on the right-hand side of your page, with a long list of nested HTML tags housing the information currently displayed in your browser window. In the upper-left of this menu, there’s a small box with an arrow icon in it. Once clicked, the box will illuminate in blue (notice in the screenshot below). This will allow you to cursor over the elements in the page to display both the tag associated with that item, and to bring your inspection window directly to that item’s place in the HTML for the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0mhFgkSG0M-4i4gtfg0Slw.png" /></figure><p>In the screen show above, I’ve cursored over one of the job postings to show how the entire job’s contents is held within a &lt;div&gt; tag, with attributes including “class = ‘row result’”, “id=’pj_7a21e2c11afb0428&#39;”, etc,. Luckily, we will not need to know every attribute of every tag to extract our information, but it is helpful to know how to read the basic structure of a page’s HTML.</p><p>Now, let’s turn to python to extract the html from the page and look to building our scraper.</p><h3>Building the Scraper Components</h3><p>Now that we’ve looked at the basic structure of the page and know a little about it’s basic HTML structure, we can see about building code to pull out the information we’re interested in. We’ll import our libraries first. Note, I’m also importing “time”, which can be a helpful way of staggering page requests to not overwhelm a site’s servers when scraping information.</p><pre>import requests<br>import bs4<br>from bs4 import BeautifulSoup</pre><pre>import pandas as pd<br>import time</pre><p>Let’s start by pulling a single page, and working out the code to withdraw each piece of information we want:</p><pre>URL = “https://www.indeed.com/jobs?q=data+scientist+%2420%2C000&amp;l=New+York&amp;start=10&quot;</pre><pre>#conducting a request of the stated URL above:<br>page = requests.get(URL)</pre><pre>#specifying a desired format of “page” using the html parser - this allows python to read the various components of the page, rather than treating it as one long string.<br>soup = BeautifulSoup(page.text, “html.parser”)</pre><pre>#printing soup in a more structured tree format that makes for easier reading<br>print(soup.prettify())</pre><p>Using prettify makes it much easier to look through a page’s HTML coding, and will provide an output like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I9x47GWyjEmQFQHRTk6d7w.png" /></figure><p>Now, we know that our variable “soup” has all of the information housed in our page of interest. It is now a matter of writing code to iterate through the various tags (and nested tags therein) to capture the information we want.</p><p>While this is not the appropriate place to go over all of the ways in which information can be found or withdrawn from a page’s HTML, the <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">BeautifulSoup documentation</a> has a lot of helpful information that can guide one’s searching.</p><h4>Withdrawing Basic Elements of Data</h4><p>Approaching this task, I wanted to find and extract five key pieces of information from each job posting: Job Title, Company Name, Location, Salary, and Job Summary.</p><p>I know from looking at the page, that there must be 15 job postings therein. As such, I know that each function I write to withdraw a piece of information should yield 15 different items. If my output provides fewer than this, I can refer back to the page itself to see what information I’m not capturing.</p><h4>Job Title</h4><p>As noted above, I could tell that the entirety of each job posting is nested under &lt;div&gt; tags, with an attribute “class” = “row result”.</p><p>From there, I could see that job titles are listed under &lt;a&gt; tags, with attribute “title = (title)”. One can find the value of a tag’s attribute with tag[“attribute”], so I could use this to find the job title for each posting.</p><p>My function for withdrawing job title information involved three steps:</p><ul><li>pulling out all &lt;div&gt; tags with class including “row”</li><li>identifying &lt;a&gt; tags with attribute “data-tn-element”:”jobTitle”</li><li>for each of these &lt;a&gt; tags, find the value of attributes “title”</li></ul><pre>def extract_job_title_from_result(soup): <br>  jobs = []<br>    for div in soup.find_all(name=”div”, attrs={“class”:”row”}):<br>      for a in div.find_all(name=”a”, attrs={“data-tn-element”:”jobTitle”}):<br>      jobs.append(a[“title”])<br>  return(jobs)</pre><pre>extract_job_title_from_result(soup)</pre><p>This yielded an output like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1U8vT4IynI44xv5Vx9nN4w.png" /></figure><p>Perfect! All 15 jobs are represented here.</p><h4>Company Name</h4><p>Company names were a bit tricky, as most would appear in &lt;span&gt; tags, with “class”:”company”. Rarely, however, they will be housed in &lt;span&gt; tags with “class”:”result-link-source”.</p><p>I developed if/else statements to extract the company info from either of these places. Company names are output with a lot of white space around them, so inputting .strip() at the end helps to remove this when extracting the information.</p><pre>def extract_company_from_result(soup): <br>  companies = []<br>  for div in soup.find_all(name=”div”, attrs={“class”:”row”}):<br>    company = div.find_all(name=”span”, attrs={“class”:”company”})<br>    if len(company) &gt; 0:<br>      for b in company:<br>        companies.append(b.text.strip())<br>    else:<br>      sec_try = div.find_all(name=”span”, attrs={“class”:”result-link-source”})<br>        for span in sec_try:<br>          companies.append(span.text.strip())<br> return(companies)<br> <br>extract_company_from_result(soup)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Omj2YA9OpI9Ieit7vXpy4A.png" /></figure><h4>Location</h4><p>Locations are located under the &lt;span&gt; tags. Span tags are sometimes nested within each other, such that the location text may sometimes be within “class” : “location” attributes, or nested in “itemprop” : “addressLocality”. However, a simple for loop can examine all span tags for text wherever it may be and retrieve the necessary information.</p><pre>def extract_location_from_result(soup): <br>  locations = []<br>  spans = soup.findAll(‘span’, attrs={‘class’: ‘location’})<br>  for span in spans:<br>    locations.append(span.text)<br>  return(locations)</pre><pre>extract_location_from_result(soup)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2NQrhoq-rCW--bJh7B7wKg.png" /></figure><h4>Salary</h4><p>Salary was the most difficult data to extract from job postings. Most postings don’t contain any salary information at all. Among those that do, it can be in one of two different places. So, we need to write a function that can look in multiple places for information, and we need to create a placeholder “Nothing Found” value for any jobs that don’t contain salary. We want this placeholder to ensure that all of the information from any given job post can line up with all of the other pieces of relevant data when we later assemble our data into a single data frame.</p><p>Some salaries are housed under &lt;nobr&gt; tags, while others are under &lt;div&gt; tags, “class” : “sjcl” and are under separate nested &lt;div&gt; tags with no attributes. Try/except statements were particularly helpful in withdrawing this information.</p><pre>def extract_salary_from_result(soup): <br>  salaries = []<br>  for div in soup.find_all(name=”div”, attrs={“class”:”row”}):<br>    try:<br>      salaries.append(div.find(‘nobr’).text)<br>    except:<br>      try:<br>        div_two = div.find(name=”div”, attrs={“class”:”sjcl”})<br>        div_three = div_two.find(“div”)<br>        salaries.append(div_three.text.strip())<br>      except:<br>        salaries.append(“Nothing_found”)<br>  return(salaries)</pre><pre>extract_salary_from_result(soup)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p6CwQjktO4SjkcLeGQWyEQ.png" /></figure><p>As noted above, most jobs did not have any salary information included.</p><h4>Job Summary</h4><p>Finally, the job summaries. Unfortunately, the entirety of the job summaries are not included in the HTML from a given indeed page, however, we can get some information about each job from what’s provided. Selenium is a suite of tools that could allow a web scraper to click through different links on a page to withdraw this information from the full job postings. However, I did not utilize selenium for this particular effort.</p><p>Summaries are located under &lt;span&gt; tags. Span tags are sometimes nested within each other, such that the location text may sometimes be within “class” : “location” tags, or nested in “itemprop” : “addressLocality”. However, a simple for loop can examine all span tags for text wherever it may be and retrieve the necessary information.</p><pre>def extract_summary_from_result(soup): <br>  summaries = []<br>  spans = soup.findAll(‘span’, attrs={‘class’: ‘summary’})<br>  for span in spans:<br>    summaries.append(span.text.strip())<br>  return(summaries)</pre><pre>extract_summary_from_result(soup)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uYkj7DVSI7G8-_26TWB7Rg.png" /></figure><h3>Putting all the Pieces Together</h3><p>We’ve got all of the various pieces of our scraper. Now, we need to assemble them into the final scraper that will withdraw the appropriate information for each job post, keep it separate from all other job posts, and assemble all of my job posts into a single dataframe one at a time.</p><p>We can set up the initial conditions for each scrape by specifying a few pieces of information:</p><ul><li>We can detail how many results we want to scrape from each city of interest</li><li>We can assemble a list of all of the cities for which we want to scrape job postings</li><li>We can create an empty dataframe to house the scraped data for each posting. We can specify in advance the names of our columns for where we expect each piece of information to be located.</li></ul><pre>max_results_per_city = 100</pre><pre>city_set = [‘New+York’,’Chicago’,’San+Francisco’, ‘Austin’, ‘Seattle’, ‘Los+Angeles’, ‘Philadelphia’, ‘Atlanta’, ‘Dallas’, ‘Pittsburgh’, ‘Portland’, ‘Phoenix’, ‘Denver’, ‘Houston’, ‘Miami’, ‘Washington+DC’, ‘Boulder’]</pre><pre>columns = [“city”, “job_title”, ”company_name”, ”location”, ”summary”, ”salary”]</pre><pre>sample_df = pd.DataFrame(columns = columns)</pre><p>It goes without saying that the more results you want, and the more cities you look at, the longer the scraping process will take. This isn’t a huge issue if you start your scraper before you go out or go to sleep, but it’s still something to consider.</p><p>Assembling the actual scraper relates back to the patterns we noticed in the URL structure above. Because we know how the URLs will be patterned for each page, we can exploit this when building a loop to visit each page in a specific order to extract data.</p><pre>#scraping code:</pre><pre>for city in city_set:<br>  for start in range(0, max_results_per_city, 10):<br>  page = requests.get(‘<a href="http://www.indeed.com/jobs?q=data+scientist+%2420%2C000&amp;l=&#39;">http://www.indeed.com/jobs?q=data+scientist+%2420%2C000&amp;l=&#39;</a> + str(city) + ‘&amp;start=’ + str(start))<br>  time.sleep(1)  #ensuring at least 1 second between page grabs<br>  soup = BeautifulSoup(page.text, “lxml”, from_encoding=”utf-8&quot;)<br>  for div in soup.find_all(name=”div”, attrs={“class”:”row”}): <br>    #specifying row num for index of job posting in dataframe<br>    num = (len(sample_df) + 1) <br>    #creating an empty list to hold the data for each posting<br>    job_post = [] <br>    #append city name<br>    job_post.append(city) <br>    #grabbing job title<br>    for a in div.find_all(name=”a”, attrs={“data-tn-element”:”jobTitle”}):<br>      job_post.append(a[“title”]) <br>    #grabbing company name<br>    company = div.find_all(name=”span”, attrs={“class”:”company”}) <br>    if len(company) &gt; 0: <br>      for b in company:<br>        job_post.append(b.text.strip()) <br>    else: <br>      sec_try = div.find_all(name=”span”, attrs={“class”:”result-link-source”})<br>      for span in sec_try:<br>        job_post.append(span.text) <br>    #grabbing location name<br>    c = div.findAll(‘span’, attrs={‘class’: ‘location’}) <br>    for span in c: <br>      job_post.append(span.text) <br>    #grabbing summary text<br>    d = div.findAll(‘span’, attrs={‘class’: ‘summary’}) <br>      for span in d:<br>        job_post.append(span.text.strip()) <br>    #grabbing salary<br>    try:<br>      job_post.append(div.find(‘nobr’).text) <br>    except:<br>      try:<br>        div_two = div.find(name=”div”, attrs={“class”:”sjcl”}) <br>        div_three = div_two.find(“div”) <br>        job_post.append(div_three.text.strip())<br>      except:<br>        job_post.append(“Nothing_found”) <br>    #appending list of job post info to dataframe at index num<br>    sample_df.loc[num] = job_post</pre><pre><br>#saving sample_df as a local csv file — define your own local path to save contents <br>sample_df.to_csv(“[filepath].csv”, encoding=’utf-8&#39;)<br> <br></pre><p>And that should do it! After a short while (maybe longer), you’ll have you’re very own dataframe of scraped job postings. Here’s a quick look at an example output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*foA6RQ51IAAM_u378mhLhA.png" /></figure><p>In the near future, I’ll post again to describe how I used this data further to examine what sorts of jobs were available, and my attempts to classify whether jobs with no salary information would be above or below the median salary of those jobs I collected that did have salary information.</p><p>Thanks for tuning in — I hope you enjoyed the post!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96bd588dcb4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Tableau to Visualize the List of Greatest Video Games from Wikipedia]]></title>
            <link>https://medium.com/data-science/using-tableau-to-visualize-the-list-of-greatest-video-games-from-wikipedia-eee804459f29?source=rss-8e7466898cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/eee804459f29</guid>
            <category><![CDATA[gaming]]></category>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[videogames]]></category>
            <dc:creator><![CDATA[Michael Salmon]]></dc:creator>
            <pubDate>Sun, 09 Apr 2017 19:28:05 GMT</pubDate>
            <atom:updated>2017-04-09T19:28:05.132Z</atom:updated>
            <content:encoded><![CDATA[<p>A few weeks ago, I inaugurated my blog with a <a href="https://medium.com/towards-data-science/web-scraping-the-list-of-greatest-video-games-from-wikipedia-pt-1-230c64d93e8c">post reviewing a web scrape I did</a> of a <a href="https://en.wikipedia.org/wiki/List_of_video_games_considered_the_best">Wikipedia page listing the greatest video games of all time</a>. While I’ve been occupied with other ventures in the meantime, I’ve wanted to revisit the information I gathered to see if I can visualize the data in interesting ways and to practice my emergent Tableau skills.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2NuUsyfXsB9MRas46OYMzQ.png" /><figcaption>Solid Snake and Liquid Snake from Metal Gear Solid (1998). Original artwork by Yoji Shinkawa</figcaption></figure><p>For this effort, I used <a href="https://public.tableau.com/s/">Tableau Public</a>, which is a free, downloadable app with considerable depth. A caveat to using this program is that you’ll need to create a Tableau profile in order to save any of the visualizations you create. These will be hosted by the Tableau Public site, and so you’ll want to be aware of any proprietary concerns related to your data before saving your creations.</p><p>To briefly recap the data, the list of greatest games I utilized for this process consisted of 100 different games released between 1978 and 2013. This data included the year of each games’s release, the title of the game, the original platform/console on which it was released, the genre of the game, and the number of “greatest games” lists on which the game has appeared. Here is a partial snapshot of the data table’s contents:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o8UtbhZylICv8Mp0RpI-Jg.png" /><figcaption>Basic structure of the “Greatest Games” data table.</figcaption></figure><p>I initially saved this information as a dataframe in pandas, before converting and saving the data as a local csv file. Tableau Public is pretty flexible in terms of the types of files it can work with. It can import or connect to individual files (excel, JSON, etc.) or to remote data servers. For working with csv files, you’ll want to select “text file” from the starting menu.</p><p>Another great feature of Tableau Public is that it intuitively reads and understands the types of data that exist in each of your data columns (e.g., categorical, continuous, geo-spatial), which makes “drag-and-drop” visualizations that much easier to produce.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mkp5PzgwOSEdYrx8xneAMw.png" /><figcaption>The data structure of my csv file, as interpreted by Tableau. Categorical columns are labeled “Abc”, while continuous variables are labeled “#”. I generated an additional column of information “Console Developer” to ease some of my later visualizations.</figcaption></figure><p>When creating any given plot, it is as simple as opening a new Sheet (see the bottom-left corner of your Tableau window), and then dragging your variables of interest into their respective column, row, or marks fields. There are a few additional fields here that can be used in interesting ways that I won’t go into here. (For example, the “pages” field in the upper left can be used with time-series data to create a sort of flip book of data as it appears for each day/month/year of your time data.)</p><p>Here’s a quick look at the Tableau window with some of the data dropped into place. This plot depicts years on the x-axis, number of greatest games list appearances on the y-axis, with games as each individual dot, colored by original platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*avneS5xuyQGJcDxx4BkP_Q.png" /></figure><p>While Medium doesn’t yet allow easy embedding of Tableau visualizations directly into posts, I can post image captures, and then embed links in the images to take you to the interactive versions of these visualizations on the Tableau Public site. (For other Medium users, links can be embedded on images using <strong>Command+K on Mac, Control+K on PC.</strong>)</p><p>Here’s a more polished version of the above plot, which links to the interactive Tableau version:</p><figure><a href="https://public.tableau.com/profile/mike.salmon#!/vizhome/GreatestGames_1/Sheet1"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKSdCJZSSSQXIi_OD761rg.png" /></a></figure><p>Using the link, you’ll be able to cursor over any of the data points in the plot to reveal the name of each individual game, the year and original platform of its release, and the number of greatest games lists on which it has appeared (according to Wikipedia).</p><p>This initial plot is pretty interesting as a first glance. One can see a general upward trend in the number of games and number of greatest games lists on which they appear leading into the mid-late 90s and early 2000s, before trailing off again in the mid-2000s and beyond. This may be due in part to historical bias, with the late 90s and early 2000s appearing far enough in the distance for popular opinion to have coalesced regarding these games’ greatness, with other more modern classics like The Last of Us not yet having had enough time since release to cement their places in a wide range of greatest games lists.</p><p>While the clusters of dots can be somewhat telling, looking at the number of greatest games per year can be made a bit easier using a segmented bar graph. Exploring the interactive plot, games appear in descending order each year by the number of greatest games lists on which they appear.:</p><figure><a href="https://public.tableau.com/profile/mike.salmon#!/vizhome/GreatestGames_3/Sheet3"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jaHnEt0lKscRB250fRDp4Q.png" /></a></figure><p>This graph is much easier to read at face value when determining the number of “greatest games” released each year in the last several decades. Clearly, 1998 and 2001 are standout years here. The year 1998 alone saw such seminal classics as The Legend of Zelda: Ocarina of Time, Metal Gear Solid, Pokemon Red, Soul Caliber, Star Craft, Half-Life, and Resident Evil: 2, while 2001 had other historic heavy-hitters such as Halo: Combat Evolved, Super Smash Bros: Melee, Metal Gear Solid 2: Sons of Liberty, Final Fantasy X, Grand Theft Auto III, and Silent Hill 2.</p><p>After looking at the number of greatest games released by year, I wanted to take a closer look at the number of games released by console:</p><figure><a href="https://public.tableau.com/profile/mike.salmon#!/vizhome/GreatestGames/Sheet2"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DDI6UqmlcusKHYB84DNcVw.png" /></a></figure><p>Plotting games by original console of release is somewhat complicated by PC and Arcade games. This is due in part to these platforms somewhat transcending the concept of generational consoles — Arcade platforms clearly dominated before home consoles were widely adopted, and PCs are a sort of timeless platform unconstrained by the distinct time periods occupied by other home consoles. Still, it is interesting to see the first three Playstation consoles within the top six listed consoles by number of greatest games. It is also interesting to see just how many Nintendo-created consoles appear in this list.</p><p>In an effort to level the playing field with PC and Arcade, I collapsed game categories by console developers, in order to see how well separate console developers (e.g., Sony, Microsoft, Nintendo) performed relative to one another across all of their disparate platforms (e.g., Playstation 1, Playstation 2, Playstation 3 vs. NES, SNES, N64, and so on.)</p><p>I decided to plot every game in descending order by number of greatest list mentions, colored by console developer: (Note, clicking through to the interactive plot, you can scroll to the right to see all listed games)</p><figure><a href="https://public.tableau.com/profile/mike.salmon#!/vizhome/GreatestGames_5/Sheet1"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7sWDlU-Ti-ft0Ygc9WD8Jg.png" /></a></figure><p>I thought it was interesting to see how, while Sony’s individual consoles performed better than most Nintendo platforms, Nintendo-developed games are strongly represented among the most critically acclaimed games by number of greatest game list mentions.</p><p>I hope you enjoy clicking through these visualizations of the greatest games. I really enjoyed making them in the short time I’ve been working with Tableau. I also hope you’ve learned a bit about how Tableau works and how user-friendly it can be if you decide to try it out.</p><p>Let me know if you have any feedback!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eee804459f29" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/using-tableau-to-visualize-the-list-of-greatest-video-games-from-wikipedia-eee804459f29">Using Tableau to Visualize the List of Greatest Video Games from Wikipedia</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predictive Modeling with Iowa State Liquor Sales Data]]></title>
            <link>https://medium.com/data-science/predictive-modeling-with-iowa-state-liquor-sales-data-e45342081b83?source=rss-8e7466898cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e45342081b83</guid>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Michael Salmon]]></dc:creator>
            <pubDate>Sun, 02 Apr 2017 23:01:35 GMT</pubDate>
            <atom:updated>2017-04-04T17:36:50.966Z</atom:updated>
            <content:encoded><![CDATA[<p>Howdy friends,</p><p>Recently, I was tasked with building a predictive model using a large data set of liquor sales in Iowa.</p><p>As a brief bit of background, Iowa is an <a href="https://en.wikipedia.org/wiki/Alcoholic_beverage_control_state">alcohol beverage control state</a>, meaning that the state maintains a monopoly on wholesaling of alcohol throughout the State. Effectively, private retailers must purchase their alcohol from the state before selling it to individual consumers. You can learn a bit more about this at the wiki page for the <a href="https://en.wikipedia.org/wiki/Iowa_Alcoholic_Beverages_Division">Iowa Alcoholic Beverage Division</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e3XouWJprVxIDzxzLgWPpw.png" /><figcaption>The majestic Iowa flag</figcaption></figure><p>For purposes of this project, I utilized a subset of the Iowa liquor sales data, which can be found on Iowa’s <a href="https://data.iowa.gov/Economy/Iowa-Liquor-Sales/m3tr-qhgy">state-hosted open data portal</a>. While the portal contains liquor sales going back to 2012, the dataset I used comprised individual liquor sales for the entirety of 2015 and the first quarter (Jan-Mar) of 2016. The objective of the project was to use sales data from 2015 to build a model that could predict total 2016 sales based off Q1 2016 data. I analyzed the data using the Pandas library in Python.</p><p>Briefly, the dataset comprised over 2.7 million rows of data. Each observation contained information including:</p><ul><li>the date of purchase (mm/dd/yyyy)</li><li>details on the vendor making purchase (e.g., assigned store number, county and city details);</li><li>the type and brand of liquor purchased (e.g., Jack Daniel’s Tennessee Whiskey);</li><li>the bottle size (e.g., 750 ml) and number of bottles purchased;</li><li>the total sale amount, including State and retail costs per bottle; and</li><li>the total volume of liquor sold, in liters and gallons</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ahOKzGYPWsgvRQleuALY2w.png" /><figcaption>A snapshot of select columns in the original dataset. There are many additional data columns not pictured here.</figcaption></figure><h3>Initial Data Cleaning and Exploratory Analysis</h3><p>Initial data cleaning and exploratory analysis consisted of the following steps.</p><p>General Data Cleaning:</p><ul><li>Improving clarity of column names, converting data types where needed.</li><li>Removing duplicate rows</li><li>Replacing string “nan” values with true nulls</li><li>For paired columns (category_ID and category_name), if one column value had no adjacent pair, I filled it in, if able, or else changed it to null</li></ul><p>Ensuring Correct Calculations:</p><ul><li>Ensuring bottle size (e.g., 750 ml) x bottles sold = volume liters sold</li><li>Ensuring bottle retail value x bottles sold = sale dollars</li><li>I found no problems with the math, but it was good to check all the same</li></ul><h3>Preparing for Model Building</h3><p>I knew that analysis would be significantly easier if I could group large portions of the data into simpler categories. As such, I created a few new columns that would make my life significantly easier in the long-run.</p><p>For example, I created a column that would collapse date entries into their quarter and year of occurrence:</p><p>(I named my dataframe i_a for Iowa Alcohol)</p><pre>#Generating column for quarter-year<br>i_a[“Quarter_Year”] = “-”</pre><pre>i_a.ix[(i_a.Date &gt; ‘2014–12–31’) &amp; (i_a.Date &lt; ‘2015–04–01’), “Quarter_Year”] = “Q1_2015”<br>i_a.ix[(i_a.Date &gt; ‘2015–03–31’) &amp; (i_a.Date &lt; ‘2015–07–01’), “Quarter_Year”] = “Q2_2015”<br>i_a.ix[(i_a.Date &gt; ‘2015–06–30’) &amp; (i_a.Date &lt; ‘2015–10–01’), “Quarter_Year”] = “Q3_2015”<br>i_a.ix[(i_a.Date &gt; ‘2015–09–30’) &amp; (i_a.Date &lt; ‘2016–01–01’), “Quarter_Year”] = “Q4_2015”<br>i_a.ix[(i_a.Date &gt; ‘2015–12–31’) &amp; (i_a.Date &lt; ‘2016–04–01’), “Quarter_Year”] = “Q1_2016”</pre><p>I also created separate columns to count up total retail cost and total cost to the state for each transaction, to get a sense of how much profit the state was making off of its liquor sales.</p><p>When examining associations between certain categories and total sales in 2015, I could see certain items accounted for large portions of total sales compared to other, less popular options.</p><p>For example, there were over 30 different bottle sizes (volume by ml). However, the vast majority of bottles sold were either 750ml, 1000ml, or 1750ml. To make further analysis and model building easier, I created a column the would indicate whether the bottle size of liquor being purchased was either 750ml, 1000ml, 1750ml, or anything else.</p><pre>i_a[“Consolidated_Bottle_ml”] = “”</pre><pre>#After examining data, it made sense to consolidate bottle sizes into 750ml, 1000ml, 1750ml, and all others<br>i_a.ix[i_a.Bottle_Volume_ml == 750, “Consolidated_Bottle_ml”] = “750ml”<br>i_a.ix[i_a.Bottle_Volume_ml == 1000, “Consolidated_Bottle_ml”] = “1000ml”<br>i_a.ix[i_a.Bottle_Volume_ml == 1750, “Consolidated_Bottle_ml”] = “1750ml”<br>i_a.ix[i_a.Consolidated_Bottle_ml == “”, “Consolidated_Bottle_ml”] = “Other_ml”</pre><p>When plotting the relative total sales associated with each of these categories, it is plain to see how dominant these bottle sizes are among all others:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*IIzLCZXKUbMk-4JS7PGfzw.png" /></figure><p>Similarly, I noticed broad trends among liquor categories indicating that many purchases were either varieties of whiskey or vodka. To parse these out, I conducted a text search of category strings to indicate whether the liquor being purchased was some sort of whiskey, vodka, or anything else:</p><pre>#Creating “Is_” columns as booleans for each type in the list above<br>#After further analysis, only whiskey and vodka seem to warrant their own categories, with everything else in “Other”</pre><pre>i_a[“Is_WHISK”] = i_a[‘Alcohol_Category_Name’].str.contains(“WHISK”)<br>i_a[“Is_VODKA”] = i_a[‘Alcohol_Category_Name’].str.contains(“VODKA”)</pre><pre>#Creating new column to identify consolidated alcohol type</pre><pre>i_a[“Consolidated_Category”] = “”</pre><pre>#Assigning values as either Whiskey, Vodka, or Non-Whiskey-Vodka, depending on true values in &quot;Is_&quot; columns. If the category is neither whiskey, nor vodka, or is null, assigning it to this other category.</pre><pre>i_a.ix[i_a.Is_WHISK == True, “Consolidated_Category”] = “WHISKEY”<br>i_a.ix[i_a.Is_VODKA == True, “Consolidated_Category”] = “VODKA”<br><br>i_a.ix[((i_a.Is_WHISK == False) &amp; (i_a.Is_VODKA == False)) | (i_a.Alcohol_Category_Name.isnull()), “Consolidated_Category”] = “NON-WHISKEY-VODKA”</pre><p>Again, you’ll notice how prominant the two categories of whiskey and vodka are compared to all other hard alcohols:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*Odw2idlIWf6ohO8Ht_Gnng.png" /></figure><p>Now that I could see where the majority of sales dollars were grouped, I felt I had enough information to begin building a model that could</p><h3>Transforming the Data Set</h3><p>To build my model, I knew that I would need to predict an entire year’s sales using only a quarter’s worth of sales data.</p><p>To do this, I wanted to transform my data set to group the data by store number, and then build a model that would predict the total yearly sales of each store based off of a single quarter’s worth of data.</p><p>I knew that I would want to keep the number of predictive variables to a minimum, and so I wanted to choose those variables that seemed to account for the most variance in total sales amounts.</p><p>Specifically, I wanted to predict total yearly sales based of Q1 2015 sales dollars associated with four variables: 750ml bottles, 1000ml bottles, 1750ml bottles, and non-whiskey/non-vodka liquor purchases.</p><p>To accomplish this in pandas, I created a series of separate dataframes using groupby statements, and then concatenated these various dataframes together to create my final modeling set.</p><p>Where I grouped by multiple variables, using the “.unstack()” function helped to store each variable of interest as its own column, rather than nesting results hierarchically. I also created a separate dateframe of just 2015 data (i_a_2015) before beginning this process.</p><p>(There are a few extra variables that I didn’t include in my final model, that were initially added for curiousity’s sake.)</p><pre>#Use the data from 2015 to make a linear model using as many variables as you find useful to predict the yearly sales of all stores.</pre><pre>#We don’t want to use ALL of the data from the variables of interest — we only want about 1/4 of it (as we’ll only have a quarter’s worth of data from 2016 when generating predictions for all of 2016.)</pre><pre>#create dataframe grouped by store for 2015 that will be used for building regression model</pre><pre>#We DO still keep ALL of the sales data from 2015 as the target variable, as we want to predict the final 2016 totals<br>#for all stores</pre><pre>store_sales_2015 = pd.DataFrame(i_a_2015.groupby(“Store_Number”).Sale_Dollars.sum())</pre><pre>store_sales_2015.rename(columns={‘Sale_Dollars’:’Sale_Dollars_Total_2015&#39;}, inplace=True) </pre><pre>#Drawing out bottles_sold; keeping Q1 data and dropping Q2:Q4</pre><pre>add_Q1_bottle_sold = pd.DataFrame(i_a_2015.groupby([“Store_Number”, “Quarter_Year”]).Bottles_Sold.sum().unstack())</pre><pre>#alter Q1_2015 column name for easier interpretability</pre><pre>add_Q1_bottle_sold.rename(columns={‘Q1_2015’:’Bottles_Sold_Q1_2015&#39;}, inplace=True)</pre><pre>add_Q1_bottle_sold.drop([‘Q2_2015’, “Q3_2015”, “Q4_2015”], axis=1, inplace=True)</pre><pre>#Finding the “mode” of the stores’ counties, which will be the only item associated with each store</pre><pre>add_store_county = pd.DataFrame(i_a_2015.groupby(“Store_Number”).County.agg(lambda x:x.value_counts().index[0]))</pre><pre>#Creating df of only Q1_2015 to make extracting of bottle sizes and other multi-variable columns easier</pre><pre>i_a_Q1_2015 = i_a[i_a[“Quarter_Year”] == “Q1_2015”]</pre><pre>#Drawing out bottles_sizes sales volume for 750ml, 1000ml, 1750ml, and other_ml for Q1_2015;</pre><pre>add_Q1_bottle_sizes = pd.DataFrame(i_a_Q1_2015.groupby([“Store_Number”, “Consolidated_Bottle_ml”]).Sale_Dollars.sum().unstack())</pre><pre>add_Q1_bottle_sizes.rename(columns={‘1000ml’:’1000ml_Sales_Q1_2015&#39;, ‘1750ml’:’1750ml_Sales_Q1_2015&#39;,’750ml’:’750ml_Sales_Q1_2015&#39;,’Other_ml’:’Other_ml_Sales_Q1_2015&#39;}, inplace=True)</pre><pre>#Drawing out consolidated_category sales volume for whiskey, vodka, and others for Q1_2015;</pre><pre>add_Q1_consolidated_category = pd.DataFrame(i_a_Q1_2015.groupby([“Store_Number”, “Consolidated_Category”]).Sale_Dollars.sum().unstack())</pre><pre>add_Q1_consolidated_category.rename(columns={‘NON-WHISKEY-VODKA’:’NON-WHISKEY-VODKA_Sales_Q1_2015&#39;, ‘VODKA’:’VODKA_Sales_Q1_2015&#39;,’WHISKEY’:’WHISKEY_Sales_Q1_2015&#39;}, inplace=True) #saves changes</pre><pre>#drawing out total Q1 sales from initial i_a_2015 df, dropping Q2, Q3, and Q4</pre><pre>add_Q1_total_sales = pd.DataFrame(i_a_2015.groupby([“Store_Number”, “Quarter_Year”]).Sale_Dollars.sum().unstack())</pre><pre>add_Q1_total_sales.rename(columns={‘Q1_2015’:’Q1_2015_total_sales’}, inplace=True) #saves changes </pre><pre>add_Q1_total_sales.drop([‘Q2_2015’, “Q3_2015”, “Q4_2015”], axis=1, inplace=True)</pre><pre>#creating final dataframe for modeling by concatenating all other dfs</pre><pre>model_df = pd.concat([store_sales_2015, add_Q1_bottle_sold, add_store_county, add_Q1_bottle_sizes, <br> add_Q1_consolidated_category, add_Q1_total_sales], axis=1)</pre><pre>#there are some NaN values where stores didn’t buy certain bottle sizes or liquor types — convert to zeros</pre><pre>model_df.fillna(value=0, inplace=True)</pre><p>After all of this, we had our dataframe of total 2015 sales and select Q1 sales variables that could be used for prediction. Here’s a quick look:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8sLwkQNuXI0kgWn1ENZ7fw.png" /><figcaption>A snapshot of our dataframe used for modeling. The Index column on the far left is each store’s assigned number. Not all columns are pictured here.</figcaption></figure><p>Examining a correlation heat map of my variables of interest, I can see that there are moderately strong associations between these variables, which is not entirely surprising (especially given that non-whiskey/non-vodka liquors will inherently overlap with bottle sizes).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZkaDWcAkTCPrqTSiLqHdPg.png" /></figure><h3>Building a Predictive Model</h3><p>To train and test my model, I conducted a 70:30 train-test split of the 2015 model set. This means that I could fit my model using 70% of the data in my modeling dataframe, and then test the accuracy of my model against the remaining 30% of data that was not used in fitting the model.</p><p>In the code below:</p><p>X = modeling dataframe with store data from only those variables listed in heatmap above</p><p>y = dataframe containing only 2015 total sales by store</p><pre>from sklearn import linear_model<br>from sklearn.model_selection import train_test_split</pre><pre>#Conducting 70:30 train-test split for purposes of building and testing model. Set random state = 1 for reproducibility.</pre><pre>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)</pre><pre>#instantiate linear model for ease of coding below</pre><pre>lm = linear_model.LinearRegression() </pre><pre>#generate fit of model base on training set<br>#fit method expects two arguments of x’s and y’s</pre><pre>model = lm.fit(X_train, y_train) </pre><pre>#generate predicted values of y_test from X_test based off of training set</pre><pre>predictions = lm.predict(X_test)</pre><pre>#plotting predicted ys against y values in test set </pre><pre>plt.scatter(y_test, predictions) <br>plt.xlabel(“True 2015 Total Sales”)<br>plt.ylabel(“Predicted 2015 Total Sales”)</pre><p>The resulting model had an R² =.956, meaning that the model accounted for approximately 95.6% of the variance in store’s yearly 2015 sales, based off of their Q1 sales data from my chosen variables. Here’s a plot of the predicted values against the actual values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jXdF9QsrF9iCPLZcv6PwXA.png" /></figure><p>I examined the model further with cross-validation and regularization, but this blog post is long enough already, and they did not add much to the model that wasn’t already there.</p><h3>Generating Predictions for 2016</h3><p>The final step, was to reproduce a similar modeling dataframe (info by store) for the Q1 2016 I had available, and then generate predictions for each store’s total 2016 sales. I utilized very similar code to the huge code block above to generate the 2016 predictor set.</p><p>After generating the predictions, I appended the predictions (LM_Predictions) to each store’s line, like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PDa8sT0DCiErUfIPBcn1iQ.png" /></figure><h3><strong>Final Results</strong></h3><p>The linear model I built predicted total 2016 liquor sales of $292 million, which is a 3% increase compared to 2015 ($284 million).</p><p>Where the state’s profit in 2015 was 33% of total sales ($95 million), the state’s estimated profit for 2016 was estimated at $98 million. Pretty cool, stuff!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e45342081b83" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/predictive-modeling-with-iowa-state-liquor-sales-data-e45342081b83">Predictive Modeling with Iowa State Liquor Sales Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>