Analytics and AI in Travel North America Summit, a conference organised by eye for travel, took place for the first time in San Francisco this year. The summit promised to bring “the Silicon Valley of travel together with analytics leaders”, and was heavily focussed on sharing innovations in personalisation, prediction and reducing data fragmentation across the travel industry. The following post covers some of the highlights from the show:

Organising data science teams

Many of the talks touched on both how to scale and organise data science teams in your organisation. The universal opinion on this was that the data science team must not exist in a silo and should be integrated into cross functional teams. A specific use case and a sponsor must be available up front in order for the models to be successfully implemented. The result of not having this structure is that models are often either not used, or need heavy refactoring to fit the operational use.

Recommendation and personalisation approaches

There were some common themes around how to successfully use data to make recommendations and create personalised experiences. The need to develop fast through iterative experimentation was a common theme. Many of the talks suggested that personalisation should start with small proof of concept experiments starting with best customers first. An interesting point was around the cold start problem in machine learning — where you do not yet have labelled data. It was suggested that a solution to this problem is to start with basic assumptions, start making recommendations or tailored offers, and learn from the data generated to create profiles.

AI impact on travel

There was some discussion around how AI is enhancing both customer and employee experiences in the travel sector both through automation and personalisation. Tata Crocombe, Managing Director of Aitutaki Lagoon Private Island Resort, gave a talk about how he sees AI being used in hotel operational automation. Examples included automating some of the more routine duties in night auditing and applying AI to understand the requirements of stakeholders. Ultimately the main takeaway from this was that we can automate the “boring work” with AI so that “humans can do the fun stuff”.

Overall I found this conference extremely useful mainly from an execution strategy perspective rather than technical implementation of AI and machine learning. It was also interesting to hear about how other travel companies are using these technologies. The conference was reasonably small, I think there were around 150 people there, and this size setting meant that I got to meet and talk to some really great people too.

Highlights from an Analytics and AI in Travel Conference was originally published in vickdata on Medium, where people are continuing the conversation by highlighting and responding to this story.

The role of a data scientist requires a broad set of skills. Ideally you need a sound knowledge of maths and statistics. Good programming skills in at least one language. A solid understanding of machine and deep learning, and ideally some domain knowledge, and/or business acumen.

In the following article I am going to describe a number of different routes to move into a data science career. This post will focus on an applied data science role within a business or other large scale enterprise, as opposed to something like a research data science role.

Data analyst to data scientist

I am going to be biased and list out my own personal route first. I transitioned over the period of about three years from a data analyst role to data scientist. Having spent time in a data analyst role you will already have a really good foundation of understanding around data. How to extract it, clean it, some of the gotchas that you may experience when pulling data together for reporting such as missing or erroneous values. You will also gather an understanding of how to extract insights from data and develop a high level of data curiosity.

To transition from data analyst to data scientist you will need to develop good programming skills in at least one language. My personal opinion here is that if you are just starting out in programming then python would be the best one to learn. In a poll by KDNuggets Analytics in 2018, 65.6% of data scientists surveyed selected python as a tool that they used, up 11% on the same survey carried out in 2017. So python is currently the most popular language for data scientists and this popularity is continuing to grow. Python is highly accessible and fairly easy to learn for beginners in programming. It also has a very active community around it in the data science space, examples include NumFocus and Pydata.

Chances are that, unless you have already have a highly numerate degree, you will also need to brush up on your knowledge of maths and statistics. You will need to have an enough of an understanding to perform complex data analyses, and have a reasonable understanding of what is going on behind machine learning algorithms for example.

Graduate degree in maths/engineering/computer science

Chances are that if you have an advanced degree in one of these subjects you will already have a lot of the required skills for data science. You might need to learn programming if you don’t already have this, or more maths and statistics if your degree was computer science based for example.

The learning curve here is likely to be adopting typical business practices such as writing reproducible code, version control and corporate working methodologies e.g. Agile. Developing domain knowledge of the industry you are working within and interpreting, sometimes vague, business questions into useful projects.

Software engineering

I personally think this a great role to transition from. From my experience, data science teams often lack the knowledge of software development best practices, and being able to write the most efficient, production ready code. If you have already spent time working as a software engineer, then the chances are you will have a solid foundation in the sort of programming that data science teams often lack.

For a transition here you may need to learn one of the popular data science programming languages — as mentioned previously probably python, as well as SQL. You would need to develop a solid understanding of data, extracting, cleansing, processing and analysing. A foundational knowledge of maths and statistics, especially as applied to data analysis and machine/deep learning, is also needed.

Other routes

I am hugely passionate about diversity in tech and I believe very much that this also includes diversity of backgrounds. Not coming from a “traditional” route can have its advantages, your experience or outlook on life may give a fresh perspective on processes or problems within a team. I have listed three of the most common routes for entering a career in data science in this article, however there are of course many more. If you don’t have a relevant degree, or experience in data, or programming, I don’t believe that this is a barrier, it just means that it may take you a little longer to get there.

Career Paths to Data Science was originally published in vickdata on Medium, where people are continuing the conversation by highlighting and responding to this story.

I have been working quite a lot recently with encoding categorical variables in Python for machine learning. I wanted to write a post covering some of the things that I have learnt along the way. Kaggle have a data set from the Austin Animal Center in which almost all variables are categorical. The following post covers some examples of how we might process these into features that will be useful in a predictive model.

The data set can be downloaded here, I am using the train.csv file in my workflow. The data comprises a number of characteristics about animals that were admitted to the Austin Animal Center between 1st October 2013 and March 2016. You are asked to predict the outcome for each animal of which there are 5 possibilities;

• Adoption
• Transfer
• Return to owner
• Euthanasia
• Died

Basic data exploration

Firstly I have read in the train.csv file and produced a simple bar plot to visualise the distribution for the outcome types.

import pandas as pd
import matplotlib.pyplot as plt
% matplotlib inline

train = pd.read_csv('train.csv')
train['OutcomeType'].value_counts().plot.bar()

We can see that there are differences in class sizes, in particular, the classes died and euthanasia are, thankfully, much smaller than the others. This could prove difficult to predict these outcomes without handling the imbalances. However, I am not going to cover that here as I would like to focus more on processing the variables.

train.dtypes

You can see from the above that all columns in the data are non-numeric. Each one will need some pre-processing performed before I can use them in a classifier.

In order to get an idea of how I might process them I run the following to get a count of unique values in each column. We can see from the below that some columns, such as “Breed” have a large number of unique values. These will need some more attention in comparison to others.

columns = train.columns
for column in columns:
    print(column)
    print(train[column].nunique())

Processing categorical data

Generally there are three main ways to convert categorical data into numeric. These are as follows:

1. One hot encoding

One hot encoding takes categorical data and makes new columns for each unique value. A 0 or 1 denotes wether or not the row has that value. Lets look at an example using our animal shelter data. Take the column “AnimalTtype”, in this column we have two unique values, cat and dog. When we apply a one hot encoder transformation, two new columns are produced, each containing 1’s and 0’s. This technique works well for columns such as this where the data is of low cardinality. But if the columns are high cardinality or the order of the categories has a meaning, that can be interpreted in a numerical fashion, then more complex approaches should be used.

2. Label encoding

Label encoding simply maps the unique values to numbers. For example, if we were to apply this to the “SexuponOutcome” column, then each unique value would become a number that could be mapped back to the string representation. This approach however has significant drawbacks for handling categorical variables. The problem is that with categorical data there isn’t actually a relationship between the rows, but if we simply convert it to numbers, the classifier may infer relationships that are not present. For example, if “Neutered Male” was given 1 and “Spayed Female’” 2, the classifier may infer that ‘Spayed Female’ is more important because it has a larger number.

3. Converting to ordinal or continuous variables

You may find that some categorical columns actually have a numerical meaning. In this data set that is true for the
AgeuponOutcome feature, where a higher value should be treated as being greater than others. This isn’t really categorical data, and therefore should be transformed into the numerical representation of the age. I will discuss the method I used for this later on in the post.

Missing data

Before processing the variables I am going to do a little cleaning of the data. I need to check for any missing values and handle them appropriately. Firstly I run the following code to check for missing values.

train.apply(lambda x: sum(x.isnull()/len(train)))

This gives the following output.

You can see that we have a high number of missing values for two of the features, “Name” and “OutcomeSubtype”. But only a small number of missing values for “SexuponOutcome” and “AgeuponOutcome”. I am therefore going to have to use a number of different methods for handling this missing data.

Looking at the data I can see that the “OutcomeSubtype” is a further categorisation of the label we are trying to predict. Therefore in a real world use case for this predictive model it is unlikely that we would have this feature in the data. I am therefore going to drop this from my features.

train = train.drop('OutcomeSubtype', axis=1)

Missing data in the “Name” field could well turn out to have some impact on the outcome of an animal. Particularly in an outcome such as “return to owner” so rather than seek to fill the missing data with a value I will instead turn it into a new feature “has_name”. This will have a 0 where a name is missing and a 1 where it is present. To do this I fill all missing data with a 0 in the “Name” column. I then use this to create a new column called “has_name”. Finally I drop the “Name” column as this is unlikely to be useful in the final model.

train['Name'] = train[['Name']].fillna(value=0)
train['has_name'] = (train['Name'] != 0).astype('int64')
train = train.drop('Name', axis=1)

Finally I will handle the missing values in “SexuponOutcome” and “AgeuponOutcome”. As these are both categorical and have only a small amount of missing data the simplest method will be to fill with the most commonly occurring value. The following code does this.

train = train.apply(lambda x:x.fillna(x.value_counts().index[0]))
train.apply(lambda x: sum(x.isnull()/len(train)))

We can see from the output that we now have no missing values and we have created a new feature.

Finally I am going to drop the AnimalID column as that will not be useful in the model.

train = train.drop('AnimalID', axis=1)

High cardinality

As previously described the most sensible method to convert categorical variables into numerical data is with one hot encoding. However, there is a problem when you have a feature that has high cardinality, or in other words has a large number of unique values. We have two examples of this in our data set — “Breed” and “Color”. If we were to simply use one hot encoding for these features we would end up creating 1,380 new features from “Breed” and 366 from “Color”. This would be unlikely to create a good performing model.

There are a number of ways to handle this situation and I will talk through them in the next section.

1. Most popular values

One way to handle this is to encode each of the most commonly occurring values, and place the remaining values into a label called “other”. Lets take “Color” as an example — we can run the code below which takes any values occurring fewer than 300 times and places them into the “other” label. We use this to create a new feature called “top_colors”.

color_counts = train['Color'].value_counts()
color_others = set(color_counts[color_counts < 300].index)

train['top_colors'] = train['Color'].replace(list(color_others), 'Others')
print(train['top_colors'].nunique())

This reduces the number of unique values in the column from 366 to 28. This is a much more manageable number for one hot encoding.

2. New features

Another method is to create a meaningful categorisation of the values. I will give an example of this here, with the “Breed” column. We can see from the data that some of the values contain the word “Mix” whilst others don’t. We can assume with some limited knowledge of animals that this suggests that mix means that the breed is a cross-breed. This could prove to be a useful feature so we can use the code below to create this.

import re
train['breed_type'] = train.Breed.str.extract('({})'.format('|'.join(['Mix'])), 
                        flags=re.IGNORECASE, expand=False).str.lower().fillna('pure')

I also did something similar with the “Color”. Creating a new feature called “multi_colors” containing a categorisation as to wether the animal has a single colour or not.

train['multi_colors'] = train['Color'].apply(lambda x : 1 if '/' in x else 0)

3. Numerical representation

In this data set the “AgeuponOutcome” feature is not really categorical data. The order of the values has a meaning. Therefore the best way to handle this is to convert it into its numerical representation. I am going to convert this feature into the age in days. I found this fantastic function in this Kaggle kernel which does this perfectly. This code creates a new feature and drops the original column.

def age_converter(row):
    age_string = row['AgeuponOutcome']
    [age,unit] = age_string.split(" ")
    unit = unit.lower()
    if("day" in unit):
        if age=='0': return 1
        return int(age)
    if("week" in unit):
        if(age)=='0': return 7
        return int(age)*7
    elif("month" in unit):
        if(age)=='0': return 30
        return int(age) * 4*7
    elif("year" in unit):
        if(age)=='0': return 365
        return int(age) * 4*12*7

train['age_numeric'] = train.apply(age_converter, axis=1)
train = train.drop('AgeuponOutcome', axis=1)

The remaining columns for the first iteration of this model can be directly converted using one hot encoding. I am using pandas get_dummies to convert all categorical variables in the code below.

train = train.drop(['Breed','Color', 'DateTime'], axis=1)

numeric_features = train.select_dtypes(include=['int64', 'float64']).columns
categorical_features = train.select_dtypes(include=['object']).drop(['OutcomeType'], axis=1).columns

dummy_columns = pd.get_dummies(train[categorical_features])

final_train = pd.concat([dummy_columns, train],axis=1)

final_train = final_train.drop(['AnimalType', 'breed_type', 'SexuponOutcome', 'top_colors'], axis=1)

We now have fully numerical data now lets see how this performs in a classifier.

Training a classifier

In following code I am specifying the features X and target y, and using scikit-learn train_test_split to create the training and testing data.

X = final_train.drop('OutcomeType', axis=1)
y = final_train['OutcomeType']

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=1)

I am then using a simple RandomForestClassifier to find out roughly how the model will perform.

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, log_loss

clf = RandomForestClassifier(n_estimators=100, max_depth=2,
                            random_state=0)
rf_model = clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
print(accuracy_score(y_test,y_pred))

y_prob = rf_model.predict_proba(X_test)
print(log_loss(y_test, y_prob))

This model does not perform particularly well, so it looks like there is more work to be done on feature engineering, model selection or optimisation. I am not going to cover this in depth here but as one final thing lets take a peek at the feature importances for this model. This may inform any further feature transformations or engineering work.

import numpy as np
features=X.columns
importances = rf_model.feature_importances_
indices = np.argsort(importances)

plt.figure(figsize=(10,20))
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='b', align='center')
plt.yticks(range(len(indices)), features[indices])
plt.xlabel('Relative Importance')
plt.show

The output show that features relating to the presence or absence of a name, age, sex and animal type are of particular importance.

There is likely to be a lot more work we can do here to improve the performance of this model. For example there are many more possibilities for transforming the categorical variables than I have listed here. As with a lot of data science work much of this is down to trial and error. I hope that in this article I have given a broad introduction into the possibilities of how to handle categorical variables in a machine learning project.

Predicting Animal Shelter Outcomes was originally published in vickdata on Medium, where people are continuing the conversation by highlighting and responding to this story.

Key concepts in maths and statistics for data science, and where to learn them.

Maths and statistics form the basis of most data science techniques. As a data scientist you will need a good foundational knowledge of these core concepts in order to understand how to perform exploratory data analysis, forecast future events, or apply machine learning and deep learning well.

This is part 3 of a series of posts detailing the roadmap I have been using to learn data science. Part 1 covered programming skills, and part 2 focussed on learning how to perform data analysis. In the following post I am going to list the core mathematical and statistical concepts that I have on my roadmap, as well as some excellent resources I have found to learn them.

Data science is a broad and complex subject so the range and breadth of knowledge in maths and statistics is wide. I think however there are core areas that should be learned to create the foundation to be able to start to understand the more nuanced mechanics behind the various areas of data science. This roadmap assumes no more than high school level maths knowledge and is designed to give a basic foundation of the key topics.

Statistics

As a data scientist you might use statistics to summarise and identify patterns in data, design robust experiments or measure the performance of machine/deep learning models.

Here is a list of the key concepts you need to know:

Descriptive/Summary statistics

• How to summarise a sample of data
• Different types of distributions
• Skewness, kurtosis, central tendency (e.g. mean, median, mode)
• Measures of dependence, and relationships between variables such as correlation and covariance

Experiment design

• Hypothesis testing
• Sampling
• Significance tests
• Randomness
• Probability
• Confidence intervals and two-sample inference

Machine learning

• Inference about slope
• Linear and non linear regression
• Classification

Resources: I am a big fan of the statistics course on Khan academy for learning the basics. The SciPy lecture notes are another great resource to learn these concepts in Python. I also highly recommend reading the book Think Stats — available for free online.

Calculus

Calculus is defined by Wikipedia as “the mathematical study of continuous change.” In other words calculus can find patterns between functions, for example, in the case of derivatives it can help you to understand how a function changes over time.

Many machine learning algorithms utilise calculus to optimise the performance of models. If you have studied even a little machine learning you will probably have heard of Gradient descent. This functions by iteratively adjusting the parameter values of a model to find the optimum values to minimise the cost function. Gradient descent is a good example of how calculus is used in machine learning.

What you need to know:

Derivatives

• Geometric definition
• Calculating the derivative of a function
• Nonlinear functions

Chain rule

• Composite functions
• Composite function derivatives
• Multiple functions

Gradients

• Partial derivatives
• Directional derivatives
• Integrals

Resources: One of the best resources I have come across to learn these principles is this machine learning cheatsheet which also covers linear algebra, regression, and the maths behind neural networks. I also really love this blog post that provides a gentle introduction to calculus with practical examples.

Linear Algebra

Many popular machine learning methods, including XGBOOST, use matrices to store inputs and process data. Matrices alongside vector spaces and linear equations form the mathematical branch known as Linear Algebra. In order to understand how many machine learning methods work it is essential to get a good understanding of this field.

What you need to learn:

Vectors and spaces

• Vectors
• Linear combinations
• Linear dependence and independence
• Vector dot and cross products

Matrix transformations

• Functions and linear transformations
• Matrix multiplication
• Inverse functions
• Transpose of a matrix

Resources: This blog post by Ritchie Ng covers matrices and vectors really well. If you want a more in depth overview of the field this is a good free book that gives an extensive coverage of linear algebra.

In later posts in this series I will be providing a roadmap for other aspects of learning data science including data engineering, and machine learning.

Maths and Statistics, A Complete Roadmap for Learning Data Science — Part 3 was originally published in vickdata on Medium, where people are continuing the conversation by highlighting and responding to this story.

Get up and running with some code to perform text classification in Python.

I recently embarked on a work project to perform sentiment classification on some customer feedback. This was my first time working with text data for machine learning and I found it quite difficult to get started with such a complex area. I wanted to write a guide for anyone else new to natural language processing (NLP) in python to help others get a quick overview of the end to end process without having to delve into all the complexities right at the beginning.

This post is for anyone that would like to get up and running with some code to perform an NLP task in Python without going into all the details. My preferred learning style is to get something up and running first, and then delve into the nuts and bolts of how it all works. I hope this helps others with a similar learning style.

The Data

For this post I am going to be using a data set taken from the Analytics Vidhya website, this can be downloaded here. This is a nice simple data set to practice on and has the added advantage of being part of an ongoing competition with a leaderboard. So you can quickly get an idea of how well your chosen workflow is performing.

The dataset consists of a test and train set. The training set comprises a list of 31,962 tweets, a corresponding id and label 0 or 1 for each tweet. The particular sentiment you are asked to identify in this problem is wether or not the tweet is racist or sexist (in which case it will be labelled as 1).

So to start with I have imported the data sets, and am returning a little information about them.

import pandas as pd

train = pd.read_csv('train.csv')
print("Training Set:"% train.columns, train.shape, len(train))
test = pd.read_csv('test_tweets.csv')
print("Test Set:"% test.columns, test.shape, len(test))

Text Cleaning

Most text data will likely need some processing in order for the chosen machine learning algorithm to perform well. In this case each text document is a tweet and therefore will contain lots of characters that will not be meaningful to any machine learning algorithm. You can see below from just viewing the first few rows of the data that the tweets contain characters such as #, @ and punctuation marks.

In order to remove these I am using the Python re library, this provides regular expression matching operations. The following function successfully cleans up most of these characters. Additionally this function makes everything lower case.

This is generally a good idea as many text classification tools rely on counting the occurrences of words. If both upper and lower case versions of the same word are found in the text then the algorithm will count them as different words even though the meaning is the same. Of course this does mean that where the capitalised versions of a word exists, that does have a different meaning. For example the company Apple vs the fruit apple. This could result in poorer performance for some data sets. This is one area of NLP where you may try different methods to see how they affect the overall performance of the model.

import re

def  clean_text(df, text_field):
    df[text_field] = df[text_field].str.lower()
    df[text_field] = df[text_field].apply(lambda elem: re.sub(r"(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)|^rt|http.+?", "", elem))  
    return df

test_clean = clean_text(test, "tweet")
train_clean = clean_text(train, "tweet")

The output after cleaning looks like this. Not perfect but we can see how it performs later in our model.

Handling imbalanced classes

If we count the number of tweets for each label we can see that there are a significantly larger number of tweets labelled as 0. In fact only 7% are classified as sexist/racist. This is problematic as if we provide an algorithm with this data there is a high chance that it will default to predicting all labels as 0.

There are a number of methods you can use to handle this. One approach is to use either upsampling or downsampling. In the case of upsampling we use a function that repeatedly takes samples, with replacement, from the minority class until the class is the same size as the majority. With replacement means that the same sample can be used multiple times.

from sklearn.utils import resample

train_majority = train_clean[train_clean.label==0]
train_minority = train_clean[train_clean.label==1]

train_minority_upsampled = resample(train_minority, 
                                 replace=True,    
                                 n_samples=len(train_majority),   
                                 random_state=123)
train_upsampled = pd.concat([train_minority_upsampled, train_majority])
train_upsampled['label'].value_counts()

When downsampling we use the same function to take samples from the majority class, without replacement, until it is the same size of the minority class. Without replacement means that each sample is only sampled once.

train_majority = train_clean[train_clean.label==0]
train_minority = train_clean[train_clean.label==1]
 
train_majority_downsampled = resample(train_majority, 
                                 replace=True,  
                                 n_samples=len(train_minority),   
                                 random_state=123)
train_downsampled = pd.concat([train_majority_downsampled, train_minority])
train_downsampled['label'].value_counts()

I have included my code for this above. I tried both upsampling and downsampling and achieved a better result with upsampling so I have used that in my workflow.

CountVectoriser

Machines are not capable of reading text in the same way as humans do. In order for a machine learning algorithm to determine patterns in text it must first be converted into a numeric structure. One of the most common techniques for this is called Bag of Words, or BoW.

A BoW model splits the words in a piece of text into tokens disregarding grammar and word order. The model also counts the frequency in which a word occurs in the text, and assigns a weight proportional to this frequency. The output is a matrix of term frequencies where each row represents the text and each column a word in the vocabulary.

Sci-kit learn has a number of built in functions to perform this type of modelling. But for this walkthrough I am going to be using one of the most simple functions which is CountVectoriser. This function works quite well with the default settings so I will use those for the first iteration of my model.

TfidfTransformer

As mentioned earlier in the post the BoW model which we are using to process the text has three steps. CountVectoriser accomplishes the first two, splitting the words into tokens and counting the frequency. We can use another scikit-learn function called TfidfTransformer to apply the frequency weighting.

In any text document there will be a number of words that appear very frequently such as I, we, and get. If we were to build a model without weighting these words they would overshadow less frequent words during training. By weighting these high frequency words we can assign, for example, more importance to less frequent but perhaps more useful words.

Training

For simplicity and reproducibility I am going to use a sci-kit learn pipeline with a SGDClassifier. The below code creates a pipeline object that when used will apply each step to the data.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import SGDClassifier

pipeline_sgd = Pipeline([
    ('vect', CountVectorizer()),
    ('tfidf',  TfidfTransformer()),
    ('nb', SGDClassifier()),
])

Before training the model I am splitting the training data into a training and test set.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(train_upsampled['tweet'],                                    train_upsampled['label'],random_state = 0)

The code below fits the model to the training data and computes the F1 score.

model = pipeline_sgd.fit(X_train, y_train)

y_predict = model.predict(X_test)

from sklearn.metrics import f1_score

f1_score(y_test, y_predict)

You will notice above that I am using the upsampled data set. This is not ideal as the model performs vastly better on the training data than on the new test data. For example on the training data I achieved (after some further optimisation) an F1 score of 0.97 but this translated to a score of 0.75 once I made my submission. Downsampling because of the low volume of data for the minority class performed even more poorly. There are other more complex techniques for handling imbalanced classes such as weight balancing but that subject in itself warrants a whole other post.

Natural language processing is a vastly complex subject and there is so much more that I could cover in this article. My aim here is to give enough information, and code, to get up and running with your first text classification model. The final score is not too bad for a first attempt but there is much scope for improvement. There are many other models besides BoW for processing the text data for example that may work better. There is more work that could be done with the re library to further clean the data, and perhaps the removal of stop words may help to improve the model further.

Detecting Hate Speech in Tweets: Natural Language Processing in Python for Beginners was originally published in vickdata on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are so many great resources out there to learn data science and analysis for free. Over the last year I have read quite a few data science books and I wanted to share some of the best here. If you are studying, or practicing data science, and haven’t read these books I really think they are worth adding to your reading list for 2019. Below is a list of the top 10 I have found most useful to me over the last few years that are currently available online.

Automate the boring stuff

I really love this book it is a simple introduction to getting started with python from a practical point of view. Although not a specifically data science related book it includes most of the basic concepts around using python for data science. Including flow control, functions, web scraping, working with csv and json files, and running programs. It is very much aimed at absolute beginners so a great book for getting started with python. As well as step by step instructions for each technique, at the end of each chapter there are also practice questions and problems.

Data science at the command line

I started using python for data analysis purely in Jupyter Notebooks. However, over time I found that using the command line enabled me to be much more efficient in my work. For example I can very quickly obtain data, run programs and search through files all by typing commands and pressing enter in the terminal window. This book is a highly accessible and comprehensive guide to data science at the command line. In each chapter it covers, alongside working examples, how to obtain, clean, explore, model and interpret data via the command line.

Think stats

Think Stats 2e

This is a really practical overview of statistics for data science. The book uses a data set from the National Institute of Health throughout to explain the core concepts in probability and statistics necessary for data science and analysis. This is another highly practical book, and includes lots of example python code, and simple programs to explain the concepts. This is much more lightweight than a lot of the more theoretical textbooks you may find on this subject, and I found this really suited my learning style.

Python data science handbook

This is a really comprehensive guide to python for data science. This builds from beginner to advanced concepts. There is a chapter on iPython which really made such a difference to my efficiency as a data science practitioner. This book also covers Numpy, data manipulation with Pandas, visualisation methods, and Machine Learning. The Machine Learning chapter in particular is really good, and covers both the practical implementation of the various libraries, and the nuts and bolts of how they work.

R for data science

R for Data Science

I mainly work in python but I still find it is really useful to have at least a working knowledge of R. I have often found that if a good library for a particular method is not available in python, R usually has one. This book is a really comprehensive guide to doing data science with R, and covers everything from data visualisation and transformation, to the R workflow, to data modelling.

Probabilistic Programming and Bayesian methods for hackers

GitHub - CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers: aka "Bayesian Methods for Hackers": An introduction to Bayesian methods + probabilistic programming with a computation/understanding-first, mathematics-second point of view. All in pure Python ;)

In the authors own words this book is an attempt to “bridge the gap between Bayesian mathematics and probabilistic programming”, and I believe it does this very well. As with Think Stats it moves away from heavily theoretical textbooks and offers practical use cases for Bayesian inference, and the approach is a computational understanding first, and a mathematical understanding second. It is another python based book with lots of practical examples, and uses predominately the PyMC libraries.

Machine learning yearning

Machine Learning Yearning

This book has been released in draft by Andrew Ng this year. It is designed to teach data scientists how to structure Machine Learning projects, and set direction for a data science team. It is a good overview of when and how to use Machine Learning, and how to handle the complexities involved in implementing AI in the real world.

Ethics and data science

Ethics and Data Science

There has been a lot in the news this year relating to bias in machine learning applications, and data protection and privacy concerns. I read this book as I wanted to ensure that I had the required knowledge to practice good data science. This book covers how to put ethical principles into data science projects. It includes a really good checklist to go through when designing a project as well as lots of suggestions for building ethics into a general data culture. Another resource released this year along very similar lines was the deon command line tool from drivendata.org. This tool allows you to build an ethics checklist into data science projects. This is definitely something I will be incorporating into my work in the new year.

Deep learning

This is an excellent book now available to read for free online. It covers applied maths for Machine Learning, and has a large emphasis on deep learning in particular. It covers the mathematics behind key concepts in deep learning such as convolutional networks, regularisation and recurrent and recursive nets. It is very much a theory based book but gives a deep level of understanding into the subject. It does also include chapters on the practical implementation of these techniques.

Rules for machine learning

This is really an ebook/paper and only about 24 pages long. However, I have to include it here as it is such a great resource and I found it by chance on twitter this year. This covers some best practices from Google in how to implement a machine learning project. It emphasises the importance of data engineering to create great features and a solid data pipeline over machine learning expertise.

These books have been really useful to me over the last couple of years, I am always amazed at the quantity and quality of free resources available online. I am sure that I will continue to refer back to these into 2019 and beyond, and hopefully find some more brilliant resources to share. Happy New Year!

10 Free Data Science Books You Must Read in 2019 was originally published in vickdata on Medium, where people are continuing the conversation by highlighting and responding to this story.

Machine learning competitions can be a really useful way to learn how to validate and improve models. I found that data science practice problems on Analytics Vidhya were a great starting point. These allow you access to simple datasets on which to practice your machine learning skills and benchmark yourself against others. I think they offer a great introduction to approaching these problems before perhaps moving onto something a bit more challenging such as Kaggle competitions.

I initially found it challenging to know how to build a model and make a submission. I decided to write this post to help anyone else who may be starting out and experiencing the same problem.

The competition I am going to cover in this post is the Loan Prediction Practice Problem. In this competition you are asked to create a model to predict whether or not a person would be approved for a home loan, given a number of fields from an online search form. In the following post, I am going to walk through step by step how to get started with building the first model for this task using Python in JupyterLab. I am not going to go into great detail about each step but wanted to show the end-to-end process of how to approach a classification problem.

Getting started

For this problem, you are given two CSV files, train.csv and test.csv (they actually have longer file names when downloaded but the first step I took was to rename them to make handling them much easier). The data sets can be downloaded here. We are going to use the training file to train and test the model, and we then use the model to predict the unknown values in the text file. The first steps are to read in the files and perform some preliminary analysis to determine what pre-processing we will need before attempting to train a model.

In the below code I am reading in the renamed CSV files using the pandas read_csv function. I have also dropped the Loan_ID column from the train set as I won’t need this for training the model.

import pandas as pd

train = pd.read_csv('train.csv')
test = pd.read_csv('test.csv')
train = train.drop('Loan_ID', axis=1)

Once the files have been imported I run the following function to get some more information about the dataset I am going to be working with.

def describe_data(df):
    print("Data Types:")
    print(df.dtypes)
    print("Rows and Columns:")
    print(df.shape)
    print("Column Names:")
    print(df.columns)
    print("Null Values:")
    print(df.apply(lambda x: sum(x.isnull()) / len(df)))

describe_data(train)

This gives me the following information about the data set;

• That we have a number of columns that are non numeric which will need some pre-processing before they can be used to build a model.
• That we have a relatively small data set consisting of 614 rows.
• That there are no null values present in the data.

Pre-processing

Most machine learning models are unable to handle non-numeric columns, and missing values in data. It is therefore necessary to perform a number of pre-processing steps.

To make the process simpler I am going to be using a scikit-learn pipeline. These pipelines allow you to apply a series of transformative steps, and fit and predict on an estimator with just one line of code. These are highly reusable, cut down on the need to apply pre-processing steps multiple times on train and test sets, and make workflows reproducible and easy to follow.

Before creating the pipeline I need to specify which columns are numerical features and which are categorical features. I am doing this using the pandas select_dtypes function.

numeric_features = train.select_dtypes(include=['int64', 'float64']).columns
categorical_features = train.select_dtypes(include=['object']).drop(['Loan_Status'], axis=1).columns

The next step is to create two transformer pipelines, one for the numerical features and one for the categorical features. In each pipeline is a SimpleImputer to handle null values. We don’t actually need this for our dataset as we don’t have any null values but it is good practice to always include one as in a real world example you do not want your prediction to fail because of a null value. For the numeric transformer I am simply imputing the nulls with the median value for the column, and for the categorical tranformer I am imputing them with the value “missing”.

In the numeric transformer I have also included a scaler function. This ensures that all the values in each feature are on the same scale. For the categorical transformer I am using OneHotEncoder to transform each unique value in the categorical columns into a new column containing a 0 or 1 dependant on wether or not the value is present.

from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.pipeline import Pipeline

numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())])

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))])

I then use the ColumnTransfomer to concatenate both the numeric and categorical transformers into an object called preprocessor.

from sklearn.compose import ColumnTransformer

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)])

Training the model

Once you have your transformation pipeline you can combine this with a classifier to create the full machine learning pipeline. This can be used to both train (fit) and predict on new data.

Before doing this we need to split the train data into a training and test set so that we can get an idea as to how well our model is predicting on new data. To do this I first specify the features (X) and the target (y). I then use the train_test_split function and have chosen to use 20% of the data for the test set.

from sklearn.model_selection import train_test_split
X = train.drop('Loan_Status', axis=1)
y = train['Loan_Status']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

As this is a simple walkthrough and I am not showing the full process of obtaining the best performing model I am going to use a simple LogisticRegression model. The below code creates a logistic regression model that performs the defined transformations before fitting or predicting.

from sklearn.linear_model import LogisticRegression
lr = Pipeline(steps=[('preprocessor', preprocessor),
                      ('classifier', LogisticRegression(solver='lbfgs'))])

I then fit the model on the training set, and obtain an accuracy score using the built in score method. This returns a score of 0.82. A glance at the leaderboard tells us that this is not a bad initial score but there is still some work to do as the top competitors are currently scoring in excess of 0.90.

lr.fit(X_train, y_train)
print("model score: %.3f" % lr.score(X_test, y_test))

To finish I will show you how to make your first submission. We now need to use the test.csv file that we read in earlier. Firstly we are going to create a new data frame with the Loan_ID column dropped.

test_no_id = test.drop('Loan_ID', axis=1)

We then use our model to predict on this new data.

test_predictions = lr.predict(test_no_id)

Finally we need to build the submission file, which we know from looking at the sample submissions available on the website, needs to consist of a csv file containing the Loan_ID’s and corresponding predictions. In the below code I have obtained the Loan_ID’s from the original test data frame. I am then building a new data frame by combining with the test_predictions.

Loan_ID = test['Loan_ID']
submission_df_1 = pd.DataFrame({
                  "Loan_ID": Loan_ID, 
                  "Loan_Status": test_predictions})

Finally exporting this as a csv file using the pandas to_csv function. You can now upload this as a submission.

submission_df_1.to_csv('submission_1.csv', index=False)

Once you have made a submission you will get a score on the website. You can see here that the accuracy score for the submission is much lower than the score on the test set from the training data. This suggests that I have an overfit model which means that the model has learned patterns in the training data that are specific to that dataset and not necessarily applicable on the wider data.

In this post I have described how to prepare data, train a simple model and make your first submission to a machine learning competition. In another blog post I will talk through the steps you can take to improve on this score and move further up the leaderboard.

How to Enter Your First Machine Learning Competition was originally published in vickdata on Medium, where people are continuing the conversation by highlighting and responding to this story.

Over the last few months I have been working with the Google Cloud Machine Learning (ML) Engine to deploy machine learning models into production. Google’s vision is to make machine learning and deep learning accessible to all from engineers to analysts to data scientists. Over the last few years they have released a variety of cloud based tools designed to lower the barriers of entry to using Artificial Intelligence (AI).

My experience so far has been positive ML Engine does seem to abstract a lot of the complexity of deploying models away for example. In the following post I want to review the various tools that Google have or are releasing in an effort to democratise AI.

BigQuery ML

The primary purpose of BigQuery ML is to make machine learning accessible to data analysts by enabling the user to develop machine learning models with standard sql. Additionally it allows for fast prototyping, by allowing users to run a model directly where the data already sits, the viability of a machine learning project can be assessed with limited time investment. This is currently in a beta release, and only supports three types of models — linear regression, binary logistic regression and multiclass logistic regression.

It can be used through a number of tools that data analysts will already have extensive knowledge of. These include spreadsheets with a connection to BigQuery, the BigQuery web UI, statistical software and the Google visualisation tools Cloud Datalab and Data Studio.

I have recently tried using this both in Datalab and the BigQuery web UI and found it very straight forward to get it working. In essence it is as simple as writing a sql query similar to the one shown below. You will need to create the data set in your project first by following the instructions in the documentation. Running this extracts the data and trains the model selected in OPTIONS using this data. You can view training statistics and evaluate the performance of the model directly in the web UI which is impressive. You can access predictions via a different sql query.

My hope is that in the future they will build in support for other open source libraries such as scikit-learn. Currently the limitation of only having regression based models available limits the ability to develop the most accurate models on real world data.

ML Engine

Aimed at both developers and data scientists this solution offers a managed service to train and deploy machine learning models. A number of machine learning libraries are available including scikit-learn, XGBoost, Keras and Tensorflow (more on this later in the post).

I have recently been trialling using ML Engine to train a classification model with scikit-learn. I have been able to serve the model in a production environment without having to involve engineers so on the surface this appears to be a way to democratise deploying models for data scientists.

The process we have followed is relatively straightforward. I created a packaged version of the scikit-learn code for training the model and predicting on new data. This was placed on cloud storage. A cli command needs to be run for each part of the process to create, train and then serve the model. You can then send online prediction requests to the served model.

Cloud AutoML

This product is aimed primarily at developers, enabling those with limited knowledge of machine learning to deploy a number of pre-built models for specific use cases. Current offerings include AutoML Vision for image classification, AutoML Natural Language for text classification (I am currently trialling this for sentiment analysis) and AutoML Translation.

With these products you import your prepared data from Google Cloud Storage using the UI for the AutoML product you are using. You train the model using options in the UI. The performance is evaluated via the command line, and predictions are accessed with the UI.

Tensorflow

Tensorflow, created by the Google Brain team is an open source library designed to simplify large scale deep learning. This tool enables both experts and non experts to leverage the learning power of Artificial Neural Networks. In recent years these networks have powered a lot of the growth in advanced AI including the development of self driving cars and facial recognition.

The Tensorflow library includes a set of pre-made estimators that act as building blocks to use neural networks for classification, regression, as well text and image identification. Similar to other open source libraries for machine learning the code is written in python and follows a similar train, predict and evaluate set of functions. Tensorflow can be also used with ML Engine to serve these models in a production environment.

I have been fortunate to have access to use these Google tools with data already in BigQuery as part of my job. However for anybody that wants to try them Google offers free tiers for most of these products. Additionally BigQuery has a variety of public datasets that can easily be made available to practice with. Although my use of these tools is limited so far it does seem that they offer a way to take some of the pain away from developing and deploying both machine and deep learning. I am hopeful that with better support for a wider variety of machine learning libraries, and versions, that these tools will open up the use of AI to more job roles, and enterprises.

How Google are Democratising AI was originally published in vickdata on Medium, where people are continuing the conversation by highlighting and responding to this story.

Last week scikit-learn released version 0.20.0, one of the features in this release I am most excited about is the ColumnTransformer. This function allows you to combine several feature extraction or transformation methods into a single transformer. Say, you are working on a machine learning problem, and you have a dataset containing a mixture of categorical and numerical columns. Rather than having to handle each of these separately, and perhaps writing a function to then apply this to new data. These can now be combined into a transformer which can easily be reapplied, and extended.

This is a part of my machine learning workflow I have been keen to simplify, so over the weekend I took a simple data set I had lying around on my laptop, and had a go at applying this to a classification problem.

The data set I am using was taken from the Analytics Vidhya loan prediction competition and can be downloaded here. The purpose of this competition is to predict wether or not a loan application will be successful based on a number of customer features. This contains both categorical and numerical variables, and is a nice simple data set to practice using the new ColumnTransformer.

Data Preparation

To start with I am using pandas to read in both files. Using the dtypes function I can see that there are both numerical and categorical features present in the dataset.

import pandas as pd
train = pd.read_csv('train.csv')
test = pd.read_csv('test.csv')
print(train.shape, test.shape)
print(train.dtypes)

Before going further I am going to drop the Loan_ID column as that will not be used in the model. I am also filling any null values with the most commonly occurring value for each column. There are of course a number of methods I could choose for this but as I am just trying out a new function I am not too worried about the accuracy of the model for now.

train = train.drop(['Loan_ID'], axis=1)
test =  test.drop(['Loan_ID'], axis=1)

train = train.apply(lambda x:x.fillna(x.value_counts().index[0]))
test = test.apply(lambda x:x.fillna(x.value_counts().index[0]))

Next I’m going to specify the features (X) and target (y). I then use the sklearn train_test_split function to divide the training data into test and train ready to train and validate the model.

feature_set = train.drop(['Loan_Status'], axis=1)
X = feature_set.columns[:len(feature_set.columns)]
y = 'Loan_Status'

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    train[X], train[y], random_state=0)

ColumnTransformer

The next step is to apply transformation to columns to optimise them for use in the classification model. Usually I would write a function that applies these transformations, so that I can re-use it on both the test and train set, and any holdout data I may have for later validation. However, I am going to try using the ColumnTransformer to simplify these steps.

For simplicity, again I am not too worried about the accuracy of this model, I am going to transform the categorical columns using the sklearn OneHotEncoder. I will also normalize the numerical columns using the Normalizer function.

The ColumnTransformer takes a list of tuples specifying the transformers, and the corresponding columns on which the transformation needs to be applied. The columns can either be entered as strings specifying the column names in a pandas data frame, or as in the code I have used below, as integers which are interpreted as the column positions.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import Normalizer, OneHotEncoder

colT = ColumnTransformer(
    [("dummy_col", OneHotEncoder(categories=[['Male', 'Female'],
                                           ['Yes', 'No'],
                                            ['0','1', '2','3+'],
                                            ['Graduate', 'Not Graduate'],
                                            ['No', 'Yes'],
                                            ['Semiurban', 'Urban', 'Rural']]), [0,1,2,3,4,10]),
      ("norm", Normalizer(norm='l1'), [5,6,7,8,9])])

You will notice in the code above I have used the categories argument of the OnHotEncoder function. This takes a list of all possible categories in each column as a list of lists. This produces one hot encoded columns for all categories even if data does not exist for that category in the column. The reason for doing this is that when using the ColumnTransformer function on new data. If it doesn’t contain the same categories in each feature then the array produced will not be the same shape as the data used to train the model, and you will get an error.

I apply the transformer to the training data as shown below. The first few rows of the output, which is a list of lists containing numerical arrays, is shown beneath the code.

X_train = colT.fit_transform(X_train)
X_train

To transform the X_test data you simply apply the column transformer again.

X_test = colT.transform(X_test)

Training the Model

The data is now ready to be used in any scikit-learn classifier. For simplicity I have just used a RandomForestClassifier model with the default parameters.

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report
random_forest = RandomForestClassifier()
random_forest.fit(X_train, y_train)
y_pred = random_forest.predict(X_test)
print(classification_report(y_test, y_pred, target_names=['Y', 'N']))

Predicting New Data

To test how the ColumnTransformer would work if we were to use this model to make predictions on previously unseen data. I took a sample of rows from the test csv file I read in earlier. I then simply re-use the column transformer to apply the preprocessing steps.

test_samp = test[:15]
test_samp = colT.transform(test_samp)

random_forest.predict(test_samp)

Applying the Random Forest model to predict the loan status gives the following output.

Having used this new tool I have to say I am impressed. It has really simplified my workflow and I am going to be using this a lot in my work from now on. It is really simple to add new transformation steps into the ColumnTransformer, something which I will definitely be looking at over the next few weeks.

Easier Machine Learning with the New Column Transformer from Scikit-Learn was originally published in vickdata on Medium, where people are continuing the conversation by highlighting and responding to this story.