“Every act of communication is a miracle of translation.”
― Ken Liu

TLDR: We trained a model that can translate sentences from West African Pidgin (Creole) to English — and vice versa — without showing it a single parallel sentence (a Pidgin sentence and its English equivalent) to learn from. You can skip to the Results section at the end of the article to see some example translations by our model and the link to the code on github.

Introduction

Translation is an important area of research in Artificial Intelligence, and, most of all, communication. Many Machine Translation works have focused on popular languages like English, French, German, Chinese and so on. However, little work has been done on African languages.

Over 1000 languages are spoken across West and Central Africa, with over 250 of them being Nigerian. Despite the obvious diversity amongst these languages, one language significantly unifies them all — Pidgin English. There are over 75 million speakers in Nigeria alone, however, there is no known Natural Language Processing work on this language.

The problems this research addresses are the following:

1. Provision of a Pidgin corpus and training of Pidgin Word Vectors
2. Cross-lingual embedding of Pidgin and English
3. Unsupervised Machine Translation from Pidgin to English

1. Obtaining Corpus and Training Word Vectors

In total, we obtained a corpus consisting of 56048 sentences and 32925 unique words by scraping pidgin news websites. Below are examples of sentences in the corpus:

I. dis one na one of di first songs wey commot dis year for nigeria but as dem release am, yawa dey.

II. dem say na serious gbege if dem catch anybody with biabia for inside di campus

We initialized word vectors with Glove and fine tuned on the corpus with a CBOW model trained with 8 negative samples, window size of 5, dimension of 300 and a batch size of 3000 for 5 epochs.

2. Training Cross-Lingual Embeddings

Given the absence of parallel data, we performed Unsupervised Translation, which relies on cross-lingual embeddings. A dictionary consisting of 1097 word pairs was scraped and manually edited for supervised alignment. Evaluation of methods was done on a validation set of 108 pairs.

Alignment of the word vectors was performed with the Procrustes method in which an orthogonal weight matrix is learned to map source to target word vectors (Conneau et al., 2018) and with the Retrieval Criterion following (Joulin et al., 2018). The latter method outperformed the former achieving a Nearest Neighbor accuracy of 0.1282, compared to 0.0853 for the former and a baseline of 0.009, which is the probability of selecting the right nearest neighbor from the validation set of 108 pairs.

Given a word, below are some examples of its three English nearest neighbor words after alignment and their cosine similarity:

pikin — child (0.7461), infant (0.5493), children (0.5357)

presido — president (0.9173), vice (0.6589), presidents (0.5875)

wahala — problem (0.7265), problems (0.6983), trouble (0.6906)

3. Unsupervised NMT

For this, we used the Transformer with 10 attention heads. There are 4 encoder and 4 decoder layers with 3 encoder and decoder layers shared across both languages.

For a decoder to work well, its inputs should be produced by the encoder it was trained with or they should come from a similar distribution as that encoder. Hence, we make sure the encoder encodes sentences from source and target languages to the same latent space. This ensures the decoder can translate regardless of the input source language. We perform this enforcement by adversarial training following (Lample et al., 2018a) where we constrain the encoder to map the two languages to the same feature space. We do this by training a discriminator to classify encodings of source and target sentences. The encoder is trained to fool the discriminator such that latent representations of both source or target are indistinguishable.

We also make sure the same latent space is used for both language modelling and translation so that the language model can be transferred nicely to the translation task.

At each training step, we perform the following:

1. Discriminator training which aims to predict the language of an encoded sentence.
2. Denoising autoencoder training on each language (this is equivalent to training a language model as the model learns useful patterns for reconstruction and becomes immune to noisy input sentences)
3. On-the-fly back translation such that a given sentence is translated with the current translation model M and we then attempt to reconstruct it from the translation while leveraging the trained language model in step 2 above.

Thus, our final objective function is:

The discriminator loss is minimized in parallel.

We trained for 8 epochs on a V100 (approx 3 days). To select the best model, we evaluated on a test parallel set of sentences.

Results

The best performing model achieves a BLEU score of 7.93 from Pidgin to English and 5.18 from English to Pidgin on a test set of 2101 sentence pairs.

Below are some translations made by our model.

NB: [pd — Pidgin, en — English]

From these results, we can see that the language model helps with performing translations that are not necessarily word-for-word, but also grammatically correct as in translation 2.

Code, trained translation model, aligned pidgin word vectors and more translations by the model can be seen in the project’s Github repository — https://github.com/keleog/PidginUNMT

Conclusion

There has been some interesting NLP work done on African languages, however it barely scratches the surface in comparison to more popular languages. This is the very first NLP work - that we know of - performed on West African Pidgin. We hope that this work spurs the exploration of more unsupervised research for African languages.

Acknowldgements

• Special thanks to my amazing colleagues at Instadeep for constant support and constructive feedback throughout the project.
• Thanks to Naijalingo.com for permission to scrape their website to create a Pidgin to English dictionary.
• Thanks to Deepquest AI and AI Saturday's Lagos for computation support.
• Test set was obtained from the JW300 dataset [6] and preprocessed by the Masakhane group here

References

1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data.

2. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Herv Jegou. 2018. Word translation without parallel data. In ICLR

3. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In ICLR.

4. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018b. Phrase-based & neural unsupervised machine translation. In EMNLP.

5. A. Joulin, P. Bojanowski, T. Mikolov, H. Jegou, E. Grave, Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion, In EMNLP.

6. Željko Agić, Ivan Vulić: “JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages”, In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.

7. J. Tiedemann: “Parallel Data, Tools and Interfaces in OPUS.” In Proceedings of the 8th International Conference on Language Resources and Evaluation, 2012.

Update: Edit about BLEU scores made on 7/11/2019after final testing of model on more parallel sentences.

How we trained a translation model from West African Pidgin to English without parallel sentences was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Give a girl the right shoes, and she can conquer the world.” — Marilyn Monroe

Shoes have an incredible history. Anthropological research has revealed that humans may have started wearing shoes some 40,000 years ago. The oldest leather shoe found has been dated to be about 5500 years old, which gives you a fair idea of how old footwear is. However, shoes have greatly changed in the present modern times. Nike is now making auto-lacing shoes that can be laced from your smartphone and other fancy stuff!

But that’s not why you are here 😊 Yesterday, I discovered this dataset on Women’s shoes prices which seemed quite interesting. I decided to do some minor analysis and see what I could get out of it. The data set is largely a mess, so I had to do a bit of cleaning.

The data set contains information on 10,000 shoes spread across 110 brands. To ensure the stability of results, I considered only brands with at least 15 listed products which should be enough to give us a fair idea about their general pricing.

For each shoe, there are two columns in the data that provide information about its price: minimum shoe price and maximum shoe price. We shall focus on these two price columns, along with two other main features: brand and colors. This means we will see how the prices vary across different brands and different colors.

Let’s begin.

Brand

I was interested to see how prices vary across the different brands. The median minimum shoe price across various brands can be seen in the bar plot below.

The shoe brand with the highest median minimum price is Brooks at around $120. Other brands with high such prices are Salomon and Altra.

The lowest median minimum prices come from the following brands — Lauren Conrad, Apt 9 and, guess who? (hint: my former celebrity crush), J Lo by Jenifer Lopez.

Some brands with averagely priced minimum prices include Vans and Dr. Scholls. shoes and 2 lips shoes.

Any sneaker heads in the house?? I have got you!

Among sneaker brands, we can see that the highest priced sneaker brand is Asics followed by Adidas and Nike. We then have Skechers, New Balance, Vans and Fila.

Median maximum prices by brand

Also, the median maximum shoe price across the various shoe brands can be seen below as well:

We can see that Altra and Salmon take top place with a median maximum price of $129 dollars. The other brand in the top three is Brooks, ensuring the top three is unchanged from the median minimum.

Asics and Rampage no longer appear in the top ten. New entries to the top ten brands with expensive shoe prices include Dr Scholls and Style Charles.

One of my favorite brands appears among the top ten — Clarks :)

The shoes with the least median maximum prices include Hush Puppies, Union bay and Lauren Conrad

Range of prices by brand

I noticed something interesting from the range (subtracting the lowest from the highest) of both maximum and minimum prices across the brands.

As we can see, sneaker brands lead the pack. New balance has the highest range between minimum prices. The second is Asics while the third is Adidas.

Also, in the range of maximum prices, sneaker brands are totally killing it! The top four spots are dominated by sneaker brands — NB, Nike, Asics and Adidas.

I am not sure why sneaker brands seem to have the highest price range, but I could make a guess.

While many sneaker brands are available for moderate prices as indicated by the median maximum and minimum prices, they seem to offer some niche products that people want to pay more for. When we think high end niche sneakers, we immediately think basketball and celebrity signature sneakers. Some Jordans, LeBrons and Adidas Pharrells could cost well over a thousand dollars.

I suspect the slides usually made by sneaker brands are responsible for the low minimum prices, thus increasing the range. Notice that other sneaker brands like Vans and Fila do not have high ranges? That could be because these brands are mostly focused on more affordable shoes. Vans is a top streetwear sneaker that mostly looks to offer basic comfortable shoes that primarily differ in color rather than athletic benefits, niche construction or styles.

Colors

Women love colors and seem to know a whole lot about colors than men. In fact, some research works (here and here ) have indicated that the wiring of the visual cortex of the brain of women help them distinguish color better.

In our data set, we will consider only colors that appear in at least 40 shoes.

Curious about how the prices of women’s shoes vary according to color? Let’s take a look.

Medium Minimum Price by Color

We see here that the highest median minimum price belongs to shoes with Dark Gray and Stucco colors (I had no idea there was a color called stucco until now!)

Many of the colors have relatively average median minimum prices with the lowest going to shoes with Burgundy and Mushroom colors.

Median Maximum price by color

We can see that Whiskey colored shoes have the highest median maximum prices. This is closely followed by Slate and chestnut colored shoes. At the bottom of the pile, we have blush and pewter colored shoes

Range of prices by color

Here, we can see that tan has the highest range of maximum prices. Followed by black and blue.

The lowest range belongs to Lux Navy and white sand. Lurking somewhere at the bottom is…wait for it…White black! How many types of blacks are there????

Anyway, there you have it — a very minor analysis of women’s shoe prices by brands and colors. This was a fun one for me. I learned a lot about colors I never knew existed. I also learned that there are several types of blacks (besides regular black and matte black) and that people actually google “What colors go well with pewter and stucco?”!

I hope you enjoyed reading this! Do not forget to clap if you did :)

All the plots were done with Plotly, so they are interactive!!!

For other analysis and interaction with the plots, check the entire rendered notebook here and this is a link to the notebook on GitHub. The data set can be found here.

“ Death is not extinguishing the light; it is only putting out the lamp because the dawn has come.

— Rabindranath Tagore”

One of my recent penchants has been reading about other religions, particularly finding intersections between their beliefs and Christianity. For one, I refuse to believe that there is only one “true religion” and the practitioners of other religions are going to hell.

I lost a friend last year, and it definitely put things into perspective for a while. I realized that every other thing pales in comparison when death is involved. However, death is constant; we only just further on the queue. Some people, however, yearn for death and would not mind jumping the queue. Almost everyone who believes in a supernatural being also believes in an afterlife that follows death. This afterlife is, supposedly, eternal. The sad part is that we will never find out if such really exists until we die.

So, what do different religions say about death and an afterlife? In my search, I read some papers and also stumbled upon this wonderful documentary on Netflix called — The Story of God. It is a show where Morgan Freeman goes around the world asking religious leaders pertinent questions about God and life.

In the very first episode, he explored what different religions believe about the afterlife. I’ll do a brief summary of what some religions and science believe.

Ancient Egypt

Ancient Egypt was one of the first civilized places in the world. It was a largely polytheistic society. There was belief in three major afterlife scenarios — the underworld, eternal life and rebirth/reincarnation.

Ancient Egyptians believed that the body would be important in the afterlife, hence the reason for mummification. Also, they believed the journey to the afterlife was super long, so they buried the dead with food and drinks to help their journey. Weapons were also seen in many tombs to help the dead fight enemies in their afterlife journey. Other things like sacred spells, clothes, ointments and boats were also present to help with their journey to the afterlife. It is worth noting that it was important for the dead to proceed to eternal life with the gods so they could preserve humanity, particularly ensuring the sun never ceased.

Hinduism

I was particularly excited about reading this one because I had no previous idea what Hinduism was about.

Hindus believe that the soul is immutable and can take on different coverings or shells in the form of a body in different cycles of rebirth known as Samsara. The could be human form, animals or other forms of life. The assignment of these bodies depends on the performance or karma of the person in the last cycle of rebirth.

Hence the soul is constantly reborn in different bodies until it achieves a state called Moksha or Nirvana. Moksha is a state of liberation obtained through a virtuous and moral life, material prosperity and emotional fulfilment (dharma, artha and kama).

Buddhism

The Buddhists maintain the same belief in rebirth, albeit with several differences. There is no permanent soul in Buddhism. Also, the paths for attaining Nirvana are different in both religions.

Science

There have been numerous attempts to understand the afterlife in science. One that stood out for me was Dr. Sam Parnia, a critical care physician, who has studied over a hundred cardiac arrest survivors — people who have tasted death for a while.

Sam says a common observation is that many of them claim to have seen a light and felt peace. They claim to have felt themselves leaving their bodies behind in transition to a calm atmosphere. Many others claimed to have seen their dead loved ones welcoming them with open arms.

I also stumbled upon this website where people shared similar visions of their dying friends / relatives.

However, many Near Death Experience (NDE) experts have criticized Sam’s methodology. Also Sam has himself said that NDEs may well be an illusion.

Cyber-consciousness is becoming a common research area amongst neuroscientists. Cyber consciousness is largely defined as “a software system that can persuade observers into thinking it has an internalized sense of who it is, what it is and what its place in the world is”.

That bit of consciousness is what separates humans from machines and what, supposedly, transcends to the afterlife. This means that if machines can be given consciousness, we could well be on our way to achieving immortality by attempting to transfer the consciousness of a human to a machine.

In the book, Heavens on Earth by Michael Shermer, several important questions were raised as regards the possibility of transferring consciousness.

They include — How do we define self-identity? Shall I insist that it is inseparable from my body? If I upload my mind to a computer, how shall I conceive of my original self versus the copy?

Other scientists like Sean Carroll, a cosmologist and physics professor at the California Institute of Technology, have refuted the possibility of an afterlife.

According to Carroll, “Claims that some form of consciousness persists after our bodies die and decay into their constituent atoms face one huge, insuperable obstacle: the laws of physics underlying everyday life are completely understood, and there’s no way within those laws to allow for the information stored in our brains to persist after we die.”

Carroll is of the opinion that for an afterlife to exist, consciousness must be something that is entirely separated from our physical body, and that is impossible.

Conclusion

I am primarily Christian, so I believe in the afterlife per the Christian doctrine. I think that the afterlife in other religions is very interesting. I believe there are "several truths" and one just has to stick to the truth they are most comfortable with, while — most importantly — respecting the truth of others.

I look forward to reading about how other beliefs vary across various religions. I also look forward to seeing my friend in the afterlife ☺

PS — I scoured the internet for afterlife beliefs in African traditional religion, but I couldn’t find anything concrete. I am particularly interested in intersections between African traditional religion and Christianity. African traditional religion has been repackaged to us as black magic, and that shouldn’t be. Any references as regards this will be appreciated.

References

Guasch-Jané, M., Andrés-Lacueva, C., Jáuregui, O. and Lamuela-Raventós, R. (2006). First evidence of white wine in ancient Egypt from Tutankhamun’s tomb. Journal of Archaeological Science, 33(8), pp.1075–1080.

Burial customs in ancient Egypt: life in death for rich and poor
W Grajetzki (2003)

https://m-cacm.acm.org/news/178647-seeking-human-cyber-consciousness/fulltext

• Investigating the afterlife
• There is NO life after death: Scientist insists afterlife is IMPOSSIBLE

As data scientists, we will come across various types of datasets. One such dataset is an imbalanced data set. This type of dataset always poses a problem for beginner data scientists as most machine learning algorithms are bad at handling it.

What is an imbalanced dataset?

An imbalanced dataset is one where the number of observations belonging to one group or class is significantly higher than those belonging to the other classes.

This occurs in cases such as credit card fraud detection where there might be only 1000 fraud cases in over a million transactions, representing a meager 0.1% of the dataset. The identification of rare diseases is another possible case of dealing with imbalanced data.

Challenges of dealing with Imbalanced datasets

Machine Learning algorithms are very likely to produce faulty classifiers when they are trained with imbalanced datasets. These algorithms tend to show a bias for the majority class, treating the minority class as a noise in the dataset. With many standard classifier algorithms, such as Logistic Regression, Naive Bayes and Decision Trees, there is a likelihood of the wrong classification of the minority class.

There is also a problem of vanity metrics in measuring the performance of algorithms on imbalanced datasets. If we have an imbalanced dataset containing 1% of a minority class and 99% of the majority class, an algorithm can predict all cases as belonging to the majority class. The accuracy score of this algorithm will yield an accuracy of 99% which seems impressive, but is it really? The minority class is totally ignored in this case and this can prove expensive in some classification problems, such as the case of a credit card fraud, which can cost individuals and businesses lots of money.

Methods of Handling Imbalanced Datasets

There are two major methods of handling imbalanced datasets and they are discussed below:

Oversampling: This method involves reducing or eliminating the imbalance in the dataset by replicating or creating new observations of the minority class. There are four types of oversampling techniques:

• Random Oversampling: In this case, new instances of the minority class are created by randomly replicating existing samples in order to increase the minority count in the dataset. This method however can lead to overfitting as it simply replicates the already existing instances of the minority class.
• Cluster-based oversampling: Here, the K-means algorithm is applied separately to both the majority and minority instances. This helps identify the clusters in the dataset. After the identification, each cluster is oversampled such that all clusters have the same number of observations. Again, there is a risk of overfitting the model with this method.
• Synthetic Oversampling: This method helps to avoid overfitting. In this method, a small subset of minority is chosen and synthetic examples of this subset are created to balance up the overall dataset. This adds new information to the dataset and increases the overall number of observations.
• Modified Synthetic Oversampling: This is just like the synthetic oversampling method, however this one makes room for the noise and inherent distributions of the minority class.

2. Under-sampling: In this method, the imbalance of the dataset is reduced by focusing on the majority class. One popular type is explained below:

• Random Undersampling: In this case, existing instances of the majority class are randomly eliminated. This technique is not the best because it can eliminate information or data points that could be useful for the classification algorithm.

Measuring algorithm performance on an Imbalanced Dataset

Since we have established that accuracy is a poor measure of performance on imbalanced datasets, how then do we measure performance? In order to establish this, let us first define some terms:

False Positive (FP): This is used to describe positive predictions that are actually negative in real life. An example will be predicting a credit card transaction is fraudulent when in truth it is not.

True Positive (TP): This is used to describe positive predictions that are actually positive in real life.

False Negative (FN): This is used to describe negative predictions that are actually positive in real life. An example will be predicting a credit card transaction is not fraudulent when in truth it is.

True Negative (TN): This is used to describe negative predictions that are actually negative in real life.

Now that we have that out of the way, let us look at possible performance measures.

Some Measures and indicators of model performance include —

Precision: This is an indicator of the number of items correctly identified as positive out of total items identified as positive. Formula is given as: TP/(TP+FP)

Recall / Sensitivity / True Positive Rate (TPR): This is an indicator of the number of items correctly identified as positive out of total actual positives. Formula is given as: TP/(TP+FN)

Precision can be seen as “how useful the results are”, and recall is “how complete the results are”

Specificity / True Negative Rate (TNR): This is an indicator of the number of items correctly identified as negative out of total actual negatives. Formula is given as: TN/(TN+FP)

F1 Score: This is a performance score that combines both precision and recall. It is a harmonic mean of these two variables. Formula is given as: 2*Precision*Recall/(Precision + Recall)

Matthew Correlation Coefficient Score: This is a performance score that takes into account true and false positives, as well as true and false negatives. This score gives a good evaluation for imbalanced dataset. Formula is given as:

Area Under ROC Curve: An ROC curve (receiver operating characteristic curve) is a graph that shows the performance of a classification algorithm at all classification thresholds. This curve plots TPR vs. FPR.

Area Under Precision-Recall Curve: A Precision-Recall curve is graph that shows performance by plotting precision against recall.

Confusion Matrix: This is a graphical representation of the TP, FP, FN and TN. A generalized confusion matrix is given below:

Generally, the best measures for imbalanced datasets are: Matthew Coefficient Correlation Score, F1 Score and The Area under the Precision-Recall Curve.

Still, it is important that you understand the business implication of the model when looking to choose a performance evaluation metric. Ideally, there would be trade offs between false positives and false negatives in real life. For instance, if you are classifying against credit card fraud, you may prefer to have a false positive than have a false negative. That is, you would rather predict that a transaction is fraudulent when it actually is not, than predict that a transaction is not fraudulent when it is. Bear in mind that excessive false positive predictions could lead to bad customer experience, so you may have to consider that as well. Also, if you are building a movie recommendation model, for instance, you may prefer to have false negatives than false positives. That is, you would rather tell a person they do not like a movie when they actually do, than tell a person they will like a movie when they do not.

Essentially, the business implications of these metrics need to be taken into account before settling for one.

Practical Example: Credit Card Fraud Prediction

This practical example utilizes an anonymized credit card transactions dataset.

The ratio of non-fraudulent transactions to fraudulent transactions was a whooping — 99.83% to 0.17%

I proceeded to modelling, selecting Logistic Regression as my algorithm. I trained the algorithm with the unbalanced dataset. Below was my result:

Notice that the accuracy is very high, despite the presence of false positives and false negatives. However, the F1 score and MCC score tell a better story of the actual model performance

I then proceeded to balance the dataset.

I used SMOTE (synthetic minority oversampling technique) to balance the dataset.

I then trained the algorithm with the balanced dataset, tested with the original data and obtained these results:

As we can see, this model is much better than the former one.

There you have it! I hope you enjoyed reading this half as much as I enjoyed writing it. If you did, please clap for the post and share it. You should also follow me and check out my other posts.

Handling Imbalanced Datasets: Predicting Credit Card Fraud was originally published in Coinmonks on Medium, where people are continuing the conversation by highlighting and responding to this story.

As beginner data scientists, one of the words we hear very often is — ensemble, but what does this really mean? We mostly hear about it from Kaggle competitions, although nobody has cared enough to really explain it to us. In this post, I will try my best to explain ensemble learning to you in very simple terms.

What is an Ensemble in Machine Learning?

An ensemble model is a collection of machine learning models that are each trained on a dataset, and then combined by some logic into a single model which is more robust and accurate than each of these individual models. The intuition behind this is that each model follows a different approach or utilizes a different technique, thus providing new information and marginal increases to the performance of the overall model.

In our daily lives, we create ensembles without even realizing it. Say we want to rate or form an opinion about the recently released Drake Album — Scorpion. We can go to various music websites — say Rollingstone, Metacritic, Pitchfork and All Music. Each of these websites provides a rating of Scorpion, and we can combine these ratings to decide on a personal rating to give the album.

We can also decide to ask our friends who have listened to the album whether or not it is good. We can ask five friends, and if four of them say we should listen, we can infer that maybe the album is good.

This is basically what Ensembles are about — combining opinions (or in this case — models)

Ensemble models are very popular and have been known to be very effective in Kaggle Competitions. They are also a good way to achieve appropriate variance-bias tradeoff.

Types of Ensemble Models

1. Bagging: Bagging stands for Bootstrap AGGregating. As this implies, Bagging combines bootstrapping and aggregating into the formulation of an ensemble. Bootstrapping basically involves random sampling with replacement. Bagging implements learning models on small population sizes or samples, and then aggregates the results from each of these models into one ultimate model. Bagging helps to reduce the variance error in models.

2. Boosting: This is an iterative approach to combining models. Boosting performs several iterations of training on the dataset and each successive learner improves on the weakness of the previous learner. Thereafter, the boosting technique combines all the individual learners into one single strong learner. Boosting helps to reduce bias error, although it may over fit the training data, leading to a high variance when tested on other datasets.

3. Stacking: This is an ensemble approach that involves using a learner to combine and collect outputs from different individual learners on the same data set. This combination is then presented as the final model.

4. Blending: This is an ensemble approach that is similar to stacking, but utilizes a validation data set from the training data to make predictions.

Ensemble Aggregation Techniques

1. Voting: In this technique, which is mostly for classification problems, each prediction by a machine learning model is treated as a vote. The final prediction for each data point is obtained from a combination of all votes. There are two types of voting methods — Hard and Soft:

a. In the hard voting method, the mode prediction is the one that is selected by the ensemble. For instance, if there are three classifiers and two states to predict and, for a data point,

Classifier 1 predicts A

Classifier 2 predicts A

and Classifier 3 predicts B

Then, the output of the ensemble for that data point will be the most popular prediction, which is A.

b. In the soft voting method, the ensemble takes into consideration the level of certainty of each model. Using the same example above, assuming for the same data point:

Classifier 1 predicts A with 55%, leaving B with 45% probability

Classifier 2 predicts A with 51%, leaving B with 49% probability

and Classifier 3 predicts B with 90%, leaving A with 10% probability

Calculating the average of each class predictions, class A yields 38.7%, while class B yields 61.3%. Even though more classifiers predicted class A, the ensemble will output class B because of the high level of certainty of the average prediction. Essentially, Soft voting considers the certainty level of each voter, rather than simply considering the binary choice of each voter.

2. Averaging: This is similar to the soft voting approach and can be used for both classification and regression problems. The average of predictions from each regressor is used to make the final prediction. In the case of classification, the average of probabilities as has been shown above is considered. Some classifiers do not output probabilities, such as the SVC. Hence, when using such classifiers, you have to set their Probability parameter to be “True”

3. Weighted Average: This is just like the averaging method, except some models are assigned more weight or preference than others. For instance, if you know a model performs well in a particular problem, you will give it more weight than a regular performing model. This also comes into play in real life. If we wanted to predict the outcome of a football game, predictions from some pundits will carry more weight than those from other pundits.

Popular Examples of Bagging and Boosting Algorithms

Bagging algorithms:

1. Random forest

2. Bagging Regressor

3. Bagging Classifier

Boosting algorithms:

1. GBM
2. XGB
3. CatBoost
4. LightGBM
5. AdaBoost

Talk is Cheap; show me the damn Code:

I have written code examples of each of the ensemble aggregation techniques, and the Bagging Classifier and Regression in this notebook.

Just found this in my draft — Written on August 20, 2015 after I read The Art of Thinking Clearly by Rolf Dobelli. I did not edit it, just published as I found it. Enjoy! (or not).

“Facts do not cease to exist because they are ignored” — Aldus Hexley

In our daily activities, we tend to think of things in such a way that they conform to our preexisting notion. This is a very irrational manner of thinking and it forms the basis of the confirmation bias.

“Black Swan logic makes what you don’t know far more relevant than what you do know” — Nassim Taleb

Nassim Taleb further accentuates this in his book “the black swan”. For many years in the old world, people thought all swans were white. With each white swan they saw, their thoughts were further confirmed (or so they thought). But the sight of the first black swan shattered their supposed knowledge of the color of swans.

There seems to be a subtle overlap between the confirmation bias and the clustering illusion. The latter is man’s proclivity to search for patterns where none exists. However, both of them have a final goal of confirming our preexisting thoughts.

Rolf Dobelli sights a very practical example of the clustering illusion in his book, “the art of thinking clearly”. During the second world war, the Germans bombed London using the V1 self-navigating rockets. With every attack, the impact sites were plotted on a map and Londoners thought they had discovered a pattern. They then made theories about which parts of the city were safest. However, after the war, statistical analysis confirmed that the distribution was totally random.

“It is easier to prove something is right than it is to prove the same thing is wrong.” — Some guy on Quora :)

Now, the confirmation bias and clustering illusion may sound a tad bit simple but their effects are far beyond that. If anything, their effects are quite insidious and only conspicuous in retrospect. However, a major question lingers:

How does(n’t) our knowledge of confirmation bias affect our confirmation bias and how we attribute events to confirmation bias?

Well, truth is this is a subject of worry and I don’t think any definite answer can be given to this question.

Finally, it is an almost Herculean task to rid ourselves of the confirmation and clustering illusion. However, like everything in life, it takes constant practice. So the next time you think you’ve noticed a pattern, try to disprove it. Also, the next time you want to confirm a certain thought, try to disprove it as well. Only when it cannot be disproved can it truly be confirmed. After all, that is what science is all about!

We have explored classification algorithms with a practical case study in part 1. This Part 2 will be focused on regression algorithms. So, let us cut right down to the chase.

Regression Algorithms

Regression problems are those that involve the prediction of continuous variables. For example, predicting the weight of students based on given data, predicting the scores of students, predicting house prices, etc. This is unlike classification problems whereby we are bothered with grouping observations.

Again, there are several algorithms that can be used for regression problems. We will explore them in detail by using a practical case study.

Case Study: Big Mart Sales Prediction (Link)

Problem:

The data scientists at BigMart have collected 2013 sales data for 1559 products across 10 stores in different cities. Also, certain attributes of each product and store have been defined. The aim is to build a predictive model and find out the sales of each product at a particular store. Using this model, BigMart will try to understand the properties of products and stores which play a key role in increasing sales.

Approach:

It was basically the same as the classification problem. I made some hypotheses before starting out. I posited some factors that could influence the item outlet sales and decided to validate or ditch them during my EDA. Also, I decided to understand the importance of feature scaling on various algorithms, so I had two datasets — a scaled one and an unscaled one. Thereafter, I compared different regression algorithms as the evaluation was the RMSE.

Solution (Github Link):

I explored the data extensively. This led to the discrediting of some of my hypotheses. For instance, I posited that items with high visibility should have high outlet sales. However, the opposite was the case. I noticed several other attributes in the data, all of which are comprehensively explained in my notebook.

I filled in missing variables based on insight from my EDA. I also handled inconsistent entries in the fat content column. I then created some new features — The first was the age of the outlets. (2018 — Establishment year), the second was whether or not the item type is a consumable and the third was whether the item type is a food, drink or non-consumable.

I then converted the categorical variables to numerical ones. Also, the item visibility column had several zeros, which may have represented missing values. So I replaced with the median of item visibility of similar outlets and item identifier. I then scaled the data for and saved it for later comparison with unscaled one.

I then proceeded to modelling and evaluation. I used the following algorithms for modelling:

· Linear Regression: This is an algorithm that works by trying to fit a line to your data points. This line is usually the line of best fit such that it has the minimum distance between most, if not all, data points. The higher the distance between the line and the data points, the greater the error. This is a very simple algorithm and works well when variables are linearly related. However, it punishes co-linearity between variables. Linear regression is prone to overfitting and this is why there exists regularization. Linear regression cannot also handle several input features and cannot handle non-linear relationships.

· Lasso Regression: This is an algorithm that is just like the linear regression, but comes regularized. LASSO stands for Least Absolute Shrinkage and Selection Operator. This algorithm punishes the absolute size of coefficients. This means it can offer an automatic selection of feature as some coefficients can be shrunk to zero, meaning it drops such features.

· Ridge Regression: Ridge regression also penalizes coefficients, but it does so by penalizing the squared size of these coefficients. Unlike Lasso, ridge actually does not force features to zero.

· Gradient Boosting Regressor: Boosting is an ensemble method that trains several weak learners sequentially. Every successive weak learner focuses on learning from the mistakes of the previous learner. Thereafter, the boosting technique combines all the weak learners into one single strong learner.

· Random Forest Regressor: As described earlier, this algorithm combines several strong decision tree learners by bagging. Bagging is a method where several strong learners are trained in parallel. These learners are then combined and their flaws are smoothened out.

· Support Vector Regressor: As described earlier, this algorithm aims to distinctly separate the input variable space. SVR finds the coefficients that yield the highest separations. The distance between the separation hyperplane and the closest data point is referred to as the margin. The aim is to get the highest margin, which yields the most distinct separation.

· K nearest neighbour Regressor: For a data point to be predicted, this algorithm looks to find the most similar data points or neighbours to that data point and uses to it to form the outcome variable. In this case, it may be the mean output variable. The similarity between data points is usually determined by Euclidian distances or Hamming distances. As a result of this, the scales of features will be strong determinants of these distances.

Conclusion

After submitting the solutions to the Analytics Vidhya platform, the following were noticed:

The worst performing algorithm was the KNN on the unscaled data (remember KNN works well with scaled data cos it is distance dependence). However when used on the scaled data, the KNN algorithm performed well. Reducing the RMSE from 2482 to 1246. The Lasso algorithm also performed better on the scaled data than on the unscaled one. RMSE reduced from 2401 to 1202. Same with Ridge regression, Random Forest, Linear regression and SVR. The decrease in SVR was not as significant as the rest though. RMSE reduced from 1814 to 1757. The best performing algorithm was the GradientBoosting Regressor. On the normal data, it had a score of 1155. On the scaled data, however, it gave an RMSE score of 1153. On the competition leaderboard, I am in position number 540 out of over 16 000 registered competitors (top 3.2%).

PS: my notebook is very explanatory, check it out.

Introduction

There is a lot of buzz around machine learning algorithms these days, and rightly so. Machine learning is arguably the most glorified alongside its fellow new technologies like cloud computing, iOT, AR, VR and Blockchain. In order to harness the full potential of machine learning algorithms, it is important to know the best scenarios and case studies in which they should be applied. I set out to understand these algorithms in detail. I conducted classification and regression tasks, and compared the results obtained by these algorithms. I will be meticulously explaining how I went about this and what I discovered along the way.

Classification Algorithms

Classification problem are those that require the prediction of states. Essentially, classification problems require you to predict what group or class or state an entity falls in. For example, prediction of whether or not a credit card transaction is fraudulent one, predicting whether or not a tweet is hate speech, and so on.

The simplest classification problems have just two classes or are binary classification problems. However, classification problems can have more than two classes and these are often referred to as multi-class classification problems.

There are several algorithms that can be used for classification problems. We will explore them in detail by using a practical case study.

Case Study: The Dream House Finance Loan Problem (Link)

About Company

Dream Housing Finance Company deals in all home loans. They have presence across all urban, semi urban and rural areas. Customer first apply for home loan after that company validates the customer eligibility for loan.

Problem

Company wants to automate the loan eligibility process (real time) based on customer detail provided while filling online application form. These details are Gender, Marital Status, Education, Number of Dependents, Income, Loan Amount, Credit History and others. To automate this process, they have given a problem to identify the customers segments, those are eligible for loan amount so that they can specifically target these customers. Here they have provided a partial data set.

Approach

My approach was simple. I sought out to understand the data and put myself in the shoes of the company manager. This way I would be able to think beyond the technical aspects and apply some domain knowledge in the manipulation of my data. I spent quality time exploring my data, cleaning it and creating new features before modelling.

Solution (Github Link)

I first detected extreme outliers using the Tukey method and dropped them (two in total). There after I performed some exploratory data analysis. I began with describing the statistical properties of the numerical variables in the data set. I observed the presence of some outliers based on these descriptions.

I then proceeded to perform visualizations. I started with strip plots of the target variable (loan status) against the numerical variables. I made several observations, particularly with the applicant and co-applicant incomes. I then employed the use of hues to visualize three variables at once — two categorical and one numerical. This revealed the variation of loan eligibility status across all numerical variables by gender, etc.

Next, I used count plots to compare categorical variables. I noticed that applicants with credit histories seemed to have higher chances of obtaining loans when compared to those without. I also noticed observations between other categorical variables, all of which are well explained in my notebook.

I detected and filled in missing values where appropriate. I then converted the categorical variables to numerical variables since some of our classification algorithms need that. I detected some new outliers and log transformed them.

I created some new features based on observations from my exploratory data Analysis. I initially created family income which is a summation of applicant and co-applicant income. Thereafter, I created — loan amount/loan amount term, family income/loan amount, family income/loan amount term, applicant income/loan amount, applicant income/loan amount term.

The first will show how much the person has to release per month to pay up the loan. The second and fourth will allow us know how much of the family’s and person’s monthly income is the loan amount. The third and fifth will let us know how much the family and person have to spend monthly over the course of the loan term. There is no information as regards whether the income is the total income over the loan amount term or yearly, but I assumed the former.

I then proceeded to modelling where I identified the important features according to the gradient boosting classifier. I used the following algorithms for modelling:

· Logistic Regression: Even though it has the word regression, this is a classification algorithm that works by trying to fit a line to your data points. This algorithm is based on a logistic function. This function makes probability-like predictions between zero and one that determine the class of a data point. Logistic regression is a very simple algorithm and works well when there is a strong linear relationship between variables and the outcome variable. However, co-linearity between input variable affects this model. It is also prone to overfitting and regularization is advised.

· Linear Discriminant Analysis: This is an algorithm that is used to classify by making predictions by calculating statistical properties of data points. These statistical properties, usually mean and variance are then used to make predictions. This algorithm assumes a Gaussian distribution for datasets, so it is very sensitive to outliers. It can also be used as a dimension reduction algorithm. LDA is also good with multi-class classification problems. It is advisable to standardize your data and ensure it is normalized.

· Gradient Boosting Classifier: Boosting is an ensemble method that trains several weak learners sequentially. Every successive weak learner focuses on learning from the mistakes of the previous learner. Thereafter, the boosting technique combines all the weak learners into one single strong learner.

· Naive Bayes Classifier: This is a machine learning classification algorithm that assumes independence between input variables. This algorithm, which is based on the Bayesian theorem, assumes that the presence or outcome of a feature is not based on the presence or outcome of another.

· Random Forest Classifier: This algorithm combines several strong decision tree learners by bagging. Bagging is a method where several strong learners are trained in parallel. These learners are then combined and their flaws are smoothened out.

· Support Vector Classifier: This algorithm aims to distinctly separate the input variable space. SVC finds the coefficients that yield the highest separations. The distance between the separation hyperplane and the closest data point is referred to as the margin. The aim is to get the highest margin, which yields the most distinct separation.

Conclusion

Upon submitting to the competition platform, the Linear Discriminant Analysis, Logistic Regression and Gradient Boosting classifiers gave the best results on the test data after being fitted with the normal data with accuracies of over 78%. However, on the pre-processed train data, the Naive Bayes and SVC gave the best accuracies of about 72%. This gives us an idea of what algorithms are affected by outliers the most.

I also ended up with a peak accuracy of 78.472%, placing me amongst the top 4% out of about 27k registered participants.

Here is part 2 explaining the regression bit.

PS: the notebook in my github is very well explained. Check it out.

I am interested, but can you provide a definition for data cleaning?

Sure, I got you.

According to Wikipedia, Data cleansing or data cleaning is the process of detecting and correcting (or removing) corrupt or inaccurate records from a record set, table, or database and refers to identifying incomplete, incorrect, inaccurate or irrelevant parts of the data and then replacing, modifying, or deleting the dirty or coarse data.

Scary, right?

Now, forget that very long sentence. Let me break it down for you in a simple sentence.

Data cleaning is basically the act of identifying and correcting false, incomplete or irrelevant parts of data.

Sounds better, yeah?

Okay, but what is the importance of this data cleaning?

According to an article on Forbes, by the year 2020, about 1.7 megabytes of new information will be created every second for every human being on the planet. This means that different types and sources of data will be at play, and that could prove to be messy.

Yes, there is usually a lot of buzz around the data science algorithms and not data cleaning. This is simply because data cleaning is the least fun part of data science.

If data science is the sexiest job of the 21st century, then data cleaning is the most unsexy part of data science.

The importance of data cleaning in training a model cannot be overstated. No matter the algorithm you use — if your data is bad, you will get bad results. Professional data scientists know this and have revealed that data cleaning takes up to 70% of the time spent on a data science project.

Better and cleaner data outperforms the best algorithms. If you use a very simple algorithm on the cleanest data, you will get very impressive results. However, if you use the best algorithm on messy data, you would most likely not get the desired result.

Essentially, it is garbage in, garbage out.

Enough of all this! How do I perform this data cleaning thing?

Jeez. Chill out! We were just getting to that. Before we get started, it is advised you open the code examples here — so you can follow along well.

1.Dealing with Missing Values: This is perhaps the most common trait of unclean data. These values usually take the form of NaN or None. Missing values can also lead to our models predicting NaN values, which we would not want. So, how do we tackle these missing values?

The data used in the code example contains information about building permits issued in San Francisco from Jan 1st, 2013 to Feb 25th, 2018.

After reading in your data, you need to first establish that there are missing values in the dataset.

It is important to also check the percentage of data missing. This will give you an idea of how to handle the missing values. We can also visualize how much data is missing per column.

There are so many missing values!

Now that we have established that there are missing values in the dataset, how do we handle them?

In order to handle missing data, it is important to identify the cause of the missing data. This will guide us in deciding on the best way to handle them.

There are several causes of missing data values. Some values could be missing because they do not exist, others could be missing because of improper collection of data or poor data entry. For example, if someone is single and a question applies to married people, then the question will contain a missing value. In cases like this, it would be wrong to fill in a value for that question.

In our data, for instance, Street Number Suffix does not always exist, so it would not be ideal to fill it up. However, the missing locations, zipcodes and permit expiration date can be filled up.

There are several ways to fill up missing values, and we will explore some of them here.

We can decide to drop the rows or columns with missing values. This is only advised when the percentage of missing values in a row or column is so high that it does not make sense to keep the column or row.

For instance, the TDIF compliance column and the Voluntary Soft-Story Retrofit have near 100% missing values, so it makes sense to drop them. We can also decide to drop columns where the missing values are at least 60% of the data

The same can also be done for the rows of any data set by just specifying that the axis = 0

If we are dealing with numerical values, we can decide to fill all null variables with 0. We can also decide to fill in all NA values with the mean of all the values in the column, or the median. All of these decisions depend on the type of data,what you want to accomplish with the data, and the cause of the missing values.

As you can see, the missing values have been replaced with 0

Finally, we can also decide to fill up missing values with whatever value comes directly after it in the same column. We will also specify that any missing value which does not have any value directly under it will be filled with 0.

2. Inconsistent Data Entry: This occurs when there are different unique values in a column which are meant to be the same. There are also other types of inconsistent data entries, but this will be our focus here. The data setto be used here contains information about Suicide Attacks in Pakistan from 1995 to 2016. As usual, we will first import all relevant libraries and read in our data.

We will handle the inconsistent entries in the Province column. To check for all unique values, we do this:

We can guess that ‘Balochistan’ are probably meant to be the same ‘Baluchistan’. A simple Google search suggests that we are right. Also, ‘FATA’ and ‘Fata’ are most likely meant to be the same.

One of the ways to remove data inconsistencies is by removing whitespaces before or after entry names and by converting all cases to lower cases. For white spaces, if we had something like ‘ fata’ and ‘fata ’, they would be considered two unique values.

Nice, we can see that fata and FATA have been unified

We still have a problem though. We need to unify ‘balochistan’ and ‘baluchistan’ entries.

Niceee. Seems like we have solved it all.

NB: If we have a large number of inconsistent unique entries, we cannot check with our eyes like we just did for the closest matches. We can use the Fuzzy Matching. This helps to identify which strings are most likely to be the same. A package — Fuzzy Wuzzy — does this well. The FuzzyWuzzy package to be used takes in two strings and returns a ratio. The closer the ratio is to 100, the more likely we will unify the strings.

3. Handling Dates and Times: We are often faced with dates in our datasets. And, as beginners, this can often prove daunting. It is even worse when there are inconsistent formats of dates. For instance — dd/mm/yy and mm/dd/yy in the same columns. Also, our date values might not be in the right data type, and this will not allow us effectively perform manipulations and get insight from it. Fear not though, as we will solve all of that here.

As usual, we will first import all relevant libraries and read in our data.

The data to be used here contains information about a record of the date, time, location, depth, magnitude, and source of every earthquake with a reported magnitude 5.5 or higher since 1965. All compiled by The National Earthquake Information Center (NEIC).

If we take a look at the Numpy documentation, the datatype O matches with the Object data type, which is not the date time data type

We proceed to convert our date datatypes to date time data types. We will need to import the datetime package to handle this.

As we can see, the data type has correctly changed to the datetime data type, and now we can perform various date time manipulations on the column because of its data type.

If all your dates are in a specific format, you can easily feed in that format into the datetime when converting. However, all our dates are not in a specific format, so we asked the function to automatically infer the format.

Hence, we need to be sure all our dates converted well since we asked the function to infer the format. A simple mistake in data entry could lead to a wrong inference.

To do this, we will plot a barchart of the months and ensure that the highest value is 12.

As you can see, the highest day is 12, so we are good to go.

We can also do the same for the days of the months. The highest value should be 31, and there should be a dip there since not all days of the month have 31 days.

Nice, our hypothesis is correct. Seems like we got this one right.

4. Scaling and Normalization: Let us take them one by one:

Scaling is important if one wants to specify that a change in one quantity is not equal to another change in another. This is particularly useful if you want to work with algorithms that treat a change in a numerical feature as an equal change in another numerical feature. Scaling ensures that just because some features are big it won’t lead to using them as a main predictor. For instance, if the age and the salary of a person are being used in prediction, some algorithms will pay attention to the salary more because it is bigger. Also, an increase in the age by 1, will be considered similar to an increase in the salary by 1. This does not make sense, so it makes sense to scale your data.

Normalization basically involves transforming or converting your dataset into a normal distribution. This is useful when you want to use algorithms that work well with normal distributions. Some algorithms like SVM converge far faster on normalized data, so it makes sense to normalize your data to get better results.

We would normalize a data and visualize it before and after normalization.

The data to be used here contains information about over 300 000 kickstarter projects.

Nice, we can see that our data now resembles a normal distribution

Conclusion

There are several other forms of data cleaning that might need to be done on data. However, this is a good primer for absolute beginners. The need to properly arrange and tidy up your data before the formulation of any model cannot be overstated. I hope this article has helped spark that interest in you! :)

Very well explained code examples are in this github repo.

PS: For further reading on Handling missing data — you can read this brilliant article.

Get Best Software Deals Directly In Your Inbox

A Practical Guide To Data Cleaning For Beginners was originally published in Coinmonks on Medium, where people are continuing the conversation by highlighting and responding to this story.

your eyes are a kaleidoscope of everything
i want to get lost in them
piece by piece
till i become you

maybe then i would understand your intricacies
maybe then i would fathom your awkwardness
contemplating you is like thinking in eight
dimensions

is it the parabolic curve of your lips and how you never stutter?
or the wisdom i garner with every word you
utter?

is it the smell of your perfume that always
lingers?
or the sensation that your touches trigger?

is it your immaculate smile and how it never
fails to dazzle?
or how you hold me at ransom without flinching a muscle?

is it how i’m still addicted to you with every
dose?
or how your flaws complement you? only heaven knows

contemplating you is a journey
down an undulating road
that meanders
into oblivion