SHAP

SHAP (an acronym for SHapley Additive exPlanations) uses the explanations on shapely values — measures of contributions each feature has in the model. The idea is to get insights into how the machine learning model works. In python, there are a variety of model that can help achieve this. But here, we will consider only SHAP library.

SHAP can be applied for multiple forms data prediction. Here is a code snippet for a binary classification problem. There are more options to visualize SHAP results. The most common one is a summary plot which is a neat way to visualize every feature’s importance and their impact on prediction.

You get a nice visualization for the impact of each feature:

LIME

LIME is short for Local Interpretable Model-Agnostic Explanations. Each part of the name reflects something that we desire in explanations. Local refers to local fidelity — i.e., we want the explanation to really reflect the behavior of the classifier “around” the instance being predicted. This explanation is useless unless it is interpretable — that is, unless a human can make sense of it. Lime is able to explain any model without needing to ‘peak’ into it, so it is model-agnostic. We now give a high level overview of how lime works. More details can be found in the famous by Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin.

Here is a code snippet on how to use LIME for text classification. I created a probability predicting function for a given text. LIME identifies key words that can predict the sentiment of a given text.

Here is the outcome of the code:

In conclusion, SHAP and LIME provide explanation to any type of model. I cannot make recommendations as to which to use. Because it is largely dependent on the details of explainability and the type of visualization that either of these two great libraries can provide.

Frequentists’ Approach to Linear Regression

Linear regressions are often presented as well drawn problems from a uniformly distributed population. This assumptions conforms well with the central limit theorem. In brief, the classical representation of linear regression looks something like this:

Where ε and y terms are sampled from a uniform distribution:

The terms X and y represent the feature matrix and the target that we want to estimate with our model. Classical regression uses Ordinary Least Squares (OLS) where a single fit is calculated that minimizes the L2 loss function (also called residual sum) represented as follows.

Finding the β estimates (β hat) values that can minimize the loss could be computed using the following equation.

Inverting a matrix is computationally costly when the sizes are large. Some machine learning tools take different route in doing this. Scikit learn uses iterative gradient descent approach while statsmodel employs a direct method, techniques probably like Moore–Penrose inverse that avoid matrix singularity.

The flaw of this method is that it does not account for priors and updating the posterior, unlike the Bayesian statistics. In addition, in the classical method, only one fit is given as a solution to predict future cases which is unrealistic in the real world. The other shortcoming is that the β terms predicted using OLS have high variance, meaning that small changes in x can lead to high change in the coefficients, another undesirable effect.

The Bayesian Approach

Unironically, the Bayesian approach uses Bayes’ theorem while also considering the central limit theorem. The following picture shades light on what terminologies are used in describing Bayesian statistics.

When used for linear regression, the equation looks like this.

MAP stands for Maximum a posteriori estimation, which is to say maximizing the posterior probability. The direct interpretation is that what are the β estimates that can maximize the prediction, given features and target values. The conditional probability can be replaced its Bayesian equivalent.

The denominator has no β term thus can be ignored which reduces our equation to this.

Taking the log,

Now, let’s treat each term separately. Just as the frequentist approach, the conditional probability of y given β is assumed to be Gaussian. Therefore, the probability becomes:

Taking the log of the function,

Now for the second part. The prior in this case is hard to generalize. Making assumptions on βs without targets really does not make sense. But we are going to take a leap of faith here and claim that β is also a normally distributed with 0 mean and τ standard deviation as such.

Then the probability becomes:

Taking the log, the equation is changed to:

Putting it all together, our MAP becomes:

Since all the terms are negative, the ARGMAX is changed to ARGMIN with positive values. The constants can be taken out and the function is written in a vectorized form:

Again substituting σ²/τ² by some constant λ, the function now becomes:

For those familiar with classical linear regression, this is what is called Ridge regression (also called L2 regularization). Better predictions can be made by penalizing the β estimators by some constant. The Bayesian explanation of this is,when λ is increased, τ is reduced, squeezing the β normal distribution towards the center. This leads to smaller β estimations.

Let’s take a few steps back and considered a different distribution instead, like a Laplace distribution instead. The β now becomes:

The probability and log of probability are now changed to:

Now the MAP function becomes a Lasso or L1 regularization:

It is important to establish the theoretical background of some regression techniques the reasoning of which is strongly linked to Bayesian approach.

Computing Bayesian Linear Regression

Now that the mathematical formulation is covered, how do we implement Bayesian linear regression. Remember the vectorized forms of β MAP estimates are summations of vectors. This means that methods used for integrating continuous functions for unknown functions can be applied here. The most common one is usually used in tandem with Bayesian statistics is the Markov Chain Monte Carlo (MCMC) method. Samples are drawn from likelihood and prior to compute the MAP distribution. Here it is important to note that unlike the classical linear regression, β is computed as a distribution, and not as a vector of coefficients.

Libraries and code samples

PyMC3 provides excellent tools for MCMC estimation. Some notable GitHub repositories that help to understand the implementation of PyMC3 are written by Michael Pyrcz (video lectures also found here) and a blog by Will Koehrsen.

Conclusion

Bayesian statistics has wide variety of applications. But in this blog, I tried to show how some of the methods used in the frequentist approach have Bayesian background. Although not described in detail, interested readers are encouraged to explore the resources provided here and other materials in relation to Bayesian linear regression. The blog might be updated with complex applications of Bayesian regression, hopefully soon. Finally, I would like to thank Ritvikmath for his excellent video on how to derive necessary equations for Bayesian linear regression.

Tabulated data is the bread-and-butter of any data scientists. Virtually all data lakes and warehouses use data tabulation to wrangle data and extract key features to process. One of the most commonly used data tabulation methods are Dataframes. They afford flexibility in collecting and cleaning data from delimited text files, spreadsheets and queries from database. The most commonly used dataframe is Pandas, a python package which performs sufficiently well for limited data. However, what if the data is too large for Pandas to handle but also too small for distributed file management systems such as Spark? Polars seeks to bridge that gap.

Polars is written in Rust for a more robust performance and Apache Arrow(2) as memory model. PyPolars (currently updated to just Polars) is a python wrapper around Polars. It has API similar to Pandas which makes it a lot easier to transition.

Installation

Installing Polars is simple. Polars can be installed with pip as follows:

pip install polars

Unfortunately, Polars is not available on Anaconda for now. Updates will be made if this changes.

Disclaimer: Create and activate a new environment to install Polars as the stable version has not been released yet.

Importing files with Polars

Importing Polars is as smooth as importing Pandas.

import polars as pl

Let’s read one of the datasets offered with Polars.

df = pl.read_csv('my_data.csv')

The latest version of Polars ( v=0.8.10) supports csv, ipc, json , parquet and sql.

To check if your data is loaded, you can use head just as Pandas.

df.head()

Similarly, the last 10 entries, the shapes and type of the dataframe can be checked using the following codes:

df.tail(10)

df.shape

type(df)

The current version does not provide options to import compressed delimited files or read the first n-rows of the file. For a loaded Polars dataframe, describe and dtype provide information on the types of data by columns. Column names can be checked with df.columns .

Features in df.describe() can be passed to Pandas for a better display with:

df.describe().to_pandas()

Accessing Table Elements

Polars can access table rows directly through row index similar to pandas.DataFrame.iloc function as follows:

df[:10] #access the first ten rows

Columns can be directly referenced by name.

df['name'] #find column 'name'

Indices can be selected by passing list of index to the dataframe

df[[1,4,10,15],:]

Slicing through indices can be done using the built-in function slice

df.slice(0,5) #Slices the df beginning at index 0 and 5 rows down

Polars also enables slicing dataframe with conditional boolean.

df[df['sale']>=10]

Polars also has.value_counts, .unique and .dtypes functions

df['name'].value_counts() #returns unique values with numbers of occurrences

df['name'].unique() #returns a list of unique values in the column

df.dtypes() #returns the data types

Groupby and sort is also supported by Polars:

df.groupby('country').sort('products',reverse=True)

apply() and map() in Polars

Similar to Pandas, Polars supports .apply() and .map() functions.

df['values'].map(lambda x: 'high' if x>=5 else 'low')

df['values].apply(my_func)

The only caveat is that dictionaries cannot be passed to these functions.

mydict = {k:v for v,k in enumerate(df['catagories'].unique())}

df['catagories'].map(mydict) #This will throw an error

Plotting with Polars

Polars, just as Pandas, has built-in plotting functions. It also allows integration with Matplotlib and Seaborn.

import matplotlib.pyplot as plt

fig,ax=plt.subplots(figsize=(10,5))

ax.hist(df['name'])

plt.show()

Eager and Lazy APIs

Polars (by extension, Pandas) employs eager evaluation by default, meaning that functions are mapped to each data in real time. But as the table size grows larger, lazy implementation is more efficient in loading and transforming selected values. A sample implementation looks like as follows:

from polars.lazy import *

lazy_df = pl.scan_csv('large_dataset.csv',ignore_erros=True)

filt_lazy_df = (lazy_df.filter(col('name').str_lengths() > 5).filter(col('country').str_contains(r'US-Germany-France').is_not()))

#Filter names longer than 5 characters that are not in US, Germany or France

Similar to Spark’s lazy implementation, filt_lazy_df is evaluated when collect function is called.

lazy_df.collect()

As mentioned earlier, the most appealing aspect of Polars is its ability to transform large datasets. h2oai has tables of benchmark performances between different dataframes. According the benchmark, groupby applied on a 1,000,000,000x9 50GB file was reported as follows:

Final Thoughts

Polars has promising prospects with its fast performances on very large datasets that may be too big for Pandas. It has similar implementation to Pandas, supports mapping and applying functions to series in dataframes. Plots are easy to generate and integrate with some of the most common visualization tools. Moreover, it allows for lazy evaluation without Resilient Distributed Datasets (RDDs).

Overall, Polars can provide data scientists and enthusiasts with better tools to import data into a dataframe. There are a lot functionalities that Pandas can do which are not currently present on Polars. In such cases, casting the dataframe to Pandas is highly recommended.

The intention of this blog is to introduce the data science community to Polars in a rather crude manner. Detailed documentations and other resources are provided below for more in-depth use cases and implementation of Polars. Ritchie Vink, the creator of Polars, wrote a detailed blog on how Polars operates.

Additional Resources

Polars User Guide

Introduction to Polars: One of the fastest Dataframe library at the moment

Polars: The fastest DataFrame library you’ve never heard of

PyPolars Python Tutorial (Data Analysis with PyPolars & Pandas)

Or any other blogs or tutorials on Polars