Powerful library for creating and manipulating n-dimensional arrays, along with linear algebra operations and random value generation.
!pip3 install numpy #Download numpy if needed. np.__version__ # check verison of the numpy library; '1.23.5'
import numpy as np # importing the numpy library arr = np.array([1, 4, 2, 5, 3]) #numpy array created | array([1, 4, 2, 5, 3]) arrf = array([1, 4, 2, 5, 3], dtype = float) # will shape items as float arr.ndim # will give the dimension type(arr) # will show type of array arr.shape # it will give the rows and columns arr.size # will give the no of items in the array
Difference between numpy array and list
Data type should be same of all the elements in a numpy array whereas it can be different in case of lists.
a= [1,4,'2',5,False] # its a list sample_numpy_array = np.array([1, 2, 3, 4, 5, 6]) # numpy array
Broadcasting: NumPy operations are usually done on pairs of arrays on an element-by-element basis. If we add an element of string data type in a numpy array# then it will upcast the data type to string of all the elements. https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html
# Typecasting (list to array) a = [1,2,3,4,5,6] y = np.array(x) # this will convert into array type(y) # will display the datatype y*2 # now it will do broascasting y>10 # will give true or false as result depending upon if individual item > 10 or not y[y>10]
# Create a matrix using numpy a = [[1,2,3], [4,5,6], [7,8,9]] a #call it ; [[1, 2, 3], [4, 5, 6], [7, 8, 9]] a = np.array(a) a #array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
a[0][1] # call individual 2 d elements; 2
# Create a 3x3 array of random integers in the interval [0, 10) np.random.randint(0, 10, (3, 3)) # array([[1, 1, 9], [4, 7, 8], [2, 3, 9]]) np.random.randint(0, 10, (3, 3)) # array([[3, 5, 7], [4, 3, 9], [7, 3, 3]])
np.random.seed(0) # used to generate same matrix again np.random.randint(0, 10, (3, 3)) #array([[5, 0, 3], [3, 7, 9], [3, 5, 2]]) np.random.seed(0) np.random.randint(0, 10, (3, 3)) # array([[5, 0, 3], [3, 7, 9], [3, 5, 2]]) d = np.random.random((2,2)) # create an array with random integers
# Create a 2 X 10 integer array filled with zeros np.zeros((3,4), dtype=int) # array([[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]) # Create a 4 X 5 integer array filled with zeros np.zeros((4,5), dtype=int)# array([[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]])
f = np.arange(0,50,2) # create a sequence of integers with specific values g = np.linspace(0,5,10) # evenly spaced sequence at a specific interval; # linearly spaced vector from 1-5, total 10 numbers.
ary = np.array([[2,3,4,5],[6,5,4,3],[4,5,8,9]]) # reshapeing a 3*4 array to 2*2*3 array newary = ary.reshape(3,2,2,order = 'c') # Will reshape but keep the size same
The SciPy library depends on NumPy, which provides convenient and fast N-dimensional array manipulation. It Collection of tools for scientific computing, including advanced calculus functions and optimization algorithms.
import scipy # import the scipy library py.__version__ # check the version of scipysci; '1.10.0'
# import the derivative from scipy from scipy.misc import derivative # define the function def my_function(x): return x**2 + x + 1 # calculate the first derivative of the function at x = 2 derivative(func= my_function, x0=3) #7.0 ;Function: f(x) = x**2 + x + 1= 2x+1 derivative(func= my_function, x0=3,n=2) #Now, calculate the second derivative;2.0
PERMUTATION AND COMBINATIONS # COMBINATIONS C(n,r) = n! / (r! * (n-r)!) from scipy.special import comb # Value of 4C2 com = comb(4, 2) print(com) #6.0
# PERMUTATIONS: Value of 4P2 P (n,r) = n! ÷ (n-r) from scipy.special import perm per = perm(4, 2) print(per) #12.0
LINEAR ALGEBRA from scipy import linalg # import linear algebra module and numpy import numpy as np matrix = np.array([[1, 5, 2], [3, 2, 1], [1, 2, 1]]) # square matrix matrix # array([[1, 5, 2], [3, 2, 1], [1, 2, 1]])
linalg.det(matrix) #-1.9999999999999998 # pass values to det() function # inverse of a matrix linalg.inv(matrix) # array([[ 0. , 0.5, -0.5], [ 1. , 0.5, -2.5], [-2. , -1.5, 6.5]])
Pandas
Pandas: Open source library for data manipulation and analysis. We can read and write differnt formats of file like CSV, JSON, EXCEL, HTML, etc.
We can summarize the data. We can filter and modify the data based on multiple conditions. We can merge multiple files.
If you are using jupyter with python3: !pip3 install pandas pd.__version__ #'1.5.3' ;# check the version of the pandas library
import pandas as pd # import the library data = pd.read_csv('big_mart_sales.csv') # read the file data.head(3) # view the data data.shape # shape of the data; (8523, 12) data.columns # check the column names; Index(['Item_Identifier', 'Item_Weight', 'Item_Fat_Content', 'Item_Visibility', ..., 'Item_Outlet_Sales'], dtype='object') data.dtypes # check the data types in the columns data.isna().sum() # check the null values in the data data.describe() # summary of the data Eg: means, std, count, min, max, 25%, 75%,50%, max If you got an error while running the above cell, import it by using the following command.
MATPLOTLIB
Core data visualization library in Python for creating static, interactive, and publication-quality plots and charts.
!pip3 install matplotlib # If you are using jupyter with python3 matplotlib.__version__ # '3.7.0'
import matplotlib.pyplot as plt # import matplotlib for plotting %matplotlib inline # draw charts import numpy as np #for numbers import pandas as pd # data setup
# LINE PLOT x = np.linspace(0, 10, 1000) # create random data plt.plot(x, np.cos(x), label='Cosine'); # plot
It provides high-level commands to create a variety of plot types useful for statistical data exploration, and even some statistical model fitting.
It provide choices for plot style and color defaults, defines simple high-level functions for common statistical plot types, and integrates with the functionality provided by Pandas DataFrames.
import seaborn as snssns.set() data = np.random.multivariate_normal([0, 0], [[5, 2], [2, 2]], size=2000) # create random data data = pd.DataFrame(data, columns=['x', 'y']) # DENSITY PLOTS for col in 'xy': sns.kdeplot(data[col], fill=True) # shown below for col in 'xy': sns.histplot(data[col],fill=True) # stacked plot with X and Y overlap for col in 'xy': sns.displot(data[col]) # separate X and Y
It provides simple and efficient tools for pre-processing and predictive modeling
Steps to build a model in scikit-learn.
Import the model
Prepare the data set
Separate the independent and target variables.
Create an object of the model
Fit the model with the data
Use the model to predict target.
import sklearn # import the scikit-learn library !pip3 install scikit-learn # If you are using jupyter with python3
sklearn.__version__ # check the version; '1.2.1'
# read the data set and check for thre null values import pandas as pd data = pd.read_csv('Datafile.csv') data.isna().sum() # check the null values. from sklearn.impute import SimpleImputer # import the SimpleImputerfor imputing the missing values
# First we will create an object of the Imputer and define the strategy. impute_weight = SimpleImputer(strategy= 'mean') # create the object of the imputer for Item_Weight and Outlet_Size impute_size = SimpleImputer(strategy= 'most_frequent') # fit the Item_Weight imputer with the data and transform
# Fit the objects with the data. impute_weight.fit(data[['Item_Weight']]) data.Item_Weight = impute_weight.transform(data[['Item_Weight']]) impute_size.fit(data[['Outlet_Size']])
# Transform the data data.Outlet_Size = impute_size.transform(data[['Outlet_Size']]) data.isna().sum() # check the null values again
Now, after the preprocessing step, we separate the independent and target variable and pass the data to the model object to train the model.
If we have a problem in which we have to identify the category of an object based on some features. For example whether the given picture is of a cat or a dog. These are classification problems.
Or, if we have to identify a continous attribute like predicting sales based on some features. These are Regression Problems
# some of the very basic models scikit learn has. from sklearn.linear_model import LinearRegression from sklearn.tree import DecisionTreeClassifier from sklearn.pipeline import Pipeline
StatsModels
Statsmodels is a Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests, and statistical data exploration.
!pip3 install statsmodels # If you are using jupyter with python3: statsmodels.__version__ # check the version ; '0.13.5'
import statsmodels.api as sm # statsmodels from sklearn import datasets import numpy as np import pandas as pd data = datasets.fetch_california_housing() features = pd.DataFrame(data.data, columns= data.feature_names) # create a dataframe features.head() target = pd.DataFrame(data.target, columns=["MEDV"]) # target in another DataFrame target.head()
CREATE A LINEAR REGRESSION MODEL X = features # define X and Y y = target["MEDV"]
BUILD A MODEL
# Note the difference in argument order model = sm.OLS(y, X).fit() predictions = model.predict(X) # make the predictions by the model model.summary() # Print out the statistics
Visualizations with Matpotlib
- Setting up
# importing required libraries import numpy as np import pandas as pd
# importing matplotlib import matplotlib.pyplot as plt
# display plots in the notebook itself %matplotlib inline
# Matplotlib basics height = [150,160,165,185] # list of height weight = [70, 80, 90, 100] # list of weight
plt.plot(height, weight); # plot() method to draw plot plt.title("Relationship between height and weight") # add title plt.xlabel("Height") # label x axis plt.ylabel("Weight") # label y axis
# draw multiple lines calories_burnt = [65, 75, 95, 99] # list of calories_burnt plt.plot(calories_burnt) # draw the plot for calories burnt plt.plot(weight); # draw the plot for weight
plt.legend(labels=['Calories Burnt', 'Weight'], loc='lower right'); # add legend in the lower right part of the figure plt.xticks(ticks=[0,1,2,3], labels=['p1', 'p2', 'p3', 'p4']); # set labels for each of these persons plt.figure(figsize=(15,5)) # figure size in inches plt.plot(calories_burnt, marker= 'o') # draw the plot with specific mark plt.plot(weight, marker = '*') # draw the plot with specific mark plt.plot(weight,'y--', marker='*') # draw the plot with specific color and dotted line/ "go" --> green circles
# Multiple plots # subplots() method to create more plots; nrows and ncols give subplot no. fig, ax = plt.subplots(nrows=2, ncols=2, figsize=(6,6))
ax[0,0].plot(calories_burnt,'go') # plot on 0 row and 0 column ax[0,1].plot(weight) # plot on 0 row and 1 column
# set titles for subplots ax[0,0].set_title("Calories Burnt") ax[0,1].set_title("Weight")
# set ticks for each of these persons ax[0,0].set_xticks(ticks=[0,1,2,3]); ax[0,1].set_xticks(ticks=[0,1,2,3]);
# set labels for each of these persons ax[0,0].set_xticklabels(labels=['p1', 'p2', 'p3', 'p4']); ax[0,1].set_xticklabels(labels=['p1', 'p2', 'p3', 'p4']);
# Extra notes: This method creates two objects: figure and axes which we store in variables fig and ax. # also can create common axis using sharex and sharey parameters like
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,6), sharex=True, sharey=True) ax[0].plot(calories_burnt,'go') # plot on 0 row and 0 column/1D parameters used
Graphs with Seaborn
# importing required libraries import seaborn as sns sns.set() sns.set(style="darkgrid")
import numpy as np import pandas as pd # importing matplotlib import matplotlib.pyplot as plt %matplotlib inline import warnings warnings.filterwarnings("ignore") plt.rcParams['figure.figsize']=(10,10)
Load dataset
data_BM = pd.read_csv('dataset/datafile.csv') # read the dataset data_BM = data_BM.dropna(how="any") # drop the null values data_BM.head() # view the top results
1. Line Chart
<<With Matpotlib>> #line chart to denote the mean price per item. Line chart is the default plot used with time series data price_by_item = data_BM.groupby('Outlet_Establishment_Year').Item_Outlet_Sales.mean() price_by_item
# mean price based on item type price_by_item = data_BM.groupby('Item_Type').Item_MRP.mean()[:10]
x = price_by_item.index.tolist() y = price_by_item.values.tolist()
plt.figure(figsize=(14, 8)) # set figure size plt.title('Mean price for each item type') # set title plt.xlabel('Item Type') # set axis labels plt.ylabel('Mean Price')
plt.xticks(labels=x, ticks=np.arange(len(x))) # set xticks ax[0].set_xticks(ticks=[0,1,2,3]); plt.plot(x, y, marker = 'o');
<<With Seaborn>> # line plot using relplot directly with relplot() by setting kind="line" sns.lineplot(x="Item_Weight", y="Item_MRP",data=data_BM[:50]); # or # hue semantic # different line plots for different categories of the Outlet_Size sns.lineplot(x="Item_Weight", y="Item_MRP",hue='Outlet_Size',data=data_BM[:150]);
Press enter or click to view image in full size
2. Bar Chart
<<With Matpotlib>> sales_by_outlet_size = data_BM.groupby('Outlet_Size').Item_Outlet_Sales.mean() # used for categorical variables. # sales by outlet size sales_by_outlet_size.sort_values(inplace=True) # sort by sales
x = sales_by_outlet_size.index.tolist() y = sales_by_outlet_size.values.tolist()
plt.xlabel('Outlet Size') # set axis labels plt.ylabel('Sales') plt.title('Mean sales for each outlet type') # set title plt.xticks(labels=x, ticks=np.arange(len(x))) # set xticks plt.bar(x, y, color=['red', 'orange', 'magenta']);
<<With Seaborn>> # barplot function sns.barplot(x="Item_Type", y="Item_MRP", data=data_BM[:5])
3. Histogram
<<With Matpotlib>> # Distribution of Item price, plotted within a range against its frequency. # Very commonly occurring graphs in probability and statistics and form the basis for various distributions . plt.title('Item MRP (price) distribution') # title plt.xlabel('Item_MRP') # xlabel plt.ylabel('Frequency') # ylabel plt.hist(data_BM['Item_MRP'], bins=50, color='lightblue'); # plot histogram
<<With Matpotlib>> # Distribution of sales. Box plot shows the three quartile values of the distribution along with extreme values. # Boxplot corresponds to an actual observation in the data. data = data_BM[['Item_Outlet_Sales']] red_diamond = dict(markerfacecolor='r', marker='D') # create outlier point shape plt.title('Item Sales distribution') # set title plt.boxplot(data.values, labels=['Item Sales'], flierprops=red_diamond); # make the boxplot
<<With Matpotlib>> #Density distribution of Item weights and Item price data = data_BM[['Item_Weight', 'Item_MRP']] fig, ax = plt.subplots() # generate subplots plt.xticks(ticks=[1,2], labels=['Item Weight', 'Item MRP']) # add labels to x axis plt.violinplot(data.values); # make the violinplot
<<With Matpotlib>> #Relative distribution of item weight and it's visibility # It depicts the distribution of two variables using a cloud of points. plt.xlabel('Item_Weight') # set label of axes plt.ylabel('Item_Visibility')
<<With Seaborn>> # scatter plot using relplot() sns.relplot(x="Item_MRP", y="Item_Outlet_Sales", data=data_BM[:200], kind="scatter"); # or # hue Semantic sns.relplot(x="Item_MRP", y="Item_Outlet_Sales", hue="Item_Type",data=data_BM[:200]);
7. Bubble Plots
<<With Matpotlib>> # Relative distribution of sales, item price and item visibility # Bubble plots let you understand the interdependent relations among 3 variables. plt.xlabel('Item_MRP') # set label of axes plt.ylabel('Item_Outlet_Sales') plt.title('Item Outlet Sales vs Item MRP (price)') # set title plt.scatter(data_BM["Item_MRP"][:100], data_BM["Item_Outlet_Sales"][:100], s=data_BM["Item_Visibility"][:100]*1000, c='red');
You can also create plots based on category in seaborn.
We have created scatter plots for each Outlet_Size
# subplots for each of the category of Outlet_Size sns.relplot(x=”Item_Weight”, y=”Item_Visibility”,hue=’Outlet_Size’,style=’Outlet_Size’,col=’Outlet_Size’,data=data_BM[:100]);
Press enter or click to view image in full size
2. Advance categorical plots in seaborn
Get Manpreet Buttar’s stories in your inbox
Join Medium for free to get updates from this writer.
The default representation of the data in catplot() uses a scatterplot.
When you generalize joint plots to datasets of larger dimensions, you end up with pair plots. This is very useful for exploring correlations between multidimensional data, when you’d like to plot all pairs of values against each other.
iris = sns.load_dataset("iris") iris.head() sns.pairplot(iris, hue=’species’, height=2.5); # Visualizing the multidimensional relationships among the samples is as easy as calling sns.pairplot:
