Supervised machine learning algorithm — Linear regression

We might have heard the linear equation to form a line y=mx+c during our schooling days. It is used to plot a linear line on a graph.

For example, consider the data plotted in a graph below.

where ,

X : Value of X

m : Slope ( coefficient ) of the line. It determines how much the line goes up or down with increase in X as it gets multiplied with X value.

c : Constant value also called as y-intercept. It is the point in which the line coincides with the y-axis ( i.e when x=0). In this case, as seen from graph the line coincides with y-axis at y=1 when x=0.

y : Value of y which is result of (m*x) + c.

Linear Regression

So, what does the above explanation regarding linear equation have to do with machine learning? Turns out this concept is used in supervised machine learning to predict the target value (y) given the input values (X,m,c). In this article, we will see about basics of linear regression and its implementation using python.

In practical cases however there will be more features needed to predit accuracte output. If there are 2 features, the formula will be

y =X₁m₁+X₂m₂+c

and if there are n features, the formula will be

y =X₁m₁+X₂m₂+X₃m₃+….Xₙmₙ+c

as the coefficient (m) is referred to as weights(w) and the y-intercept(c) is referred to as bias(b), in machine learning this formula is often written as

y =X₁w₁+X₂w₂+X₃w₃+….Xₙwₙ+b, However their meaning does not change.

Some of the pratical application of linear regression are,

1. analyzing the impact of study time on exam scores ( which we will be seeing in this article ).
2. predicting sales based on advertising spend.
3. estimating house prices based on features like square footage.
4. forecasting weather patterns.
5. evaluating the influence of fertilizer on crop yield.

The Linear regression model works by following steps.

1. Find coefficient and bias of the provided data.
2. Find the linear line that fits the provided data.
3. Fine tune by minimizing the error to find best fit line.
4. Use the best fit line to predict the output for the given input.

1. Find coefficient and bias of the provided data.

• The coefficient (slope) (w) represents the effect of each feature on the target variable, and the bias (intercept) (b) represents the value of the target variable when all features are zero.

2. Find the linear line that fits the provided data

The model attempts to fit a linear equation (as mentioned before) of the form: ŷ =X₁w₁+X₂w₂+X₃w₃+….Xₙwₙ+b where:

• ŷ (read as y hat) is the predicted target value,
• X₁​,X₂, X₃, Xₙ are the features,
• w₁ ,w₂,w₃,wₙ are the coefficients (weights),
• b is the bias (intercept).

The goal is to find the values for w₁,w₂,w₃,wₙ and b that best fit the data.

3. Fine tune by minimizing the error to find best fit line.

In general, the available data is split into training data and test data.

Get Santhosh Sivan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Training data is used to test the model and the Test data is used to test the accuracy of the model. The machine learning model will learn from the existing available data ( It can be easily done using scikit learn which will be discussed below ).

Error can be written as

ŷ (y hat)- predicted target value , y — actual target value , i — ith data point.

The above equation , however may result in positive or negative error. As we only want to know how much the error is, we will simply square it to make the error a positive value regardless of whether ŷ-y result in positive or negative value.

So, from the above equation, we can get error of one data point. To get error from all the data points J(w,b) , we add all the errors and then divide it by total number of data points(m) . It’s called as Mean square error.

We will be using the above formula as error in this article. However, You may see sometimes the error is also written as follows. To make the formula simple when performing gradient descent (will be discussed in future article) to minimize the error by differentiating it.

4. Use the best fit line to predict the output for the given input.

• Once the model has been trained (i.e., the coefficients and bias that produce the least error have been determined), you can use the best-fit line to make predictions for new, unseen data.
• The model predicts the target value (ŷ) for any given set of input features (X₁​,X₂, X₃, Xₙ​) by applying the learned coefficients and bias to the linear equation.

Enough theory, let’s see code to perform the above steps.

we will be using this dataset for the tutorial. There are some data preprocessing steps needs to be done inorder to perform linear regression and display the plot. I will skip coding required for those steps. However you can take a look on my github if you are interested.

The data when plotted looks like this. There are almost 100 datas and therefore it looks crowded.

from sklearn.linear_model import LinearRegression 

model = LinearRegression()
model.fit(X_train, y_train)

print(f"Intercept: {model.intercept_}")
print(f"Coefficient: {model.coef_[0]}")

y_pred = model.predict(X_test)
results = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred})

Printing result aquired on the above code results in the following image

When the Actual and predicted values are visualized in bar plot, It can be seen that the predicted values are pretty accurate.

from sklearn.metrics import mean_squared_error

mse = mean_squared_error(y_test, y_pred)

print(f"Mean Squared Error: {mse}")

# prints "Mean Squared Error: 13.881520927483525"

There are many techniques like feature engineering, Hyperparameter tuning and much more to minimize the error. However this might give you a good image on how to perform simple linear regression.

Finally, the linear line that fits the test data looks like this which fits the data pretty good.

This line will be used to predict output for new unknown datas in future. For example as shown in the figure, if a student studies for 5 hours, that student is likely to score around 50 marks.

Finally , If you add one more feature (coffe per day) to the above, the plot will look like this 3D. However, there are some differences,

1. the steps to perform Linear regression remains same. It only has to be done 2 times to find 2 seperate coefficients for Study hours and Coffee per day.
2. The linear regression line gets converted to plane because of increase in dimension (feature)

Limitations:

1. Non-Linearity: If the actual relationship between variables is non-linear, a linear model won’t capture the true pattern. For example, a quadratic or exponential relationship wouldn’t be well-represented by a straight line. This could lead to underfitting, where the model fails to make accurate predictions.
2. Inflexibility: Linear models can’t adapt well to complex data with intricate patterns or multiple peaks. For example, data showing a U-shape, periodic behavior, or growth trends over time would not be adequately modeled using a linear assumption.
3. Over-Simplification: Sometimes, the assumption of linearity oversimplifies the underlying data, leaving out important features that may influence the output in more complex ways.

Overcoming the Limitations:

If the relationship between variables is non-linear, there are several approaches to model such data. Following are some of the examples

• Polynomial Regression
• Decision Trees/Random Forests
• Neural Networks

In conclusion, linear regression remains one of the most widely used and fundamental techniques in statistical modeling and machine learning. Its simplicity, interpretability, and efficiency make it an ideal choice for understanding the relationship between variables and making predictions. Understanding the assumptions, limitations, and proper application of linear regression is crucial for making informed decisions in data analysis and predictive modeling.