09 Ultimate Guide to Least Squared Error
This is a short story created to complement another story that will be published in the future. In fact, I have written an article titled “Ultimate Guide to Least Square Estimation” which is about deriving the formula of beta hat which is the optimal solution in linear regression based on projection in maths, which is, however, not related to this very article since this is going to talk about why the sum of squared error can be minimised, so, this is also the theorectical background supporting the method of finding the leaset square estimate in previous article.
Anyway, this is going to be a short complementary easy essay!
Let’s start.
Then
If B is symmetric (B’ = B), above second derivative becomes 2B.
Get WONG Hoi Ching, Stephen’s stories in your inbox
Join Medium for free to get updates from this writer.
In linear regression least square error is the sum of squared error 𝜀⃗ ’ 𝜀 ⃗ = ( 𝑦⃗ − X𝛽⃗ )’ ( 𝑦⃗ − X𝛽⃗ ) where 𝜀ᵢ is the difference between the i-th observation and the i-th predicted value by the regression model. Notice that the i-th entry in X𝛽⃗ is the dot product of the i-th row in X and the vector 𝛽⃗, it is a linear combination of the 1×1 entries in 𝛽⃗, thus the i-th entry in 𝜀 ⃗ is 𝜀ᵢ which is a function of all parameters.
Generally speaking, each 𝜀ᵢ is a deviation of the predicted value from the ground truth, this deviation is controlled by the parameters, better parameters lead to lower deviation. Therefore, every entries in 𝜀 ⃗ should be a function of parameters, at the end 𝜀⃗ ’ 𝜀 ⃗ is also a function of all parameters!
So, why can 𝜀⃗ ’ 𝜀 ⃗ be minimised?
Notice that
Thus
Finally, we have the Hessian positive semi-definite, which suggests that the function (i.e., the sum of squared error) can be minimised with respect to 𝛽⃗.
