Parametric Regression
In our previous article, we explored the differences between Parametric and Non-Parametric models.
“Parametric models have a fixed number of parameters learned from training”. This means we must:
- Need a Loss Function: Since we’re “learning” parameters, we need a way to measure what “good learning” looks like. How else would we know if our slope and intercept are optimal?
- Must Choose Function Class: We’re committing to a “fixed” structure (linear, quadratic, etc.), so we must decide this upfront.
- Require Optimization: We need a systematic way to find the best parameter values that minimize our loss.
Non-parametric methods like kernel regression work completely differently. They don’t learn fixed parameters during training; instead, they keep the training data and make predictions by directly comparing new inputs to stored examples. Since there are no parameters to optimize, they don’t need a loss function or optimization step for their core algorithm. However, a loss function can still be useful for evaluating how well the model performs, but it’s not essential for making predictions like it is in parametric models.
Linear Regression
Linear regression is like drawing the best-fitting straight line through a scatter plot of data points. We assume the relationship between input features (x₁, x₂, etc.) and the output (ŷ) is a straight line.
Here,
- w₀: The bias term (y-intercept) — where the line crosses the y-axis
- w₁, w₂, …, wₐ: The weights (slopes) — how much each feature influences the prediction
- x₁, x₂, …, xₐ: The input features
Mapping to Standard Linear Algebra Form
In the classic equation ax + by + cz = result:
Linear Regression: w₀×1 + w₁×x₁ + w₂×x₂ + w₃×x₃ + … = ŷ
Classic Form: a×x + b×y + c×z + d×w + … = resultMapping:
- a, b, c, d, … = w₀, w₁, w₂, w₃, … (our weights/coefficients)
- x, y, z, w, … = 1, x₁, x₂, x₃, … (our features, including dummy x₀=1)
- result = ŷ (our prediction)
What are we solving?
We’re finding the weights (w₀, w₁, w₂, …) that best predict house prices based on features (size, rooms, etc.).
Originally, we have this equation: ŷ = w₀ + w₁x₁ + w₂x₂ + … + wₐxₐ
where,
- w₀ = bias (intercept)
- w₁, w₂, … = slopes (weights)
This has two different types of terms:
- A standalone bias term (w₀)
- Feature terms (w₁x₁, w₂x₂, etc.)
The problem is that this mixes a standalone term (w₀) with product terms (w₁x₁, etc.) — Can’t write as a clean vector operation!
The Trick: Adding x₀ = 1
We create an artificial “dummy” feature x₀ and set it to 1.
Before the trick:
- Features: x₁, x₂, x₃, …
- Weights: w₀, w₁, w₂, w₃, …
- Equation: ŷ = w₀ + w₁x₁ + w₂x₂ + w₃x₃ + …
After adding x₀ = 1:
- Features: x₀=1, x₁, x₂, x₃, …
- Weights: w₀, w₁, w₂, w₃, …
- Equation: ŷ = w₀x₀ + w₁x₁ + w₂x₂ + w₃x₃ + …
Sincex₀ = 1we have,w₀x₀ = w₀ × 1 = w₀, so the equation is the same!
Now We Can Use the Dot Product!
We can express this as an inner product (dot product):
x and w both are column vector. To transpose the w, so that we can perform the dot product. When we transpose the column vector w, we get a row vector.This is equivalent to, ŷ = wᵀx
By convention, both w and x are column vectors. However, we cannot directly compute the dot product between two column vectors using matrix multiplication rules. To make the operation valid, we take the transpose of w. This turns w into a row vector wᵀ, so the multiplication wᵀx is well-defined and produces a scalar.
Why This Is Useful: Instead of treating the bias term w₀ as a special case, we can now treat everything uniformly as a single vector operation. This makes the math cleaner and the code simpler. We don’t need separate handling for the bias term.
The key insight is that multiplying w₀ by 1 doesn’t change its value, but it allows us to include it in the vector multiplication!
We can explore this concept interactively with Linear Regression that lets you adjust parameters in real-time.
- We start with
X, the independent variables (our data), andy, the dependent variable we want to predict. - To model this relationship, we need to find the weights w. The goal is to determine w such that the predicted values fit the data well. But not every choice of w will work. We need a systematic way to measure how good or bad our predictions are.
That’s why we introduce a loss function: it quantifies the error between the predicted values and the actual values. By finding the w that minimizes this loss function, we ensure the model fits the data as closely as possible.
Loss Function
A loss function measures how well (or poorly) our model’s predictions match the actual data.
- If the model predicts values close to the true y, the loss is small.
- If the predictions are far from the true y, the loss is large.
The goal in machine learning is to find parameters w that minimize this loss.
For linear regression, the most common choice is the Least Squares Loss.
where:
- yₙ: the true (observed) value for data point n. It is the actual value
- wᵀxₙ: the model’s prediction for that data point
- The difference (yₙ - wᵀxₙ) is the error (residual)
- Squaring ensures errors are always positive and penalizes large errors more heavily
- The factor 1/2 is for mathematical convenience (it cancels during differentiation)
Ideally, we want the model’s prediction (wᵀxₙ) to exactly match the true value (yₙ) for every data point n, which would mean a perfect fit. However, in practice, this is rarely possible because real-world data contains noise and variability. Instead, we aim to find weights w that minimize the overall error across all data points.
This means: pick the weights w* that give the smallest possible error across all data points.
- argmin stands for “argument of the minimum.”
- It asks: “Which value of w makes L(w) as small as possible?”
The Mean Squared Error (MSE) is just the average of the squared errors:
Both L(w) and MSE(w) give the same optimal weights because one is just a scaled version of the other.
MSE (Mean Squared Error) measures the average error per data point, making it independent of dataset size and suitable for fair comparisons. In contrast, SSE (Sum of Squared Errors, L(w)) is the total error, which can be misleading: a good model tested on a large dataset will naturally have a higher SSE than a poor model tested on a few points. By dividing SSE by the number of samples (N), MSE provides a normalized metric for true apples-to-apples comparison of model performance, regardless of dataset size.
Now the key question is: how do we find the optimal weights w* that minimize the error across all data points? Once we determine w*, our model can make predictions using y = wᵀx so that for any new input x, we can generate the best possible estimate of y.
Finding these optimal weights efficiently is crucial because simply guessing or trying all possibilities is impossible in practice. To solve this, we need a systematic optimization method that adjusts w step by step to reduce the loss function.
This brings us to our next topic: Gradient Descent, a powerful algorithm for finding w* by iteratively minimizing the loss function.
Gradient Descent
Gradient Descent is an iterative optimization algorithm used to find the values of parameters (like w) that minimize a loss function.
- The gradient of the loss function tells us the direction of steepest increase (the highest rate of increase).
- By moving in the opposite direction of the gradient, we step toward the minimum of the loss.
- Repeating this process iteratively updates the weights until we (ideally) converge to the optimal w*.
The Trick or rule to remember is that the Gradient always points uphill.
Interpretation:
- The gradient points uphill, that is, towards the right
- The ground goes uphill when you move to the right. That is, as weight increases, the L(w) increases as well.
- So, if we want to minimize L(w), we should move LEFT, the opposite direction of the gradient
- The positive sign (+) tells us: “moving right makes loss worse.”
The Gradient points toward the right, and we move toward the left. That is the opposite of the gradient.
Interpretation:
- The gradient points uphill, that is, towards the left in this case.
- The ground goes downhill when you move to the right. That is, as weight increases, the L(W) decreases.
- So, if we want to minimize L(w), we should move RIGHT, following the downward slope
- The negative sign (-) tells us: “moving right makes loss better.”
The Gradient points toward the left, and we move toward the right. That is the opposite of the gradient.
We define a loss function L(w) to measure error.
The gradient of the loss tells us the slope, i.e., in which direction the loss increases fastest:
So, the gradient points uphill. To minimize the loss, we must go in the opposite direction.
Convergence means the weights w stop changing significantly, because the gradient (slope) is near zero.
Get Prajun Trital’s stories in your inbox
Join Medium for free to get updates from this writer.
Gradient Descent updates the weights iteratively:
Here:
- η is the learning rate (step size),
- ∇L(wt) is the gradient at the current point
- We subtract because we want to move downhill toward the minimum.
Learning rate(η) controls how big each step is:
- If η is too large, → overshoot, divergence, oscillation.
- If η is too small → convergence is very slow.
Modern optimizers (e.g., Adam, RMSprop, AdaGrad) adjust η automatically during training.
Example
1 for the intercept/bias term)X (The Feature Matrix)
w (The weight Vector)
y (The Target Vector)
Xᵀ (The Transposed Feature Matrix)
Step-by-step calculation of the gradient ∇L(w) using the matrix formula and the sample data.
The formula is: ∇L(w) = −Xᵀ (y − Xw)
Step 1: Calculate Predictions (Xw)
Step 2: Calculate Errors (y — Xw)
(For simplicity, let’s call the 5 error values in this vector e₁, e₂, e₃, e₄, and e₅.)
Step 3: Calculate Final Gradient (-Xᵀe)
Finally, we multiply the transpose of the feature matrix (Xᵀ) by the error vector (from Step 2) and add the negative sign.
Step 4: Update the Weights (wₙₑw = wₒld − η · ∇L(w))
Take the current “old” weights and subtract a small portion of the gradient to get the “new” weights.
here,
η = The learning rate ∇L(w)= The gradient vector we just calculated
This gives the new, improved set of weights, and then we repeat the entire process (Steps 1–4) with these new weights. We keep on repeating this iterative process until the loss is minimal, stops changing significantly, or we reach a set number of iterations.
Some Issues with Gradient Descent
1. Zig-zag
- Happens when the learning rate is too large.
- Instead of smoothly descending, the updates overshoot across the valley of the loss function.
- This causes the path to “zig-zag” back and forth, especially in narrow or steep ravines.
- Convergence becomes slow or may even fail if the step size is not reduced.
2. Exploding Gradient
- Occurs when gradients become very large during backpropagation (common in deep or recurrent neural networks).
- Weight updates blow up, leading to numerical instability (NaNs, infinities).
- The model fails to learn and may diverge.
- Causes: poorly initialized weights, large learning rate, long sequences in RNNs.
- Solutions: gradient clipping, better initialization, smaller learning rate, normalization techniques.
3. Vanishing Gradient
- Opposite of exploding: gradients become very small as they are propagated back.
- Weight updates shrink toward zero → early layers stop learning.
- Very common in deep networks with activation functions like sigmoid/tanh.
- Training becomes extremely slow or stalls completely.
- Solutions: ReLU/Leaky ReLU activations, careful initialization, residual connections (ResNets), batch normalization.
The blue curve represents the loss function: L(w) = (w−2)² + 1
This has a minimum at w = 2 (marked with a red dot).
At several example points (w = −2, 0, 1.5, 2.5, 3.5), we:
- Calculate the slope (gradient)
- Draw the tangent line (orange dashed)
- Show an arrow (green) pointing in the descent direction (opposite to the slope)
When the slope is calculated at a given point, the gradient reveals both the steepness of the curve and the direction in which the cost is increasing.
- If the slope is positive, the curve is rising to the right, so the algorithm moves left to descend.
- If the slope is negative, the curve is rising to the left, so the algorithm moves right to go downhill.
This information guides the algorithm on which direction to move. With each step, it gets closer to a nearby valley — whether that’s a local or global minimum. Once the slope becomes nearly zero, the curve is flat, and the algorithm stops — indicating that it has reached a minimum point.
In the 1D case, like the graph shown above, there is only one variable — let’s say w. The slope at any point on the curve is simply the derivative of the cost function with respect to w, written as:
This value is a single number that tells the algorithm whether to move left (if the slope is negative) or right (if the slope is positive) to reduce the cost.
In the multi-dimensional case, which is typical in real-world models like linear regression or neural networks, there are many parameters, such as: w = [w1,w2, w3,…, wn].
Now, instead of one slope, we calculate a gradient vector,
This gradient points in the direction of the steepest increase in all dimensions. To minimize the cost, we move in the opposite direction of the gradient:
Here, η is the learning rate, which controls the step size.
So yes, gradient descent always involves calculating slope — in one dimension, it’s a single derivative, while in multiple dimensions, it’s a vector of partial derivatives (the gradient). It’s simply a higher-dimensional version of the same concept.
A function is convex if, when drawing a straight line between any two points on its curve, that line always stays above or exactly on the curve. We can imagine it like a smooth bowl. If we connect two points on the edge of the bowl with a string, the string will never dip below the surface of the bowl. It has a single global minimum. Gradient descent will always converge to this global minimum, no matter the starting point.
Example: The function f(x) = x².
A function is non-convex if it has regions that curve both upwards and downwards. It can have multiple local minima and saddle points. Gradient descent may become stuck in a local minimum, depending on the initial point.
Example: The function f(x) = sin(3x) + 0.3x².
- We use the linear function (first-order Taylor expansion) to approximate the loss surface locally, guiding updates using only slope information.
- We use a quadratic function (second-order Taylor expansion)
- The Hessian acts like a curvature-aware adjustment to the step size in each direction, guiding updates more efficiently toward the minimum.
- For quadratic functions, the minimum can be found directly since the curvature is simple and well-defined
- In high-dimensional settings, we generalize curvature information using the Hessian matrix, which captures second-order partial derivatives of the loss function.
- Computing the inverse of the Hessian is computationally expensive, making full second-order methods mostly theoretical.
- In practice, we use quasi-Newton methods (e.g., BFGS, L-BFGS) that approximate the Hessian efficiently without explicit inversion.
Various Optimization Algorithms and Packages
1. Gradient Descent — The first order that we discussed earlier.
2. Stochastic Gradient Descent (SGD)
- Definition: Instead of using the full dataset to compute the gradient, SGD updates parameters using a single randomly chosen sample (or a small mini-batch) at each step.
- Effect: Introduces randomness into the updates, which helps escape local minima but also makes the path noisier.
- Use Case: Particularly effective for large datasets, since it avoids the cost of computing the gradient over the entire dataset in each iteration.
3. Adam — Default best one.
Definition: Combines ideas from Momentum that keeps track of past gradients (histories)
Effect: Provides adaptive learning rates for each parameter, leading to faster and more stable convergence.
Use Case: Often considered a default choice in deep learning, as it works well across a wide range of problems without much tuning. Prevents the Zig-zag issue as well.
Analytical Solution
- Instead of iterative gradient descent, we can solve for the optimal weights directly using calculus (closed-form solution).
- This is called the Normal Equation.
- Works well for small datasets, but becomes infeasible for very large datasets (matrix inversion cost).
1. Loss Function
2. Gradient Calculation
3. Optimal Solution
4. Prediction
This is exact but computationally expensive for large datasets due to inverse calculation → in practice, we use gradient descent instead.
- XᵀX is always symmetric and positive semi-definite, but not always invertible.
- When it is not invertible, we replace the standard inverse with the pseudo-inverse to still obtain a valid solution.
Projection View
Here
- xd becomes X,
- inner products (xd^⊤ xd) become the Gram matrix (X^⊤X)
- and the projection scales across all columns at once instead of a single line.
Some Examples
- Figure (1): The Line of best fit has a positive slope
- Figure (2): The Line of best fit has a positive slope, but an outlier pulls the line to the right. Because MSE squares the residuals, making large errors dominate.
- Figure (3): The extreme outlier is very far out, causing the slope to flip.
- Figure (4): Linear regression does not do a good job. However, it does capture a positive slope. It is not simply possible to express the two-clustered dataset using a linear regression.
🔙 Back to all notes
𝕏 Let’s Connect
Backlinks:
Machine Learning
