Logistic Regression Example: Predicting Churn From Support Tickets, Worked by Hand

Chart: the fitted S-curve (blue) stays between 0 and 1. A straight line through the same points (grey) doesn’t. Created by the author with Matplotlib.

Eight customers. For each one I know how many support tickets they opened last month and whether they cancelled. A customer with 5 tickets walks in: how likely are they to leave? This logistic regression example answers that with a number (87.3%), and gets there by hand: the sigmoid, the log loss, one gradient descent step, the fitted coefficients and what they mean in plain odds.

Then I check every number in NumPy and scikit-learn, and break the model on purpose so you can see a failure that is easy to miss: the weights that never stop growing.

The eight customers are made-up example data, kept tiny so the arithmetic fits on a page. None of it comes from a real company.

What is logistic regression in simple terms?

Logistic regression is a model for yes/no outcomes. It takes a weighted sum of the inputs, z = b0 + b1·x, and squeezes it through the sigmoid function σ(z) = 1 / (1 + e^(−z)), which turns any number into a probability between 0 and 1. The weights are chosen to make the observed outcomes as likely as possible.

That last sentence is maximum likelihood, which I worked through on coins and counts in my maximum likelihood estimation example. Logistic regression is the same idea with a slope attached. The Wikipedia article on logistic regression has the formal version, and chapter 4 of Jurafsky and Martin’s free textbook Speech and Language Processing, “Logistic Regression and Text Classification”, is the clearest long-form treatment I know.

The data: 8 customers, one feature

Customer  Tickets last month (x)  Churned (y)
A         0                       0
B         1                       0
C         2                       0
D         3                       1
E         4                       0
F         5                       1
G         6                       1
H         7                       1

More tickets, more churn. Mostly. Customer D left after 3 tickets and customer E stayed after 4, so no single cut-off splits the groups perfectly. That overlap matters later. (Hold on to it.)

Why not fit a straight line?

You could. Least squares on these points gives y = −0.0833 + 0.1667·x. At 0 tickets it predicts a −8.3% chance of churn. At 10 tickets it predicts 158%. Neither is a probability.

A probability has to live between 0 and 1, and a line can’t stay inside that band forever. The sigmoid can. That’s the whole reason logistic regression exists.

The sigmoid, by hand

σ(z) = 1 / (1 + e^(−z)). Plug in a few values:

z   e^(−z)  σ(z)
−4  54.598  0.0180
−2  7.389   0.1192
0   1       0.5000
2   0.1353  0.8808
4   0.0183  0.9820

Big negative z: close to 0. Big positive z: close to 1. At z = 0 you get exactly 0.5. Notice the symmetry, too: σ(−2) = 1 − σ(2).

So the model is p = σ(b0 + b1·x). Two numbers to find. Here’s how.

The loss: why log loss and not squared error?

For one customer with outcome y and predicted probability p, the log loss (also called binary cross-entropy) is

ℓ = −[ y·ln(p) + (1 − y)·ln(1 − p) ]

If the customer churned (y = 1), only −ln(p) survives. Predict p = 0.9 and you pay 0.105. Predict p = 0.1 and you pay 2.303. Confident and wrong costs a lot.

This isn’t an arbitrary choice. Each outcome is a coin flip with its own probability p, so the likelihood of all eight outcomes is a product of p’s and (1 − p)’s. Take the log, flip the sign, and divide by n: that’s the mean log loss. Minimising log loss and maximising likelihood are the same job.

Squared error would work numerically, but paired with a sigmoid it gives a bumpy, non-convex surface. Log loss with a sigmoid is convex: one valley, no traps. The Wikipedia page on cross-entropy covers the link to information theory if you want it.

Step 0: start at zero

Set b0 = 0 and b1 = 0. Then z = 0 for every customer, so every p = 0.5.

Each customer’s loss is −ln(0.5) = 0.6931, so the mean log loss is 0.6931 too. That’s the “I have no idea” baseline. Any useful model has to beat it.

Gradient descent, one step by hand

The gradient of log loss has a famously tidy form. Using dσ/dz = σ(z)(1 − σ(z)), the chain rule collapses to:

∂ℓ/∂z = p − y

That’s it. The error signal is prediction minus truth. Averaged over the data:

• ∂L/∂b0 = (1/n)·Σ(p − y)
• ∂L/∂b1 = (1/n)·Σ(p − y)·x

At step 0, p − y is 0.5 for the four customers who stayed and −0.5 for the four who left:

x    y  p    p − y  (p − y)·x
0    0  0.5  0.5    0
1    0  0.5  0.5    0.5
2    0  0.5  0.5    1.0
3    1  0.5  −0.5   −1.5
4    0  0.5  0.5    2.0
5    1  0.5  −0.5   −2.5
6    1  0.5  −0.5   −3.0
7    1  0.5  −0.5   −3.5
Sum          0      −7.0

So ∂L/∂b0 = 0/8 = 0 and ∂L/∂b1 = −7/8 = −0.875.

With a learning rate of 0.1, each weight moves against its gradient:

• b0 = 0 − 0.1·0 = 0
• b1 = 0 − 0.1·(−0.875) = 0.0875

The intercept doesn’t move on the first step because the classes are balanced (four and four). The slope goes positive: more tickets, more churn. The mean log loss drops from 0.6931 to 0.6332. One step, already better than guessing.

I also checked this gradient numerically (nudge each weight by 0.000001 and measure the change in loss). At the new weights both methods give (0.075232, −0.500106). If you ever write this from scratch, run that check. It catches sign errors instantly.

Logistic regression example in Python: the fitted model

Keep stepping and the weights settle. Here’s the whole loop next to scikit-learn:

import numpy as np
from sklearn.linear_model import LogisticRegression
x = np.array([0, 1, 2, 3, 4, 5, 6, 7], dtype=float)  # tickets last month
y = np.array([0, 0, 0, 1, 0, 1, 1, 1])              # 1 = churned
X = np.c_[np.ones_like(x), x]                       # column of 1s for b0
w = np.zeros(2)
for step in range(10_000):
    p = 1 / (1 + np.exp(-X @ w))
    w -= 0.1 * X.T @ (p - y) / len(y)              # gradient of mean log loss
print(w.round(4))
sk = LogisticRegression(C=np.inf).fit(x.reshape(-1, 1), y)
print(np.r_[sk.intercept_, sk.coef_.ravel()].round(4))
print(sk.predict_proba([[5]])[:, 1].round(4))

Output:

[-4.488   1.2823]
[-4.4881  1.2823]
[0.8725]

My loop and scikit-learn agree to three decimals (the last digit differs because scikit-learn stops at its default tolerance). The fitted model is

Get Rayan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

p = σ(−4.488 + 1.282·tickets)

and the mean log loss falls from 0.6931 to 0.3129.

C=np.inf matters here. The scikit-learn LogisticRegression docs state that C=np.inf gives unpenalized logistic regression. Leave it out and you get the default C = 1.0, an L2 penalty, and different numbers: −3.0374 and 0.8678 on this data. Forget that default and you can spend an hour "debugging" a gradient that was right all along.

A by-hand check you can do on the fitted model

At the optimum the gradient is zero, which means Σp = Σy and Σp·x = Σy·x. The fitted probabilities are 0.011, 0.039, 0.127, 0.345, 0.655, 0.873, 0.961 and 0.989. They add to 4.0, the number of churners. Weighted by tickets they add to 21.0, which matches 3 + 5 + 6 + 7 = 21. The model reproduces the data’s totals exactly. Pretty neat for two numbers.

How do you interpret logistic regression coefficients?

Read them as odds.

Odds are p / (1 − p). Take the log and you get exactly the linear part: ln(odds) = b0 + b1·x. So b1 is how much the log-odds rise per extra ticket, and e^(b1) is how much the odds get multiplied.

Here e^(1.2823) = 3.60. Each extra support ticket multiplies the odds of churn by about 3.6.

Check it by hand:

• 2 tickets: z = −4.488 + 1.2823·2 = −1.923, odds = e^(−1.923) = 0.146
• 3 tickets: z = −0.641, odds = e^(−0.641) = 0.527
• 0.527 / 0.146 ≈ 3.6 (3.605 with unrounded numbers)

That number is an odds ratio. It’s constant across the whole range. The change in probability isn’t: going from 2 to 3 tickets moves p from 0.127 to 0.345 (+21.8 points), while going from 6 to 7 moves it from 0.961 to 0.989 (+2.8 points). Same odds ratio, very different jumps. If you report coefficients to a non-technical team, report the odds ratio and one or two concrete probabilities. Skip the log-odds.

The new customer with 5 tickets

z = −4.488 + 1.2823·5 = 1.923, and σ(1.923) = 1 / (1 + e^(−1.923)) = 1 / (1 + 0.1462) = 0.872 (0.8725 with the unrounded weights).

So about an 87.3% chance of churn. That matches predict_proba above.

Where is the decision boundary?

The model says “churn” when p ≥ 0.5, which happens when z ≥ 0:

−4.488 + 1.2823·x = 0, so x = 4.488 / 1.2823 = 3.5 tickets.

The boundary lands exactly halfway between customers D and E, the two who break the pattern. With a 0.5 threshold the model gets 6 of 8 customers right (75%). It misses D (p = 0.345, actually churned) and E (p = 0.655, actually stayed).

Here’s the part I think most tutorials skip. A plain cut at 2.5 tickets gets 7 of 8 right, and so does a cut at 4.5. Logistic regression didn’t pick either, because it maximises the likelihood of all eight outcomes, not the count of right answers. It hedges on D and E instead of sacrificing one of them. If accuracy at one threshold is all you care about, that’s a trade-off you should know you’re making.

The threshold doesn’t have to be 0.5. If losing a customer costs ten times more than a retention call, lower it.

What goes wrong: perfect separation

Now swap D and E, so customers with 0 to 3 tickets all stay and 4 to 7 all leave. The classes are perfectly separated at 3.5. Sounds like the easy case. It’s the broken one.

Chart: the same gradient descent on both datasets. Created by the author with Matplotlib.

Running the same gradient descent on the separated data:

Steps    b0       b1     Mean log loss
1,000    −5.312   1.602  0.1232
10,000   −13.030  3.769  0.0366
100,000  −26.937  7.735  0.0052

The loss keeps falling and the weights keep growing. There’s no finite answer. A steeper sigmoid always fits a perfect split a little better, so the maximum likelihood estimate runs off to infinity. Statisticians call this separation.

scikit-learn with C=np.inf on the separated data stopped at −62.0 and 17.63 on my machine (version 1.9.1). Those numbers mean nothing. They're wherever the solver happened to stop.

The fix is the penalty you just turned off. With the default C = 1 the coefficients become a sensible −4.0956 and 1.1701. With C = 100, −19.7751 and 5.6498. The smaller C is, the harder the weights get pulled toward zero. In real data with many features, separation is common (one rare category that’s always “yes” is enough), which is why most libraries regularise by default.

Why is logistic regression called regression?

Because it is a regression: a linear regression on the log-odds. The classifier part only appears when you add a threshold. Statisticians use it for inference (odds ratios with confidence intervals). ML people use it as a classifier and a baseline. Same math.

It’s also the building block of a neural network. One neuron with a sigmoid output, trained on log loss, is exactly this model. Swap the sigmoid for a softmax and you have multiclass logistic regression, the final layer of most classifiers. Chapter 4 of An Introduction to Statistical Learning (free PDF, with R and Python editions) covers both the statistics view and the multiclass version.

What I’d take from this example

• Use the sigmoid to keep predictions between 0 and 1. A straight line gave −8.3% and 158% on the same data.
• The gradient of log loss is (p − y) times the input. Check it numerically once.
• Report e^(b1) as “odds multiply by 3.6 per ticket”, plus one or two concrete probabilities.
• Decision boundary: x = −b0 / b1. Pick the threshold from costs, not habit.
• If weights keep growing and loss heads to zero, suspect separation and add a penalty.
• In scikit-learn, C=1.0 is on by default. Set C=np.inf only when you want the textbook, unpenalized answer.

The next step from here is the same model with more than one feature, where the boundary becomes a line or a plane. The arithmetic doesn’t change. Only the vectors get longer.