How AI learns
Introduction
Fifty years ago, if someone told you a machine would write your emails, detect cancer earlier than any radiologist, or pass a bar exam in the 90th percentile, you’d have had no framework to process the claim. Now most people accept what AI can do. Far fewer understand how. That gap matters more than it used to.
What AI Actually Is
The most persistent misconception is that AI thinks. It doesn’t. A modern AI system is, at its core, a very large mathematical function, specifically, a parametric function with billions of tunable weights. Feed it a sentence, it predicts the most statistically probable continuation. Feed it an image, it predicts a label. Feed it a patient’s medical history, it predicts which treatments correlate with better outcomes in similar cases.
The sophistication lies entirely in how those parameters get set. The dominant approach is a neural network: layers of simple computational units, each taking a weighted sum of its inputs and passing it through a nonlinear function. What makes modern models powerful isn’t any single unit, it’s depth. A 2017 transformer like the original GPT has 117 million parameters. GPT-4 is estimated somewhere above 1 trillion. The scale itself creates emergent capabilities nobody explicitly programmed.
Training begins with those parameters set essentially at random. The model makes a prediction. You measure how wrong it was using a loss function, a number that quantifies the error. Then you compute the gradient of that error with respect to every parameter (via backpropagation, a chain-rule calculation applied backwards through the entire network), and nudge each parameter fractionally in the direction that reduces the error. Do that billions of times across a dataset large enough to fill a hard drive farm, and the parameters converge toward something that looks like intelligence. Nobody wrote the rules. The rules emerged from the data.
How It Actually Learns and Why That’s Risky
The model doesn’t learn facts. It learns statistical correlations. It learns that certain tokens (the subword units text gets broken into) co-occur in certain patterns, that certain pixel intensity gradients correspond to certain object boundaries, that certain combinations of biomarkers predict certain diagnoses. The result is a high-dimensional statistical map of everything in the training corpus. Its extremely detailed but completely unconscious.
Which means the model is only as good as its data, in ways that aren’t always obvious. A classic case: in the early 2010s, image classifiers trained to detect wolves in photos were inadvertently detecting snow. Wolves in the training set almost always appeared in snowy backgrounds. The model had learned a spurious correlation that was invisible in its accuracy metrics until someone tested it on wolves photographed in green fields. The model failed completely. It had learned the right answer for the wrong reasons.
This problem doesn’t go away at scale. A 2019 study found that chest X-ray classifiers trained on data from one hospital performed substantially worse at another, because different hospitals position patients differently, use different equipment, and serve different demographic populations. The model had learned hospital-specific artefacts alongside the actual pathology. Accuracy on the training distribution told you almost nothing about accuracy elsewhere.
The technical term for this is distribution shift is the difference between the environment a model was trained in and the environment it’s deployed in. Most real-world AI failures, when you trace them back, involve some version of this. The model encounters inputs that look slightly different from anything in its training data and extrapolates badly, with complete confidence.
Get Ron Batunskiy’s stories in your inbox
Join Medium for free to get updates from this writer.
What Makes It Different From Traditional Software
A traditional program is explicit: if X, then Y. The logic is entirely legible. When it breaks, you can read the code and find out why. Neural networks invert this. Instead of writing the rules, you show examples and let the system infer rules from them. This is why AI can do things that would be impossible to hard-code. No programmer could write explicit rules for recognising every face, in every lighting condition, at every angle. A model trained on hundreds of millions of face images figures this out anyway, in ways that are often not interpretable even to the people who built it.
That last part is understated. In 2017, researchers at Google Brain published a finding that adversarial examples: images perturbed by a tiny amount of noise, imperceptible to the human eye, could reliably fool state-of-the-art image classifiers into confidently misidentifying them. A photo of a panda, with pixel values adjusted by a fraction of a percent, was classified as a gibbon with 99.3% confidence. The perturbations exploit features in the model’s high-dimensional representation that have no analogue in human perception. What the model actually learned is not what it appears to have learned.
Interpretability research exists precisely to address this problem. Techniques like SHAP values, attention visualisation, and mechanistic interpretability try to make model reasoning legible. But this remains an unsolved problem at the frontier. We can often identify which input features drove a particular output. We cannot yet reliably verify that a model’s internal reasoning is sound in the way that reasoning should mean.
Where It’s Good and Where It Isn’t
The core competency is pattern recognition at scale. Identifying anomalies in medical imaging, flagging fraud in financial transaction streams, predicting equipment failure from sensor data, synthesising research across thousands of papers. Anything involving the detection of structure in large datasets, done reliably and at volume.
It does not reason from first principles. It has no model of the world, only a model of its training data. When a language model produces something that looks like causal reasoning: “if you heat water, it boils because the increased thermal energy allows molecules to escape the liquid phase”, it’s not running a physics simulation. It’s pattern-matching against text where that kind of reasoning appeared, successfully enough that the output looks right. Most of the time this works. When it doesn’t, it fails silently, producing fluent, confident, plausible-sounding nonsense. This is called hallucination, and it’s not a bug being fixed so much as a structural property of how these systems work. Large language models are trained to predict likely text, not to verify truth. These are different objectives.
AI shouldnt be trusted with tasks requiring judgment in novel situations like legal reasoning about cases with no precedent, clinical decisions in patients with rare overlapping conditions, engineering problems at the edge of the known. Not because AI can’t be useful as a tool in those domains, but because a human in the loop who understands the system’s failure modes can catch what the model can’t catch itself.
AI in Medicine: What It Actually Looks Like
The clearest place to see all of this play out in practice is healthcare. Over the past year I worked directly with the CEO of a venture-backed healthcare AI company, and it reframed how I think about most of this.
The system being built used an approach called k-nearest neighbours. Every patient gets represented as a point in a mathematical space, each dimension corresponding to something measurable about them: age, weight, lab results, diagnosis history, current medications. When a new patient enters the system, the algorithm finds the historical patients most mathematically similar to them and asks what happened to those people. Which treatments worked? Which made things worse? A doctor with forty years of experience has seen maybe three hundred thousand patients. A well-built system draws on tens of millions. The goal isn’t to replace the doctor’s judgment, but rather to give the doctor better raw material to exercise that judgment on. What stayed with me from those conversations wasn’t the sophistication of mathematics. It was how seriously the people building it thought about failure. What happens when the training data underrepresents certain ethnic groups? How do you make the reasoning visible enough that a clinician can push back? Where does confidence become dangerous? That has really proven to me that what makes an AI stand out is the safety and care, not impressive results.
Why This Matters
AI is already in clinical use, already shaping hiring decisions, already determining what you read, what you’re recommended, what you’re charged for insurance. And most people using it have no real model of what it is, what it was trained on, or where its failure modes live.
The model has no way of telling you when it’s operating outside its competence. It will answer with the same tone and apparent confidence regardless. That job, knowing when to trust the output, knowing when to push back, knowing when the answer is subtly wrong in ways that compound, falls to whoever is using it. And doing that job requires understanding, at least roughly, what the thing actually is. Not because it makes you a better engineer. Because the alternative is delegating judgment to a system that cannot judge, and not knowing you’ve done it.
