Finding Meaning in Models
I strongly believe lessons exist everywhere if you’re willing to look hard enough. Here are some lessons I took from my degree by viewing the concepts I studied through a philosophical lens.
Simpson’s Paradox — a statistical phenomenon where an association between two variables in a population emerges, disappears or reverses when the population is divided into subpopulations.
Example: Take a quick look at this visualization of Berkeley Graduate Admissions Data from the 1970s.
It would seem like women are admitted at a much lower rate than men. (30.4% vs. 44.5%)
Let us now separate the data by department.
This plot allows us to understand that women disproportionately applied to more competitive departments while men disproportionately applied to less competitive departments. The first plot averaged admission rates without accounting for difficulty. Even if women outperform men within each department, the overall average can still flip it.
Get Ruth Sharon’s stories in your inbox
Join Medium for free to get updates from this writer.
What this taught me was that there is no single truth without specifying context. And what’s true on a micro scale may not be true on the macro scale.
Reward Hacking — what occurs when a reinforcement learning (RL) agent exploits flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task.
Much of alignment research in language models centers around making these models do what humans want or ensuring our goals align. That is the main goal. A reward signal is a measure. These could be positive human feedback, safety markers, or preference rankings. But the model may try to maximize rewards without fulfilling the goal or find ways around it by means of agreeableness, task shortcutting, or not admitting uncertainty. Essentially, it finds loopholes in the reward function, which is the function that quantifies the model’s performance. This reminds me of one of my favourite laws — that of Goodhart’s. It states that when a measure starts becoming a target, it ceases to be a good measure.
This taught me the reality of shallow goals. If your goal doesn’t mean anything to you, you may end up achieving things that make you feel unsatisfied. This can look like chasing validation (think academic tests/exams), productivity, and milestones instead of true growth , purpose, and learning.
Masked Self-Attention — a mechanism that ensures that the prediction for each next word should only depend on the preceding words.
During training, a transformer is fed a full token sequence at once with the objective being next word prediction. If a token gets to peek into the future to see what comes after itself, it is a lot like cheating in this context. It doesn’t have to understand context, meaning, or language. This is also called information leakage from the future. While the model may do well during training, it may fail when generating text in the future. Thus, when a causal mask is applied to the attention score matrix, each token attends only to itself and the ones preceding it. This ensures that the training loss doesn’t drop too low and prevents the models from learning shortcuts.
Something about an inaccessible future stirred something in me. This one wasn’t easily visible. But after I thought about it for a while, I saw it as actions losing meaning if their outcomes are already known. And it also confirms how the real world works. We know the future will come but we aren’t aware of what it will contain.
This is a very messy and non-exhaustive list of interesting parallels I could draw. I plan on updating the list as I study more!
