CTT, IRT & the Things We Don’t Realize We’re Measuring
This week’s topic is CTT versus IRT, but the social scientist in me wants to write about a fun little experiment I did a couple of days back. Afflicted by this dilemma to choose between the two, I am trying to bring the two together and weave them into a story that doesn’t just help readers differentiate CTT from IRT but does so while also teaching an important life lesson which is, as it turns out, endorsed by my little quasi-experimental adventure.
Classical Test Theory (CTT) and Item Response Theory (IRT) are two widely used approaches in modern psychometrics. Yes! CTT is not classical enough to have become completely obsolete, neither has it been replaced by IRT. So, IMO, instead of seeing CTT and IRT as the older versus newer approach to psychometrics, we should view them as two complementary approaches that can be used together without one replacing the other. Each has its own assumptions, strengths and purposes.
Speaking of assumptions… if I may deviate for a second…
A couple of days ago, I conducted what I, as a social scientist, would very generously like to call a tiny quasi-experimental exercise to test how often women hold negative assumptions about themselves which slip through rather subconsciously in their day-to-day speech.
The task was simple. I asked a few females from my family, including me, with ages ranging from the early teens to the late 30s and early 40s to play a simple game. Every time they said anything negative, condescending or demeaning about themselves; anything that devalued, belittled or put themselves down, they would lose 1 point.
We started at 1 PM and the game had to end at 11 PM, so we had less than 12 hours of close self-reflection. Whoever lost 5 points first lost the game, and whoever finished with the highest score won.
Simple. Right?
Well… not quite.
The majority of us made a negative remark about ourselves within the first 15 minutes of starting the game, and none of the participants, including myself, could reach the finish line. We all ended up losing all five points in under two hours.
That too, when we were highly alert and actively monitoring what was coming out of our mouths.
Imagine how frequently we demean ourselves when there is no game, no score to lose and no conscious effort to monitor what we are saying.
It’s almost as natural as breathing or blinking.
We do it without noticing. Sometimes we don’t even process what we have said. But as a social scientist, I know just how important our words are whether chosen consciously or slipping out subconsciously. In fact, the things that come out when we are not consciously monitoring ourselves can sometimes tell us a lot about the thoughts, assumptions and feelings sitting underneath.
Self-criticism and self-deprecation are unfortunately so normalized for women that many of us don’t even realize how much or how frequently we throw shade at ourselves and often for things that are completely beyond our control or things that don’t even matter.
Like the participant who remarked:
“My hair is just untamable.”
Or the one who said:
“Such fancy clothes are just not meant for me, given my age.”
The exercise started as a fun little game and a challenge to ourselves. It ended up being a rather uncomfortable eye-opener.
Is this a gendered phenomenon? Does it extend to men as well? Is it rooted in cultural expectations around humility and modesty, familial upbringing, social conditioning, or something else entirely?
There is so much we still don’t know.
But one thing was quite striking.
There seems to be something fundamentally wrong with the way many of us have trained our minds to think about ourselves. We dislike, dismiss and disrespect ourselves with a frequency that is, quite frankly, scary.
Get Nasrina Siddiqi’s stories in your inbox
Join Medium for free to get updates from this writer.
And no, this is NOT a scientific experiment in the true sense of the word, nor is this tiny family exercise remotely enough to establish a psychological claim.
But it did tell me something worth paying attention to.
Sometimes, we don’t realize how negative our self-perceptions are until someone asks us to count them.
Until someone decides to take this up as a proper research topic, let’s continue the game.
Play it with your family and friends. And share what you learn.
Now that my hunger to tell that story has been satiated, let’s get back to CTT versus IRT and what actually makes the two different.
The two differ in a number of ways, and discussing all of them would require a dedicated article of its own. So, in this one, I am only going to emphasise the key differences that a beginner must understand to be able to differentiate between the two.
For starters, CTT is fundamentally centred around the observed test score and its reliability, while item-level analysis is still very much a part of CTT. We routinely examine individual items for things like difficulty, discrimination and distractor functioning, but these statistics are properties of how the items behave in a particular sample.
This is where IRT takes a rather different route.
IRT explicitly models the probability of a person responding to an individual item as a function of their position on a latent trait continuum. So instead of simply asking, “How did this item perform in this sample?”, IRT allows us to ask, “At what level of the underlying trait does this item become more or less likely to be endorsed or answered correctly, and how sharply does it discriminate between different levels of the trait?”
Another extremely important difference is how CTT and IRT conceptualize item parameters such as difficulty and discrimination.
In CTT, item difficulty for a dichotomous item is essentially the proportion of respondents who answer it correctly. So if 80% of my sample gets an item right, the item has a difficulty index of .80. The catch is that this statistic depends heavily on the sample.
Give the same item to a much stronger group and it may suddenly look extremely easy. Give it to a weaker group and the very same item may look much harder.
IRT approaches difficulty differently.
In the commonly used 1PL/Rasch and 2PL models, the difficulty parameter is located on the latent trait scale and, in the 2PL model, corresponds to the point where the probability of a correct response is 50%.
So instead of simply telling us how many people in this particular sample got an item right, IRT tells us where that item sits along the underlying ability continuum.
And that is a pretty important conceptual shift.
Notice, however, that IRT parameters are not magically immune to samples. Their theoretical appeal is that, when the model fits and is appropriately estimated, the item parameters are intended to be invariant across populations more than CTT item statistics are. The estimates still depend on data quality, sample size and model assumptions.
These are the two most important differences a beginner should be aware of. Once these are clear, the other differences start to make much more sense.
For instance, why CTT is often more practical with relatively smaller samples, whereas IRT generally needs larger samples for stable item parameter estimation. Or why IRT can sometimes achieve highly efficient measurement with fewer, well-targeted items, whereas adding items in a CTT-based test can substantially improve reliability — provided, of course, that those additional items are actually good items.
But these are secondary differences, and they start making much more sense once the main conceptual distinction is understood.
The topic can easily be extended into another post, but I feel this one has served its intended purpose: to develop a basic understanding of how CTT and IRT differ.
And perhaps, in the process, remind ourselves of a completely unrelated but oddly relevant lesson:
Sometimes the things we measure least deliberately are the things we should probably be paying the most attention to.
Let me know if you would like me to write an extended article talking about the other differences, and I will be happy to come back with another complementary story.
