I experienced this first-hand while reading about Cognitive Diagnostic Model (CDM) analysis. In simple terms, CDM looks at which combination of skills or abilities may be needed to answer each item correctly, and then uses this to build a profile of what an individual has or hasn’t mastered.

As an assessment expert, I am trained to ensure that constructs are clearly defined, distinct from one another, and that each item has a clear measurement purpose. Ideally, an item should measure one construct rather than a combination of different constructs.
So CDM initially seemed like a contradiction.

If an item can require more than one skill to be answered correctly, doesn’t that mean it is measuring more than one construct? Isn’t that what we worry about with double-barrelled items?

But here’s where it gets interesting.

There is a distinction between how many constructs an item is intended to measure and how many attributes may be required to respond successfully to it.

Take a leadership assessment. The items should stay within the chosen leadership framework. But effective leadership may involve a combination of attributes such as influence, decision-making, accountability and perspective-taking.

In a CDM framework, we can specify which attributes are required for success on each item. Some items may require one attribute; others may require a combination of two or more.

So, instead of only producing a cumulative leadership score, a CDM can potentially provide a more fine-grained profile of which attributes an individual is classified as having mastered and where further development may be needed.

That has interesting implications for assessments designed not just to categorise candidates, but to provide targeted feedback and personalised upskilling.

It took me a while, a whole lot of reading, and a little back-and-forth with GPT to understand this distinction.

The more I think about it, the more interesting I find the idea that clarity of constructs and multiple attributes contributing to successful performance do not necessarily contradict each other.

Sharing this so anyone else exploring CDM doesn’t have to struggle as much with this “apparent” contradiction.

Cognitive Diagnostic Model Analysis and Mental Sets in Learning was originally published in Psochometry on Medium, where people are continuing the conversation by highlighting and responding to this story.

I wanted to find out what my tolerance level is when it comes to consuming horror content. So, I took one of the biggest risks of my life and decided to watch Darna Mana Hai. For those who are deprived of the privilege of watching Bollywood movies, Darna Mana Hai is an anthology of allegedly scary stories. Allegedly being the keyword here!

I walked out of the room with inexplicable confidence and a feeling of achievement I hadn’t anticipated. To the surprise of everyone who had always seen me as a squeamish little girl, I wasn’t feeling scared! That proves something about how much horror content I can consume and my ability to watch scary stuff without feeling unnerved.

I took this challenge to the next level and asked my friend to replicate the process and see if she could finish the movie without feeling scared.

There was one difference, though!

My friend chose to watch The Exorcism of Emily Rose. Unlike me, she couldn’t finish watching it alone and came out looking shook and seriously disconcerted.

My verdict! I have a higher tolerance for horror content than my friend.

It’s that easy. Isn’t it? It’s NOT.

The fundamental issue here is my sheer dismissal of the fact that the two movies are poles apart in terms of their scary quotient.

I essentially compared our reactions to two very different stimuli and treated the outcomes as though they were directly comparable.

Something similar can happen in psychometric assessments that involve multiple test forms or different sets of items for ability testing. Even when different forms are designed to measure the same underlying construct, differences in item characteristics, particularly item difficulty and discrimination, can introduce differences in observed scores.

A candidate taking a more difficult form may obtain a lower raw score than another candidate with the same underlying ability simply because they received a more difficult set of items.

This calls for appropriate statistical methods to establish comparability across different test forms, so that irrespective of which question paper set or version a candidate attempts, their score can be meaningfully compared with scores obtained from other forms.

In psychometric parlance, this process is known as Test Equating.

There are different equating designs that can be used to establish equivalency across test forms, two of the commonly discussed ones being the common-person design and the common-item design.

Going back to my experiment, can my friend contest that she and I have the same tolerance for horror, but the reason I finished my movie while she couldn’t is simply that we watched two movies with very different scary quotients?

We can use one of the two designs i.e. common-person and common-item design to establish equivalency between different test forms (in this case, the two movies).

In the former, the same group of people (test-takers) takes two different forms of a test.

Their performance on both forms provides the common reference needed to establish the relationship between the two forms. If the relationship between their scores across the two forms is consistent, we have evidence that allows us to link the forms and place their scores on a common scale.

Going back to my experiment, I would have used something analogous to the common-person design if I had watched both movies and my reactions across the two movies had given us a reliable basis for establishing how scary the two movies were relative to each other.

If Darna Mana Hai and The Exorcism of Emily Rose had produced comparable reactions in me, that would give us some evidence about their relative scary quotient. We could then use that relationship to make a more meaningful comparison when my friend watches one of them.

Of course, this is where my brilliant experimental design falls apart.

Because I watched Darna Mana Hai and my friend watched The Exorcism of Emily Rose, we have absolutely no common-person reference.

So much for my career in horror research.

That’s one way to psychometrically bring down my friend.

The other design is the common-item design, where some items are common across two different test forms. These are called anchor items.

The idea is rather elegant.

Suppose two groups of test-takers take two different forms, but both forms contain a set of common items. These common items act as the link between the two forms.

Why?

Because the performance on the common items gives us information that helps establish the relationship between the two groups and the two forms. Once that common link is established, differences in performance on the remaining, unique items can be interpreted in the context of the differences between the forms rather than simply being attributed to differences in ability.

And this is particularly powerful in IRT because the common items can be used to place the item parameters from the different forms onto a common underlying scale.

That’s like a cloudburst of information.

Let’s understand this with our example.

Hypothetically speaking, imagine that some of the scary scenes across the two movies were identical. These would be our anchor scenes.

If both my friend and I watched those scenes without blinking an eye, they would provide us with a common reference point for comparing our reactions.

Now, suppose that despite showing similar reactions to these common scenes, my friend ran out of the room halfway through The Exorcism of Emily Rose, while I happily finished Darna Mana Hai.

We would then have some evidence that the two movies differ in their level of scariness, rather than immediately concluding that my friend has a lower tolerance for horror.

In other words, the common scenes help us separate differences between the people from differences between the movies.

And that is essentially the problem equating is trying to solve with tests.

If two groups take different forms of an assessment, we don’t want differences in their scores to be driven simply by the fact that one group happened to receive an easier or more difficult form.

The common items provide the bridge that allows the forms to be linked and their scores to be interpreted on a common scale.

So, strictly speaking, it would not be psychometrically appropriate for me to conclude that my friend is pigeon-hearted and I am not.

Which I am certain she is. But I need to replicate the experiment with the equivalency caveat in mind before I can prove it psychometrically.

Science is such a buzzkill.

How to Psychometrically Prove My Friend Is Faint-Hearted was originally published in Psochometry on Medium, where people are continuing the conversation by highlighting and responding to this story.

Instant noodles were an instant success when they were originally launched in India back in the late 80s-early 90s. I remember tasting Maggi (a popular instant noodle brand in India) back in the late 90s as a 7–8-year-old kid who was exposed to only limited variations of tastes and flavours and had a rather selective palate. I did not like it. No offence to Maggi lovers, and I am one of them now! But the 8-year-old me didn’t like it: maybe it was the worm-like appearance, the slippery texture or the flavour my taste buds were not accustomed to, or all of these, I couldn’t take another bite.

More than two decades later, my aversion to instant noodles has been replaced with a profound love. I can hog on them at any time of the day and, just like that, instant noodles have become my comfort food. One thing that has most certainly contributed to building this love is Maggi’s ability to taste better as we put more and more ingredients into the pot.

It has evolved from being just another brand of instant noodles to becoming a snack that can be blended with almost any ingredient; veggies, cheese, spices and garnishes. From a standalone food to being served as a side or a filling, the amount of experimentation that’s been done with Maggi is just astonishing. And over the years, having tasted different variations of Maggi, I have come to believe that the more ingredients you add, the tastier it gets.

So, in an article supposedly focused on different psychometric models rooted in Item Response Theory, why are we talking about instant noodles?There is one thing in common between the two, and that is: different ingredients can be added to enhance the final product.

We have already named the ingredients we can put in instant noodles. Similarly, in the psychometric evaluation of a test, different elements can be incorporated into the model to extract more information from the analysis and develop a more comprehensive understanding of the items and how they function.

These include item difficulty (popularly known as the b parameter), item discrimination (the a parameter), pseudo-guessing (the c parameter) and an upper asymptote that may reflect carelessness or inattentive responding (often represented by d).

Let’s start with the Rasch model. Without getting too technical, the best way to explain this model is that it estimates item difficulty while assuming that all items have equal discrimination. It also does not include item-specific pseudo-guessing or upper-asymptote parameters. So, among the item characteristics being discussed here, it mainly estimates one parameter, i.e. item difficulty, and hence is commonly referred to as the One-Parameter Logistic (1PL) model.

When we move to 2PL models, they have an additional ingredient or an extra layer of assessment that measures how much and how effectively an item can differentiate between respondents at different ability levels; a key characteristic of many quality assessments. I mean, at the end of the day, items are used to measure people’s ability levels, but if an item is responded to in an identical manner by all different kinds of respondents, irrespective of their ability level, what purpose is it serving! Adding this element makes the assessment more informative because now we not only know how easy or difficult each item is but also how sharply it can differentiate between respondents at different levels of the underlying ability.

Add another key element and the model becomes more comprehensive. That’s what 3PL is all about. It includes a third parameter called pseudo-guessing, which represents the lower asymptote of the item characteristic curve. In simple terms, it allows for the possibility that even respondents with very low levels of the underlying ability may have some probability of getting an item right. This is particularly relevant for some multiple-choice items, where a correct response may occur through guessing rather than because of the underlying ability.

The somewhat opposite phenomenon occurs when a generally high-achieving candidate gets an item wrong because they weren’t paying attention, were careless or simply misread the question. This again does not necessarily tell us anything about their ability level on the construct being measured. The 4PL model extends the 3PL by allowing the upper asymptote of the item characteristic curve to vary, thereby accommodating the possibility of unexpected incorrect responses even among respondents with high ability levels. That’s 4PL for you. This parameter is often associated with carelessness, slipping or inattention.

So, does the instant-noodle logic still hold true? Absolutely! But with one important caveat! Just as adding the right ingredients can make Maggi richer and tastier, adding relevant parameters can give us more information and a more comprehensive understanding of how the items function.

But, unlike cooking, where we can keep experimenting with ingredients, every additional parameter in a psychometric model requires sufficient data for reliable estimation. So, the idea is to add the ingredients that are needed and supported by theory and evidence, rather than simply throwing everything into the pot. After all, too many unnecessary ingredients can spoil the noodles too!

The Unexpected Connection Between IRT Models and Instant Noodles was originally published in Psochometry on Medium, where people are continuing the conversation by highlighting and responding to this story.

This week’s topic is CTT versus IRT, but the social scientist in me wants to write about a fun little experiment I did a couple of days back. Afflicted by this dilemma to choose between the two, I am trying to bring the two together and weave them into a story that doesn’t just help readers differentiate CTT from IRT but does so while also teaching an important life lesson which is, as it turns out, endorsed by my little quasi-experimental adventure.

Classical Test Theory (CTT) and Item Response Theory (IRT) are two widely used approaches in modern psychometrics. Yes! CTT is not classical enough to have become completely obsolete, neither has it been replaced by IRT. So, IMO, instead of seeing CTT and IRT as the older versus newer approach to psychometrics, we should view them as two complementary approaches that can be used together without one replacing the other. Each has its own assumptions, strengths and purposes.

Speaking of assumptions… if I may deviate for a second…

A couple of days ago, I conducted what I, as a social scientist, would very generously like to call a tiny quasi-experimental exercise to test how often women hold negative assumptions about themselves which slip through rather subconsciously in their day-to-day speech.

The task was simple. I asked a few females from my family, including me, with ages ranging from the early teens to the late 30s and early 40s to play a simple game. Every time they said anything negative, condescending or demeaning about themselves; anything that devalued, belittled or put themselves down, they would lose 1 point.

We started at 1 PM and the game had to end at 11 PM, so we had less than 12 hours of close self-reflection. Whoever lost 5 points first lost the game, and whoever finished with the highest score won.

Simple. Right?

Well… not quite.

The majority of us made a negative remark about ourselves within the first 15 minutes of starting the game, and none of the participants, including myself, could reach the finish line. We all ended up losing all five points in under two hours.

That too, when we were highly alert and actively monitoring what was coming out of our mouths.

Imagine how frequently we demean ourselves when there is no game, no score to lose and no conscious effort to monitor what we are saying.

It’s almost as natural as breathing or blinking.

We do it without noticing. Sometimes we don’t even process what we have said. But as a social scientist, I know just how important our words are whether chosen consciously or slipping out subconsciously. In fact, the things that come out when we are not consciously monitoring ourselves can sometimes tell us a lot about the thoughts, assumptions and feelings sitting underneath.

Self-criticism and self-deprecation are unfortunately so normalized for women that many of us don’t even realize how much or how frequently we throw shade at ourselves and often for things that are completely beyond our control or things that don’t even matter.

Like the participant who remarked:

“My hair is just untamable.”

Or the one who said:

“Such fancy clothes are just not meant for me, given my age.”

The exercise started as a fun little game and a challenge to ourselves. It ended up being a rather uncomfortable eye-opener.

Is this a gendered phenomenon? Does it extend to men as well? Is it rooted in cultural expectations around humility and modesty, familial upbringing, social conditioning, or something else entirely?

There is so much we still don’t know.

But one thing was quite striking.

There seems to be something fundamentally wrong with the way many of us have trained our minds to think about ourselves. We dislike, dismiss and disrespect ourselves with a frequency that is, quite frankly, scary.

And no, this is NOT a scientific experiment in the true sense of the word, nor is this tiny family exercise remotely enough to establish a psychological claim.

But it did tell me something worth paying attention to.

Sometimes, we don’t realize how negative our self-perceptions are until someone asks us to count them.

Until someone decides to take this up as a proper research topic, let’s continue the game.

Play it with your family and friends. And share what you learn.

Now that my hunger to tell that story has been satiated, let’s get back to CTT versus IRT and what actually makes the two different.

The two differ in a number of ways, and discussing all of them would require a dedicated article of its own. So, in this one, I am only going to emphasise the key differences that a beginner must understand to be able to differentiate between the two.

For starters, CTT is fundamentally centred around the observed test score and its reliability, while item-level analysis is still very much a part of CTT. We routinely examine individual items for things like difficulty, discrimination and distractor functioning, but these statistics are properties of how the items behave in a particular sample.

This is where IRT takes a rather different route.

IRT explicitly models the probability of a person responding to an individual item as a function of their position on a latent trait continuum. So instead of simply asking, “How did this item perform in this sample?”, IRT allows us to ask, “At what level of the underlying trait does this item become more or less likely to be endorsed or answered correctly, and how sharply does it discriminate between different levels of the trait?”

Another extremely important difference is how CTT and IRT conceptualize item parameters such as difficulty and discrimination.

In CTT, item difficulty for a dichotomous item is essentially the proportion of respondents who answer it correctly. So if 80% of my sample gets an item right, the item has a difficulty index of .80. The catch is that this statistic depends heavily on the sample.

Give the same item to a much stronger group and it may suddenly look extremely easy. Give it to a weaker group and the very same item may look much harder.

IRT approaches difficulty differently.

In the commonly used 1PL/Rasch and 2PL models, the difficulty parameter is located on the latent trait scale and, in the 2PL model, corresponds to the point where the probability of a correct response is 50%.

So instead of simply telling us how many people in this particular sample got an item right, IRT tells us where that item sits along the underlying ability continuum.

And that is a pretty important conceptual shift.

Notice, however, that IRT parameters are not magically immune to samples. Their theoretical appeal is that, when the model fits and is appropriately estimated, the item parameters are intended to be invariant across populations more than CTT item statistics are. The estimates still depend on data quality, sample size and model assumptions.

These are the two most important differences a beginner should be aware of. Once these are clear, the other differences start to make much more sense.

For instance, why CTT is often more practical with relatively smaller samples, whereas IRT generally needs larger samples for stable item parameter estimation. Or why IRT can sometimes achieve highly efficient measurement with fewer, well-targeted items, whereas adding items in a CTT-based test can substantially improve reliability — provided, of course, that those additional items are actually good items.

But these are secondary differences, and they start making much more sense once the main conceptual distinction is understood.

The topic can easily be extended into another post, but I feel this one has served its intended purpose: to develop a basic understanding of how CTT and IRT differ.

And perhaps, in the process, remind ourselves of a completely unrelated but oddly relevant lesson:

Sometimes the things we measure least deliberately are the things we should probably be paying the most attention to.

Let me know if you would like me to write an extended article talking about the other differences, and I will be happy to come back with another complementary story.

CTT, IRT & the Things We Don’t Realize We’re Measuring was originally published in Psochometry on Medium, where people are continuing the conversation by highlighting and responding to this story.

Before starting this post, I was doing some back-and-forth with my AI subordinate.

Yes, I call it subordinate because that’s the role it plays in my work.

It takes my ideas and builds a story around them. It adds visual elements to bring those ideas to life; you know, for the optics. 😄 It helps me retrieve and connect ideas I’ve worked on previously and, most importantly, it gracefully accepts its mistakes when I point them out.

And I have absolutely no qualms about admitting that I use AI to optimise my work and deliver better, more efficiently.

Has it made me passive or overly reliant, to the point where my thinking is losing its human element or the personal touch that makes ideas more authentic and humane?

I don’t think so.

If anything, AI has enabled me to spend more time on the part that actually requires me- thinking, questioning, challenging assumptions, and forming ideas in the first place.

It helps me polish those ideas, structure them into a coherent narrative, and present them in ways that resonate with different groups of readers.

And it does all of that in a fraction of the time it would have taken me to do it manually.

For me, AI isn’t replacing the thinking. It’s amplifying it.

The ideas are still mine. The judgement is still mine. The final call is still mine.

AI simply helps me get from “I have an idea” to “Here’s how I want to say it”- faster and, sometimes, much better.

And honestly, I’m quite happy to have a subordinate that never asks for a promotion. 😉

Sorry for digressing!

It’s a habit I developed during my teaching days. And if I am being honest, I actually like digressing sometimes.

Because when you get carried away, you end up in a space that wasn’t planned or prepared for, but it opens a new window, connecting ideas in ways you couldn’t have arrived at through all that careful planning.

And that’s often where the magic happens. That cognitive space where we move from ideation to creativity.

Okay. Enough of this.

Let’s get back to today’s topic.

I wanted to write about EAP reliability. As always, I was doing some fact-checking before freezing the draft I had already built. I asked my AI subordinate to differentiate EAP from more conventionally used reliability coefficients such as omega and alpha.

And here’s what it came back with:

“Alpha and omega ask: “How consistently do these items work together?” EAP reliability asks: “How precisely are we estimating each person’s latent ability?””

The response was, of course, accompanied by gazillions of related pieces of information about IRT and CTT, precision and consistency, and the whole shebang. Because AI likes being comprehensive. And that’s wonderful.

But comprehensive information is only useful when it is accurate. And that’s where AI needs to be put in the back seat and humans should be able to take the lead.

That’s where our own expertise and judgement come into the picture.

So, being the skeptic that I am about AI-generated information, I cross-questioned it.

The question wasn’t the best piece of writing I had ever produced, decorated with fancy words and elegant metaphors. It was raw. Rough around the edges. But it still packed the idea I was attempting to put across in a fairly articulated fashion.

That’s another feature of AI I personally admire.

You don’t have to carefully craft every question to get a useful response. You can shoot off a roughly drafted thought, and AI will position it within the broader context of the discussion and everything else you have discussed previously and return a response that is tailored to you.

As for accuracy, though, you need a sharp eye to spot the anomalies.

Some of us might see this as a fair deal: you get truckloads of information, and all you have to do is put your human judgement to work to determine how much of it is factually correct.

I personally subscribe to this view.

But there are others who would see it as a disadvantage- having to eyeball large amounts of information and look for errors and inconsistencies, with the possibility of missing subtle errors as the volume of information increases.

But whether voluminous information generated by AI is a curse or a gift is a debate beyond the scope of this post.

You see.

I digressed again. 😄

Anyway, I shot back with a follow-up question:

“alpha is also essentially a measure of precision. From a CTT point of view alpha tells us what the ratio of true to error scores is. The higher the alpha the better the precision because error gets reduced”.

Notice how the phrasing almost makes me sound like a miffed little school kid who isn’t convinced about something because it goes against what their favourite teacher taught them.

But AI doesn’t take offence.

That’s a respite.

Instead, AI very gracefully accepted its defeat (the school kid in me grins in ALL CAPS 😁) and responded:

“Yes — your correction is right, and my earlier wording drew too sharp a distinction between CTT reliability and IRT reliability.”

I felt a profound sense of achievement.

So, one thing we certainly know by now is that EAP is a reliability coefficient that represents measurement precision.

Now let’s go back to the question I originally asked AI:

“What makes EAP different from other conventional reliability coefficients?”

EAP, short for Expected A Posteriori, is an IRT-based estimate of overall measurement reliability.

The important distinction is not that EAP measures “precision” while alpha and omega measure “consistency.” Alpha and omega are also measures of reliability and, therefore, measurement precision.

The real advantage of the IRT framework is that it allows us to go a step further.

Rather than assuming that measurement precision is uniform across the entire ability continuum, IRT allows us to examine how measurement precision changes at different levels of the latent trait.

In simpler words: a test can be very precise for candidates around one ability level and considerably less precise for candidates at another. Or, in other words, it is quite possible for a test to measure low-ability candidates very precisely but high-ability candidates rather poorly or vice versa.

And this is where things get interesting.

The overall EAP reliability gives us a summary of the test’s reliability. But the IRT information function and conditional standard errors allow us to see where along the ability continuum that precision is coming from.

Think of it like a weighing machine designed to weigh accurately up to 100 kg. Anything beyond that, and the machine starts giving increasingly inaccurate readings.

You wouldn’t necessarily throw the machine away.

You’d simply understand its range.

And you certainly wouldn’t use it to make a high-stakes decision about whether someone weighs 105 or 120 kg if you already know the machine becomes unreliable beyond 100 kg.

The same principle applies to assessment.

If your test is highly precise around the average ability level but provides poor measurement precision at the upper end, you need to think carefully before using it to make fine-grained or high-stakes decisions about high-ability candidates.

So, two lessons to take away from this rather long and somewhat digressive post:

1. EAP is an IRT-based measure of overall reliability and precision. The IRT framework, through information functions and conditional error, also allows us to understand how measurement precision varies across different ability levels.

2. AI is a magnificent work buddy as long as we delegate it subordinate roles and take the information it offers with a healthy pinch of salt.

And perhaps there is a third lesson:

Never underestimate the value of a miffed school kid who keeps asking questions. 😄

EAP, Measurement Precision & My Tug-of-War with ChatGPT was originally published in Psochometry on Medium, where people are continuing the conversation by highlighting and responding to this story.

Most people who have studied statistics are familiar with the four levels of measurement; nominal, ordinal, interval, and ratio. These describe how an observed variable is measured and determine the kinds of mathematical and statistical operations that can be meaningfully performed.

What is discussed far less often, however, is the response format of an assessment item and the type of data it generates. Although these concepts are related, they describe two different aspects of measurement and should not be confused.

Let’s understand the difference. Suppose we want to measure the construct Neuroticism. The construct itself remains the same, but it can be operationalized in different ways:

• A measure may simply classify individuals as “Neurotic” or “Non-neurotic.” Here, the observed variable is categorical.
• Another measure may classify individuals as Low, Moderate, or High Neuroticism, producing ordered categories.
• Alternatively, a neuroticism questionnaire may produce a numerical total score (e.g., 18, 24, or 31), with higher scores indicating higher levels of neuroticism. In practice, these total scores are often treated as approximately interval for statistical analysis, although they are derived from ordinal item responses.

These examples relate to the level of measurement of the observed variable.

Response format, however, is a different concept.

Instead of asking how the final score is represented, response format asks how many response options does each item provide?

For example, suppose a neuroticism questionnaire consists of 20 Yes/No items. An individual item might ask:

I often worry about things that may never happen.

The respondent can answer only Yes or No. Since each item has exactly two response categories, the response format is dichotomous.

After responding to all 20 items, the individual receives a total score ranging from 0 to 20. This total score is then used to classify the person’s level of neuroticism (e.g., Low, Moderate, or High) or may be treated as a continuous score for statistical analysis.

Notice what has changed: the item responses remain dichotomous, but the final score is no longer binary. In other words, the response format of the items and the measurement level of the resulting score are two different concepts.

Why does this distinction matter?

Because the response format determines which Item Response Theory (IRT) model is appropriate. Dichotomous items are typically analysed using dichotomous IRT models such as:

• Rasch Model (1PL)
• One-Parameter Logistic Model (1PL)
• Two-Parameter Logistic Model (2PL)
• Three-Parameter Logistic Model (3PL)
• Four-Parameter Logistic Model (4PL)

Whereas Polytomous items require polytomous IRT models, including:

• Graded Response Model (GRM)
• Partial Credit Model (PCM)
• Generalized Partial Credit Model (GPCM)
• Rating Scale Model (RSM)

Key Takeaways

• Levels of measurement (nominal, ordinal, interval, ratio) describe how an observed variable or score is measured.
• Dichotomous and polytomous describe the response format of individual assessment items.
• Choosing the appropriate psychometric or IRT model depends primarily on the response format.

Although this distinction may seem subtle at first, it becomes critically important when selecting the appropriate statistical procedures for scale development and psychometric validation.

Dichotomous versus Polytomous data was originally published in Psochometry on Medium, where people are continuing the conversation by highlighting and responding to this story.

7. Observed Behaviour > Self-Reported Behaviour-

While many conventional assessment instruments claim to measure behaviour, the extent to which they actually capture behaviour, rather than people’s perceptions of their own behaviour, is debatable.

Asking someone to rate themselves on a scale primarily reflects their understanding of themselves, rather than providing direct evidence of how they are likely to behave.

Behavioural assessments are so named because they measure behavioural attributes such as personality, attitudes, motivation, and cultural fit (as opposed to cognitive tests that measure abilities such as reasoning, problem-solving, and decision-making). They are not necessarily behavioural because they observe actual behaviour.

This distinction is often overlooked.

Game-based assessments, on the other hand, don’t just measure behavioural constructs, they do so by capturing demonstrated behaviour within realistic, job-relevant scenarios rather than relying solely on individuals’ beliefs about how they would behave.

This is one of the key advantages game-based assessments offer over text-based SJTs. Instead of asking candidates to imagine themselves in a situation, they place them in one. Rather than asking how they respond under pressure, these assessments require candidates to make decisions in timed, interactive tasks that simulate the kinds of challenges they are likely to encounter on the job.

As a result, game-based assessments provide richer and more nuanced behavioural insights than traditional self-rating scales, enabling organizations to make hiring decisions based on demonstrated behaviour rather than self-perceptions.

8. Lower Risk of AI-Assisted Cheating-

Copy. Paste. Ask AI. Submit.

That’s how easily a conventional text-based assessment can sometimes be compromised.

Candidates can feed questions into an AI tool, generate polished responses, and return answers that may have little to do with their actual capabilities or behavioural tendencies.

Now imagine an assessment where the candidate has to navigate a changing scenario, make decisions in real time, solve problems, and adapt to new information.

That’s where game-based assessments have an advantage.

Well-designed gamified tasks make it considerably harder to outsource the assessment to AI because the interaction itself is the assessment.

AI is evolving. So should our assessments.

Gamified Talent Assessment Tools- 3/3 was originally published in Psochometry on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why should hiring managers consider gamified assessments over conventional rating scales and text-based SJTs?

4. Lower practice effects –

If your organization uses the same set of assessment questions repeatedly, candidates may eventually become familiar with the items. Instead of carefully evaluating each question, they begin responding on autopilot with reduced cognitive engagement. This phenomenon, commonly referred to as practice effect, can compromise response validity and, ultimately, assessment accuracy.

Well-designed game-based assessments offer a compelling advantage. Rather than relying on repetitive rating-scale items, they immerse candidates in realistic, job-relevant scenarios that require critical thinking, decision-making, creativity, and problem-solving. The tasks are designed so that there is no obvious “best” response to infer or memorize.

Many game-based assessments also allow scenarios, visual elements, or task sequences to be randomized, making it far more difficult for candidates to benefit simply because they have encountered the assessment before.

The result? Less influence of item familiarity, greater cognitive engagement, and more authentic behavioural data, leading to more accurate hiring decisions.

5. Higher ecological validity-

Ecological validity refers to the extent to which an assessment reflects real-world situations and predicts how candidates are likely to behave in natural settings.

It’s one thing to give yourself the highest possible score on a question asking how good you are at problem-solving. But being able to solve actual problems as part of an assessment is a completely different ball game.

If an assessment incorporates realistic, job-relevant challenges and a candidate performs well on those tasks, it is likely to be a much stronger indicator of role fit than a score derived from a self-rating scale measuring problem-solving ability.

Well-designed game-based assessments immerse candidates in scenarios that closely resemble the day-to-day situations encountered in the role. The closer these simulations are to real workplace challenges (i.e. the higher their simulation fidelity), the greater the ecological validity of the assessment.

The result? Employers can make hiring decisions with greater confidence because they’re evaluating demonstrated behaviour in realistic contexts and not just candidates’ opinions about themselves.

6. Higher Engagement –

When was the last time you were genuinely excited about taking a self-rating assessment?
How many of us look forward to selecting “Strongly Agree” to a question like: “Does your manager involve you in important decision-making?” (Pun intended! 😉)

Let’s be honest; traditional assessments can often feel repetitive and monotonous. Reading through 30 or 40 text-based questions, many of which may not seem relevant, isn’t exactly an engaging experience.

If this resonates with you, you’re not alone.

A growing body of research suggests that candidates generally prefer game-based and immersive assessments over conventional rating scales. More importantly, higher engagement means candidates are more likely to remain attentive throughout the assessment, reducing careless responding and improving the quality of the data collected.

At Psochometry, we believe assessments shouldn’t feel like a chore. That’s why we design immersive, job-relevant tasks that feel more like solving real workplace challenges than completing another questionnaire. The result is an assessment experience that not only keeps candidates engaged but also captures richer behavioural evidence for more informed hiring decisions.

Gamified Talent Assessment Tools- 2/3 was originally published in Psochometry on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why should hiring managers consider gamified assessments over conventional rating scales and text-based SJTs?

The answer isn’t because they’re more “fun.” It’s because, when designed scientifically, they can produce better quality assessment data.

Gamified assessments can offer several advantages over traditional self-report questionnaires and text-heavy SJTs. This article highlights some of the advantages of game-based assessments:

1. Lower construct transparency — 

Gamified items reduce construct transparency or the ability to discern the underlying construct being measured. Achieving this is more challenging with text-based items, where the choice of words, sentence framing, and wording can make it evident what construct the item is intended to assess. Making it more difficult for respondents to infer what is being measured promotes greater authenticity in responses, as it reduces their ability to manipulate answers to present themselves as more competent or deserving.

2. Reduced Social Desirability Bias –

Every person taking an assessment as part of a hiring process makes every possible effort to be seen as a high-potential, deserving candidate. This tendency gives rise to what is known as social desirability bias or the proclivity to choose responses that are viewed more positively rather than those that truly represent an individual’s psychosocial attributes.

This challenge has bewildered test developers for a long time, and multiple methods have been introduced to minimize it. These include designing response options in a way that the highest-rated choice does not stand out, or providing multiple equally desirable but unrelated alternatives. However, as long as assessments rely heavily on direct self-reporting, some degree of social desirability bias is likely to persist.

This is where game-based assessment can become a game changer. When designed around realistic scenarios and behavioural decision-making, game-based tasks can make impression management more difficult and reduce some of the social desirability bias commonly observed in traditional self-report assessments.

Scenario-based, gamified role-play exercises allow respondents to choose how they would respond in specific situations. These choices provide valuable insights into core personality traits and underlying psychological attributes.

The caveat, however, is that gamified items must be developed with great care and caution. They need to be thoroughly validated, standardized, and evaluated to ensure that they measure what they are intended to measure.

3. Elimination of central tendency bias associated with rating scales–

Most assessment scores do not perfectly represent the underlying traits or behaviours they are intended to measure. One reason is that they are often contaminated by response biases or systematic tendencies to respond in ways that are unrelated to the construct being measured, thereby reducing assessment precision.

One such bias is central tendency bias.

It refers to the tendency of respondents to repeatedly choose the middle or neutral option on a rating scale instead of selecting the response that most accurately reflects their true behaviour or opinion. This is particularly common in Likert-type scales (e.g., 5-point or 7-point agreement-disagreement scales).

Why does this happen? Sometimes respondents are genuinely neutral. However, in many cases, they may choose the middle option because they are uncertain, reluctant to commit to an extreme response, or simply because it feels like the safest and least effortful choice.

This presents a challenge for assessment developers. It becomes difficult to distinguish between respondents who are truly neutral and those who are merely defaulting to the midpoint, introducing measurement error into the assessment.

One common solution is to use an even-numbered rating scale (such as a 4-point scale) that removes the neutral option altogether. However, this isn’t always appropriate. In many situations, neutrality is a legitimate response and forcing individuals to take a side may introduce a different form of measurement error.

This is where well-designed game-based assessments offer a distinct advantage.

Instead of asking candidates to rate themselves on generic statements, they place them in immersive, job-relevant situations where behaviour is demonstrated through decisions and actions rather than self-reported ratings. Because these tasks typically don’t rely on agreement-disagreement scales, they largely avoid the central tendency bias associated with traditional rating scales.

Consider this example of a traditional Item -

“Do you help your teammates when they are unable to solve a problem?”
Strongly Disagree
Disagree
Neutral
Agree
Strongly Agree

Now imagine a game-based task instead-

You’re part of a team trying to complete a mission. Several teammates are struggling to progress to the next level. You must decide whether to spend your limited resources helping them move forward or continue independently. Every teammate who falls behind weakens the team’s overall performance.

Notice the difference.

One asks candidates to describe how they think they behave. The other requires them to demonstrate how they choose to behave in a realistic context.

And that’s precisely where game-based assessments can offer richer behavioural insights than conventional rating scales.

Gamified Talent Assessment Tools- 1/3 was originally published in Psochometry on Medium, where people are continuing the conversation by highlighting and responding to this story.

Folks who use psychometric assessments for talent decisions, how often do you worry about a test being too long? In my experience, the length of a scale is often the biggest deal-breaker for most corporate clients. It makes sense not to want candidates spending hours on an initial screening assessment. That said, brevity should not come at the cost of precision.

I have spent hours in client engagement meetings explaining why it is not advisable to assess five competencies with a 10-item tool. Even if the final tool is expected to be compact, there must be an ample item pool that is piloted on a sample to ensure quality and rigor.

Think of it this way-

Suppose a candidate is asked if they like planning their tasks in advance, and they select Strongly Agree. Should you conclude that they are genuinely organized? Not necessarily. A single response can be influenced by many factors unrelated to the trait you’re trying to measure; such as momentary inattention, misreading the question, misunderstanding the wording, or accidental clicks.

This is why psychometric assessments rarely rely on a single item to measure an important characteristic. Instead, they include multiple items that assess the same underlying construct from slightly different angles. If a respondent answers these items consistently, we gain greater confidence that the score reflects the trait being measured rather than random error or chance.

In measurement science, this principle is fundamental: more high-quality information generally leads to more reliable inferences. Each well-designed item provides an additional piece of evidence, allowing the assessment to distinguish genuine patterns from noise.

This is the primary reason psychometricians recommend using sufficiently large item pools. A larger pool doesn’t simply make an assessment longer, it increases the amount of evidence available to support a score, resulting in more dependable measurement.

There are additional technical reasons why larger item pools improve assessment quality, but those involve concepts from modern psychometric theory that are beyond the scope of this article.

The bottom line is that if you ask your assessment partner to create a 10-item scale that captures 4–5 different constructs and they don’t flag this requirement as problematic, you’d be better off finding another vendor.

Psychometry Explainer #4 was originally published in Psochometry on Medium, where people are continuing the conversation by highlighting and responding to this story.