Inter-rater Reliability in R

Last Updated : 23 Jul, 2025

Inter-rater reliability (IRR) is a measure of the degree of agreement among raters or judges. It is an essential aspect of any research involving subjective assessments, ensuring that the data collected is consistent and reliable. In this article, we will explore various methods to calculate inter-rater reliability using the R Programming Language.

What is Inter-rater Reliability?

Inter-rater reliability refers to the degree of agreement among different observers or raters. It is a critical component of research that involves qualitative data, coding, or any scenario where multiple individuals are assessing the same data. High inter-rater reliability indicates that the raters have a similar understanding and interpretation of the criteria being assessed.

Importance of Inter-rater Reliability

  • Consistency in Data: Ensures that the data collected is not dependent on a single rater's perspective, thus making the findings more robust.
  • Validity: Enhances the validity of the study by confirming that the assessment criteria are well-understood and consistently applied.
  • Bias Reduction: Reduces potential biases that can arise from individual rater’s subjectivity.

Methods to Measure Inter-rater Reliability

Several statistical methods can be used to measure inter-rater reliability. The choice of method depends on the type of data and the number of raters. Some common methods include:

  • Cohen’s Kappa: Used for categorical data with two raters.
  • Fleiss’ Kappa: An extension of Cohen’s Kappa for more than two raters.
  • Intraclass Correlation Coefficient (ICC): Used for continuous data and can handle multiple raters.
  • Krippendorff's Alpha: Applicable for various data types and multiple raters.

Process of Inter-rater Reliability Analysis in R

R offers a powerful environment for calculating various inter-rater reliability coefficients. Here's a breakdown of the process:

Install and load Required Package

The irr package provides functions for calculating most common inter-rater reliability measures. Install it using the following command in your R console:

R 
install.packages(c("irr", "psych", "irrNA", "DescTools"))
library(irr)
library(psych)
library(irrNA)
library(DescTools)

Cohen’s Kappa

Cohen’s Kappa is a statistic that measures inter-rater agreement for categorical items. It accounts for the possibility of the agreement occurring by chance.

Let's create a simple dataset with two raters evaluating five items:

R 
ratings <- data.frame(
  rater1 = c(1, 2, 1, 2, 1),
  rater2 = c(1, 2, 1, 1, 2)
)

# Calculate Cohen's Kappa
kappa_result <- kappa2(ratings)
print(kappa_result)

Output:

 Cohen's Kappa for 2 Raters (Weights: unweighted)

 Subjects = 5 
   Raters = 2 
    Kappa = 0.167 

        z = 0.373 
  p-value = 0.709

The value of Cohen’s Kappa ranges from -1 to 1:

  • Values ≤ 0 indicate no agreement.
  • 0.01–0.20: Slight agreement.
  • 0.21–0.40: Fair agreement.
  • 0.41–0.60: Moderate agreement.
  • 0.61–0.80: Substantial agreement.
  • 0.81–1.00: Almost perfect agreement.

Fleiss’ Kappa

Fleiss’ Kappa is used when there are more than two raters. It generalizes Cohen’s Kappa to multiple raters.

Assume we have three raters rating five items:

R 
ratings <- matrix(c(
  1, 2, 1,
  2, 2, 1,
  1, 1, 1,
  2, 1, 2,
  1, 2, 2
), nrow = 5, byrow = TRUE)

# Calculate Fleiss' Kappa
fleiss_kappa <- kappam.fleiss(ratings)
print(fleiss_kappa)

Output:

 Fleiss' Kappa for m Raters

 Subjects = 5 
   Raters = 3 
    Kappa = -0.0714 

        z = -0.277 
  p-value = 0.782

Intraclass Correlation Coefficient (ICC)

The ICC measures the reliability of ratings for continuous data. It can also be used for more than two raters.

Consider continuous ratings from three raters on five items:

R 
ratings <- data.frame(
  rater1 = c(4.5, 3.2, 5.0, 3.7, 4.2),
  rater2 = c(4.0, 3.1, 4.8, 3.5, 4.0),
  rater3 = c(4.7, 3.3, 4.9, 3.6, 4.1)
)

# Calculate ICC
icc_result <- ICC(ratings)
print(icc_result)

Output:

Intraclass correlation coefficients 
                         type   est F-val df1 df2    p-val lwr.ci upr.ci
Single_raters_absolute   ICC1 0.927  39.1   4  10 4.47e-06     NA     NA
Single_random_raters     ICC2 0.928  71.8   4   8 2.63e-06     NA     NA
Single_fixed_raters      ICC3 0.959  71.8   4   8 2.63e-06     NA     NA
Average_raters_absolute ICC1k 0.974  39.1   4  10 4.47e-06     NA     NA
Average_random_raters   ICC2k 0.975  71.8   4   8 2.63e-06     NA     NA
Average_fixed_raters    ICC3k 0.986  71.8   4   8 2.63e-06     NA     NA

 Number of subjects = 5     Number of raters = 3

ICC values range from 0 to 1:

  • < 0.5: Poor reliability.
  • 0.5–0.75: Moderate reliability.
  • 0.75–0.9: Good reliability.
  • 0.9: Excellent reliability.

Krippendorff’s Alpha

Krippendorff’s Alpha is a versatile measure that can handle missing data and different types of measurement scales.

For a dataset with missing values:

R 
ratings <- data.frame(
  rater1 = c(1, 2, 1, NA, 1),
  rater2 = c(1, 2, 1, 1, 2),
  rater3 = c(1, 2, 1, 2, 1)
)

# Calculate Krippendorff's Alpha
alpha_result <- kripp.alpha(t(ratings), method = "nominal")
print(alpha_result)

Output:

 Krippendorff's alpha

 Subjects = 5 
   Raters = 3 
    alpha = 0.422

Conclusion

Inter-rater reliability is crucial for ensuring the consistency and validity of subjective assessments. R provides robust tools to calculate various inter-rater reliability statistics, making it easier for researchers to analyze their data and draw reliable conclusions. By using the irr, psych, irrNA, and DescTools packages, you can compute Cohen’s Kappa, Fleiss’ Kappa, ICC, and Krippendorff’s Alpha for your datasets, covering a wide range of scenarios and data types.

Comment

Explore