Examining inter-rater reliability of evaluators judging teacher performance : proposing an alternative to Cohen’s Kappa

Search for this publication on Google Scholar
Lambert, R., Holcomb, S., & Bottoms, B. (2021). Examining inter-rater reliability of evaluators judging teacher performance : proposing an alternative to Cohen’s Kappa. Ceme Technical Report.
Download PDF

Analytics

45 views ◎
10 downloads ⇓


Description

Faculty, The validity of the Kappa coefficient of chance-corrected agreement has been questioned when the prevalence of specific rating scale categories is low and agreement between raters is high. The researchers proposed the Lambda Coefficient of Rater-Mediated Agreement as an alternative to Kappa to address these concerns. Lambda corrects for chance agreement based on specific assumptions about raters and the rater-mediated assessment process including raterspecific tendencies for strict or lenient ratings. Actual ratings of teacher profiles from an interrater reliability exercise confirmed the shortcomings of Kappa. The rater data also demonstrated the robustness of Lambda-1, Lambda-2, Gwet’s AC1, and Gwet’s AC2 to the data conditions that are problematic for Kappa. All four alternative chance-corrected agreement coefficients showed less variability across the 45 raters than Kappa. However, AC-2 was undetermined for 39 of the 45 raters. Simulation data demonstrated the robustness of the Lambda Coefficient of Rater-Mediated Agreement to the data conditions that are problematic for Kappa.

Details
Title

Examining inter-rater reliability of evaluators judging teacher performance : proposing an alternative to Cohen’s Kappa

Author(s)

Lambert, Richard
Holcomb, Scott
Bottoms, Bryndle

UNC Charlotte Affiliate Type

Faculty

Extent

1 online resource

Date Published

2021

Genre

technical reports

Resource Type

text

Abstract

The validity of the Kappa coefficient of chance-corrected agreement has been questioned when the prevalence of specific rating scale categories is low and agreement between raters is high. The researchers proposed the Lambda Coefficient of Rater-Mediated Agreement as an alternative to Kappa to address these concerns. Lambda corrects for chance agreement based on specific assumptions about raters and the rater-mediated assessment process including raterspecific tendencies for strict or lenient ratings. Actual ratings of teacher profiles from an interrater reliability exercise confirmed the shortcomings of Kappa. The rater data also demonstrated the robustness of Lambda-1, Lambda-2, Gwet’s AC1, and Gwet’s AC2 to the data conditions that are problematic for Kappa. All four alternative chance-corrected agreement coefficients showed less variability across the 45 raters than Kappa. However, AC-2 was undetermined for 39 of the 45 raters. Simulation data demonstrated the robustness of the Lambda Coefficient of Rater-Mediated Agreement to the data conditions that are problematic for Kappa.

Subjects--Topics

Educational evaluation
Teachers--Rating of

Is Part Of

CEME technical report

Language

English

Publisher's Version

https://ceme.charlotte.edu/sites/ceme.charlotte.edu/files/media/IRR%20la...

Permalink

http://hdl.handle.net/20.500.13093/work:874

Rights

This item may be protected by copyright and other related rights. Atkins Library provides access to this item for educational and research purposes only; other uses require the permission of the copyright holder.