Lambert, Richard
Holcomb, Scott
Bottoms, Bryndle

Faculty

1 online resource

2022

technical reports

text

The validity of the Kappa coefficient of chance-corrected agreement has been questioned when the prevalence of specific rating scale categories is low and agreement between raters is high. The researchers proposed the Lambda Coefficient of Rater-Mediated Agreement as an alternative to Kappa to address these concerns. Lambda corrects for chance agreement based on specific assumptions about raters and the rater-mediated assessment process including raterspecific tendencies for strict or lenient ratings. Actual ratings of teacher profiles from an interrater reliability exercise confirmed the shortcomings of Kappa when used within the teacher performance evaluation process. Rater data also demonstrated the robustness of Lambda and Gwet’s AC-1 to the data conditions known to be problematic for Kappa. All alternative chancecorrected agreement coefficients evaluated showed less variability across the 57 raters than Kappa. Simulation results demonstrated the robustness of the Lambda Coefficient of RaterMediated Agreement to the data conditions that are problematic for Kappa.

Educational evaluation
Teachers--Rating of

CEME technical report

English

https://ceme.charlotte.edu/sites/ceme.charlotte.edu/files/media/NCME%202...

http://hdl.handle.net/20.500.13093/work:867

This item may be protected by copyright and other related rights. Atkins Library provides access to this item for educational and research purposes only; other uses require the permission of the copyright holder.