Lambda Coefficient of Rater-Mediated Agreement: Evaluation of an Alternative Chance-Corrected Agreement Coefficient

Search for this publication on Google Scholar
Holcomb, T. (2022). Lambda Coefficient of Rater-Mediated Agreement: Evaluation of an Alternative Chance-Corrected Agreement Coefficient. Unc Charlotte Electronic Theses And Dissertations.
Download PDF

Analytics

19 views ◎
7 downloads ⇓


Abstract

In this study, the performance of the Lambda Coefficient of Rater-Mediated Agreement was evaluated with other chance-corrected agreement coefficients. Lambda is grounded in rater-mediated assessment theory and was developed as an alternative to Kappa (Cohen, 1960) and other chance-corrected agreement coefficients. Lambda has two variations, a general form that is calculated similarly to how most chance-corrected agreement coefficients are calculated, such as Kappa (Lambert et al., 2021). The general form of Lambda is referred to as Lambda-1. Lambda-2 differs from Lambda-1 in the calculation of the proportion of expected chance agreement. Lambda-2 uses known population proportions when available and applies those proportions in the calculation of expected chance agreement. In total, six coefficients were calculated using generated data by varying the amount and location of agreement and disagreement between ratings across two-, three-, and four-point rating scales. The exact agreement specifications ranged from 75% to 95% across 115 planned data conditions. The simulations adjusted prevalence indices according to exact agreement specifications (Xie, 2013). Results demonstrated the robustness of Lambda-1 and Lambda-2 to data conditions that are problematic for other coefficients. Both variations of Lambda produced benchmark agreement results that maintained meaning that may be diminished by other coefficients.

Details
Author

Holcomb, Timothy

Title

Lambda Coefficient of Rater-Mediated Agreement: Evaluation of an Alternative Chance-Corrected Agreement Coefficient

Physical Description

1 online resource (278 pages) : PDF

Date

2022

Degree Granting Institution

University of North Carolina at Charlotte

Abstract

In this study, the performance of the Lambda Coefficient of Rater-Mediated Agreement was evaluated with other chance-corrected agreement coefficients. Lambda is grounded in rater-mediated assessment theory and was developed as an alternative to Kappa (Cohen, 1960) and other chance-corrected agreement coefficients. Lambda has two variations, a general form that is calculated similarly to how most chance-corrected agreement coefficients are calculated, such as Kappa (Lambert et al., 2021). The general form of Lambda is referred to as Lambda-1. Lambda-2 differs from Lambda-1 in the calculation of the proportion of expected chance agreement. Lambda-2 uses known population proportions when available and applies those proportions in the calculation of expected chance agreement. In total, six coefficients were calculated using generated data by varying the amount and location of agreement and disagreement between ratings across two-, three-, and four-point rating scales. The exact agreement specifications ranged from 75% to 95% across 115 planned data conditions. The simulations adjusted prevalence indices according to exact agreement specifications (Xie, 2013). Results demonstrated the robustness of Lambda-1 and Lambda-2 to data conditions that are problematic for other coefficients. Both variations of Lambda produced benchmark agreement results that maintained meaning that may be diminished by other coefficients.

Genre

doctoral dissertations

Subjects--Topics

Educational tests and measurements
Educational evaluation

Degree

Ph.D.

Keywords

Chance-Corrected Agreement Coefficient
Interrater Reliability
Rater Mediated Assessment
Rater Mediated Assessment Theory
Reliability

Subject Area

Educational Leadership

Advisor(s)

Lambert, Richard

Committee Members

Kim, Stella
Westine, Carl
Cash, Anne

Degree Note

Thesis (Ph.D.)--University of North Carolina at Charlotte, 2022.

Rights Statement

This Item is protected by copyright and/or related rights. You are free to use this Item in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s). For additional information, see http://rightsstatements.org/page/InC/1.0/.

Rights Holder Information

Copyright is held by the author unless otherwise indicated.

Identifier

Holcomb_uncc_0694D_13056

Permalink

http://hdl.handle.net/20.500.13093/etd:2962