A Systematic Approach To Interrater Reliability In Early Childhood Teacher Performance Evaluations

1 online resource (112 pages) : PDF

2022

University of North Carolina at Charlotte

Teacher evaluations are routinely conducted across the United States for licensure and professional development supports. However, there is limited research on the interrater reliability of these evaluation assessment systems, despite federal recommendations (Graham et al., 2012). This research explores the systematic approach to interrater reliability utilized by the Early Educator Support (EES) Office in North Carolina. The EES Office supports the Birth-through-Kindergarten (B-K) teacher licensure of over 900 early educators in both private and public sectors. The evaluators employed undergo extensive trainings and hold a B-K license themselves. As part of the training, the evaluators undergo an interrater reliability activity that requires them to act as raters who rate ten fictitious teacher profiles, using the North Carolina Teacher Evaluation Process (NCTEP) rubric. This research aimed to understand the rater response process. In this study, three Many Facets Rasch Models are used to understand rater patterns of strictness, leniency and potential bias based on the race of teacher profile. Additionally, two of the models are compared to understand the extent that these rater response patterns are exhibited in their real caseloads of actual early educators. In conclusion, the group of raters do show evidence of strictness, leniency, and bias, however it is mostly exhibited by a small number of individual raters. It is possible to use the results to inform the professional growth of these evaluators, so that all early educators served by the EES Office receive valid, fair, and reliable teacher evaluations. Furthermore, it depicts a systematic approach to interrater reliability that could be used by other evaluation and assessment systems across the country.

doctoral dissertations

Educational evaluation

Ph.D.

Educator Evaluation
Many Facets Rasch Models
Teacher Licensure

Educational Leadership

Lambert, Richard

Westine, Carl
Shore, Rebecca
Anderson, Kelly

Thesis (Ph.D.)--University of North Carolina at Charlotte, 2022.

This Item is protected by copyright and/or related rights. You are free to use this Item in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s). For additional information, see http://rightsstatements.org/page/InC/1.0/.

Copyright is held by the author unless otherwise indicated.

Bottoms_uncc_0694D_13078