Great article this week in “Educational Measurement” about remote proctoring.
The abstract says:
“The main role of remote proctors is to make judgments about test takers’ behaviors and decide whether these behaviors constitute rule violations. Variability in proctor decision making, or the degree to which humans/proctors make different decisions about the same test-taking behaviors, can be problematic for both test takers and test users.”
No kidding. This has driven me bonkers since 2020.
“Our results show that (1) proctors systematically differ in their decision making and (2) these differences are trait-like (i.e., ranging from lenient to strict), but (3) systematic variability in decisions can be reduced.”
The reports I have received from test-takers over the years suggest that when done on a wide scale proctoring is sometimes inconsistent. This issue has negatively impacted many of the test-takers I’ve communicated with.
Says the article:
“Taken together, a lack of proctor training and incomplete information about test-taking behaviors provide the foundation upon which more extraneous factors can influence proctors’ decisions. In fact, a considerable amount of research has demonstrated that human decision making is highly variable and due to a variety of idiosyncratic factors.”
It concludes that:
“Reducing variability in proctor decision making not only improves test takers’ outcomes but also strengthens test security: honest, well-intentioned test takers are more likely to receive certified test scores, and dishonest test takers are less likely to receive certified scores. Test users (e.g., university admissions) can also have greater confidence that certified test results are attributable to test takers’ abilities (e.g., ability to use the English language in an academic setting) when proctor decision making is more consistent.”
Alas, after four years of what I view as sub-par remote proctoring, I fear that many large testing firms don’t feel any urgency to improve this aspect of test-taker experience.