An interesting article about the Duolingo English Test in “Language Testing” this month.
Researchers shared responses to DET items (speaking only) with relevant stakeholders (faculty, administrators, students) and asked them to judge the comprehensibility and “academic acceptability” of the speakers (based solely on what they heard in the responses, of course). These judgements correlated highly with the DET scores. Test-takers with higher-scoring responses were judged to have more comprehensibility and potential academic acceptability.
I suppose it is not a great surprise that people with strong English were judged to be more comprehensible that people with weak English… but this is the sort of research that must be done if we are going to rely on automated scoring for high-stakes testing.