It has been argued that the new “Writing for an Academic Discussion” task requires test takers to write fewer words overall, every grammar or language use mistake they make will have a greater impact on their score. I figured I would run a few tests in the e-rater to see if this is the case.
First up, this response got a score of 5.0:
“I like the ideas noted by Claire and Kelly, but I feel that the only way to truly solve this problem is to build better schools. Parents want their children to be educated at the best possible places. It is almost impossible to find great schools in the country. The government currently has money for good things like science laboratories and nice libraries in cities so they should not ignore rural areas all the time. When rural schools like the one I attended lack even basic educational supplies like computers and sports equipment, parents who are concerned about their kids go to bigger places.”
If I submit the same response with all of the commas removed I get a score of 4.0.
If I submit the original response but with ONE spelling mistake (truely) I get a score of 4.0.
So does a small number of individual errors have the potential to impact the score of a test taker? It would seem so.
However, it isn’t as clear cut as it seems. While the human raters give only whole number scores, it seems like there the e-rater is using decimals under the hood. I imagine that my original response is a “low 5” answer (maybe just a 4.5, rounded up).
Here is an answer where I retain the spelling mistake, but beef up my vocabulary usage:
“I respect the ideas noted by Claire and Kelly, but I feel that the only way to truely solve this problem is to construct better schools. Parents want their children to be educated at the best possible facilities. It is almost impossible to find great schools in the countryside. The government currently provides funding for beneficial things like science laboratories and lavish libraries in cities, so they should not ignore rural areas all the time. When rural schools like the one I attended lack even basic educational supplies like computers and sports equipment, parents who are concerned about their kids depart for bigger places.”
This time, my score goes back to 5.0. I can use the same technique to overcome the penalty for not using commas. Actually, I can use the same technique to overcome the problem of not using conjunctions, which I have written about elsewhere. I bet it works both ways.
Decimals, right? I gain a few tenths of a point for improving my vocabulary and that compensates for the few tenths of a point I lost for the spelling mistake. We’ve seen a similar thing in some implementations of the SpeechRater AI.
Anyway. This might be useful for people preparing for the test. Test-takers ignore the quirks of automated scoring at their own peril.