An interesting article about the Duolingo English Test in “Language Testing” this month.

Researchers shared responses to DET items (speaking only) with relevant stakeholders (faculty, administrators, students) and asked them to judge the comprehensibility and “academic acceptability” of the speakers (based solely on what they heard in the responses, of course). These judgements correlated highly with the DET scores. Test-takers with higher-scoring responses were judged to have more comprehensibility and potential academic acceptability.

I suppose it is not a great surprise that people with strong English were judged to be more comprehensible that people with weak English… but this is the sort of research that must be done if we are going to rely on automated scoring for high-stakes testing.

Here’s the March update to the TOEFL/Duolingo Score Requirement tracker.  I didn’t post it here in March since I’m losing interest in it, but I did get a request.  The point here is to discover if schools update their Duolingo cut scores in light of update conversation tables (from Duolingo).  In March, Rice University increased their requirement by ten points.  Also, updated wording on UCLA’s page seems to suggest that they will permanently accept Duolingo scores.

You may have to scroll sideways to see the most recent numbers.

I’ll do an April update towards the end of the month.

Just published in “Language Testing” are the results of a study into the predictive validity of the Duolingo English Test. It notes:

1. At the postgraduate level, DET scores were related to academic results. Students with higher DET scores had more academic success. At the undergraduate level, DET scores were not related to academic success. The same is true of TOEFL and IELTS scores.
2. Students accepted with DET scores enjoyed lower academic success than those accepted with TOEFL and IELTS scores.
3. Given the above, perhaps higher DET cut scores are needed. Or perhaps students arriving with DET scores may need additional academic support to reach their full potential.
4. The lower academic success of DET test-takers may have something to do with the test itself. Or it may have nothing to do with the test. There are many things that affect academic success.

I note:

Perhaps early access to the results of this study encouraged Duolingo to adjust its score conversion tables. They did that some months ago and schools that pay attention to that sort of thing have already adjusted their cut scores, as I’ve reported over the past eight or nine months.

The cost of taking the Duolingo English Test has increased to $59. According to customer support, Duolingo is “gradually rolling out this new price.” I can already see the new price in my account. On April 2, the price was $49.

A new article in British newspaper i is critical of the Duolingo English Test.  It reports that:

“Professor John Heathershaw from the University of Exeter linked the acceptance of ‘things like Duolingo tests’ to lower English language standards and described it as a ‘major issue’ when he appeared before the Foreign Affairs Select Committee last week.”

And:

“Aston University in Birmingham is one of the many UK institutions that adopted Duolingo at the height of the pandemic but has now dropped it. A spokesperson told i that the decision was taken because of concerns about student performance.”

And quoting a spokesman from that university:

“There is no evidence that those students on Duolingo were failing or, indeed, fraudulent, but just not performing at the same level as peers on their course, so we have chosen to remove acceptance for the test.”

–

I’m going to babble a bit now.  Pardon me.

Older tests like the TOEFL and IELTS are designed around a particular conception of validity. For instance, the makers of the TOEFL would likely argue that we can look at a TOEFL score and make inferences about the test-taker’s ability to do well in university classes because the tasks on the TOEFL resemble (and, more importantly, involve the same knowledge, skills and processes) as the tasks done by students at post-secondary institutions.

Similarly, IDP would tell you that the IELTS General Training Module is a great test for immigration because it includes tasks that resemble what we do in everyday life.  Meanwhile, the TOEIC leans heavily on memos and invoices and emails that people encounter in an office environment.

You can read about this concept of validity in Carol Chapelle’s 2008 doorstopper about the creation of TOEFL iBT.

The Duolingo English Test is a little bit different.  It is certainly a test of one’s English abilities.  That much is obvious.   But is it appropriate for university admissions? While questions of an academic nature have been added to the DET in recent months much of the test score is still determined by “describe this picture” tasks and “fill in the missing letters” tasks that don’t closely resemble things done on a university campus.

Duolingo might argue that such tasks totally suit the purpose of the test and that they really do require the relevant skills and knowledge.  They could be right.  Who knows?

What interests me is that if we reject the idea that validity requires that test items closely resemble tasks performed in real-world contexts we can go ahead and discard all of the older tests and use the Duolingo Test for all possible purposes.  Will the receiving institutions bite?  That remains to be seen.

Duolingo has just published its quarterly earnings report. You can get the lowdown on their site.

Revenues from the Duolingo English Test hit a new high of 8.4 million dollars for the quarter.  That’s up from revenues of 8.1 million dollars in Q4 of 2021. 

At $49 a pop, we can assume that the test was taken about 171 thousand times in the quarter.  The real number is likely higher than that due to freebies and discounts (at one point, Duolingo offered a deal for users who bought two instances of the test).

Here are the historic revenues of the test:

• Q4 2022 – 8,410,000
• Q3 2022 – 8,192,000
• Q2 2022 – 8,036,000
• Q1 2022 – 8,080,000
• Q4 2021 – 8,095,000
• Q3 2021 – 6,695,000
• Q2 2021 – 4,833,000
• Q1 2021 – 5,035,000
• Q4 2020 – 4,197,000
• Q3 2020 – 5,607,000
• Q2 2020 – 4,598,000
• Q1 2020 – 753,000

This suggests the test was taken about 667,000 times (plus freebies and discounts) in 2022.

Just in case you didn’t know: the reading passages on the Duolingo English Test are generated by GPT. Yes, the same GPT that Powers ChatGPT.

Look:

The original document is available here.

The Duolingo English Test has removed suggested word counts and the word counter from writing questions. Instead of saying “Respond to the question in at least 50 words” the test now says “Write about the topic for 5 minutes.”

That’s great. Anyone who has been reading my blog for some time will know that misleading statements on tests about word counts drive me bonkers.

You can see this change both on the real test and on the free practice test provided on the DET website.

My last post reminded me of some advice I’ve given numerous times. Which is that if you are running a large-scale assessment, you should link your free practice material to user accounts. That way you have a never-ending faucet of useful and actionable data.

This is what Duolingo has done all along. Test watchers have noticed they way they stick new types of questions into the practice test now and then. Some of these have later gone on to appear on the actual test. Others have not.

The new 60-minute practice test is just an extension of this. I look forward to taking it.

It is somewhat surprising that the providers of legacy tests aren’t doing this already. I understand that it is easier for Duolingo to do this because most of their questions are machine generated, but if you’ve got access to decades worth of retired test forms you can just rotate through them as needed.

If you go to your Duolingo English Test account and click to take the 15 minute practice test, you will now be offered an opportunity to take a full 60 minute practice test. All sections of the practice test are scored. A nice opportunity if you are preparing for the DET. I don’t know for sure, but my guess is that you can take the full version only once.

I updated the TOEFL and Duolingo Score requirement tracker for December, and discovered that Carnegie Mellon reduced its Duolingo English Test score requirement down to 125.  This after increasing it to 135 in November.  Weird.

l updated the Duolingo/TOEFL score requirement tracker for November.  I was actually surprised to spot a couple of changes.  Carnegie Mellon increased their Duolingo score requirement by 10 to a total of 135 points.  That ties them with Columbia for the highest Duolingo score requirement.

I also noticed that the University of British Columbia now requires students to submit a waiver if they are going to use Duolingo Scores in their admission.

According to SEC filings from a few days ago, revenue for the Duolingo English Test in Q2 2022 was $8,192,000.  That’s a 22% increase over the same period a year ago.

At $49 a pop, we might extrapolate that the test was taken 167,000 times in the quarter. The actual number is probably a little bit higher due to discounts and free tests.

Here are the historic revenues (all USD):

Q3 2022 – 8,192,000
Q2 2022 – 8,036,000
Q1 2022 – 8,080,000

Q4 2021 – 8,095,000
Q3 2021 – 6,695,000
Q2 2021 – 4,833,000
Q1 2021 – 5,035,000

Q4 2020 – 4,197,000
Q3 2020 – 5,607,000
Q2 2020 – 4,598,000
Q1 2020 – 753,000