Just published in “Language Testing” are the results of a study into the predictive validity of the Duolingo English Test. It notes:

  1. At the postgraduate level, DET scores were related to academic results. Students with higher DET scores had more academic success. At the undergraduate level, DET scores were not related to academic success. The same is true of TOEFL and IELTS scores.
  2. Students accepted with DET scores enjoyed lower academic success than those accepted with TOEFL and IELTS scores.
  3. Given the above, perhaps higher DET cut scores are needed. Or perhaps students arriving with DET scores may need additional academic support to reach their full potential.
  4. The lower academic success of DET test-takers may have something to do with the test itself. Or it may have nothing to do with the test. There are many things that affect academic success.

I note:

Perhaps early access to the results of this study encouraged Duolingo to adjust its score conversion tables. They did that some months ago and schools that pay attention to that sort of thing have already adjusted their cut scores, as I’ve reported over the past eight or nine months.

A new article in British newspaper i is critical of the Duolingo English Test.  It reports that:

“Professor John Heathershaw from the University of Exeter linked the acceptance of ‘things like Duolingo tests’ to lower English language standards and described it as a ‘major issue’ when he appeared before the Foreign Affairs Select Committee last week.”

And:

“Aston University in Birmingham is one of the many UK institutions that adopted Duolingo at the height of the pandemic but has now dropped it. A spokesperson told i that the decision was taken because of concerns about student performance.”

And quoting a spokesman from that university:

“There is no evidence that those students on Duolingo were failing or, indeed, fraudulent, but just not performing at the same level as peers on their course, so we have chosen to remove acceptance for the test.”

I’m going to babble a bit now.  Pardon me.

Older tests like the TOEFL and IELTS are designed around a particular conception of validity. For instance, the makers of the TOEFL would likely argue that we can look at a TOEFL score and make inferences about the test-taker’s ability to do well in university classes because the tasks on the TOEFL resemble (and, more importantly, involve the same knowledge, skills and processes) as the tasks done by students at post-secondary institutions.

Similarly, IDP would tell you that the IELTS General Training Module is a great test for immigration because it includes tasks that resemble what we do in everyday life.  Meanwhile, the TOEIC leans heavily on memos and invoices and emails that people encounter in an office environment.

You can read about this concept of validity in Carol Chapelle’s 2008 doorstopper about the creation of TOEFL iBT.

The Duolingo English Test is a little bit different.  It is certainly a test of one’s English abilities.  That much is obvious.   But is it appropriate for university admissions? While questions of an academic nature have been added to the DET in recent months much of the test score is still determined by “describe this picture” tasks and “fill in the missing letters” tasks that don’t closely resemble things done on a university campus.

Duolingo might argue that such tasks totally suit the purpose of the test and that they really do require the relevant skills and knowledge.  They could be right.  Who knows?

What interests me is that if we reject the idea that validity requires that test items closely resemble tasks performed in real-world contexts we can go ahead and discard all of the older tests and use the Duolingo Test for all possible purposes.  Will the receiving institutions bite?  That remains to be seen.

 

Duolingo has just published its quarterly earnings report. You can get the lowdown on their site.

Revenues from the Duolingo English Test hit a new high of 8.4 million dollars for the quarter.  That’s up from revenues of 8.1 million dollars in Q4 of 2021. 

At $49 a pop, we can assume that the test was taken about 171 thousand times in the quarter.  The real number is likely higher than that due to freebies and discounts (at one point, Duolingo offered a deal for users who bought two instances of the test).

Here are the historic revenues of the test:

  • Q4 2022 – 8,410,000
  • Q3 2022 – 8,192,000
  • Q2 2022 – 8,036,000
  • Q1 2022 – 8,080,000
  • Q4 2021 – 8,095,000
  • Q3 2021 – 6,695,000
  • Q2 2021 – 4,833,000
  • Q1 2021 – 5,035,000
  • Q4 2020 – 4,197,000
  • Q3 2020 – 5,607,000
  • Q2 2020 – 4,598,000
  • Q1 2020 – 753,000

This suggests the test was taken about 667,000 times (plus freebies and discounts) in 2022.

The Duolingo English Test has removed suggested word counts and the word counter from writing questions. Instead of saying “Respond to the question in at least 50 words” the test now says “Write about the topic for 5 minutes.”

That’s great. Anyone who has been reading my blog for some time will know that misleading statements on tests about word counts drive me bonkers.

You can see this change both on the real test and on the free practice test provided on the DET website.

My last post reminded me of some advice I’ve given numerous times. Which is that if you are running a large-scale assessment, you should link your free practice material to user accounts. That way you have a never-ending faucet of useful and actionable data.

This is what Duolingo has done all along. Test watchers have noticed they way they stick new types of questions into the practice test now and then. Some of these have later gone on to appear on the actual test. Others have not.

The new 60-minute practice test is just an extension of this. I look forward to taking it.

It is somewhat surprising that the providers of legacy tests aren’t doing this already. I understand that it is easier for Duolingo to do this because most of their questions are machine generated, but if you’ve got access to decades worth of retired test forms you can just rotate through them as needed.

I updated the TOEFL and Duolingo Score requirement tracker for December, and discovered that Carnegie Mellon reduced its Duolingo English Test score requirement down to 125.  This after increasing it to 135 in November.  Weird.

School

Spring 2022

DET / TOEFL

August 11
DET / TOEFL

September 4
DET / TOEFL

October 10

DET / TOEFL

November 16

DET / TOEFL

December 14

DET / TOEFL

MIT

120 / 90

120 / 90

120 / 90

120 / 90

120 / 90

120 / 90

U of Toronto

120 / 100

120 / 100

120 / 100

120 / 100

120 / 100

120 / 100

Cornell

120 / 100


120 / 100

120 / 100

120 / 100

120 / 100

120 / 100

UBC

125 / 90

125 / 90

125 / 90

125 / 90

125 / 90*

125 / 90*

Emory

120 / 100

120 / 100

120 / 100

130 / 100

130 / 100

130 / 100

U of Arizona

100 / 70

100 / 70


100 / 70

100 / 70

100 / 70

100 / 70

Carnegie Mel.

125 / 102

125 / 102


125 / 102

 125 / 102

135 / 102

125 / 102

Brown

125 / 100

125 / 100

125 / 100

130 / 100

130 / 100

130 / 100

U of Utah

105 / 80

105 / 80

105 / 80

105 / 80

105 / 80

105 / 80

Rice

120 / 100

120 / 100

120 / 100

120 / 100

120 / 100

120 / 100

UCLA

120 / 100


120 / 100

120 / 100

120 / 100

120 / 100

120 / 100

Columbia

125 / 105


125 / 105


135 / 105

135 / 105

135 / 105

135 / 105

Dalhousie

115 / 90

115 / 90

115 / 90

115 / 90

115 / 90

115 / 90

City College of SF

85 / 56

85 / 56

85 / 56

85 / 56

85 / 56

85 / 56

De Anza College

95 / 61

95 / 61

95 / 61

95 / 61

95 / 61

95 / 61

Imperial College London

115 / 92

115 / 92

115 / 92

115 / 92

115 / 92

115 / 92

U of Chichester

95 / 79

95 / 79

95 / 79

 -- / 79

 -- / 79

 -- / 79

 

l updated the Duolingo/TOEFL score requirement tracker for November.  I was actually surprised to spot a couple of changes.  Carnegie Mellon increased their Duolingo score requirement by 10 to a total of 135 points.  That ties them with Columbia for the highest Duolingo score requirement.

I also noticed that the University of British Columbia now requires students to submit a waiver if they are going to use Duolingo Scores in their admission.

School

Spring 2022

DET / TOEFL

August 11
DET / TOEFL

September 4
DET / TOEFL

October 10

DET / TOEFL

November 16

DET / TOEFL

MIT

120 / 90

120 / 90

120 / 90

120 / 90

120 / 90

U of Toronto

120 / 100

120 / 100

120 / 100

120 / 100

120 / 100

Cornell

120 / 100

120 / 100

120 / 100

120 / 100

120 / 100

UBC

125 / 90

125 / 90

125 / 90

125 / 90

125 / 90*

Emory

120 / 100

120 / 100

120 / 100

130 / 100

130 / 100

U of Arizona

100 / 70

100 / 70

100 / 70

100 / 70

100 / 70

Carnegie Mel.

125 / 102

125 / 102

125 / 102

 125 / 102

135 / 102

Brown

125 / 100

125 / 100

125 / 100

130 / 100

130 / 100

U of Utah

105 / 80

105 / 80

105 / 80

105 / 80

105 / 80

Rice

120 / 100

120 / 100

120 / 100

120 / 100

120 / 100

UCLA

120 / 100

120 / 100

120 / 100

120 / 100

120 / 100

Columbia

125 / 105

125 / 105

135 / 105

135 / 105

135 / 105

Dalhousie

115 / 90

115 / 90

115 / 90

115 / 90

115 / 90

City College of SF

85 / 56

85 / 56

85 / 56

85 / 56

85 / 56

De Anza College

95 / 61

95 / 61

95 / 61

95 / 61

95 / 61

Imperial College London

115 / 92

115 / 92

115 / 92

115 / 92

115 / 92

U of Chichester

95 / 79

95 / 79

95 / 79

 -- / 79

 -- / 79

According to SEC filings from a few days ago, revenue for the Duolingo English Test in Q2 2022 was $8,192,000.  That’s a 22% increase over the same period a year ago.

At $49 a pop, we might extrapolate that the test was taken 167,000 times in the quarter. The actual number is probably a little bit higher due to discounts and free tests.

 

Here are the historic revenues (all USD):

Q3 2022 – 8,192,000
Q2 2022 – 8,036,000
Q1 2022 – 8,080,000

Q4 2021 – 8,095,000
Q3 2021 – 6,695,000
Q2 2021 – 4,833,000
Q1 2021 – 5,035,000

Q4 2020 – 4,197,000
Q3 2020 – 5,607,000
Q2 2020 – 4,598,000
Q1 2020 – 753,000

The Duolingo English Test now has a “faster results” option.  For an additional fee of $40, test takers can get their results in just 12 hours, instead of the usual 48 hours.  This could be useful for people with really tight deadlines.  Read about it here.

This is a great example of how Duolingo is building a better “Test Taker Experience,” a topic I’ve written about extensively here on the blog.  That said, the development surprised me, as it isn’t something I’ve included in my lists of suggestions for testing companies.  It is a welcome development, nonetheless.

One of Duolingo’s great strengths is its nimbleness, and ability to implement things like this really quickly.  For legacy testing providers implementing positive change sometimes seems akin to turning an aircraft carrier around.  Note how the IELTS didn’t get a home version until a few months ago, or how the TOEFL didn’t get automated speaking scoring until 2019.

A couple days ago the Daily Princetonian published an open letter urging Princeton University to accept the Duolingo English Test. The author’s heart is in the right place, but the article is full of incorrect and inaccurate statements about language testing. Perhaps someone from ETS or IDP Education Ltd should contact the paper about those.

Worth noting are the following errors:

1.  TOEFL and IELTS test-takers are not charged twenty dollars to send their results “to each and every university” they apply to. TOEFL includes four free score reports. IELTS includes five free score reports. The PTE includes unlimited free score reports.

2.  I think the author mixed up the “Pearson Test of English” and the “Preliminary English Test”. The latter is not used for university admissions. It was also renamed some years ago.

3.  Russians are not required to leave their country to take the TOEFL. The test can be taken from their homes, inside of Russia.

4.  The Educational Testing Service is not run by Princeton alum Robert Murley.

5.  The Educational Testing Service is not headquartered in Princeton. It is headquartered in nearby Lawrence Township.