It appears that the Fulbright Scholarship program for Pakistan has stopped accepting TOEFL scores. Starting with the 2024 cycle, shortlisted candidates will need to submit a Duolingo English Test score.  This is a big deal, as Pakistan has the largest Fulbright program in terms of funding (according to the Fulbright Pakistan office).

More details here. The official requirements are listed here. Let me know if you spot changes to Fulbright scholarships in other countries.

Duolingo’s Q3 numbers report revenue of 10.6 million dollars from the Duolingo English Test for the quarter. That’s an increase of 30% over Q3 of last year. Note that the price of the test increased by 20% between those two quarters.

At $59 a piece, we can assume that that the test was taken about 179,000 times in the quarter. That said, the real number is probably somewhat higher since the company sometimes offers discounts and freebies. Meanwhile, I estimate that the test was taken about 167,000 times in Q3 of last year.

This list shows growth in revenue (from the test) since the company went public:

Q3 2023 – 10,600,000
Q2 2023 – 9,800,000
Q1 2023 – 9,970,000*
Q4 2022 – 8,410,000
Q3 2022 – 8,192,000
Q2 2022 – 8,036,000
Q1 2022 – 8,080,000
Q4 2021 – 8,095,000
Q3 2021 – 6,695,000
Q2 2021 – 4,833,000
Q1 2021 – 5,035,000
Q4 2020 – 4,197,000
Q3 2020 – 5,607,000
Q2 2020 – 4,598,000
Q1 2020 – 753,000

*Q1 of 2023 seems to be the high point for the number of tests taken, at about 203,000 ($49 each).

The overall numbers coming out of Duolingo as a whole were pretty rosy and shares are up up and away.

Here is a nice new video about how the Duolingo English Test is proctored.

The video refers to the common misconception that the Duolingo test isn’t proctored by humans. It highlights how the AI is used during the test administration, and how three levels of human proctors are used following the test administration.

There are a couple of quick references to the possibility of “human error” in live proctoring (which is what I’ve been going on about in this space forever) but for the most part it talks about the test security benefits of the model.

As the testing industry (hopefully) moves towards more instances of this type of asynchronous proctoring, firms will have to do a really good job convincing stakeholders that the tests can be trusted. I was happy to see that Pearson did a good job describing their approach to test security in the materials accompanying the launch of the new Versant by Pearson test.

It won’t come as a surprise to most of you that there are people who ought to know better that have no idea that these sorts of tests include humans in the proctoring process.

Now in open access at “Language Testing” is an article by Ramsey Cardwell, Steven Nydick, J.R. Lockwood and Alina von Davier on building concordances between English tests. The article describes the unique method they used to create a concordance table for the DET, TOEFL and IELTS tests. Worth reading, if you are into that sort of thing.

Concordance tables can be amusing. Certain institutions are somewhat lazy about updating their score requirements. Though the folks at Duolingo updated their concordance tables back in August of 2022, many instructions haven’t bothered to adjust their requirements to match them. Pearson updated the concordance tables for the PTE back in 2020 and, again, many institutions haven’t updated their requirements. The tables had to be changed since the tests themselves had changed… but some institutions now have score requirements that are probably lower than they think they are.

Meanwhile, as I’ve noted here, the score requirements for Canada’s Student Direct Stream don’t seem to match any score concordance table I’ve ever seen. Perhaps the folks at IRCC are depending on a score conversion that hasn’t been made available to the public.

I’ve got my fingers crossed that the REOI from the Australian Department of Home Affairs will nudge ETS into producing a new concordance table for the TOEFL and IELTS test. The current one seems to be based on data gathered in 2008 and 2009 and the content and grading of the TOEFL test has changed significantly since that time.

Sometime within the last few months, the Harvard Business School MBA Admissions Board increased its Duolingo English Test score requirement from 130 points to a whopping 145 points. That’s the highest Duolingo score requirement I’ve ever seen.

For comparison, they require 109 on the TOEFL and 7.5 on the IELTS.

I scored 150 when I last took the Duolingo Test.

Duolingo just reported its Q2 results! Revenue from the Duolingo English Test was 9.80 million dollars in the quarter, up from 8.03 million dollars in Q2 of last year. Note that the cost of taking the test is now $59, up from $49 in 2022.

At $59 a pop, we can assume that the test was taken about 166,254 times in the quarter. The real number is probably a bit higher, given the presence of occasional discounts and freebies. This is about the same as Q2 of last year.

I was a bit surprised by the drop in both revenue and test-takers compared to Q1 of this year, when the company reported revenue of 9.97 million (from perhaps 203,000 tests).

That said, historic SEC filings suggest that Q2 is generally soft for the DET (Q2 saw the fewest DET test-takers in both 2021 and 2022).

I finally found a moment to dig into the most recent Duolingo quarterly earnings. Revenue from the Duolingo English test for the three months ending March 31 of this year was 9.97 million, up from 8.08 million for the same three months in 2022.

Assuming that the test costs $49 a pop, we can surmise that it was taken 203,510 times in the quarter. The actual number is likely higher considering occasional discount pricing and freebies.

The test continues to grow and grow and grow. Here are historic revenues:

• Q1 2023 – 9,970,000
• Q4 2022 – 8,410,000
• Q3 2022 – 8,192,000
• Q2 2022 – 8,036,000
• Q1 2022 – 8,080,000
• Q4 2021 – 8,095,000
• Q3 2021 – 6,695,000
• Q2 2021 – 4,833,000
• Q1 2021 – 5,035,000
• Q4 2020 – 4,197,000
• Q3 2020 – 5,607,000
• Q2 2020 – 4,598,000
• Q1 2020 – 753,000

Duolingo’s share price is up as well – 115% Year to Date.

Note that the price of the test increased at the beginning of April of this year.

ETS head Amit Sevak took to Duolingo’s hometown newspaper (The Pittsburgh Tribune) to level some criticisms at the popular app and related language test. For instance:

“Learning a new language is one of the most valuable skills one can learn. We owe it to the next generation to help them measure and evaluate their progress the right way. Because when the bar for evaluating proficiency is too low, students are set up to fail. We’re already seeing examples of this in UK university admissions — students who chose to submit their Duolingo English test scores as proof of English proficiency were performing worse than their peers.

Language education cannot become a race to the bottom. The goal shouldn’t be to make sure everyone can order a beer in multiple languages. We need to make sure people have effectively learned the language of the classroom and the office so they can succeed in global settings.”

And later:

“While other English-language assessments may be less expensive or easier to pass, do we really want to lower the bar for something as fundamental as the English language?”

It seems like a reasonable argument. I have my doubts about the validity of some language tests. Not just the Duolingo Test.

On the other hand, Mrs. Goodine is a paid-up subscriber to the Duolingo app and she seems to really like it.

Look for Duolingo’s response in the Lawrence Township Pennysaver sometime next week.

An interesting article about the Duolingo English Test in “Language Testing” this month.

Researchers shared responses to DET items (speaking only) with relevant stakeholders (faculty, administrators, students) and asked them to judge the comprehensibility and “academic acceptability” of the speakers (based solely on what they heard in the responses, of course). These judgements correlated highly with the DET scores. Test-takers with higher-scoring responses were judged to have more comprehensibility and potential academic acceptability.

I suppose it is not a great surprise that people with strong English were judged to be more comprehensible that people with weak English… but this is the sort of research that must be done if we are going to rely on automated scoring for high-stakes testing.

Here’s the March update to the TOEFL/Duolingo Score Requirement tracker.  I didn’t post it here in March since I’m losing interest in it, but I did get a request.  The point here is to discover if schools update their Duolingo cut scores in light of update conversation tables (from Duolingo).  In March, Rice University increased their requirement by ten points.  Also, updated wording on UCLA’s page seems to suggest that they will permanently accept Duolingo scores.

You may have to scroll sideways to see the most recent numbers.

I’ll do an April update towards the end of the month.

Just published in “Language Testing” are the results of a study into the predictive validity of the Duolingo English Test. It notes:

1. At the postgraduate level, DET scores were related to academic results. Students with higher DET scores had more academic success. At the undergraduate level, DET scores were not related to academic success. The same is true of TOEFL and IELTS scores.
2. Students accepted with DET scores enjoyed lower academic success than those accepted with TOEFL and IELTS scores.
3. Given the above, perhaps higher DET cut scores are needed. Or perhaps students arriving with DET scores may need additional academic support to reach their full potential.
4. The lower academic success of DET test-takers may have something to do with the test itself. Or it may have nothing to do with the test. There are many things that affect academic success.

I note:

Perhaps early access to the results of this study encouraged Duolingo to adjust its score conversion tables. They did that some months ago and schools that pay attention to that sort of thing have already adjusted their cut scores, as I’ve reported over the past eight or nine months.

The cost of taking the Duolingo English Test has increased to $59. According to customer support, Duolingo is “gradually rolling out this new price.” I can already see the new price in my account. On April 2, the price was $49.

A new article in British newspaper i is critical of the Duolingo English Test.  It reports that:

“Professor John Heathershaw from the University of Exeter linked the acceptance of ‘things like Duolingo tests’ to lower English language standards and described it as a ‘major issue’ when he appeared before the Foreign Affairs Select Committee last week.”

And:

“Aston University in Birmingham is one of the many UK institutions that adopted Duolingo at the height of the pandemic but has now dropped it. A spokesperson told i that the decision was taken because of concerns about student performance.”

And quoting a spokesman from that university:

“There is no evidence that those students on Duolingo were failing or, indeed, fraudulent, but just not performing at the same level as peers on their course, so we have chosen to remove acceptance for the test.”

–

I’m going to babble a bit now.  Pardon me.

Older tests like the TOEFL and IELTS are designed around a particular conception of validity. For instance, the makers of the TOEFL would likely argue that we can look at a TOEFL score and make inferences about the test-taker’s ability to do well in university classes because the tasks on the TOEFL resemble (and, more importantly, involve the same knowledge, skills and processes) as the tasks done by students at post-secondary institutions.

Similarly, IDP would tell you that the IELTS General Training Module is a great test for immigration because it includes tasks that resemble what we do in everyday life.  Meanwhile, the TOEIC leans heavily on memos and invoices and emails that people encounter in an office environment.

You can read about this concept of validity in Carol Chapelle’s 2008 doorstopper about the creation of TOEFL iBT.

The Duolingo English Test is a little bit different.  It is certainly a test of one’s English abilities.  That much is obvious.   But is it appropriate for university admissions? While questions of an academic nature have been added to the DET in recent months much of the test score is still determined by “describe this picture” tasks and “fill in the missing letters” tasks that don’t closely resemble things done on a university campus.

Duolingo might argue that such tasks totally suit the purpose of the test and that they really do require the relevant skills and knowledge.  They could be right.  Who knows?

What interests me is that if we reject the idea that validity requires that test items closely resemble tasks performed in real-world contexts we can go ahead and discard all of the older tests and use the Duolingo Test for all possible purposes.  Will the receiving institutions bite?  That remains to be seen.