The Duolingo English Test blog has a new feature on “jagged profiles” in language assessment.  This refers to a language user who is quite strong at one or more aspects of the language and quite weak at some other aspect(s).

I was quite happy to read the following:

“Are jagged profiles common? In a word, yes! Because language assessments, and especially high stakes tests like the DET, usually evaluate multiple language skills at the same time including reading, listening, speaking, and writing, jagged profiles are often detected as a result of such tests. For example, a test taker might score high in reading comprehension but struggle with writing or speaking tasks, and consequently earn lower scores for those skills. This is a common scenario with test takers who have jagged profiles, because we have long known that production-based skills develop later than perception-based skills in L2 learning.”

Regular readers know that jagged profiles are one of the criteria used to justify the cancellation of scores on the TOEFL iBT Test.  ETS might argue that jagged profiles do not result in cancellation all by themselves, but only in combination with other factors.  That said, I’ve long called for jagged profiles to be removed from the equation entirely.  I’ve voiced that opinion more loudly since the shortening of the TOEFL iBT a few months ago, as the removal of variable (unscored) questions reduced the amount of data available to officials in the Office of Testing Integrity who are making these sorts of decisions.

Before the formal appeals process for such cancellations was removed, affected test-takers often reached out to me for assistance in planning their appeals.  Usually, that involved helping those test-takers explain the reason for their jagged profile.  I was occasionally successful in having cancellations overturned.  But usually not. You may recall the story I related last month about an autistic test-taker having his score cancelled (without a refund) in part because of his low speaking score. I reached out to him this week for an update, and was sad to hear that his scores have not been reinstated.

I wrote a moment ago about changes to some question types on the Duolingo English Test coming April 2.  A more important (to me) change is coming in July.  At that time, the test will get speaking, writing, listening and reading subscores.  These will not replace the existing DET subscores (literacy, comprehension, conversation, and production) but will be provided in addition to them.

This change will please many institutions, I think. 

Subscore concordance tables (linking DET scores to TOEFL and IELTS) will be released in October.  Again, there seems to be quite a lot of interest in this sort of thing.  An old blog post about the 2022 change to DET concordance tables is one of my most popular posts.

Keep an eye on the official Duolingo English Test blog for more details in the near future.

Duolingo has scheduled an institutional-facing webinar for this coming Monday.  You can probably get an invite by asking.

Some of the questions on the Duolingo English Test will change on April 2.   Look for:

  • A two-part writing question.  The test-taker will have five minutes to respond to a writing prompt.  Then a follow-up prompt on the same topic will be presented and the test-taker will have three minutes to respond to it.
  • A new “fill in the missing word” question.  The test-taker is given a sentence containing one unfinished word and must complete the word.  I’m not sure how this is different from the existing C-Test item (other than the fact that the test-taker will get just one sentence at a time).
  • A slightly redesigned “select the real word” task.  Test-takers will get the items one at a time instead of a bunch all at once.

These question types should all be present in the free practice test now.  

(source)

 

Duolingo has now posted Q4 results for 2023. Revenue from the Duolingo English Test was $10,819,000. That’s up a shade from Q3 of 2023, when revenues were about $10.6 million.

Since the test costs $59 a piece, it was probably taken about 183,372 times in the quarter, up from about 179,000 times in Q3 of 2023 (and about 171,632 times in Q4 of 2022).

My math says that the test was taken about 732,604 times in 2023.  I believe that represents 10% growth compared to 2022. That’s just an estimate, though. The real number is different since some people pay less than $59 (there is a discount when two registrations are purchased at the same time) and others pay more (results can be processed faster for an extra fee).

For those interested in tracking the revenues over time, here are my numbers. The asterisk in 2023 indicates the last quarter when the test cost $49.

  • Q4 2023 – 10,819,000
  • Q3 2023 – 10,600,000
  • Q2 2023 – 9,800,000
  • Q1 2023 – 9,970,000*
  • Q4 2022 – 8,410,000
  • Q3 2022 – 8,192,000
  • Q2 2022 – 8,036,000
  • Q1 2022 – 8,080,000
  • Q4 2021 – 8,095,000
  • Q3 2021 – 6,695,000
  • Q2 2021 – 4,833,000
  • Q1 2021 – 5,035,000
  • Q4 2020 – 4,197,000
  • Q3 2020 – 5,607,000
  • Q2 2020 – 4,598,000
  • Q1 2020 – 753,000

It appears that the Fulbright Scholarship program for Pakistan has stopped accepting TOEFL scores. Starting with the 2024 cycle, shortlisted candidates will need to submit a Duolingo English Test score.  This is a big deal, as Pakistan has the largest Fulbright program in terms of funding (according to the Fulbright Pakistan office).

More details here. The official requirements are listed here. Let me know if you spot changes to Fulbright scholarships in other countries.

Duolingo’s Q3 numbers report revenue of 10.6 million dollars from the Duolingo English Test for the quarter. That’s an increase of 30% over Q3 of last year. Note that the price of the test increased by 20% between those two quarters.

At $59 a piece, we can assume that that the test was taken about 179,000 times in the quarter. That said, the real number is probably somewhat higher since the company sometimes offers discounts and freebies. Meanwhile, I estimate that the test was taken about 167,000 times in Q3 of last year.

This list shows growth in revenue (from the test) since the company went public:

Q3 2023 – 10,600,000
Q2 2023 – 9,800,000
Q1 2023 – 9,970,000*
Q4 2022 – 8,410,000
Q3 2022 – 8,192,000
Q2 2022 – 8,036,000
Q1 2022 – 8,080,000
Q4 2021 – 8,095,000
Q3 2021 – 6,695,000
Q2 2021 – 4,833,000
Q1 2021 – 5,035,000
Q4 2020 – 4,197,000
Q3 2020 – 5,607,000
Q2 2020 – 4,598,000
Q1 2020 – 753,000

*Q1 of 2023 seems to be the high point for the number of tests taken, at about 203,000 ($49 each).

The overall numbers coming out of Duolingo as a whole were pretty rosy and shares are up up and away.

Here is a nice new video about how the Duolingo English Test is proctored.

The video refers to the common misconception that the Duolingo test isn’t proctored by humans. It highlights how the AI is used during the test administration, and how three levels of human proctors are used following the test administration.

There are a couple of quick references to the possibility of “human error” in live proctoring (which is what I’ve been going on about in this space forever) but for the most part it talks about the test security benefits of the model.

As the testing industry (hopefully) moves towards more instances of this type of asynchronous proctoring, firms will have to do a really good job convincing stakeholders that the tests can be trusted. I was happy to see that Pearson did a good job describing their approach to test security in the materials accompanying the launch of the new Versant by Pearson test.

It won’t come as a surprise to most of you that there are people who ought to know better that have no idea that these sorts of tests include humans in the proctoring process.

Now in open access at “Language Testing” is an article by Ramsey Cardwell, Steven Nydick, J.R. Lockwood and Alina von Davier on building concordances between English tests. The article describes the unique method they used to create a concordance table for the DET, TOEFL and IELTS tests. Worth reading, if you are into that sort of thing.

Concordance tables can be amusing. Certain institutions are somewhat lazy about updating their score requirements. Though the folks at Duolingo updated their concordance tables back in August of 2022, many instructions haven’t bothered to adjust their requirements to match them. Pearson updated the concordance tables for the PTE back in 2020 and, again, many institutions haven’t updated their requirements. The tables had to be changed since the tests themselves had changed… but some institutions now have score requirements that are probably lower than they think they are.

Meanwhile, as I’ve noted here, the score requirements for Canada’s Student Direct Stream don’t seem to match any score concordance table I’ve ever seen. Perhaps the folks at IRCC are depending on a score conversion that hasn’t been made available to the public.

I’ve got my fingers crossed that the REOI from the Australian Department of Home Affairs will nudge ETS into producing a new concordance table for the TOEFL and IELTS test. The current one seems to be based on data gathered in 2008 and 2009 and the content and grading of the TOEFL test has changed significantly since that time.

Duolingo just reported its Q2 results! Revenue from the Duolingo English Test was 9.80 million dollars in the quarter, up from 8.03 million dollars in Q2 of last year. Note that the cost of taking the test is now $59, up from $49 in 2022.

At $59 a pop, we can assume that the test was taken about 166,254 times in the quarter. The real number is probably a bit higher, given the presence of occasional discounts and freebies. This is about the same as Q2 of last year.

I was a bit surprised by the drop in both revenue and test-takers compared to Q1 of this year, when the company reported revenue of 9.97 million (from perhaps 203,000 tests).

That said, historic SEC filings suggest that Q2 is generally soft for the DET (Q2 saw the fewest DET test-takers in both 2021 and 2022).

I finally found a moment to dig into the most recent Duolingo quarterly earnings. Revenue from the Duolingo English test for the three months ending March 31 of this year was 9.97 million, up from 8.08 million for the same three months in 2022.

Assuming that the test costs $49 a pop, we can surmise that it was taken 203,510 times in the quarter. The actual number is likely higher considering occasional discount pricing and freebies.

The test continues to grow and grow and grow. Here are historic revenues:

  • Q1 2023 – 9,970,000
  • Q4 2022 – 8,410,000
  • Q3 2022 – 8,192,000
  • Q2 2022 – 8,036,000
  • Q1 2022 – 8,080,000
  • Q4 2021 – 8,095,000
  • Q3 2021 – 6,695,000
  • Q2 2021 – 4,833,000
  • Q1 2021 – 5,035,000
  • Q4 2020 – 4,197,000
  • Q3 2020 – 5,607,000
  • Q2 2020 – 4,598,000
  • Q1 2020 – 753,000

Duolingo’s share price is up as well – 115% Year to Date.

Note that the price of the test increased at the beginning of April of this year.

ETS head Amit Sevak took to Duolingo’s hometown newspaper (The Pittsburgh Tribune) to level some criticisms at the popular app and related language test. For instance:

“Learning a new language is one of the most valuable skills one can learn. We owe it to the next generation to help them measure and evaluate their progress the right way. Because when the bar for evaluating proficiency is too low, students are set up to fail. We’re already seeing examples of this in UK university admissions — students who chose to submit their Duolingo English test scores as proof of English proficiency were performing worse than their peers.

Language education cannot become a race to the bottom. The goal shouldn’t be to make sure everyone can order a beer in multiple languages. We need to make sure people have effectively learned the language of the classroom and the office so they can succeed in global settings.”

And later:

“While other English-language assessments may be less expensive or easier to pass, do we really want to lower the bar for something as fundamental as the English language?”

It seems like a reasonable argument. I have my doubts about the validity of some language tests. Not just the Duolingo Test.

On the other hand, Mrs. Goodine is a paid-up subscriber to the Duolingo app and she seems to really like it.

Look for Duolingo’s response in the Lawrence Township Pennysaver sometime next week.