Here’s an interesting new Duolingo English Test thing. Some test takers are now required to use their phone as a second camera while taking the test.

After scanning a QR code on the DET desktop app, the test taker must prop their phone against an object on their desk so that the camera can capture a live image of their monitor and keyboard for the duration of the test.

This has not yet been rolled out to all test takers.

I really ought to take the test sometime next month.

Duolingo has just reported its financials for the second quarter of 2024. Revenue for the Duolingo English Test was 10.7 million dollars, up from 9.8 million in Q2 of 2023.  Note that the price of the test increased from $59 to $65 in this time frame.

If we divide the revenue by the per-test cost, we find that the DET was taken about 164,500 times in the quarter, about a 1% drop from last year.  Note that this is a crude estimate as some test takers pay more than $65 (to get faster results) while some test takers pay less (by purchasing a bundle).  Others take the test via Duolingo’s Access Program, and get vouchers from receiving institutions.   A more precise number of test takers (for 2023) was given at DETcon 2024, but I suppose that is covered by my NDA.

This flat quarter comes after a very strong Q1.

Using the same math as above, we can estimate that DET’s test volume is up about 5% in the first two quarters of 2024 (compared to the same period in 2023).  Perhaps I’ve buried the lede here as Pearson reported a 10% decline in test volume during the same period, and IDP Education is expecting a 15-20% decline in the volume of IELTS test takers for the year ending June of 2024.

It seems that growth is still happening at Duolingo, but we must pay careful attention to the final six months of 2024.

Ten years ago today, Duolingo launched a project called “Test Center.” Shortly thereafter, “Test Center” morphed into the Duolingo English Test.  That test is now taken more than 800,000 times per year.  That’s quite an accomplishment.

Many people think of the DET as a fairly new product, but ten years is a long time. When I write about the legacy testing firms being technologically far behind their competition (Duolingo and Pearson, mostly) I mean that they are more than a decade behind.  It will not be easy to close that gap. It may be impossible.

Here’s a TechCrunch article from 2014 where the project was announced.  The message of 2014 is similar to the message of today – testing monopolies are bad for test takers and are prone to fraud.

The success of the DET is mostly due to the work of some very bright researchers, engineers and psychometricians (former ETS employees, in many cases).  But the single-mindedness of Duolingo co-founder Luis von Ahn seems to play a pretty big part.  That guy does not like the testing monopolies at all.

The DET is now a major player in the admissions process for American universities. The monopoly once enjoyed by ETS in that market has come to an end.  As recent reports have indicated, times are tough at ETS.

Currently, the folks at Duolingo are heavily promoting their test to students headed to the UK, Australia and Canada.  These efforts are partly why IDP Education recently announced that IELTS test volumes will decline by 15-20% in fiscal year 2024.

I wrote yesterday about my sample of 16 international students admitted to Columbia University.  I mentioned that 15 took the Duolingo English Test, and one took the TOEFL.

What does that mean in terms of revenues at testing firms?

Well, that one dude who was admitted with a TOEFL score took the test three times, paying $400 USD for each attempt. He generated $1200 in revenue for ETS.

It is not uncommon for students applying to Ivy League schools to take the TOEFL two or three times.

I didn’t ask, but he might have spent a few extra bucks to get his scores to a raft of backup schools.  And he may have accidentally clicked to join the “TOEFL Search Service” which is still a fairly lucrative source of revenue for ETS, I think.

The other 15 students took the Duolingo test and paid $59 per attempt.  Most of them took the test only once. There may have been a few repeaters who paid $49 per attempt (after taking advantage of the two-test deal).  They sent their scores to all of their target schools without paying additional fees.

My point is that it is quite possible that the ONE guy who took the TOEFL generated more revenue than the 15 people who took the DET.

This highlights one of the struggles that the legacy firms are facing.  They know, I’m sure, that one of these days they’ll have to introduce a cheap test.  But when is the best time to do it?  From a business perspective, it is certainly possible to introduce the next-gen test too early, especially if it means losing out on that $400 registration fee. Even while under immense pressure from competitors (and while possibly losing market share), both the TOEFL and IELTS tests are revenue generating machines.

(Notes:  the price of the Duolingo test was recently hiked to $65.  The price of the TOEFL ranges from $185 to $450 depending on the country.)

Last week I attended DETcon24 (that is, the 2024 edition of the Duolingo English Test Convention) in Pittsburgh, Pennsylvania.  That’s the annual event that Duolingo holds to help stakeholders (from universities, mostly) become familiar with the DET.  Since I’m not affiliated with a university, I was really flattered when the Duolingo folks extended an invitation.

While my scribbled notes can still be comprehended, I thought I’d share a few thoughts from the event.  They are:

1. The presentations on proctoring and security were my favorites.  I’m NDAed, so I can’t write any details.  But I’m certain that if the message about security can be repeated again and again with relevant decision makers in the room, the needle can be moved.  The Duolingo folks need to find a way to communicate this stuff to more people. Perhaps a traveling road show.

2. If you have an at-home test, I really think it is important to do security and proctoring in-house. This allows the tools to be more bespoke and customer friendly.  And, of course, to provide additional security. Needless to say, Duolingo does it this way.  Many of the issues I’ve written about here (cancellations for jagged profiles, cancellations for NVIDIA drivers, cancellations for memory usage) seem, in my opinion, to be examples of security being a blunt one-size-fits-all instrument. If you don’t do security properly people get annoyed and abandon your test (best case scenario) and innocent people suffer unduly (worst case scenario).

3. The overall test volume seems to be higher than my estimates.  Likely because the volume of vouchers given to institutions is quite high.  I’ll see if I can repeat the exact number given for 2023.

4. For some more info on the new writing tasks, read: “The Impact of Task Duration on the Scoring of Independent Writing Responses” now in preprint.

5. I spoke on a panel.  I was very nervous.  I wore a new shirt.

6. I think many decision makers still have an image of the Duolingo Test based on its 2019 iteration.  Duolingo needs to do a better job exposing more people (and more types of people) to the most recent iterations.

7. There was a comment about how and why Duolingo doesn’t really work with test prep providers.  And about not selling test prep materials.  Food for thought. Maybe testing companies shouldn’t sell prep for their tests. Someone made a comment about students having to unlearn IELTS prep. I laughed. But I love all my IELTS friends.

8. Daniel Isbell has an article due out around the end of this year (but correct me if I’m wrong about the timeline). I won’t give away the topic but it will be an absolute must-read for anyone in the test prep space. If you are interested in this kind of stuff, check out his work. He’s the very best.

9. There will be concordance matching the new individual subscores to subscores on other tests.  The 2022 overall concordance will not change. I think that the sum of the subscores will match the overall score (which is not the case with the integrated subscores).

10. An attendee made a comment about how institutions are loathe to change score requirements when concordance tables are changed, because they don’t want to be the only one. That explained a lot.  I thought it was just inertia.

11. I got a plush Duo for Mrs. Goodine. It is a surprise, so don’t tell her.

Duolingo just released its 2024 Q1 earnings report.

Revenue from the Duolingo English Test was 12.8 million dollars in the quarter.  That’s a 28% increase from the same quarter last year.  Note that the test cost $10 less at that time.

My math (revenue/$59 price tag) suggests that the test was probably taken about 217,000 times in the quarter. That’s a new high. The previous high was Q1 2023, when the test was probably taken about 203,000 times.

Those are just estimates, though. The real numbers are different since some people paid less than $59 (there is a discount when two registrations are purchased at the same time) and others paid more (results can be processed faster for an extra fee).  Still others paid nothing at all via the Duolingo Access Program.

As noted here a few days ago, the overall volume of the PTE Test declined slightly in this same period.

I believe that the DET generates about 7.6% of Duolingo’s overall revenue.  It is easy to forget that the DET is just a small piece of the company.

I saw this interesting article in The Koala yesterday. It mirrors what I’ve observed happening in Australia – since the beginning of 2024 I’ve noticed Duolingo’s Australian office bringing some really talented people into the fold. They are working hard to increase acceptance of DET scores.

Says the article:

“Reflecting on Duolingo’s entry into the Australian market, Blacker noted that Duolingo is working on gaining Australian government acceptance for visa purposes, however, he expressed optimism and cited the overwhelmingly positive response from institutions across the sector and a willingness to accept the DET where possible.”

Earlier, I predicted government acceptance of the DET by governments in 5 to 10 years. I’ve mentioned Australia as being the trickiest of all cases, and acceptance there probably coming closer to the end of that time period. But maybe we’ll see some movement before 2034.

When I mention government acceptance of DET to insider-ish people the response is generally “Nah, never going to happen.” Or “Never going to happen unless X, Y and Z happen first.”

This sort of head-in-sand approach to score use can be risky. Just ask all the ETS folk who were darn certain that American schools would stop accepting the DET once the pandemic ended.