An interesting tidbit from the Australian Financial Review:

“A survey of 11,500 prospective, applied and current students by recruitment firm IDP Education found that there has been a sharp increase in the popularity of the US as news of more restrictive policies in Australia, as well as in Canada and in the UK, have started to influence decisions.

Jane Li, IDP Education’s area director for Australasia and Japan, said the US used to rank last on a preference list of four major student destinations but has now leapt to first. Canada has gone from first to last.”

What does this mean for the business of language testing? Well, you can expect DET’s volume to increase, as you don’t really need a TOEFL, IELTS or PTE score to study in the United States. As I’ve indicated here, I work closely with a decent number of students at very good universities in the USA and most of them applied using DET scores.

Perhaps it means a slight uptick in the use of TOEFL, as that test still has strong associations with schools in the USA.

\In any case, it certainly portends a decline in the use of the IELTS test. I think the importance of Canada to IDP’s business model cannot be overstated.

I posted the following on LinkedIn last week, and IELTS tutors piled in to my page to insist that it will never happen, and that people are proud to take the IELTS instead of some other test.  Look, I could be wrong.  But I must note that no TOEFL tutors came to say that.  TOEFL tutors already know what the Duolingo English Test has done to their business model.  And the smart TOEFL tutors are planning for when the NABP starts accepting the Duolingo Test.

A final observation re: market share.

In the relatively near future, the Duolingo English Test will likely be accepted by governments around the world for visas and study permits.  This change won’t happen in 2024 or 2025, but it will probably happen within the next five years. Certainly within the next decade. When this occurs, demand for the IELTS, PTE-A and TOEFL tests will decline precipitously.

Few people will opt for a $250 test when a $65 test is accepted by the same authorities and is just as respected (or more respected) by the universities they wish to apply to.

As I indicated earlier, I assume that work on the next-gen IELTS and TOEFL tests is already underway.  If it isn’t… now is the time to break the glass and get started on tests that can compete.  Remember that it takes quite a long time to develop a test – the DET was in development for more than six years before it went mainstream in 2020, and the TOEFL iBT was developed over about twelve years before it launched in 2005.

Some anecdotal evidence of changes in language testing for university admissions.

Twice a year, first year international students (undergraduates) at Columbia University reach out to me for tutoring while they are preparing for Columbia’s placement test (the ALP Essay Test).  Basically, students whose first language is English take a test to determine if they’ll have to take language classes in addition to their normal course load.  I help them get ready for that test.  I’ve worked with about twenty such students since 2022.

Of course, I take the opportunity to quiz them about the test scores they submitted when they applied to Columbia in the first place.

Here’s the thing. All but one of my students submitted Duolingo English Test scores when they applied to Columbia.  One student submitted IELTS scores.  None submitted TOEFL or PTE-A scores.

Pre-pandemic, I think, most of them would have submitted TOEFL scores.  A few would have submitted IELTS scores.  Things have changed a lot since then.  They may continue to change.

(PS:  None of them submitted SAT scores)

Here’s an example of some fairly effective marketing from Duolingo. 

 

The UCLA student in the video says of the Duolingo English Test:

“I didn’t have to travel halfway across the city to go find a test spot. I didn’t have to book a test spot. The website didn’t crash on me. And it was definitely more affordable. I didn’t have to buy really expensive test material. I didn’t have to pay a whole bank account for me to buy a spot for me to take the test. I thought it was just one of those things that also had results that came really quickly and that was also super great so I think for me the DET was just this amalgamation of the best of everything that could happen for test taking for me.”

In thirty seconds the video hits on most of the things test-takers care about nowadays:

“I didn’t have to pay a whole bank account.”

Fees for taking so-called “legacy tests” have increased dramatically in the past few years.  Test-takers in some countries faced price hikes TWICE in 2023.  The decline of most currencies against the USD makes things more challenging. In conversation, some executives at testing firms have expressed skepticism that price is a factor when students pick an English test.  But it is.

“It was definitely more affordable. I didn’t have to buy really expensive test material.”

Speaking of affordability, many students in 2024 really don’t want to invest heavily in test prep and there is a perception that legacy tests require test-takers to spend quite a lot of money in order to get a score that matches their actual aptitude with the language. Test makers may be reinforcing this belief each time they unveil a new suite of costly preparation products.  This has been a through line in standardized testing since before I was born and with old monopolies dissolving it is now an issue in language testing.

“The website didn’t crash on me.”

Many test-takers really, really, really want to test at home. That said, some of them fear that at-home testing from legacy test providers is clunky and prone to technical problems. They also worry that if a failure occurs they will be on the hook financially for the failure.

I haven’t even mentioned the elimination of travel time or quick test results.

Anyway, that line at the end – “the amalgamation of the best of everything” – is marketing done well. Really really well.

Again, I want to emphasize that I love the legacy tests.  These posts are just an attempt to explain changes in the market that have occurred over the past four years.

Now in preprint from Daniel Isbell and Nicholas Coney is an article examining how English Language Proficiency tests are used at 146 research-intensive universities in the USA.  It examines which tests are used for admissions, what cut scores are used (and they compare across tests), and how subscores are used in admissions decisions.

The authors learned that the TOEFL iBT, IELTS, Duolingo, and PTE-A tests are most widely accepted, and in that order.

A few fun bits:

  • The TOEFL iBT is accepted for unconditional undergraduate admission at 135 schools, the IELTS at 133 schools, the Duolingo at 110 schools and the PTE-A at 61 schools.  I suppose this will be a priority of the folks at Pearson in the years ahead.  Though the test has (I think) moved into the #2 spot worldwide in terms of test taker volume, they still have plenty of room for growth in this area.
  • For unconditional graduate general admissions, the numbers are a bit different. The TOEFL is accepted at 117 schools, the IELTS at 116, the DET at 62, and the PTE-A at 54.  Obviously both the Pearson and Duo folks may wish to prioritize this area.
  • I was very pleased to see that the TOEFL CBT, which ceased to exist in 2006 is still accepted for unconditional admission to 11 undergraduate programs and 10 graduate general programs.  The TOEFL PBT, which was discontinued in 2017, is even more popular.  I suppose ETS ought to prioritize communications with score users in the years ahead.
  • As I have noted in my “score requirement tracker” posts, Duolingo cut scores have not always kept pace with revisions to their score concordance tables.

That’s just the tip of the iceberg. There is some really wonderful data here, so do check it out.

Note, of course, that the above figures may have changed since the time the data was gathered.

How closely should an English Test result reflect a test taker’s ability if they go into the test mostly blind?

If we know that the test taker is at a C1 level, is it proper to expect them to get an equivalent test score (95 on the TOEFL iBT, 7-8 on the IELTS, 76-84 on the PTE Academic, etc) if they take the test with only a cursory amount of preparation? Or is that expectation inappropriate?

I think this is probably too much to ask for, but how close should the theoretical test taker’s result be to their actual fluency in the language?

A wise test watcher recently noted that students pick their tests based on “perceived easiness.” I think that part of “perceived easiness” is the perception that the test result will reflect their actual English ability even if they don’t spend hundreds (thousands?) of dollars on test prep products.

Perhaps this accounts for the skyrocketing popularity of certain tests in recent years. Perhaps test takers feel those tests provide a better opportunity to certify their English language skills without investing too much in supplementary products.

Thoughts?

You know, if I were a wealthy man I would pay for my mother to take every English test with a CEFR concordance available. I’d see which results more closely match the fact that she’s a C2 user of the English language. I guess it doesn’t have to be my mom taking the tests, but she does seem to have a lot of time on her hands nowadays.

The Duolingo English Test blog has a new feature on “jagged profiles” in language assessment.  This refers to a language user who is quite strong at one or more aspects of the language and quite weak at some other aspect(s).

I was quite happy to read the following:

“Are jagged profiles common? In a word, yes! Because language assessments, and especially high stakes tests like the DET, usually evaluate multiple language skills at the same time including reading, listening, speaking, and writing, jagged profiles are often detected as a result of such tests. For example, a test taker might score high in reading comprehension but struggle with writing or speaking tasks, and consequently earn lower scores for those skills. This is a common scenario with test takers who have jagged profiles, because we have long known that production-based skills develop later than perception-based skills in L2 learning.”

Regular readers know that jagged profiles are one of the criteria used to justify the cancellation of scores on the TOEFL iBT Test.  ETS might argue that jagged profiles do not result in cancellation all by themselves, but only in combination with other factors.  That said, I’ve long called for jagged profiles to be removed from the equation entirely.  I’ve voiced that opinion more loudly since the shortening of the TOEFL iBT a few months ago, as the removal of variable (unscored) questions reduced the amount of data available to officials in the Office of Testing Integrity who are making these sorts of decisions.

Before the formal appeals process for such cancellations was removed, affected test-takers often reached out to me for assistance in planning their appeals.  Usually, that involved helping those test-takers explain the reason for their jagged profile.  I was occasionally successful in having cancellations overturned.  But usually not. You may recall the story I related last month about an autistic test-taker having his score cancelled (without a refund) in part because of his low speaking score. I reached out to him this week for an update, and was sad to hear that his scores have not been reinstated.

I wrote a moment ago about changes to some question types on the Duolingo English Test coming April 2.  A more important (to me) change is coming in July.  At that time, the test will get speaking, writing, listening and reading subscores.  These will not replace the existing DET subscores (literacy, comprehension, conversation, and production) but will be provided in addition to them.

This change will please many institutions, I think. 

Subscore concordance tables (linking DET scores to TOEFL and IELTS) will be released in October.  Again, there seems to be quite a lot of interest in this sort of thing.  An old blog post about the 2022 change to DET concordance tables is one of my most popular posts.

Keep an eye on the official Duolingo English Test blog for more details in the near future.

Duolingo has scheduled an institutional-facing webinar for this coming Monday.  You can probably get an invite by asking.

Some of the questions on the Duolingo English Test will change on April 2.   Look for:

  • A two-part writing question.  The test-taker will have five minutes to respond to a writing prompt.  Then a follow-up prompt on the same topic will be presented and the test-taker will have three minutes to respond to it.
  • A new “fill in the missing word” question.  The test-taker is given a sentence containing one unfinished word and must complete the word.  I’m not sure how this is different from the existing C-Test item (other than the fact that the test-taker will get just one sentence at a time).
  • A slightly redesigned “select the real word” task.  Test-takers will get the items one at a time instead of a bunch all at once.

These question types should all be present in the free practice test now.  

(source)

 

Duolingo has now posted Q4 results for 2023. Revenue from the Duolingo English Test was $10,819,000. That’s up a shade from Q3 of 2023, when revenues were about $10.6 million.

Since the test costs $59 a piece, it was probably taken about 183,372 times in the quarter, up from about 179,000 times in Q3 of 2023 (and about 171,632 times in Q4 of 2022).

My math says that the test was taken about 732,604 times in 2023.  I believe that represents 10% growth compared to 2022. That’s just an estimate, though. The real number is different since some people pay less than $59 (there is a discount when two registrations are purchased at the same time) and others pay more (results can be processed faster for an extra fee).

For those interested in tracking the revenues over time, here are my numbers. The asterisk in 2023 indicates the last quarter when the test cost $49.

  • Q4 2023 – 10,819,000
  • Q3 2023 – 10,600,000
  • Q2 2023 – 9,800,000
  • Q1 2023 – 9,970,000*
  • Q4 2022 – 8,410,000
  • Q3 2022 – 8,192,000
  • Q2 2022 – 8,036,000
  • Q1 2022 – 8,080,000
  • Q4 2021 – 8,095,000
  • Q3 2021 – 6,695,000
  • Q2 2021 – 4,833,000
  • Q1 2021 – 5,035,000
  • Q4 2020 – 4,197,000
  • Q3 2020 – 5,607,000
  • Q2 2020 – 4,598,000
  • Q1 2020 – 753,000

It appears that the Fulbright Scholarship program for Pakistan has stopped accepting TOEFL scores. Starting with the 2024 cycle, shortlisted candidates will need to submit a Duolingo English Test score.  This is a big deal, as Pakistan has the largest Fulbright program in terms of funding (according to the Fulbright Pakistan office).

More details here. The official requirements are listed here. Let me know if you spot changes to Fulbright scholarships in other countries.

Duolingo’s Q3 numbers report revenue of 10.6 million dollars from the Duolingo English Test for the quarter. That’s an increase of 30% over Q3 of last year. Note that the price of the test increased by 20% between those two quarters.

At $59 a piece, we can assume that that the test was taken about 179,000 times in the quarter. That said, the real number is probably somewhat higher since the company sometimes offers discounts and freebies. Meanwhile, I estimate that the test was taken about 167,000 times in Q3 of last year.

This list shows growth in revenue (from the test) since the company went public:

Q3 2023 – 10,600,000
Q2 2023 – 9,800,000
Q1 2023 – 9,970,000*
Q4 2022 – 8,410,000
Q3 2022 – 8,192,000
Q2 2022 – 8,036,000
Q1 2022 – 8,080,000
Q4 2021 – 8,095,000
Q3 2021 – 6,695,000
Q2 2021 – 4,833,000
Q1 2021 – 5,035,000
Q4 2020 – 4,197,000
Q3 2020 – 5,607,000
Q2 2020 – 4,598,000
Q1 2020 – 753,000

*Q1 of 2023 seems to be the high point for the number of tests taken, at about 203,000 ($49 each).

The overall numbers coming out of Duolingo as a whole were pretty rosy and shares are up up and away.