The Guardian has weighed in with an editorial suggesting that the UK government should “cherish and preserve” the British Council.  I imagine that they will soon step in to provide favorable terms for repayment of the British Council’s large debt.  They might even kick in more funding as well.  The Council does good work, and I’ve recently enjoyed learning about it via the “Our World, Connected” podcast (do check it out).  Soft power can be a valuable thing, and for very obvious reasons many populations still view western European governments with suspicion that they don’t have for, say, the government of Russia.  The British Council can help to remedy this.

But before funding is increased, CEO Scott McDonald should probably be queried about challenges related to language testing.  Even following its exit from the lucrative Indian market, revenue from the IELTS test remains a major part of how the BC funds its activities. This revenue is at risk: regulatory changes have wiped out most of the old IELTS monopolies and (more importantly) competing tests are increasingly attractive to both test takers and score users.  There is a very real chance that IELTS revenue will decline year after year.  IDP Education shareholders might have a few thoughts to share on this possibility.

So while the UK government can create stability in the short term with more funding and more favorable loan repayment terms, it is entirely possible that some future CEO will also appear, hat-in-hand, looking for a bailout.

What’s the plan, then?

A few months ago, the IELTS team received plaudits for stating that they are “cautiously curious” about AI in language testing.  And that they think AI “has a lot of potential.”  But among the plaudits there was some snark*, as Pearson has successfully been using AI in the PTE for 15 years.  And Duolingo has been doing so in the DET for a decade.  Some have suggested that the IELTS test is falling behind the times.  There was a bit more snark** late last year when the IELTS team made an “important announcement” about switching from pencils to ballpoint pens.

I spoke to a colleague with some experience in the industry a few weeks ago.  She said:  “Everyone is worried about Duolingo but do absolutely nothing about it. Not with product development, marketing, nothing.

The plan is unclear.

What’s worse is that some of the people responsible for handing out hundreds of millions of pounds of public money don’t even seem to know that tests like the DET even exist.  It is perhaps unfortunate that these issues weren’t explored in more depth when Scott McDonald appeared before the Foreign Affairs Committee last month.

*not from me!!

**a little bit from me

While browsing the “recent acquisitions” shelf at the Doksan Public Library I spotted this newish DET prep book (just beside a handsome new Korean edition of Ben Hogan’s “Five Lessons”). Given Doksan’s demographic makeup, we don’t get a lot of new test prep books, so finding it was a pleasant surprise.

A few months ago I wrote about how people working in test prep need to keep their eyes open in order to stay current.  Sometimes, I guess, that means visiting their local library.  Too many times have I heard people in test prep say things like “everyone stopped accepting the DET after the pandemic” or even “the DET isn’t very popular nowadays.”

Sometimes these statements are a sort of defense mechanism for individuals who have a livelihood tied up in the old testing duopoly.   But most of the time they are just rooted in ignorance of what is going on in the world of English testing.

(the book is decent enough, by the way)

 

 

If you are in New York from January 14 to 25 you can go see the Pulitzer Prize winning play “English” for free.  The folks behind the Duolingo English Test will pay for your tickets!

Set in Iran in 2008, the play tells the story of four students preparing to take the TOEFL test.  The action is set entirely within their classroom. It has a lot to say about how being understood (or not) can impact our well-being. “English” is a very touching work, but at times also very funny.

I’ve mentioned the play quite a few times in this space, starting in early 2022 when it debuted.  Sadly, I’ll be a continent away (as always) and won’t be able to claim my free seat… but I highly recommend that anyone with an interest in this sort of thing make the trip to Broadway.

Details here.

Here’s a new blog post from Duolingo.  It expresses the same point I’ve been making here for a month, which is that due to recent regulatory changes in Canada students are now free to choose whatever English test they desire, as long as their target school accepts it.  No longer are they forced to choose from a small number of expensive tests.

Says the green owl:

“Students can use any English language proficiency test accepted for admission by the academic institution they’re applying to – we know because we asked IRCC and they confirmed that for us!”

And:

“At just $65 USD, the DET is much more cost-effective than other tests, which range from $300 to $400 USD. Plus, there are no travel costs to go to a test center or hidden fees for sending scores to multiple schools.”

Legacy tastemakers have expressed the idea that despite the changes students will continue to take their tests.  They suggest that students like their tests because they include items that closely resemble what students do on campus.  Sometimes they say that students like their tests because the time they spend preparing for them gets them ready for their studies.  Other times they say that their test is “the gold standard” or that it is “fit for purpose.”

This is magical thinking.

Students are practical and pragmatic.  Indeed, they are some of the most pragmatic and practical people on the planet. They will choose the most affordable and accessible test almost every single time.

As I’ve mentioned here before, this is a “break glass in case of emergency” situation for firms that compete with Duolingo. They ought to have a plan already.

Eight months ago, I posted a short item here predicting that within 5-10 years, the big three commonwealth governments would begin accepting at-home English tests for visa purposes.  It remains one of the most widely-read things I’ve written.

With the termination of the SDS in Canada, I suppose my prediction has arrived ahead of schedule.  As things stand, Canada-bound students can now get a study permit without visiting a test center.

Maybe that’s not a huge deal, but the termination of the SDS also means that all tests are now on a level playing field.  Students are now free to pick any test that is accepted by their target school, without consideration of how it will impact the issuance of their study permit.

Previously, students opting for entrance via the SDS program took an IELTS, TOEFL, CAEL, CELPIP or PTE test.  And now?  They have a buffet of options to choose from.  Take the University of Toronto, which is apparently a nice school.  Undergrad applicants can meet language fluency requirements by submitting scores from the following tests:  C1 Advanced, C2 Proficiency, DET, COPE, IELTS, PTE and TOEFL.  They can also submit an IB literature result, or some qualifications from the UK that I’m not familiar with.  Whatever their choice, it won’t affect their study permit.

Yes, I know that technically this was always an option, as the SDS could be bypassed in favor of the slow stream, but few students in the key sending markets went that route.

Clearly, then, this change represents yet another challenge for the legacy testing firms.  But it also represents a challenge for Pearson and Prometric (developer of the CAEL and CELPIP tests). In my view it will be very difficult to convince a student in China to pay $300 to take the TOEFL or $310 to take the PTE when the DET costs $65 and will get them to Canada just as easily.  Likewise, it will be hard to convince a student in India to take a 165 minute IELTS when a 60 minute DET is just as good.

And decades of shabby customer service from legacy testing firms has left them without enough goodwill to convince test takers to stick with them during these trying times.

Some may wonder how legacy test makers will approach this challenge.  One strategy might be to pray daily that IRCC introduces a language test requirement into the study permit system, rendering moot all of the above.  That may happen.  Another strategy may be to focus mostly on testing students bound for graduate programs, where the DET doesn’t have widespread acceptance.  That could be a smart approach.

A more ambitious strategy would be to take this as an opportunity to innovate in terms of test content and item development.

Hear me out.

One reason why items on legacy tests seem to be set in stone is that the test makers are beholden to whatever research and validity studies they have submitted to gain acceptance by governments.  Any drastic changes to test design can result in that acceptance being revoked.  That’s why Australia’s Department of Home Affairs temporarily stopped accepting TOEFL scores starting in July of last year.

But with the shuttering of the SDS and the winding down of the SELT program in the UK, this isn’t as big a concern as it used to be.

The folks at ETS have little to lose and much to gain from reinventing the TOEFL as a cheaper, shorter and more consumer friendly test.  Doing so will no longer impact their relationship with IRCC.  And since the TOEFL isn’t accepted in the UK, they have nothing to worry about there.  Yeah, it would cost them Australia (again) but I don’t think the test is widely used by students heading to that country anyway.  And, heck, the aforementioned changes would likely increase use of the test among US-bound students.

The IELTS partnership is also looking at a brave new world right now, though changes to that test would be trickier to implement. Though they are in the same boat as ETS as far as Canada is concerned, they have much more to lose in Australia and (in the short term) in the UK.  But the HOELT is coming, and in a few years the IELTS partners will find themselves in a position where innovation is much easier to pull off.

With all that said, there is a fourth possible strategy.  That is, of course, to do nothing.

Duolingo’s quarterly financial report was published a few days ago.  It indicates that for the three months ending September 30, revenue from the DET was about $10,772,000.  If we divide that number by the cost of the DET ($65) we can estimate that the test was taken about 165,723 times in the quarter.  That’s down about 8% from the same period last year, when the same math suggests the test was taken about 179,864 times.

Keep in mind that this is an imprecise calculation.  Some people pay less than $65 by purchasing a bundle of tests.  Others pay more than $65 to get fast results.  Some people pay less (or nothing) because they get vouchers from a third party partner or through Duolingo’s Access program.

With that said, the math suggests that the test was taken 547,223 times for the nine months ending September 30.  That’s basically unchanged from the same period last year, when the test was taken about 549,231 times.  The lack of a year-on-year decline is due to a very strong Q1 in 2024.  Indeed, I think Q1 of 2024 is actually the high water mark for the test.

These numbers track what’s going on in this industry as a whole.  Pearson has suggested that PTE test volumes will likely be flat for the year.  And IDP Education reported an 18% decline in IELTS volumes for the year ending June 30, 2024.

Note that if you are looking for these sorts of numbers from Duolingo, you have to check the 10-Q forms filed by Duolingo.  Nowadays, they aren’t reported elsewhere. 

Canada has ended the Student Direct Stream (SDS).  This program was introduced by IRCC in 2018 to provide expedited study permit processing from select countries, including the key sending countries of India and China.  

Applicants opting for the SDS route were subject to requirements beyond those in the regular non-SDS study permit application.  These requirements included a language test result.

Moving forward, all students must apply through the regular study permit stream.

Notably, the regular study permit stream does not require a language test score.  It merely requires a letter of acceptance from an institution.  Institutions set their own language test requirements and issue the LOA once applicants have fulfilled them.

In terms of language testing, one imagines that this change will generate business for Duolingo’s DET, which is widely accepted by schools across the country but was not accepted for use in the SDS stream.  It may reduce volumes of more expensive tests like the TOEFL, PTE and IELTS, scores from which were accepted as part of the SDS stream.

I suppose, though, that one should keep an eye on the specific requirements of the student permit program.  Perhaps a language test requirement will be added in the future.

Note that the Nigeria Student Express program has also been eliminated.  It was similar to the SDS.

The folks at Duolingo have published an article (and matching blog post) about how much time test takers should be given to complete writing tasks. Their research suggests that shorter tasks are just as useful as longer tasks in terms of reliability and validity.

This is a controversial topic among people who take the time to look at the sorts of tasks that are included on tests of English proficiency. It has generated some discussion.

Nowadays, both test makers and test takers seem to favor test forms that are shorter (in duration) than those used in the past. Since long essay tasks require a significant amount of time to complete, they are less popular than they used to be. Recall that last year the 300(ish) word “Independent Essay” task was dropped from the TOEFL, in favor of a 100(ish) word “Academic Discussion” response, meant to simulate a message board interaction. Research provided by ETS indicates that the shorter task is just as useful as the longer one it replaced.

A separate (but related) controversy relates to how closely test items should resemble real-world tasks carried out by students in the course of their future studies. The move to shorter writing tasks means that newer tests often include items that do not simulate real academic work. Some people find this problematic. Some do not. Yet others argue that the tasks on more traditional tests never actually simulated this sort of thing in the first place.

Everyone taking the Duolingo English Test must now use a phone as a “secondary camera” during the test. This involves propping a phone against a heavy option and angling it in such a way that it can record the test taker’s screen and keyboard during the test.

This requirement seems designed to target remote-access cheaters. I guess Duolingo’s security team seem to believe they can detect when the action of a test taker’s fingers doesn’t match what is recorded from the screen.

I’ll take the DET before the end of the year to see how straightforward the setup is.

This video explains what to do (or you can read about it on the official DET blog):

Duolingo just published a report on the “Demographic and Score Properties of Test Takers,” which contains a ton of useful and interesting data about who takes the Duolingo English Test.  It covers the year ending June 30, 2024. 

The report indicates that the overall mean score is 110.5.  Comprehension is the subscore with the highest mean score (118.06), while production is the lowest (88.97).  Percentiles of overall scores and subscores are also provided.  Numbers are also provided depending on test taker intent.

I was happy to see which percentage of test takers were first timers (68%) and repeaters (32%).  I was also happy to see that Koreans students are most likely to be repeaters (42%).  Norwegians, not surprisingly, are least likely (7%).  There is some fun data about repeaters that you can dive into on your own.

There is also demographic information about age and gender and first language.

Most interesting of all is the list of test sessions by test taker ID.  I will reproduce it here:

  • India – 18.91%
  • China – 15.88%
  • Canada – 4.35%
  • South Korea – 3.51%
  • Brazil – 3.42%
  • United States – 3.34%
  • Mexico – 3.12%
  • Indonesia – 2.78%
  • France – 2.09%
  • Pakistan – 1.94%

It isn’t surprising that India and China come out on top.  Those are two very big sending countries.  I was surprised to see South Korea in the number four spot… despite being physically in Korea.  That’s an indication that test watchers must keep their eyes open and always be talking to people in all segments of the industry.

Missing is data on which countries test takers plan to study on, or the countries of receiving institutions.

Below are photos of the endcap displays of TOEFL and IELTS books at the big Kyobo Books location near Seoul City Hall.  Even more books of both types are found on the store’s shelves.  Eagle-eyed readers will spot a mixture of both official and third party publications.

When trying to account for why certain tests are popular in the East Asian market, it is really important to pay attention to the amount of prep materials on the market, especially (but not only) collections of practice tests.

To some extent, here in Korea TOEFL and IELTS have avoided the headwinds they face in other markets because there is such a healthy ecosystem of prep materials available.  The PTE is doing okay here, but it isn’t challenging IELTS to the same extent that it is in India.  Likewise, while the DET has a lot of fans here, it hasn’t supplanted the TOEFL as a test for America-bound students like it has in Europe.

People here want to take tests that they can prepare for on their own.  Young people in Korea spend a lot of time studying for tests, and are pretty good at breaking them down to their component parts and mastering them bit by bit.  But that can only be done when they have access to a ton of super accurate study materials. Reputable third party publishers here are trusted to produce practice tests that are 99% similar to the real thing. That’s one reason why some people here are sticking with the legacy tests.

In markets with less of this sort of stuff, there is less reason to stick with the legacy tests.

In part this explains the high volumes (in Korea) of somewhat older English tests like the G-TELP, TOEIC and OPIc, all of which remain extremely popular in the region.  Surprisingly so.

Test makers should keep this in mind.  They should also remember that the legacy tests can count on third party publishers to crank out a steady stream of materials on their own, but upstarts might have to shoulder some of the burden themselves.

(For the record, I didn’t spot any PTE books and I’m not sure any are available from Korea publishers at all.  I didn’t spot any DET books either, though the influential “Siwon School” company does have a line on the market.)

The Chinese market deserves a post all of its own.  So… I think I’ll save my thoughts on that one for a later date.

The Duolingo English Test blog includes a new post about how “a team of former spies with years of experience in espionage, intelligence, and covert operations” were hired “to exploit the DET in any way they could.”  It’s a good post, but only scratches the surface of what I’d like to read on the topic.

The author mentions a bunch of possible exploits that the spies considered.  Like: outside assistance, deepfakes, bribes, leaked questions, phony ID and identical twins.

Left unmentioned, though, is the issue of remote access.  Did the team of spies attempt to remotely control a test taker’s computer during a test administration?  Did the spies possess this expertise in the first place? That remains unclear.

It’s no secret that this method of cheating is the main area of concern when it comes to at-home testing.

Regular readers will recall a story from June about how Florida Man allegedly used this approach to allegedly subvert Pearson VUE’s at-home test security an alleged 820 times.  And he didn’t even have a background in the fine art of espionage.

Cheating rings advertise pretty blatantly on non-USA social media sites.  And they can usually SEO their way to top-20 placement in Google search results.  Testing firms can’t hire the cheating rings themselves to look for security gaps, but perhaps there are consultancy firms that specialize in this area and use the exact same methods as the cheaters.

I’m bullish on the potential of at-home testing.  I recently predicted that at least one commonwealth government will accept at-home test scores for visas in the next ten years.  But the story from Florida didn’t exactly fill me with optimism.

Anyhow.

One of my pipe dreams is the creation of some sort of non-profit industry association focused on improving the security of at-home testing through collaboration.  All of the big testing and proctoring firms could join (and fund it), and security experts who work in cheating hotspots could play a big role as well. Research and technological advances could be shared for the common good instead of being kept in-house.

The more I think about it, this is exactly the kind of thing the new-look ETS should take the lead on. And guys, if you’re pitching it to Amit, just tell him it will unlock a whole new stream of keynote addresses.