The Duolingo English Test blog has a new feature on “jagged profiles” in language assessment.  This refers to a language user who is quite strong at one or more aspects of the language and quite weak at some other aspect(s).

I was quite happy to read the following:

“Are jagged profiles common? In a word, yes! Because language assessments, and especially high stakes tests like the DET, usually evaluate multiple language skills at the same time including reading, listening, speaking, and writing, jagged profiles are often detected as a result of such tests. For example, a test taker might score high in reading comprehension but struggle with writing or speaking tasks, and consequently earn lower scores for those skills. This is a common scenario with test takers who have jagged profiles, because we have long known that production-based skills develop later than perception-based skills in L2 learning.”

Regular readers know that jagged profiles are one of the criteria used to justify the cancellation of scores on the TOEFL iBT Test.  ETS might argue that jagged profiles do not result in cancellation all by themselves, but only in combination with other factors.  That said, I’ve long called for jagged profiles to be removed from the equation entirely.  I’ve voiced that opinion more loudly since the shortening of the TOEFL iBT a few months ago, as the removal of variable (unscored) questions reduced the amount of data available to officials in the Office of Testing Integrity who are making these sorts of decisions.

Before the formal appeals process for such cancellations was removed, affected test-takers often reached out to me for assistance in planning their appeals.  Usually, that involved helping those test-takers explain the reason for their jagged profile.  I was occasionally successful in having cancellations overturned.  But usually not. You may recall the story I related last month about an autistic test-taker having his score cancelled (without a refund) in part because of his low speaking score. I reached out to him this week for an update, and was sad to hear that his scores have not been reinstated.

Like most of you, I’m a big fan of the “Tried and Tested Podcast” from PSI services.  In an episode from October, PSI president Janet Garcia said:

“The only thing we’re trying to test is knowledge and competence.  We’re not trying to test their digital literacy skills.  We’re not trying to test their patience.”

I like that last sentence a lot.  When trying to account for why certain language tests are gaining popularity, it’s important to remember how desirable it is to provide a smooth and seamless testing experience.  It’s been about four years since the jump to at-home testing and while everyone is getting better at it, many testing firms still struggle to do this. Indeed, test-taker patience is often tested.

I still hear stories about proctors forgetting to turn off their microphones, forcing test-takers to listen to their conversations.

I still hear about room scans being requested at the most inopportune times (and without the test being paused).

People still fail pre-test checks because one major proctoring service still doesn’t account for how modern Apple computers utilize RAM.

Note-taking is often clunky and awkward.

The list goes on.  I’ve talked about all this before, so I won’t repeat myself.  But, yeah, I love that line about not testing the test-taker’s patience. It’s so incredibly relevant.  If you want people to take your test, make the test experience as pleasant as you can.

Some may be interested to know that publication of the three official TOEFL books has been pushed back again. Per Amazon, they will be published on April 24. The bundles still have an April 22 street date, but I suppose that will be changed.

In other book news:

  1. The 18th edition of Barron’s TOEFL will be released on April 2.
  2. The print publication of “English,” Sanaz Toossi’s Pulitzer prize winning depiction of a TOEFL classroom, has been pushed back to June 25.
  3. When I reduced the price of the Kindle version of my TOEFL writing book to a buck, sales increased ten-fold. But surprisingly, about half of the sales were of the print version, which had the same old price. Maybe this is a good tip for self-publishers. If you get some attention via a cheap ebook version, people who like the looks of the book will go ahead and get the paper version.
  4. I was hoping that some of the sales would result in reviews on Amazon. Sadly, only one more person wrote a review. So, if you like the book, it would be cool if you could take a moment to write a few words on Amazon.

Password English Language Testing‘s at-home “Password Skills Plus” test is now available around the world. This comes after a long pilot project in the UK and Cyprus. The test is intended to be used for university admissions. I haven’t paid much attention to this one, but it looks pretty solid. I hope to take it in the near future.

A few key details are worth mentioning:

  1. The test takes about three hours to complete.
  2. Live proctoring is carried out by Examity (part of Meazure Learning)
  3. The fee is 110 GBP (about 140 USD), which is less than most tests in the category.
  4. I can’t find confirmation, but it looks like scoring is done by humans, given the 7-day waiting time for results and the ability to request that W and S responses be rescored.
  5. A handful of accepting institutions are listed.

At first glance, there are a few things I really like:

  1. There is a 30 minute essay that looks a lot like the old TOEFL Independent Writing Task. I miss that task a lot. I’m not alone.
  2. There are some long and challenging reading and listening questions. The test designers have really taken advantage of the test’s 3 hour duration.
  3. In case of a test cancellation (for any reason) the test-taker can pay a 50 GBP fee to have the cancellation manually reviewed. This fee is refunded if the cancellation is overturned. I think more testing companies should offer this kind of service. Listen: doing this (and keeping it in-house) would both please your customers and give you valuable insights/data.

The language testing market is getting pretty crowded, but we’ll all seen over the past few years how much students benefit from competition. For that reason, I’m always happy to learn about a new English test.

Pearson published its 2023 full year results a few moments ago.  They indicate that the PTE test was taken about 1.231 million times in 2023.  That’s an increase of 49% from 2022, when it was taken about 827 thousand times. 

Clearly, the folks at Pearson are doing something right.

I think this means the PTE is now the second most popular test of its type, behind the IELTS and ahead of the TOEFL.  I could be wrong, though, as TOEFL test volumes aren’t reported publicly. Just note that ETS head Amit Sevak has indicated a few times that the TOEFL is taken about a million times a year.

Once the British Council has released its 2023 annual report I’ll post estimated volumes of the four big English tests (and if anyone from a smaller firm can point me to stats for their own tests I’ll be happy to mention them as well).

I wrote a moment ago about changes to some question types on the Duolingo English Test coming April 2.  A more important (to me) change is coming in July.  At that time, the test will get speaking, writing, listening and reading subscores.  These will not replace the existing DET subscores (literacy, comprehension, conversation, and production) but will be provided in addition to them.

This change will please many institutions, I think. 

Subscore concordance tables (linking DET scores to TOEFL and IELTS) will be released in October.  Again, there seems to be quite a lot of interest in this sort of thing.  An old blog post about the 2022 change to DET concordance tables is one of my most popular posts.

Keep an eye on the official Duolingo English Test blog for more details in the near future.

Duolingo has scheduled an institutional-facing webinar for this coming Monday.  You can probably get an invite by asking.

Some of the questions on the Duolingo English Test will change on April 2.   Look for:

  • A two-part writing question.  The test-taker will have five minutes to respond to a writing prompt.  Then a follow-up prompt on the same topic will be presented and the test-taker will have three minutes to respond to it.
  • A new “fill in the missing word” question.  The test-taker is given a sentence containing one unfinished word and must complete the word.  I’m not sure how this is different from the existing C-Test item (other than the fact that the test-taker will get just one sentence at a time).
  • A slightly redesigned “select the real word” task.  Test-takers will get the items one at a time instead of a bunch all at once.

These question types should all be present in the free practice test now.  

(source)

 

Duolingo has now posted Q4 results for 2023. Revenue from the Duolingo English Test was $10,819,000. That’s up a shade from Q3 of 2023, when revenues were about $10.6 million.

Since the test costs $59 a piece, it was probably taken about 183,372 times in the quarter, up from about 179,000 times in Q3 of 2023 (and about 171,632 times in Q4 of 2022).

My math says that the test was taken about 732,604 times in 2023.  I believe that represents 10% growth compared to 2022. That’s just an estimate, though. The real number is different since some people pay less than $59 (there is a discount when two registrations are purchased at the same time) and others pay more (results can be processed faster for an extra fee).

For those interested in tracking the revenues over time, here are my numbers. The asterisk in 2023 indicates the last quarter when the test cost $49.

  • Q4 2023 – 10,819,000
  • Q3 2023 – 10,600,000
  • Q2 2023 – 9,800,000
  • Q1 2023 – 9,970,000*
  • Q4 2022 – 8,410,000
  • Q3 2022 – 8,192,000
  • Q2 2022 – 8,036,000
  • Q1 2022 – 8,080,000
  • Q4 2021 – 8,095,000
  • Q3 2021 – 6,695,000
  • Q2 2021 – 4,833,000
  • Q1 2021 – 5,035,000
  • Q4 2020 – 4,197,000
  • Q3 2020 – 5,607,000
  • Q2 2020 – 4,598,000
  • Q1 2020 – 753,000

Hey, so I’ve reduced the price of my TOEFL ebook to 99 cents on Amazon.  The book covers the TOEFL iBT writing section and contains some of the best practice questions and answers I’ve created over the years.  It also collects a bunch of the grammar articles that appeared on this blog before August of last year.  If that seems like the sort of the thing that might interest you, do pick up a copy on Amazon.  The book is no longer part of the Kindle Unlimited program, so even Prime members can buy a copy.  It is also available in paperback, but obviously that’s a bit more expensive.  Buying a copy will help me fulfill my dream of having the best-selling TOEFL book on Amazon, where I’m currently #6.

This has been a busy month.  Too many airplanes.

In case you missed it, check out my review of the new Princeton Review TOEFL book.  I felt let down by it.  Princeton Review should do better.

Meanwhile, when I was in Canada I finally got my hands on a copy of the newest edition of the MLA Handbook.  I often work with students preparing for their freshman year of studies in America and I always encourage them to get a printed copy of the handbook so they don’t f–k up the formatting and sourcing of their essays.  A few of the students even listen to my advice!  If you are planning on studying at a university in the USA go get a copy.  You’ll use it quite often.

I read a few normal books and articles, meanwhile.

Continuing my read-along with the Norton Library Podcast, I read “Diary of a Madman and Other Stories.”  I think this might be my favorite from the podcast series so far.   Gogol is new to me – I didn’t realize his works are so humorous.  If you want to check out the podcast, here’s a youtube link.

I used my airplane time to dig deep into my stack of unread issues of the New Yorker.  A few articles are worth passing along.

First up, I read the March 28, 2022 issue (yeah, I’m really behind).  I read The Pied Piper of Psychedelic Toads, about the consumption of a hallucinogenic substance harvested from South American toads… and the fellow who has popularized the practice.  The whole thing seems really cultish.  If you are into cults (isn’t everyone nowadays) you might find the article amusing.

I also read the September 19, 2022 issue.  I enjoyed The Enduring Allure of Choose Your Own Adventure Books. This one is for aging millennials who have fond memories of these odd little books.  Readers not familiar with them might want to skip this reading.  Another piece from the issue worth checking out is Johnson and Johnson and a new War on Consumer Protection.  This one describes how users of that company’s baby powder have suffered severe health problems, as well as the company’s efforts to avoid taking financial responsibility for its action.

Moving ever forward, I read the September 26, 2022 issue.  A fun article in this one is The Case of the Disputed Lucien Freud, which tells the story of a portrait that may or may not have been painted by the famous artist.  If this tale wasn’t so bloody convoluted, I would turn it into an integrated writing question!  TOEFL experts know that the Official Guide to the TOEFL contains an integrated writing question about a famous artwork of disputed providence

And then I read the October 3, 2022 issue.  Most interesting was Seize the Night, a long article about the famous DJ Soluman.  Since I’m really square, I’ve always wondered what it is that makes DJs so special.  I mean, they just play other people’s music, right?  Well, it seems to be more complicated than that.  Passages about music show up on the TOEFL now and then.  You’ll never get something about this sort of music, but the point of this column is to encourage people to do some challenging reading, so I think it is relevant.

Lastly, I read the March 13, 2023 issue!  Yes, I made it to 2023.  I enjoyed The Fight over Penn Station and Madison Square Garden, which describes a long conflict between the city of New York and the owners of the famous arena.  I didn’t realize until now that the famous Penn Station is located immediately underneath the arena.  That makes it quite difficult for the city to carry out a much-needed expansion of the station.

That’s all for now.  More odds and ends next month.