A bit delayed, but here is a list of all the changes (that I could spot) in the fourth edition of “Official TOEFL iBT Tests, Volume 2.” You can find the changes in Volume 1 over here.

In addition to the changes listed below, note that the independent writing task has been replaced with an academic discussion task in each test.

By the way:  this is an excellent book for TOEFL preparation.  I recommend it to literally everyone I teach.

Test One

Reading:

  • “Colonizing the Americas” removed
  • “Reflection in Teaching” removed (it had a weird question about “how is paragraph 4 related to…”)
  • “The Impact of Industrialization on Labor Systems” added

Test Two

Reading:

  • “Variations in the Climate” removed

Test Three

Reading:

  • “17th Century European Economic Growth” removed

Test Four

Reading:

  • “What Hand Did They use?” removed
  • “Water in the Desert” removed (it had a chart)
  • “The Chaco Phenomenon” added (surprisingly, more Chaco)

Listening:

  • A lecture about music history removed
  • A lecture about architecture added

Test Five

Reading:

  • “Types of Social Groups” removed (it had a chart)
  • “Biological Clocks” removed (it had only three giant paragraphs)
  • “Steel and the United States Industrial Economy” added

Listening:

  • A really dated campus encounter about going to the language lab to watch a video removed
  • A campus encounter at the financial aid office added.

I did not spot changes to the speaking questions in any of the tests.

 

Scattered reports over the past couple of weeks of problems when attempting to create a profile on the official TOEFL site. Affected users say that they get stuck on the page with “Search Service” at the top. Whatever the user selects, they just get sent back to the same page. That renders them unable to progress in the registration process.

It seems to affect only users who already have a GRE account. This has been a problem area (in my estimation) for ETS’s registration system going back many years… but this particular manifestation of the problem is new.

I mention it here because students have expressed some feelings of urgency regarding a fix, and some ETSers follow me on this platform. If something was adjusted “under the hood” maybe take a moment to ensure that it was adjusted properly. Feel free to shoot me a message if you want a bit more information.

If you are experiencing this as a user, leave a comment below.  I’ll try to find out more.

I wrote yesterday about my sample of 16 international students admitted to Columbia University.  I mentioned that 15 took the Duolingo English Test, and one took the TOEFL.

What does that mean in terms of revenues at testing firms?

Well, that one dude who was admitted with a TOEFL score took the test three times, paying $400 USD for each attempt. He generated $1200 in revenue for ETS.

It is not uncommon for students applying to Ivy League schools to take the TOEFL two or three times.

I didn’t ask, but he might have spent a few extra bucks to get his scores to a raft of backup schools.  And he may have accidentally clicked to join the “TOEFL Search Service” which is still a fairly lucrative source of revenue for ETS, I think.

The other 15 students took the Duolingo test and paid $59 per attempt.  Most of them took the test only once. There may have been a few repeaters who paid $49 per attempt (after taking advantage of the two-test deal).  They sent their scores to all of their target schools without paying additional fees.

My point is that it is quite possible that the ONE guy who took the TOEFL generated more revenue than the 15 people who took the DET.

This highlights one of the struggles that the legacy firms are facing.  They know, I’m sure, that one of these days they’ll have to introduce a cheap test.  But when is the best time to do it?  From a business perspective, it is certainly possible to introduce the next-gen test too early, especially if it means losing out on that $400 registration fee. Even while under immense pressure from competitors (and while possibly losing market share), both the TOEFL and IELTS tests are revenue generating machines.

(Notes:  the price of the Duolingo test was recently hiked to $65.  The price of the TOEFL ranges from $185 to $450 depending on the country.)

Including the last few stragglers, 16 international students at Columbia University reached out to me for help with the ALP Essay Exam (need help?  Contact me!) for this year’s fall term.  As always, I asked them what tests they used to get admitted to the school in the first place.  The results were:

  • Duolingo English Test – 15
  • TOEFL – 1
  • IELTS – 0

When I ask questions like “what went wrong with TOEFL?” I am sometimes accused of being too harsh. But clearly something has changed. Pre-pandemic, there is a good chance that every one of those students would have taken the TOEFL.

Individuals who don’t talk to test takers every day and who don’t teach this stuff one-on-one can miss trends in the industry that ought to be obvious to everyone.  Speaking of what went wrong at the legacy testing firms… that might have something to do with it.  There may be too many degrees of separation between executives and customers.

Anyway.  Of course I asked why they all took the DET.  Most mentioned the price and convenience.  One mentioned that the cute characters put her at ease.  Another said that the university told him to take the DET.

A few poor souls might sit for the placement test in August. I’ll update this post if I talk to any of them.

I have far too many unread copies of “The New Yorker.”  I have signed up for the $6 promotional deal quite a few times and now I’ve got a big pile of unread issues on my shelf.

My catch-up strategy nowadays is to read copies of the magazine when I’m on long flights.  This month I took a terribly long flight – to Pittsburgh for the Duolingo English Test Convention.  And I read six issues!  A few articles stood out:

  • In the April 4, 2022 issue (yes, 2022) I read How an Ivy League School Turned Against a Student.  It tells the story of a student who was admitted to the University of Pennsylvania… perhaps under dubious circumstances.  It is certainly a tawdry article, but interesting if you are into the world of university admissions.
  • In the same issue I enjoyed Killing Wolves to Own the Libs? About the politics of wolf culls.  They may be useful.  They may not be.  It would make a good integrated writing question, I guess.  I’ll add it to the list of pending topics!
  • In the March 20, 2023 issue I enjoyed The Little-Known World of Caterpillars.  I think the title speaks for itself.  The TOEFL often includes reading and listening content about animal species.
  • In the March 27, 2023 issue I read The Button-Pushing Impresario of Balenciaga. Fashion doesn’t show up on the TOEFL too often, but this is a fascinating deep dive into a world that I don’t know much about.  I’m not yet convinced that couture fashion is meaningful art… but I’m getting there.
  • Next, in the April 3, 2023 issue I read The Wild World of Music.  There is something here about elephants playing music.  That matches up with an integrated essay about elephant intelligence people keep sending me.
  • In the same issue I read a book review called Why the Animal Kingdom is Full of Con-Artists.  This is a perfect article.  It mentions a whole bunch of topics that would make for perfect speaking #3 topics, among them brood parasitism, kleptoparasitism and aggressive mimicry.
  • In the April 10, 2023 issue I read The Christian Liberal Arts School at the Heart of the Culture War.  Not very TOEFLy, but another article that might be of interest to people who are into the university scene in the USA.
  • Lastly, in the April 17, 2023 issue I read Family Values, about the mom who started PFLAG.  A very inspiring  story.

 

I also read a few books! 

First up, I read “Standing and Delivering: What the Movie Didn’t Tell” by Henry Gradillas.  It goes more into detail about the circumstances of the “Escalante Miracle” in East LA in the 1980s, a story that came up back in Part 24 of this column. This book is essential reading for anyone interested in the Escalante miracle of the early 1980s and can be read alongside Jay Matthews’ “Escalante: The Best Teacher in America.” Here’s what I wrote about it over on Goodreads:

Written by Henry Gradillas, the principal of Garfield High at the time Escalante was there, this book describes the challenging situation as it existed when he took over the school, and methods he used to address it. I don’t know if the lessons can be applied in 2024, but if nothing else the book is a nice time capsule from the 1980s.

It’s worth reading if you can find a copy.

Finally, I read David Graeber’s Bullshit Jobs, which is about the preponderance of meaningless jobs in the modern USA.  It’s a very funny book.

I read on Pereless that ETS is seeking to hire a “Director of Technology- Next Gen ELL.”

The job listing says:

“The Director of Technology – Next Gen ELL role provides strategic leadership, engineering and development roadmap guidance and subject matter expertise to assigned areas of Technology to drive meaningful business outcomes for the next generation English Language Learning products (Next Gen TOEFL). This role acts as the Global SBU Engineering Leader and as the main point of contact, communicating effectively with the SBU leadership, Product Management and Technology leadership, vendors, and across ETS and partners to act as a strong advocate and steward for the Next Gen TOEFL technology portfolio. “

And:

“This role “lives & breathes” all things that are next gen TOEFL, is a key SME with business commercial, operational and solution knowledge, and the single point of contact for all things technology (platforms, architecture, security, production operation) for next gen TOEFL.”

Aha!  There it is – the “Next Gen TOEFL.”  This is the first time I’ve seen ETS use those words in a public space!  Perhaps we can mark June 2024 as the date that work began on the successor to the TOEFL iBT.

Most readers are likely aware that the last time ETS began work on a “next gen” TOEFL was back in 1993.  That’s when ETS launched the “TOEFL 2000” initiative, which culminated in the launch of the TOEFL iBT twelve years later.

I suspect that the next iteration of the TOEFL will not involve 12 years of development.  Back in 1993 there were no serious competitors to the TOEFL in the USA (the University of Michigan’s MELAB is the only one that comes to mind). The IELTS was picking up steam outside of the USA, but it still enjoyed just a fraction of the total test taker volume of the TOEFL.  The competitive landscape is quite different in 2024, as I’ve written about ad nauseam in this space.

It is worth mentioning that the cost of taking the TOEFL test generally increases (in some countries) on July 1. Last year, price hikes ranged from $10 to $105. Students who are planning to take the test may want to lock in their registrations before the cost goes up.

As always, I could be wrong. Caveat Emptor.

Poets and Quants reported last month that The Ohio State University will no longer accept at-home GRE scores due to concerns about cheating.

Says the article:

“Business schools have overturned acceptances to dozens of African MBA candidates after they were suspected of cheating on the virtual GRE. The cheating scandal on the at-home GRE test, introduced during the pandemic, has caused some MBA admission officials to lose confidence in the security measures put in place by the Educational Testing Service, the administrator of the GRE.”

Programs mentioned in the article include The Ohio State University’s Fisher College of Business and the University of Minnesota’s Carlson School of Management.

An assistant Dean at Carlson is quoted as saying:

“At least 25 people in Nigeria succeeded in getting their scores through and then it scaled after they told other applicants. It is a testing issue, not a cultural or geo-political issue.”

Officials from The Ohio State University have asked ETS to audit recent results.  That audit was ongoing at the time P & Q published the article.

Inside Higher Ed has greatly expanded and updated its article on the situation at ETS.  Do check it out again if you haven’t looked since it was posted yesterday.

One thing that grabbed my attention was a note that “ETS will no longer administer the SAT.”

This comes after 77 years of ETS involvement in that test.

A College Board spokesman confirmed that “with the SAT Suite’s full transition to digital on College Board’s Bluebook testing platform, we now develop and administer the SAT and PSAT-related assessments directly.”

This represents the end of an era, I suppose, as running the SAT on behalf of the College Board was the whole reason that ETS came into existence in the first place.  The role of ETS in the running of the SAT has steadily diminished since the 2010s, but at one point College Board work represented more than 80% of ETS’s revenue, according to some sources. Indeed, for decades and decades that was the whole raison d’etre of ETS. And as recently as last year College Board work accounted for about 30% of ETS’s revenues.

As Inside Higher Ed points out, ETS will remain involved with the College Board (contributing to the AP program, I guess), but their new contract is much less lucrative than the previous one.  We’ll know the exact figure next time an ETS audit is released.  That will probably happen in January.

Inside Higher Ed  is reporting this evening that testing firm Educational Testing Service (ETS) has offered voluntary buyouts to every employee in the United States with more than two years of service at the firm. The publication refers to this as “massive downsizing.”

This news comes just a few weeks after layoffs that affected about 69 employees, which I wrote about in this space.

According to a video address by CEO Amit Sevak that was sent to employees and obtained by Inside Higher Ed, anyone “on the fence” about the package is encouraged to take it, and that “the purpose is to reduce our staff in the most gracious way we can.”  Sevak reportedly described the buyout as “an opportunity.”

Unsurprisingly, that’s not how employees see it.  Inside Higher Ed quoted a longtime ETS employee as noting:

“This is affecting people who raised their families alongside their work at ETS, people who have spent lifetimes working on a single product… it’s been an hour since the news broke and folks are earnestly sharing self-harm and suicide-prevention hotlines.”

According to Inside Higher Ed, Sevak noted in the video that involuntary layoffs may occur once the voluntary buyout has been completed.

IHE cited a decline in popularity of the GRE as one possible cause of the layoffs. They did not mention increased competition in the English language testing space, which may also be a factor.  They did, however, mention that ETS’s new contract with the College Board is “less lucrative” than in the past. I suspect this is one of the biggest factors, as the 80+ year-old deal with College Board accounted for about 30% of revenues at ETS according to their most recent tax filings.

I’ll post more as I learn it.

Given the recent goings-on at IDP Education (layoffs and downsizing), I think it’s worth repeating my standard pitch for disruption in the English testing industry.  I posted this on LinkedIn a few days ago, but I realize that some very smart people don’t use that platform.

I’ll preface this by pointing out that while on stage at the Duolingo English Test Convention a few days ago I noted that I don’t think that the DET is more valid than other tests, or that it has better items.  Test items are not what make me a fan.

With that out of the way, here’s the pitch.

At the aforementioned DETcon, Duolingo co-founder Luis von Ahn told the story of his personal experience with language testing and how it informed (and continues to inform) the development of the DET.  He has told this story many times now.  In brief, it goes like this:  while applying for schools in the United States he needed a TOEFL score.  Sadly, there were no available spots at test centers in Guatemala, where he lived, and in order to meet his deadlines he was forced to travel outside of the country to take the test.  The cost was significant.

That was in the 1990s. I wasn’t doing test prep at that time.  But when I started teaching around 2010, it was still clear to many people that the test center model was less than perfect. People were still traveling great distances to take tests.  On top of paying for test fees, bus tickets and hotels, test takers experienced lost wages due to  travel times.  A $150 test could easily become a $1500 test.

But here’s the thing: in 2024, the test center situation is even worse.

I speak to test center operators now and then.  They tell me how their relationships with legacy testing firms have changed over the past decade.  Once upon a time, they tell me, they were paid a flat rate for a scheduled test administration, no matter how many people had registered for the given date.  But now, they complain, they are paid based on the number of registrants.

In effect, fewer registrations = less profit.

Can you see where this is going? The test center operators tell me that if there aren’t enough test takers to make an administration on a particular date worthwhile, they find a way to cancel it. When this happens, test takers are left in the lurch (deadlines be damned).  I hear from test takers who have experienced this.  Sure, they get a new test date at no cost, but they can kiss goodbye to the money they’ve spent on hotels and bus tickets.

See what I mean?  A bad thing has been transformed into something even worse.

But at-home testing is here to save the day, right?

Well, when the legacy players got into the at-home business back in 2020, their product (in my opinion) was pretty clunky.  Tests were often terminated for nebulous reasons.  Instructions were badly communicated. I still remember how one at-home test told every test taker to put on their headset just before the listening section started… but terminated the test whenever someone actually followed those instructions because wearing a headset was against the rules of the at-home test.  I’m not kidding. Apparently it took years for them to figure out a way to remove those instructions from the at-home version.

And in 2024?

Again, the product appears to be even worse. Back in ‘20 when something went wrong a person taking a legacy test could usually get a free re-test. But now? If something goes wrong test takers are often told to go pound sand. Back in ‘20 when a jagged score profile was used to justify a score cancellation the test taker could appeal the decision or simply take the test again for free.  But now?  The appeals process has been eliminated and they can take it again only at their own expense. I hear these things every week from heartbroken test takers. I end up with a broken heart, too.

And don’t even get me started on the legacy testing company that recently made big cash payments to at-home test takers after reaching a settlement with the US Attorney’s Office to resolve accusations that they had violated the Americans with Disabilities Act (which was not the first time that accusation had been leveled at that firm, by the way).

But maybe the move to at-home testing has eliminated barriers to education by reducing fees for people in countries where running test centers is expensive due to logistics and security-related issues?

Not really.

Today a kid taking the TOEFL from his bedroom in Japan will pay $199.  A kid in neighboring South Korea will pay $220.  Meanwhile, a kid taking the test from his bedroom in Afghanistan, who thinks an education might be a path to a better life, will pay $230.

A kid in the Palestinian Territories will pay $270.

All for the same at-home test, delivered in the same way, proctored by the same proctors, and graded by the same raters.

I’ve never really gotten an explanation for this.  My guess is that there are iron-clad deals in place with test center owners that prevent equitable pricing for the at-home TOEFL. But it could be something else.

In any case, things are not any better than they were when I started teaching in 2010. And that, in effect, is why I support disruption in English language testing.

I’ll end by reminding my friends at legacy testing companies that this is why many influential people are excited by the idea of disruption, this is why institutions are accepting new tests, and this is why test-takers are choosing those tests.

This is why, in part, there are so many layoffs. This is why you need to start doing better.

A few points are worth mentioning that I couldn’t fit into the above:

  1. Fees for some tests have increased way beyond the rate of inflation in certain markets, despite the fact that those tests include far fewer items than in the past.
  2. I talked to one test center owner in Germany who said he sometimes runs tests at a loss because he feels guilty about canceling dates.
  3. Charging $25 to send a score to an institution is no longer justifiable.
  4. If key parts of your test have been outsourced to private equity your test is likely doomed.
  5. It costs $450 to take the TOEFL from your bedroom in Switzerland.
  6. The next Luis von Ahn will pay $230 to take the TOEFL from his bedroom in Guatemala.  Or he might just opt for the DET.

Last week I attended DETcon24 (that is, the 2024 edition of the Duolingo English Test Convention) in Pittsburgh, Pennsylvania.  That’s the annual event that Duolingo holds to help stakeholders (from universities, mostly) become familiar with the DET.  Since I’m not affiliated with a university, I was really flattered when the Duolingo folks extended an invitation.

While my scribbled notes can still be comprehended, I thought I’d share a few thoughts from the event.  They are:

  1. The presentations on proctoring and security were my favorites.  I’m NDAed, so I can’t write any details.  But I’m certain that if the message about security can be repeated again and again with relevant decision makers in the room, the needle can be moved.  The Duolingo folks need to find a way to communicate this stuff to more people. Perhaps a traveling road show.

 

  1. If you have an at-home test, I really think it is important to do security and proctoring in-house. This allows the tools to be more bespoke and customer friendly.  And, of course, to provide additional security. Needless to say, Duolingo does it this way.  Many of the issues I’ve written about here (cancellations for jagged profiles, cancellations for NVIDIA drivers, cancellations for memory usage) seem, in my opinion, to be examples of security being a blunt one-size-fits-all instrument. If you don’t do security properly people get annoyed and abandon your test (best case scenario) and innocent people suffer unduly (worst case scenario).

 

  1. The overall test volume seems to be higher than my estimates.  Likely because the volume of vouchers given to institutions is quite high.  I’ll see if I can repeat the exact number given for 2023.

 

  1. For some more info on the new writing tasks, read: “The Impact of Task Duration on the Scoring of Independent Writing Responses” now in preprint.

 

  1. I spoke on a panel.  I was very nervous.  I wore a new shirt.

 

  1. I think many decision makers still have an image of the Duolingo Test based on its 2019 iteration.  Duolingo needs to do a better job exposing more people (and more types of people) to the most recent iterations.

 

  1. There was a comment about how and why Duolingo doesn’t really work with test prep providers.  And about not selling test prep materials.  Food for thought. Maybe testing companies shouldn’t sell prep for their tests. Someone made a comment about students having to unlearn IELTS prep. I laughed. But I love all my IELTS friends.

 

  1. Daniel Isbell has an article due out around the end of this year (but correct me if I’m wrong about the timeline). I won’t give away the topic but it will be an absolute must-read for anyone in the test prep space. If you are interested in this kind of stuff, check out his work. He’s the very best.

 

  1. There will be concordance matching the new individual subscores to subscores on other tests.  The 2022 overall concordance will not change. I think that the sum of the subscores will match the overall score (which is not the case with the integrated subscores).

 

  1. An attendee made a comment about how institutions are loathe to change score requirements when concordance tables are changed, because they don’t want to be the only one. That explained a lot.  I thought it was just inertia.

 

  1. I got a plush Duo for Mrs. Goodine. It is a surprise, so don’t tell her.