The ETS 990 form for the year ending September 30, 2023 is now available.  You can read it on Propublica.

A few things are worth noting here:

  1. ETS reported revenue of 1.06 billion dollars for the year.  That’s down from 1.12 billion in 2022. Other than the pandemic-impacted 2020 financial year, you’ve got to go back to 2011 to find a year with less revenue (note that the 2015 form is for a partial year).
  2. ETS reported a  net loss of about 9 million dollars for the year.
  3. Program service revenue was 918 million dollars, which I think is the lowest since 2010.
  4. Total assets are valued at about 1.79 billion dollars, up slightly compared to last year.
  5. The organization’s top earner was former president Walt MacDonald, who was paid about 2.3 million dollars for the year.  He left ETS in June of 2022.
  6. Indeed, former employees did pretty well, financially. The form lists decent amounts of money paid to key individuals who left as long ago as early 2021.
  7. Current president Amit Sevak earned 658 thousand dollars for the year.
  8. Cambium Assessment was paid 26.8 million dollars for subcontracting under the big California contract.
  9. Prometric got 26.3 million for running test centers.
  10. ProctorU got 23.4 million for its amazing services.
  11. About 106 thousand dollars was spent on lobbying.
  12. I still don’t know how much was spent to purchase PSI, but I guess that will be in the audit (coming soon).
  13. The ETS hotel brought in revenue of 3.1 million dollars against expenses of 3.5 million dollars.  Though one popular YouTuber has described it as “sort of like a glammed up university residence,” I still dream of spending a weekend there. Apparently the nearby ETS hiking trail just got new signage, highlighting the organization’s proud history.

The standard disclaimer: Why should one care? One probably shouldn’t care. But ETS is a tax-exempt organization that still has an outsized influence on the lives of millions of young people in America and around the world. The point is not to gawk at the large numbers but just to share things that smarter people than me ought to be writing about.

Here’s the PDF version of my TOEFL score report. Interestingly, the PDF file was available the same day my scores were. Traditionally, there has been a 24-48 hour delay on the PDF file.

The overall score for my test was 119 out of 120 points. Let me know if you have any questions about how I responded to the speaking and writing tasks.

When I return home in a few weeks I’ll share the paper version. I believe that will be mailed from the hub in Seoul.

My TOEFL Score Report, 2024
My TOEFL Score Report, 2024

I saw that Nicholas Lemann has written a new book on standardized testing called “Higher Admissions: The Rise, Decline, and Return of Standardized Testing.” I’m looking forward to reading it, as Lemann’s “The Big Test” (published 25 years ago) is one of my favorite works on the topic. It was also the inaugural selection for my standardized testing book club, which is almost a real thing.

A quick mid-month column this month since I am off to India tomorrow morning and won’t have time to write between now and the end of the month (though I will have plenty of time to read magazines, as it is a long flight).

First up, I’ll remind you that I reviewed a couple of TOEFL books this month.  I really liked The TOEFL iBT Official Beginner’s Guide.  I really did not like the new Mometrix TOEFL Guide.

Next, I read a couple issues of Analog Science Fiction and Fact.  The best story I encountered was “The Last Days of Good People” by A.T. Sayre.  It tells the story of a research team sent to a distant plant to study a pre-industrial civilization being impacted by a deadly pandemic.  If you can  I’ve been reading Analog for quite a few years now, and this is one of the best things they’ve published in recent memory.  You ought to read it if you can figure out how to get a copy of the July/August 2024 issue.  A free PDF version might be available during awards season.  If that’s the case, I’ll post a link in a future column.

I read Catherine Liu’s “Virtue Hoarders.”  One day I must compile all books of this type into the the “Test Resources Left Book Club.”

Lastly, I read the April 11, 2022 (!) issue of The New Yorker.  It included a lovely profile of the actress Natasha Lyonne, to coincide with the broadcast of the second season of “Russian Doll.”  That’s one of my favorite TV shows of the past five years or so.  Check it out if you haven’t already.  It’s on Netflix.

 

I took the TOEFL iBT Home Edition a few days ago. Below are a few nitpicky comments I scribbled down right after I finished:

 

  1. To prepare, I used TOEFL TestReady. This was my first time to interact with that product as a test taker might. I like it a lot, but it desperately needs proper timers. The TOEFL, like most standardized tests, is partially a test of one’s time management skills. This should be reflected in preparation products. I’ll write much more about TestReady later on.

 

  1. I hope that technology renders room scans obsolete one day.  They are a pain.  I sense that developers might not use the same computers or have the same rooms as test takers and therefore might not realize just how much of a pain they are. The Duolingo English Test doesn’t make use of this measure, but seems to have replaced it with extremely strict gaze tracking… which is a frequent subject of test taker complaints on social media. But hope springs eternal.

 

  1. Test takers are instructed to state that they agree with the “terms and conditions” listed on the TOEFL website. I have been unable to locate them.  Next time, I’ll ask the proctor for a link. How many test takers have actually been aware of what they are agreeing to?

 

  1. I was surprised to see that the test STILL includes a message telling test takers to put on their headset. Regular readers know that using a headset during the test is against the rules and will likely cause the test results to be invalidated. These instructions ought to be removed immediately.

 

  1. ETS should get rid of the microphone calibration meter thingie.  My voice stayed in the “too quiet” range even when I spoke as loud as possible. Over the years, hundreds of test takers have reported the same. I knew to just forget about it and speak at a normal level during the test… but not everyone does. Many people respond by screaming their answers as loudly as possible, which is especially troublesome when it occurs at test centers. Surely the meter can be replaced with a calibration method that doesn’t provide any visible feedback.

 

  1. My test crashed one time, right after the first lecture played in the listening section. My proctor restarted it.

 

  1. I think I heard an Irish accent.

 

  1. One really starts to understand the whole “perceived difficulty” controversy when they sit down and take a bunch of tests. The TOEFL has always included a handful of so-called “Killer Questions” that cause even the most experienced test taker to go crosseyed. They give the test a reputation for toughness, even though you can get them all wrong and still perform at a C2 level.

 

  1. At 11:49 PM the day before the test, I received an email reminding me to add free score recipients by 10:00 PM the day before the test. Thanks, guys.

 

  1. I scored 29 in reading and 30 in listening. Which surprised me a bit, as I was really confident about my reading answers… but not confident at all about a few listening answers.

 

The next test for me? The PTE! Stay tuned!

IRCC has announced that language score requirements will be added to the PGWP application process.  University graduates will need a score equivalent to CLB level 7 and college graduates will need a CLB level 5 equivalent. A list of acceptable tests hasn’t been given, but one imagines that IELTS General, CELPIP and PTE Core scores will be accepted (as they are for PR applications at this time).  In case you are wondering, an IELTS score of 6 is equivalent to CLB level 7.

I’m not sure of the usefulness of this requirement, as most students are required to meet similar requirements while applying to schools in the first place. But including it amongst the announced changes likely helps the minister establish the tone he’s going for.

Note, also, that a PGWP application is usually followed by a PR application, something which requires similar language scores. I suppose students will use the same scores for both applications. This makes the change not a particularly big deal, from my perspective.

In some cases, though, the change could result in students taking similar tests three times – when applying to universities, when applying for a PGWP and when applying for PR.  Test companies must be happy right now.

This account of a remote proctor making a test taker run a gauntlet of increasingly inane security checks was recently shared on Reddit.  It made me chuckle.  I love how the situation continues to escalate, until the test taker is required to leave his flat and show the proctor the elevators in his building.

Any screenwriters in the audience? I’m imagining a sort of Kevin Smith movie about a bunch of young people working as remote proctors. To deal with the monotony of the job, they compete to see who can get their test takers to do the weirdest, most absurd things before they quit their tests in frustration.

I’ll visit Chandigarh next week. I’ll be there roughly September 22 to September 27. After that, I will be somewhere else. I’m interested in learning about B2B relationships involving testing firms and organizations based in India. That’s something I’ve not paid enough attention to in recent years even though it is a pretty big deal. Do give me a shout if you will be around.

The Duolingo English Test blog includes a new post about how “a team of former spies with years of experience in espionage, intelligence, and covert operations” were hired “to exploit the DET in any way they could.”  It’s a good post, but only scratches the surface of what I’d like to read on the topic.

The author mentions a bunch of possible exploits that the spies considered.  Like: outside assistance, deepfakes, bribes, leaked questions, phony ID and identical twins.

Left unmentioned, though, is the issue of remote access.  Did the team of spies attempt to remotely control a test taker’s computer during a test administration?  Did the spies possess this expertise in the first place? That remains unclear.

It’s no secret that this method of cheating is the main area of concern when it comes to at-home testing.

Regular readers will recall a story from June about how Florida Man allegedly used this approach to allegedly subvert Pearson VUE’s at-home test security an alleged 820 times.  And he didn’t even have a background in the fine art of espionage.

Cheating rings advertise pretty blatantly on non-USA social media sites.  And they can usually SEO their way to top-20 placement in Google search results.  Testing firms can’t hire the cheating rings themselves to look for security gaps, but perhaps there are consultancy firms that specialize in this area and use the exact same methods as the cheaters.

I’m bullish on the potential of at-home testing.  I recently predicted that at least one commonwealth government will accept at-home test scores for visas in the next ten years.  But the story from Florida didn’t exactly fill me with optimism.

Anyhow.

One of my pipe dreams is the creation of some sort of non-profit industry association focused on improving the security of at-home testing through collaboration.  All of the big testing and proctoring firms could join (and fund it), and security experts who work in cheating hotspots could play a big role as well. Research and technological advances could be shared for the common good instead of being kept in-house.

The more I think about it, this is exactly the kind of thing the new-look ETS should take the lead on. And guys, if you’re pitching it to Amit, just tell him it will unlock a whole new stream of keynote addresses.

I picked up Mometrix Media’s “TOEFL iBT 2024-2025 Preparation Book” a few months ago. TOEFL books by real publishers are hard to come by these days, so I think it is important to check out the small press stuff like this which frequently shows up in the Amazon charts.  As I write this, Mometrix’s book is the second best selling TOEFL book, behind only the Official Guide to the TOEFL.  But is it any good?

To cut to the chase:  this is a surprisingly terrible TOEFL preparation book.

Very little in this book is accurate.

Take the first practice reading passage, for example.  The article itself (about the succession to Elizabeth I) looks like a TOEFL reading passage to some extent, but of the 14 questions that follow only 6 resemble what one might find on the real TOEFL.  The other reading passages are just as bad.

And, yes, the passage contains 14 questions even though the real test only contains 10 questions per passage.

The practice speaking sets in the book have the same problem; though the speaking section was reduced to 4 questions back in 2019, the book follows the old 6-question model.  Though the author indicates that this was done “to help you practice as much as possible,” there is no indication of which question types were removed from the test.

Finally, let’s turn to the writing practice.  Though the independent writing question was removed from the test in 2023, it is still present in this book (which was published in 2024).  The new “writing for an academic discussion” task is not mentioned.  For what it’s worth, the provided integrated writing question isn’t even close to accurate.  In this instance, the lecture elaborates on the reading topic in a general way instead of opposing it as it would on the real test.

All-in-all, the book is focused on a practice test that is less accurate than what ChatGTP could produce with minimal prompting.

Outside of this one (inaccurate) practice test there isn’t much else in the book.  Readers will find some general study tips which are generic enough to (probably) appear in other Mometrix books, lists of suffixes and prefixes, writing strategies of a very general type, a list of idioms and a few pages about how to overcome test-related anxiety.  None of this stuff is particularly useful.  Most of it feels recycled.

I don’t recommend this book.

These Google Trends graphs for the past ten years are mildly interesting. They hint at an increasingly competitive market for English testing in India (but one impacted by forces beyond the control of testing companies). They also hint at a gradual embrace of the IELTS in East Asia markets where the TOEFL has traditionally been the number one test.

Recently I spoke to an ed-tech entrepreneur who had just returned from doing market research in El Salvador. Everyone there told him he needed to ensure that his tech could help students prepare for the TOEFL. I encouraged him to do that as well, but reminded him of trends in the rest of the world.

I’m scheduled to take the TOEFL iBT next week.  Big thanks to my friends at ETS for providing a voucher.  The test hasn’t changed since July of last year, and my test will be proctored by Meazure Learning/ProctorU, so I probably won’t have much to report post test. But do let me know if there is anything I should keep an eye out for.

A few things about the new registration process are worth noting, though.  They are:

  1. Overall, registration is now quick and painless and periwinkle.
  2. Test takers should be advised that the opt-in box for a free paper score report is no longer part of the registration process.  It must be accessed via the “test appointment details” link in the user account following the scheduling process.  By default, the box is not selected.
  3. Express delivery by courier of paper score reports (for a fee) is no longer offered.  But the TOEFL blog on Naver suggests that it will be available again in the future.

Some may wonder why anyone would want a paper score report.  Most tests no longer even provide this service. However, for some reason there remains a significant demand for official paper TOEFL (and IELTS) score reports. Indeed, one of the most frequent questions on social media I see is “where the heck is my printed score report?!”

The suspension of express shipping is worth taking note of.  Many test takers don’t realize that the free option mostly uses bog standard snail mail services from New Jersey (or a regional hub like Seoul)…  which means that in many cases paper reports never reach their destination.  Consider the many locales where regular home mail delivery is either unreliable or not present.

If asked, I would recommend eliminating the free option entirely and using the cost savings to make express delivery a bit more affordable (and to provide it at the same price both before and after completion of the test).

I would also recommend a redoubling of efforts to wean score users off of the use of printed reports.

A few days ago I completed the speaking module of the British Council’s EnglishScore Test.  This module is unlocked after completion of the “Core Skills” module.  The test, which is free, takes about 30 minutes to complete.  A verified digital certificate can be purchased for $24.99.  Like the other two EnglishScore modules, it is a mobile-only test.  You might also enjoy my comments on the core skills module, and the writing module.

Below find a copy of my certificate, as well as a combined certificate that is provided to users who purchase all three of the available certificates (core skills, writing and speaking).  The British Council provided vouchers so I could generate the certificates at no cost.

A few notes are worth mentioning:

  1. Question types will be familiar to anyone who has taken a modern English test:  read the sentences you see on the screen, repeat sentences you have heard, respond to a short prompt.  The number of questions is somewhat limited.  Perhaps it could be expanded.

 

  1. Audio quality on the “repeat sentences you have heard” questions wasn’t always fantastic. It could be better.

 

  1. The difficulty of the items is determined by your performance on the Core Skills module.  Based on your results there, you will be placed into one of three streams in both the speaking and writing modules.

 

  1. Grading is done mostly by AI, but peer review also plays a role.  I talked about the peer review process in an earlier post.  I do like the idea of peer review. That said, I spent about 20 minutes reviewing answers at the end of my test.  It was somewhat tedious.

 

  1. Results were ready in a couple of hours.  But, once again, I did not receive an email notification.  Someone should look into that, as it is promised in the app.

 

  1. Scoring tops out at C1 level.  The certificate includes subscores for communication, fluency and pronunciation.

 

  1. A printed copy of the Core Skills certificate can be purchased.  Printed copies of the other certificates are not available.

 

That’s all for the EnglishScore test!  Thanks for reading.  More tests in the fall, I hope.