I returned from vacation to find – what else – a few more complaints about the TOEFL Test.  I’ve attached some screenshots.  I know I sound like a broken record at this point, but I think this is worth paying attention to.

These reports, though shared on different platforms, seem to be about the same case.  You can see that the test was canceled due to what’s called a “jagged score profile.”  This means that the test taker performed quite well in one or more sections of the test, and quite poorly in some other sections of the test.  They also completed at least one section of the test faster than some other section(s).

In cases like this, the result is deemed “invalid” and the test result is canceled.  No refund is offered.  No opportunity to take the test again for free is offered.  There is no appeal.

Keep in mind that no evidence of cheating or malpractice on the part of the test taker is mentioned.  All that is mentioned is validity.

One of the screenshots notes:

“You don’t even have to be a linguist to see why this is utter nonsense. L2 learners *can* have wildly different levels of skills across areas, not to mention other factors like neurodivergence. This is at best discriminatory, and at worst a scam.”

TOEFL test takers have been complaining about this for some years.  I’ve highlighted it in this space many times.

Obviously, if a test score is invalid, it must be canceled.  It must not be sent to recipients.  My objection is to the idea that the test taker is forced to pay a second test fee in order to prove their ability.  Again, keep in mind that in these cases no concerns about impropriety are expressed.  The expressed concerns are merely in regards to validity.

In 2022 and earlier when a test score was canceled due to validity concerns, the test taker was given three options: a refund, a free re-test or an appeal.  In the case of an appeal, they could provide documentation explaining why their score profile was “jagged.”  A panel convened by ETS would review the evidence.  I sometimes assisted test takers in the preparation of this document.  One time I even helped a test taker win the appeal.

This was how ETS handled such cases for decades. Even as far back as the case depicted in the Academy Award nominated film “Stand and Deliver.”

In late 2022, all three options were eliminated.  No one has ever explained why that happened.  I suspect no one remaining at ETS today has any idea. It sounds like “the curse of the MBA” to me, but I could be wrong.

ETS is spending a large amount of money on marketing and rebranding.  I think those efforts will be unlikely to bear fruit due to the frequency with which concerns like these are shared on social media.

Stuff like this is part of the reason why Duolingo and Pearson have been able to grow so rapidly in recent years.  As I noted in a private conversation a few days ago:  Duolingo is loved.  PTE is liked.  IELTS is tolerated.  TOEFL? Well, that’s another matter.

Someone should update the TOEFL India site. It still states that the Aadhar card is acceptable ID when taking the TOEFL, even though ETS rescinded that policy about 13 months ago.

I still hear of people getting turned away at tests center because they lack acceptable ID. I suppose this oversight is one reason for that. The site also links to an out-of-date copy of the TOEFL Bulletin which repeats the incorrect advice.

For the record:  the Aadhar card is not acceptable ID for the TOEFL or the GRE.

Someone recently posted a complaint on Reddit.

You can click the image below to read it (and the boilerplate message from ETS) but basically their TOEFL scores were canceled because unauthorized software was detected during the test administration. The test taker is frustrated because the security specialist (proctor) failed to detect or shut down the software during the pre-test check. Their scores were canceled, and no refund was offered.  I’ve written about this many times here on the blog.

In my estimation, this is one of the biggest pain points test takers have experienced since the advent of at-home English testing in 2020. It has also been challenging for test makers because, as you know, bad stories spread very quickly over social media.

A variant of this pain point that was prevalent for the first three years of the at-home TOEFL was what some people dubbed the “insta-cancellation.” In such cases the proctor carried out all of the required pre-test checks. This included remotely accessing the test-taker’s system and manually shutting down unauthorized software. The proctor would then begin the test… which would immediately be terminated due to the detection of unauthorized software. No refund.

Fortunately, the insta-cancellation was entirely eliminated due to changes to the proctoring process introduced at the end of July 2023.

But by that time, reports of this (and similar) problems had already spread quite widely on social media. One could find them quite easily on the typical social media sites… but also on less traditional places like Trustpilot. Heck, once overseas test-takers learned about the BBB, they piled in there to make complaints as well. A bunch of people even left negative reviews on the Google Maps listing for the ETS headquarters in New Jersey. I believe that all of this had a very deleterious impact on the reputation of the at-home TOEFL.

It’s no great surprise that Duolingo’s marketing materials specifically mention their much more generous approach to unauthorized software. I would say that is evidence of some big brains in Pittsburgh, but even I could have figured it out. And, folks, I went to a test-optional university.

I don’t mention this to rag on ETS or to advocate for test takers. I mention it because there are still a lot of people scratching their heads and wondering why the market shifted so quickly, and why both students and institutions were so quick to embrace alternative English tests.  This is one of the reasons.

Someone contacted me once again to report that their TOEFL scores were cancelled (no refund) because they increased too much over the span of seven months.  This is what ETS refers to as “an extremely unusual difference between” two test dates.  Or perhaps “Inconsistent performance in your responses between” two test dates.

I wrote a very funny and sarcastic post about this (I even made a silly meme) but deleted it because I love all my friends at ETS very much and I don’t want to be insulting. But listen:

1. One one hand, the marketing folks at ETS mention potential score increases all over social media to sell test registrations to repeaters.
2. On the other hand, the product development folks have created all these wonderful new test prep products which are supposed to help people get higher scores. They enable students target the exact microfeatures that are costing them points on test day.
3. On the third hand, the OTI folks still cancel scores if they change too much.

Do you see why my hair is falling out?  I don’t think the firm should give students the tools to achieve large improvements in their scores, but also penalize students for achieving large improvements.

Leave a comment below if you have gone through something similar.

Last year I wrote about a change in the TOEFL Bulletin regarding the cancellation of scores due to invalidity. As I explained, prior to November 2022, the Bulletin explained that in cases of TOEFL scores being cancelled due to questionable validity,

“ETS notifies the test taker in writing about its concerns, gives the test taker an opportunity to submit information that addresses those concerns, considers any such information submitted, and offers the test taker a choice of options. The options may include voluntary score cancelation, a free re-test, a voucher for a future test, or arbitration in accordance with the ETS standard Arbitration Agreement.”

The above didn’t seem to apply to cases of suspected CHEATING, as in those cases scores were simply canceled outright with no opportunity to file an appeal or submit information. It seemed to apply only in cases where scores were “invalid.” More on what that can mean in a moment.

The appeal process was decent. Test takers could request a summary of the statistical evidence of invalidity. and if they chose to please their case, ETS convened a panel of three esteemed staffers to review it. Kind of like the “Stand and Deliver” situation many decades ago. Anyway, several test takers reached out to me for help. And hey…. one time a test taker who reached out even got the decision overturned.

If the appeal was denied the test taker would get a voucher to take the test again at no additional cost. Actually, they could just skip the appeal and take the voucher ASAP.

In November of 2022, the policy changed. The beginning of the above passage was modified to read,

“Score cancellation decisions are not subject to appeal to ETS. For test takers within the United States, before canceling scores based on substantial evidence of invalidity, ETS notifies the test taker in writing about…”

You can see the difference. Since then, test takers outside the US have seemingly been unable to appeal the decision to cancel scores for validity reasons. Also, there seems to be no voucher or refund offered.

I mention this now because I spotted a couple of complaints about it this afternoon – one from a TOEFL test taker and one from a GRE test taker. They are both pretty annoyed. I’ve attached the email sent by ETS to the TOEFL test taker.

As you can see, his score was canceled due to two issues. First, an inconsistency in his performance on either the reading or listening section vs his performance on the speaking section. That is… he scored very well on one section and very poorly on another. Second, “inconsistent testing times” were flagged. According to previous summaries, this means (in the words of ETS) “test takers who receive a high section score but complete the section in a short amount time [sic].”

I mention this because it is the first time I’ve encountered it since the changes to the TOEFL and GRE. And, of course, it is worth knowing about the policies and procedures of standardized testing.

Today, ETS raised the cost of sending additional TOEFL score reports to institutions. As always, test-takers may send score reports to four institutions for free (as long as they are selected before test day). Beyond that, test-takers must now pay $25 per score report. That is an increase of $5.

In comparison, Duolingo and PTE-Academic test-takers can send unlimited score reports, all free of charge.

The cost of sending GRE score reports was also increased by $5, I believe.

There was some weirdness coming out of ETS earlier this month. ETS Japan (a subsidiary of ETS) put out an “alert” about the TOEFL Home Edition on Twitter and its website.  I mentioned that on LinkedIn and how I found it puzzling.  The alert was quickly removed. 

If you are curious, here is what puzzled me about the alert:

1. It recommended system requirements quite higher than those on the ETS website and suggested that interruptions could occur if they are not met.
2. It recommended things for test-takers to avoid during the test that are not mentioned on the ETS website (though some are buried in the TOEFL Bulletin).
3. It mentioned that since 2022 testing has become more stringent due to cheating in some countries, and therefore some test-takers may not even realize they have broken a rule.
4. It recommended that people taking the TOEFL iBT for the first time opt to take it at a test center.

Now, that’s all very good stuff. Indeed, it is similar  to what I tell test-takers who ask for my opinion.  That said, the last part isn’t the most ringing endorsement of the product.  Nor is the third part, I suppose.

There was also stuff in there about test-takers only getting help from ETS Japan if unable to resolve a problem via the US office of ETS. One senses that the announcement came as a result of a certain number of complaints from local test-takers.  Perhaps it came from a feeling of frustration.

I’ve written many times about how at-home test takers have been experiencing high levels of frustration.  I’ve also expressed my feeling that testing companies are not doing all they can do (and should do) for their customers.  If you are new to a testing company and have been tasked with figuring out why your test’s market share is way below 2019… well… this is a big part of it. This isn’t a dig at ETS, as I think it is a problem with all testing companies, to some extent.

But we are on the topic of ETS, for the moment.  One of the traditional criticisms of that organization (see: David Lewis, “None of the Above”) is that entrenched as they are on 400 acres of bucolic New Jersey countryside, they don’t always have a clear picture of how test-takers in the outside world are experiencing their products. And sometimes I get the impression that there is insufficient communication between various departments and offices of the organization (hence a quickly deleted “alert” about their most popular test).

Anyway. The point of all this is to suggest that perhaps some of the people in the TOEFL program who don’t usually deal with complaints from test-takers could call up the folks in Japan and ask what prompted the alert to be posted.  And, moreover, discuss what the head office can do better moving forward.

I saw that ETS will present at this week’s AHEAD Conference on Equity and Excellence (AHEAD = Association on Higher Education and Disability). One of their presentations is titled “ETS Updates and Tips for Assisting Test Takers with Accommodation Requests.”

Regular readers of this space will know that about seven months ago ETS entered into an ADA Settlement with the U.S. Attorney’s Office “to resolve allegations of discrimination in violate of the Americans with Disabilities Act.”

Said the U.S. Attorney’s Office at that time:

“The settlement resolves allegations that Educational Testing Service (ETS), a New Jersey non-profit organization that administers standardized tests, engaged in discrimination in violation of the ADA by creating unlawful hurdles to individuals with disabilities who sought testing accommodations. Among other things, the United States alleged that ETS unlawfully denied requests for testing accommodations or failed to timely consider requests for testing accommodations, effectively denying those requests.”

And:

“This agreement compels ETS to make systemic reforms and ends an unfair process for considering requests for testing accommodations. Through this settlement, thousands of Americans with disabilities will be given a fair shot in seeking admission to higher education.”

You can read the allegations in the settlement agreement. The alleged violations affected test-takers with learning disabilities, anxiety disorders and visual impairments attempting to write the GRE and Praxis exams.

Per the settlement agreement, within the first six months of this year, relevant members of ETS’s management team must attend ADA training courses that will teach them to better understand their obligations to test-takers. A record of attendance must be provided to the United States.

Every six months for the next three years ETS must provide written updates describing its progress in meeting the requirements established in the settlement agreement.

ETS must pay financial compensation to the plaintiffs named in the settlement.

ETS denies that it violated the ADA.

You may think that this is just ol’ Goodine taking another enthusiastic swipe at the ETS. But I write this no enthusiasm. I feel nothing, but I realize that if I don’t note this no one will, so there ya go.

I love all of my friends at ETS and am enthusiastic about their mission and values. Professionals within the organization probably can provide a lot of good advice about eliminating barriers to higher education affecting learners with disabilities (especially in light of the above). But context matters, I think.

Regular readers of this space know that my pet peeve in standardized testing is the growing phenomenon of customer-no-service. Especially score cancellations that seem unjustified, unsupported or just plain mysterious.

In light of the recent SCOTUS decision re: affirmative action improvement in this area is more necessary than ever. Testing companies now have to work extra hard to ensure they aren’t canceling legitimate scores achieved by members of equity seeking groups.  If nothing else, just imagine what the New York Times would write.

Anyway. With almost all of the big standardized tests gearing up for major changes in the months ahead, someone on a different social network than this one asked me what I would do to improve this situation. Here’s how I responded (I’ve removed company names):

Short Term:

1. When tests are canceled due to the “possible” detection of malfeasance, a free re-test should be provided.  When you tell test-takers that they “may” have done something wrong they see that as an expression that your accusation lacks complete certainty.
2. Staff up the department in your organization responsible for academic integrity far beyond current levels.  Give that department the time and money necessary to complete detailed (but timely) MANUAL reviews of all decisions to cancel test scores.  AI is fine, but remember to place humans in the loop at all times.
3. Don’t depend on a third-party for-profit proctoring service for anything other than proctoring.  Don’t rely on them for decisions regarding test cancellations.
4. Provide a robust appeals process when tests are canceled due entirely to statistical reasons.

Long Term (Pie in the Sky):

1. Bring proctoring in-house.
2. Collaborate with your peers in the testing industry to create a list of “best practices” for online proctoring. Or a code of ethics. Something that can be strictly adhered to so that test-takers get the best possible experience.
3. Conduct regular independent audits of test cancellations.  Give someone outside your organization the ability to examine individual files to ensure that your decisions are justified.  This may seem like a bridge too far, but evidently it’s needed.  At the very least create a committee of employees from outside of your academic integrity department who can do this. Give them the time and resources needed to do a good job.
4. Eliminate multiple choice questions.  This will eliminate a great deal of “analog” cheating in one fell swoop. Puzzling cancellations will also decline, obviously.

Good news!  I was clicking around the ETS website (as I often do) and I discovered that people can quickly opt-out of the TOEFL Search Service or the GRE Search Service (and any other search services run by ETS) by sending an email to: etsinfo@ets.org

I learned this by reading the Privacy Policy! 

Students sometimes complain on social media about excessive contacts from recruiters after taking the TOEFL or GRE.  They report unsolicited live calls, texts and emails.  Sometimes even paper catalogs in their mailboxes.  Those poor bastards accidentally clicked to opt-in to the “TOEFL Search Service” or the “GRE Search Service” and can’t figure out how to opt-out.  To date, I’ve generally recommended that they call New Jersey… but now I’ll give them the above email address!

Some of the complaints focus on excessive contacts made by Hult International Business School.  “Poets and Quants” recently described Hult as the ninth biggest scandal of the previous decade.  But they do have very good lawyers and a top-notch social media management team, so I’ll go on the record as saying that I think that Hult is the finest school in Massachusetts, and possibly (due to its expansive of international satellite campuses) the world.

Here’s something that was posted to Reddit a few days ago. The test-taker’s previous test was canceled because he may have received assistance during the test (no refund was given):

“I’m about to retake it. I got a completely empty room, nothing on the walls, with just a desk chair and computer pushed against the wall in the corner with the door right behind me. Should I put a mirror beside the door so they can see there’s nothing else in the room? I will also be diligent to not rest my face on my hand, roll my eyes, speak words to my self… I am truly terrified. I need this to go to grad school.

[…]

Anyone has any other advice to help me make sure I don’t get unfoundedly accused again?”

If the testing company is confident that there are no false positives they should keep doing what they are doing. If, on the other hand, there could be a few of those they should figure this out.

A sort-of rhetorical question: Do you know who really loves Reddit?

A sort-of insightful answer: Students from the country ETS is targeting above all others.