Someone contacted me once again to report that their TOEFL scores were cancelled (no refund) because they increased too much over the span of seven months.  This is what ETS refers to as “an extremely unusual difference between” two test dates.  Or perhaps “Inconsistent performance in your responses between” two test dates.

I wrote a very funny and sarcastic post about this (I even made a silly meme) but deleted it because I love all my friends at ETS very much and I don’t want to be insulting. But listen:

  1. One one hand, the marketing folks at ETS mention potential score increases all over social media to sell test registrations to repeaters.
  2. On the other hand, the product development folks have created all these wonderful new test prep products which are supposed to help people get higher scores. They enable students target the exact microfeatures that are costing them points on test day.
  3. On the third hand, the OTI folks still cancel scores if they change too much.

Do you see why my hair is falling out?  I don’t think the firm should give students the tools to achieve large improvements in their scores, but also penalize students for achieving large improvements.

Leave a comment below if you have gone through something similar.

Last year I wrote about a change in the TOEFL Bulletin regarding the cancellation of scores due to invalidity. As I explained, prior to November 2022, the Bulletin explained that in cases of TOEFL scores being cancelled due to questionable validity,

“ETS notifies the test taker in writing about its concerns, gives the test taker an opportunity to submit information that addresses those concerns, considers any such information submitted, and offers the test taker a choice of options. The options may include voluntary score cancelation, a free re-test, a voucher for a future test, or arbitration in accordance with the ETS standard Arbitration Agreement.”

The above didn’t seem to apply to cases of suspected CHEATING, as in those cases scores were simply canceled outright with no opportunity to file an appeal or submit information. It seemed to apply only in cases where scores were “invalid.” More on what that can mean in a moment.

The appeal process was decent. Test takers could request a summary of the statistical evidence of invalidity. and if they chose to please their case, ETS convened a panel of three esteemed staffers to review it. Kind of like the “Stand and Deliver” situation many decades ago. Anyway, several test takers reached out to me for help. And hey…. one time a test taker who reached out even got the decision overturned.

If the appeal was denied the test taker would get a voucher to take the test again at no additional cost. Actually, they could just skip the appeal and take the voucher ASAP.

In November of 2022, the policy changed. The beginning of the above passage was modified to read,

“Score cancellation decisions are not subject to appeal to ETS. For test takers within the United States, before canceling scores based on substantial evidence of invalidity, ETS notifies the test taker in writing about…”

You can see the difference. Since then, test takers outside the US have seemingly been unable to appeal the decision to cancel scores for validity reasons. Also, there seems to be no voucher or refund offered.

I mention this now because I spotted a couple of complaints about it this afternoon – one from a TOEFL test taker and one from a GRE test taker. They are both pretty annoyed. I’ve attached the email sent by ETS to the TOEFL test taker.

As you can see, his score was canceled due to two issues. First, an inconsistency in his performance on either the reading or listening section vs his performance on the speaking section. That is… he scored very well on one section and very poorly on another. Second, “inconsistent testing times” were flagged. According to previous summaries, this means (in the words of ETS) “test takers who receive a high section score but complete the section in a short amount time [sic].”

I mention this because it is the first time I’ve encountered it since the changes to the TOEFL and GRE. And, of course, it is worth knowing about the policies and procedures of standardized testing.

Scattered reports from TOEFL test-takers in recent days claiming that their “official” reading and listening scores are different from the “unofficial” scores seen at the end of their test. This is new to me. Perhaps score equating (for new test forms) is being done some days later due to the removal of unscored R and L questions.

If you have experienced this, please leave a comment below.  If you haven’t experienced this, leave a comment as well!

Today, ETS raised the cost of sending additional TOEFL score reports to institutions. As always, test-takers may send score reports to four institutions for free (as long as they are selected before test day). Beyond that, test-takers must now pay $25 per score report. That is an increase of $5.

In comparison, Duolingo and PTE-Academic test-takers can send unlimited score reports, all free of charge.

The cost of sending GRE score reports was also increased by $5, I believe.

There was some weirdness coming out of ETS earlier this month. ETS Japan (a subsidiary of ETS) put out an “alert” about the TOEFL Home Edition on Twitter and its website.  I mentioned that on LinkedIn and how I found it puzzling.  The alert was quickly removed. 

If you are curious, here is what puzzled me about the alert:

  1. It recommended system requirements quite higher than those on the ETS website and suggested that interruptions could occur if they are not met.
  2. It recommended things for test-takers to avoid during the test that are not mentioned on the ETS website (though some are buried in the TOEFL Bulletin).
  3. It mentioned that since 2022 testing has become more stringent due to cheating in some countries, and therefore some test-takers may not even realize they have broken a rule.
  4. It recommended that people taking the TOEFL iBT for the first time opt to take it at a test center.

Now, that’s all very good stuff. Indeed, it is similar  to what I tell test-takers who ask for my opinion.  That said, the last part isn’t the most ringing endorsement of the product.  Nor is the third part, I suppose.

There was also stuff in there about test-takers only getting help from ETS Japan if unable to resolve a problem via the US office of ETS. One senses that the announcement came as a result of a certain number of complaints from local test-takers.  Perhaps it came from a feeling of frustration.

I’ve written many times about how at-home test takers have been experiencing high levels of frustration.  I’ve also expressed my feeling that testing companies are not doing all they can do (and should do) for their customers.  If you are new to a testing company and have been tasked with figuring out why your test’s market share is way below 2019… well… this is a big part of it. This isn’t a dig at ETS, as I think it is a problem with all testing companies, to some extent.

But we are on the topic of ETS, for the moment.  One of the traditional criticisms of that organization (see: David Lewis, “None of the Above”) is that entrenched as they are on 400 acres of bucolic New Jersey countryside, they don’t always have a clear picture of how test-takers in the outside world are experiencing their products. And sometimes I get the impression that there is insufficient communication between various departments and offices of the organization (hence a quickly deleted “alert” about their most popular test).

Anyway. The point of all this is to suggest that perhaps some of the people in the TOEFL program who don’t usually deal with complaints from test-takers could call up the folks in Japan and ask what prompted the alert to be posted.  And, moreover, discuss what the head office can do better moving forward.

Now in Open Access at “Language Testing” is a summary of the enquiry (aka score review) and resit policies of all our favorite language tests (Cambridge English Qualifications, Duolingo, IELTS, LanguageCert, PTE, PSI Skills for English, TOEFL, Trinity).

It was done by William S. Pearson of the University of Exeter.

Really useful information to have.

I saw that ETS will present at this week’s AHEAD Conference on Equity and Excellence (AHEAD = Association on Higher Education and Disability). One of their presentations is titled “ETS Updates and Tips for Assisting Test Takers with Accommodation Requests.”

Regular readers of this space will know that about seven months ago ETS entered into an ADA Settlement with the U.S. Attorney’s Office “to resolve allegations of discrimination in violate of the Americans with Disabilities Act.”

Said the U.S. Attorney’s Office at that time:

“The settlement resolves allegations that Educational Testing Service (ETS), a New Jersey non-profit organization that administers standardized tests, engaged in discrimination in violation of the ADA by creating unlawful hurdles to individuals with disabilities who sought testing accommodations. Among other things, the United States alleged that ETS unlawfully denied requests for testing accommodations or failed to timely consider requests for testing accommodations, effectively denying those requests.”

And:

“This agreement compels ETS to make systemic reforms and ends an unfair process for considering requests for testing accommodations. Through this settlement, thousands of Americans with disabilities will be given a fair shot in seeking admission to higher education.”

You can read the allegations in the settlement agreement. The alleged violations affected test-takers with learning disabilities, anxiety disorders and visual impairments attempting to write the GRE and Praxis exams.

Per the settlement agreement, within the first six months of this year, relevant members of ETS’s management team must attend ADA training courses that will teach them to better understand their obligations to test-takers. A record of attendance must be provided to the United States.

Every six months for the next three years ETS must provide written updates describing its progress in meeting the requirements established in the settlement agreement.

ETS must pay financial compensation to the plaintiffs named in the settlement.

ETS denies that it violated the ADA.

You may think that this is just ol’ Goodine taking another enthusiastic swipe at the ETS. But I write this no enthusiasm. I feel nothing, but I realize that if I don’t note this no one will, so there ya go.

I love all of my friends at ETS and am enthusiastic about their mission and values. Professionals within the organization probably can provide a lot of good advice about eliminating barriers to higher education affecting learners with disabilities (especially in light of the above). But context matters, I think.

Regular readers of this space know that my pet peeve in standardized testing is the growing phenomenon of customer-no-service. Especially score cancellations that seem unjustified, unsupported or just plain mysterious.

In light of the recent SCOTUS decision re: affirmative action improvement in this area is more necessary than ever. Testing companies now have to work extra hard to ensure they aren’t canceling legitimate scores achieved by members of equity seeking groups.  If nothing else, just imagine what the New York Times would write.

Anyway. With almost all of the big standardized tests gearing up for major changes in the months ahead, someone on a different social network than this one asked me what I would do to improve this situation. Here’s how I responded (I’ve removed company names):

Short Term:

  1. When tests are canceled due to the “possible” detection of malfeasance, a free re-test should be provided.  When you tell test-takers that they “may” have done something wrong they see that as an expression that your accusation lacks complete certainty.
  2. Staff up the department in your organization responsible for academic integrity far beyond current levels.  Give that department the time and money necessary to complete detailed (but timely) MANUAL reviews of all decisions to cancel test scores.  AI is fine, but remember to place humans in the loop at all times.
  3. Don’t depend on a third-party for-profit proctoring service for anything other than proctoring.  Don’t rely on them for decisions regarding test cancellations.
  4. Provide a robust appeals process when tests are canceled due entirely to statistical reasons.

 

Long Term (Pie in the Sky):

  1. Bring proctoring in-house.
  2. Collaborate with your peers in the testing industry to create a list of “best practices” for online proctoring. Or a code of ethics. Something that can be strictly adhered to so that test-takers get the best possible experience.
  3. Conduct regular independent audits of test cancellations.  Give someone outside your organization the ability to examine individual files to ensure that your decisions are justified.  This may seem like a bridge too far, but evidently it’s needed.  At the very least create a committee of employees from outside of your academic integrity department who can do this. Give them the time and resources needed to do a good job.
  4. Eliminate multiple choice questions.  This will eliminate a great deal of “analog” cheating in one fell swoop. Puzzling cancellations will also decline, obviously.

Good news!  I was clicking around the ETS website (as I often do) and I discovered that people can quickly opt-out of the TOEFL Search Service or the GRE Search Service (and any other search services run by ETS) by sending an email to: etsinfo@ets.org

I learned this by reading the Privacy Policy

Students sometimes complain on social media about excessive contacts from recruiters after taking the TOEFL or GRE.  They report unsolicited live calls, texts and emails.  Sometimes even paper catalogs in their mailboxes.  Those poor bastards accidentally clicked to opt-in to the “TOEFL Search Service” or the “GRE Search Service” and can’t figure out how to opt-out.  To date, I’ve generally recommended that they call New Jersey… but now I’ll give them the above email address!

Some of the complaints focus on excessive contacts made by Hult International Business School.  “Poets and Quants” recently described Hult as the ninth biggest scandal of the previous decade.  But they do have very good lawyers and a top-notch social media management team, so I’ll go on the record as saying that I think that Hult is the finest school in Massachusetts, and possibly (due to its expansive of international satellite campuses) the world.

Here’s something that was posted to Reddit a few days ago. The test-taker’s previous test was canceled because he may have received assistance during the test (no refund was given):

“I’m about to retake it. I got a completely empty room, nothing on the walls, with just a desk chair and computer pushed against the wall in the corner with the door right behind me. Should I put a mirror beside the door so they can see there’s nothing else in the room? I will also be diligent to not rest my face on my hand, roll my eyes, speak words to my self… I am truly terrified. I need this to go to grad school.

[…]

Anyone has any other advice to help me make sure I don’t get unfoundedly accused again?”

If the testing company is confident that there are no false positives they should keep doing what they are doing. If, on the other hand, there could be a few of those they should figure this out.

A sort-of rhetorical question: Do you know who really loves Reddit?

A sort-of insightful answer: Students from the country ETS is targeting above all others.

Test-takers from India may now pay for the TOEFL test in Rupees!  That’s really great.  To date many students have been required to use third party agents to help them process payments so this change should make a big difference.  It also looks like the fee for test-takers in India is fixed in rupees, which means the cost of the test won’t be impacted by ever-changing exchange rates.

Note, though, that local taxes are now added to the fee so it is a bit higher than it was previously.

I hope this is offered in more countries one day.