Month: October 2020
I am often asked by students taking the TOEFL Home Edition about the “Exam ID” and password screen on the “ETS Unify Test Launcher.”
Here’s the screen:
But don’t worry. This information is entered by the proctor when the test starts. The test-taker does not enter it!
Seriously, though, if I get asked about this three times a week, ETS must get asked about it three thousand times per week. They really ought to put some text on the screen that tells students what to do.
I’ve received many reports about people getting payment errors when trying to pay for the TOEFL and GRE tests on the ETS website. This is sometimes reported as “Error 481” or “Error 101” or “Error 476” (or something else). The same errors appear when students attempt to pay for the GRE on the ETS website.
Anyways, if you are having this problem, here’s what you should do:
And, of course, you should call your bank in case they are blocking the payment for some reason.
If you can’t solve the problem you can possibly register (and pay) by phone. For test centers in the USA call +1-443-751-4862. To register for a test center outside of the USA, look up your country on this contact page.
As always, the general support numbers for ETS are: +1-609-771-7100 or 1-877-863-3546. Expect to be on hold for a few hours. Good luck.
Note that if you are getting a blank screen that says “select a payment method” you should disable your adblocker.
Leave a comment down below if you have any experience with this problem.
Editor’s Note: This is part three in my series on the e-rater. You might want to read part one and part two.
Les Perelman has a new article in the Journal of Writing Assessment about current problems associated with automatic essay scoring in general, and the ETS e-rater in particular. This is stuff Perelman has written about before, but I do encourage you to read the article, even if you are familiar with his work. It is illuminating.
I want to touch on one observation made by Perelman in the article. He notes:
“Indeed, ETS researchers themselves acknowledge the susceptibility of e-rater to both coaching and gaming when discussing e-rater’s scoring mainland Chinese on average over a half a point higher than human raters (d = 0.60) on the GRE issue essay”
Perelman doesn’t mention it, but the effect is similarly pronounced in the TOEFL , (d = 0.25 on a 5 point scale) according to research from ETS.
Note, also, that Korean students experience a similar benefit, though it is not as large as the one experienced by Chinese students.
In case you are curious, here’s how this kind of TOEFL prep works.
If we assume that almost every prompt can be supported with an argument about health, we teach the student to begin their first body paragraph with a topic sentence like this:
“To begin with, _________ can improve our overall physical condition.”
They just need to fill in the blank with their choice from the prompt.
Looking at some of my sample questions, this works quite often:
Of course, not EVERY question can be answered with a comment about health, but if you teach the student five or six different sentences, you can cover most of the prompts. Like:
And so on. These will all sound a bit clunky when used (like the four above do) but they will be grammatically correct and will please the e-rater.
After that, we teach the student a totally memorized sentence to function as “explanatory” content immediately after that sentence. These just make some generalized sentences about the topic. Something like:
Just one blandly generic sentence isn’t enough to get the student flagged for being off-topic. The student must depend on their own ability to write the rest of the essay, but if they use this technique in both body paragraphs, 15 or 20 percent of their essay will be written in perfect English. That’s a nice start.
Some students might use even more memorized content, but of course that increases the risk of being flagged as off-topic.
Yeah. They say:
“Another possible explanation for the greater discrepancy between human and machine scores for essays from mainland China may be the dominance of coaching schools in mainland China that emphasize memorizing large chunks of text that can be recalled verbatim on the test. Human raters may assign a relatively low score if they recognize this memorized text as being somewhat off topic, though not so far off topic as to generate a score of 0. On the other hand, this grammatical and well-structured memorized text would receive a high score from e-rater. Although automated scoring engines can be trained to identify text that is not at all related to the assigned topic, they may not yet be sensitive enough to recognize this slightly off topic text.”
No shit. What surprises me, as a teacher, is that after saying this they just leave it hanging. Neither a solution nor a response is really offered.
They do say that:
“each essay is scored by at least one human plus e-rater. Second, if there is a discrepancy of 1.5 or more points (on the 0–5 score scale) between the human score and the e-rater score, an additional human score is obtained. The item score is then the mean of the three scores (2 human plus e-rater) unless one score is an outlier (more than 1.5 points discrepant), in which case the outlier is discarded and the remaining two scores are averaged.
But this is meaningless, as a 1.5 difference is huge. That’s a nine point difference on the 30-point scale. By using memorized content only in the independent task, a student could get a bonus of 2.25 points overall in the writing section without any alarm bells being sounded.
This is weird, because ETS uses a much better system to prevent this from being a problem in the GRE:
The GRE uses a highly conservative approach in which the machine is used only to flag discrepant human scores to signal the need for a second human rating. Specifically, if the e-rater score rounded to the nearest whole number does not agree exactly with the first human score, then a second human score is obtained; the e-rater score is never averaged with a human score
No, you cannot use the exact sentences mentioned above. Presumably ETS can write some software that will detect that you’ve copied from this website. However, wealthy students will just hire a teacher to write personalized content only for them. That content will not appear anywhere online, and it won’t even be used in anyone else’s tests. That’s how they can avoid being penalized by automatic plagiarism detection.
It is probably a bad thing that students from some ethnicities do better with the e-rater. Especially since some ethnicities, particularly African-Americans (on the GRE) and Arabic and Hindi speaking students (on the TOEFL), do worse.
There aren’t just racial implications, but class implications as well. As Perelman indicates (emphasis is mine):
“It is the following paragraph, however, that contains the most egregious instance of misinformation. “The primary emphasis in scoring the Analytical Writing section is on your critical thinking and analytical writing skills rather than on grammar and mechanics.” (Educational Testing Service, n.d.-a, n.d.-b). E-rater provides half the final score. Yet, e-rater does not emphasize “critical thinking and analytic writing skills.” Indeed, it is completely oblivious to them. Its closest approximation is its highly reductive feature of development, which is calculated by the number of sentences in each paragraph and the number of paragraphs in the essay. Furthermore, grammar and mechanics compose a significant portion of the features included in e-rater’s calculations. Low-income students will believe these statements and focus on critical thinking and analytic skills. Affluent students who have taken test preparation classes, on the other hand, will be coached to provide e-rater with the proxies that will inflate their scores.“
Anyways. That’s all I have to report. Thank you for listening to my TED Talk.
Hey, it’s my second favorite day of the year! The updated TOEFL Bulletin (for 2020-21) has been published. This edition is a few months late, as it was supposed to be made available in July.
If you are interested in what’s new this year, I’ve got you covered. Listen up:
Page 6 – Test Takers with Disabilities or Health-related Needs
This has been added:
“If seeking accommodations, submit your request as early as possible. Documentation review takes approximately four to six weeks once your request and complete paperwork have been received at ETS.”
Page 10 – Your ETS Account
These are now on the list of “Services available through your account.”
• cancel or reschedule your test
• request a score review
Page 12 – Test Dates
This line has been revised:
“The TOEFL iBT test is offered more than 60 times each year.” (previously 50)
Page 15 – Test Re-take Policy
The section in bold has been added:
“There is no limit to the number of times you can take the test, but you cannot take it more than once in a 3-day period. This applies even if you canceled your scores from a previous test. If you already have a test appointment, you cannot register for another date that is within 3 days of your existing appointment.”
Page 24 – Personal Items
The sections in bold have been added:
“Personal items other than ID documents are not allowed in the testing room and are not permitted during breaks, except for food, beverages, and medication.”
“You cannot access your phone or other device during the test session or during breaks to check messages, make a call, or check the time or for any other reason.”
Page 26 – On Test Day
The following rule has been removed:
“Referring to, looking through or working on any test or test section when not authorized to do so, or working after time has been called.”
Page 27 -Unofficial Scores
The following has been added:
At the end of the test, you’ll be able to view your scores for the Listening and Reading sections. These scores are not your official scores until your score reports are posted in your ETS account, approximately 6 days after you take the test – however, these unofficial scores will give you an idea about how you did on the test.
Page 28 – Test Question Inquiries
All of this has been removed. Weird.
“Language specialists prepare TOEFL test questions following very careful, standardized procedures developed to ensure that all test material is of consistently high quality. Each question is reviewed by several members of the ETS staff. The TOEFL Committee of Examiners (COE), an independent group of professionals in the fields of linguistics, language testing and language teaching that reports to the TOEFL Board of Trustees, establishes overall guidelines for test content and test specifications.
After new test questions have been reviewed and revised as needed, they are selectively used in trial situ ations and then assembled into tests. The tests are then reviewed using established ETS and TOEFL procedures to ensure that all possible versions of the test are free of cultural bias, and statistical analyses of each question ensure that all items provide the appropriate measurement information.”
Page 34 – ETS Score Cancellation Policies
ETS hq ain’t no sanctuary city. The following has been added:
“ETS reserves the right to share any and all information in its possession about a test taker and the terms and conditions of test taking with any third party, including but not limited to (a) any entity which ETS recognizes as an authorized user of test scores, including without limitation any entity to which ETS reports test scores at the test taker’s request, and (b) any government agency with responsibility for administration or enforcement of U.S. criminal and/or immigration laws.”
Page 35 – Holding/Cancelling Scores
The following has been added:
“The retake test policy will be enforced even if a violation is not immediately identified (e.g., inconsistent registration information). If the violation is identified after registration but before the test administration, the testing appointment will be canceled and test fees will be forfeited. If the violation is identified after test scores have been reported, the invalid scores will be canceled, score recipients will be notified of the cancellation and test fees will be forfeited.”
Interestingly
The new version includes contractions. Indeed, they are used whenever possible. Strange.
There are also a few minor changes to wording that I haven’t mentioned here.
You know, a lot of students ask for “strategies” to increase their score in the reading section of the TOEFL. Sadly, the strategies are always the same – read the questions first, avoid answer choices with superlatives, skip the hard questions and come back to them… blah, blah, blah.
Those sorts of strategies help, but only a little bit. A better approach is comprehension before strategies. That means you should try to improve your overall reading ability before taking the test. If you truly understand what you read, you won’t need to use any “strategies” on the test… or will only need to use them for a few questions.
How can you improve your reading skills? By reading more!
Today I’m happy to help by beginning my one hundred part series that will highlight a few fun things you can read to improve your reading skills. I will focus mostly on non-fiction, but I will throw in a few pieces of fiction now and then. Some of the recommendations will be available online, but others will need to be purchased. Most should be available at your local library if you are in an English-speaking country already. I will only recommend stuff that I have read within the preceding month, but I am open to recommendations!
Let’s see how it goes.
First up, I finally read Freakonomics. This is a classic, and all of your English teachers have probably recommended it already. It is also available in the English section of most libraries around the world that I have visited. This is a really fun work of non-fiction that uses economic analysis to study topics not usually looked at by economists. Like sumo wrestling. And drug dealing. There is also a section on what makes real estate agents so frustrating which made me think a lot about TOEFL teachers. If you read this book, let me know if you spot that connection as well. Truly, this is a perfect example of mainstream, fun and accessible non-fiction. This book is available via the Open Library.
Speaking of fun and easy non-fiction, I read Malcolm Gladwell’s latest book, “Talking to Strangers.” This book is about miscommunication, and our inability to interact well with people we are not familiar with. I don’t think I grasped the nuance of the central thesis here, but I still enjoyed the book. This is a “ripped from the headlines” sort of thing, and you’ll recognize a lot of the main subjects, even if you aren’t American – Sandra Bland, Amanda Knox, Khalid Sheik Mohammed, Chandler & Monica, Bernie Madoff, etc. Like “Freakonomics” it isn’t the deepest of non-fiction, but it is a really fun way to boost your reading skills. Note also that the audiobook version is also wonderful (better, probably), as it combines music, audio from actual interviews and news clips with the author’s narration of the text.
Finally, I read the mid-August issue of Science News. If you are going to subscribe to one science magazine to boost your reading skills, this is the one. Yeah, it is mostly about COVID nowadays. But there this issue includes an article about the discovery of stone artifacts which is just like a TOEFL integrated writing question! Some scientists think the stone artifacts are proof that humans reached America earlier than thought… while other scientists think they are not proof! There are even three reasons given by each side! I think you can read that article at this link. If that link doesn’t work, you can find the same story in National Geographic. Meanwhile, this very short article about dinosaurs with feathers caught my eye, since that’s been a common topic on the TOEFL for a long time.
Okay, that’s all for now, but I will be back in a few weeks with additional recommendations. I’ll try to toss some fiction into the list at that time.