Researchers shared responses to DET items (speaking only) with relevant stakeholders (faculty, administrators, students) and asked them to judge the comprehensibility and “academic acceptability” of the speakers (based solely on what they heard in the responses, of course). These judgements correlated highly with the DET scores. Test-takers with higher-scoring responses were judged to have more comprehensibility and potential academic acceptability.
I suppose it is not a great surprise that people with strong English were judged to be more comprehensible that people with weak English… but this is the sort of research that must be done if we are going to rely on automated scoring for high-stakes testing.
This month I finished reading They Say, I Say (with readings). If you are seeking a book that will help you get started along the path to better academic writing, I highly recommend it. In about 200 pages, it introduces some effective methods of presenting your ideas as part of an ongoing dialog with other relevant scholars. I’ve recently used the book in my own lessons on academic writing and it has been popular with my students. Also included is a selection of readings to stimulate discussion and writing. Fortunately, some of those readings can be found online. A few of them are especially relevant to readers who are taking the TOEFL for college admission. They include:
Should Everyone go to College? by Stephanie Owen and Isabel Sawhill argues that attending college is often a path to future prosperity, but not in every circumstance. Think carefully, kids!
The New Liberal Arts by Sanford Ungar hit close to home for me. Now, more than ever, the people who have the ability to “participate in, and help shape, civil discourse.” That’s an ability that a liberal arts education might empart. One of the brightest students I have worked with recently is majoring in architecture at a very prestigious school and is minoring in…. sociology. She knows what up. She’ll be able to move comfortably in so-called “elite” circles. Be like her, kids!
Shut Up About Harvard by Ben Casselman offers some food for thought. I totally understand why people want to go to Harvard. Those people want to not just make money, but they want to be part of the meritocracy. They want to be part of the elite group that shapes their society. I get that. But maybe we need to stop thinking so much about those particular kids and those particular schools.
Anyway. There are many more articles in the book, and most of them aren’t about education. I’ll let you find them on your own (but feel free to ask if you want me to highlight a few more options).
I also read the December 2022 issue of “History Today.” A few things grabbed my interest:
Clean Sheets, about the history of paper, could be adapted into a perfect TOEFL reading passage. It is almost the right length, too. Best of all, the article is fascinating. I didn’t realize that papermaking used to be such a difficult process.
The magazine also contains a few really wonderful long-form articles, but since they are all behind a paywall I won’t cite them here. But, hey, if you love history this is the magazine for you. It is the best of its category.
Finally, I read the August 8, 2022 issue of “The New Yorker.” I absolutely loved The Hard Sell, a long article about the door-to-door sales profession. Yes, that is still an occupation… and yes, it is weird. The article is at once an investigation of how the job works in 2022 and a character study of one particularly effective salesman. It’s the best thing I’ve read in this magazine all year. It isn’t really TOEFL adjacent, but it is a fascinating read.
When preparing for standardized tests, people are often forced to struggle with bad study tools. Books and courses often contain inaccurate questions. People teaching those tests also struggle with this problem, obviously.
To nudge publishers and course designers in the right direction as they update their books for the revised TOEFL test, I recently teamed up with Jaimie Miller to analyze existing samples of the new TOEFL writing question and produce some guidelines for good question creation. The results of our work follow.
How to Write “Authentic” Academic Discussion Prompts for the new TOEFL iBT Writing Test
If you are producing sample activities for TOEFL iBT test-takers who need to prepare for the Academic Discussion portion of the new TOEFL iBT Writing test (added to the test starting July 26, 2023) we strongly encourage you to base your activities on the analysis that we have done of the sample activities that ETS has made available. By doing so, you’re creating material that test-takers are likely to recommend to their friends.
Test-takers have this view of the information when it is time to write their Academic Discussion response:
(You may also want to download this side-by-side comparison and breakdown of the 3 samples that ETS has released so far as a reference to guide)
Section One: Instructions
Section 1 contains the instructions, which are always the same. Only the academic department (sociology, business, political science, etc) changes. Subjects in the liberal arts seem most common, but anything could be used as long as the question is accessible. The questions are unlikely to require any specific technical knowledge. For instance, there are likely to be questions about the use of social media (which most people are broadly familiar with) but questions about something like genetically modified plants (which some people aren’t familiar with) are unlikely. There is an image of the professor.
Section Two: The Question
Section 2 includes the actual question, and some context. In sample questions now available, it ranges from 39 to 56 words. It establishes the general theme of the question and provides background information to activate schemata and give the test-taker time to adjust. There is commonly a reference to “the discussion board” to make it look realistic.
After a clear line break, the professor presents 1 or 2 questions that are visually set apart in a block. The questions are academic in tone, but not challenging. Just a slight step up from the banality of the Independent writing task. Nothing technical, nothing complicated, nothing culturally or demographically inaccessible.
When they ask an open-ended question (“What do you think is the most significant effect…?” or “Which issue would you argue is more important…?”), they follow with a simple “Why?” question that encourages the test-taker to dig into reasons and examples.
When they ask a YES/NO question (“Is advertising just a way of manipulating people…?”), they follow with a second YES/NO question that takes an opposing perspective (“… or is it an important source of information…?”).
It’s common to find a comparative or superlative adjective in the question. For examples, review ETS’s 3 samples here.
The total word count for the questions ranges from 19 to 30 words in the samples currently available.
The total word count for Section 2 probably ranges from 69 to 75 words total in the samples currently available.
Section 3: The First Response
In Section 3, the first student responds to the question. Responses in the currently available samples are 39 to 59 words. In samples currently published by ETS, the first student’s responses seem to have the following characteristics:
✅ contractions (“don’t” as opposed to “do not”) occur with limited frequency
✅ “I know / I don’t think / I think”
❌ limited use of personal examples or personal stories
❌ no abbreviations (“television” not “TV”)
✅ 1 example with a generalization that is based on EITHER:
blending plausible 2nd Person generalizations with 3rd Person generalizations (“When you are watching television, you are not moving around or exercising. This is especially true for children. When children spend a lot of time watching television, they have a greater tendency to be overweight”)
OR plausible statistics with specific details and multiple numbers (“I read that in just one year, from 2018 to 2019, the number of computers, tablets and mobile phones using ad blockers increased from 142 million to 615 million”)
✅ realistic use of capital letters for emphasis in strategic places (“I think the REAL question is…”)
(The above features may or may not appear in other items of this type)
Section Four: The Second Response
In Section 4, the second student responds to the question in 53 to 59 words. In open-ended questions (“What do you think is the most significant effect…” or “Which issue would you argue is more important…?”), they mention a new idea that Student #1 didn’t mention. For YES/NO questions (“Is advertising just manipulation… or is it a source of information?”), Student #2 argues against whatever Student #1 said.
In samples currently published by ETS, student #2’s responses seem to have the following characteristics:
✅ “I think / I disagree with…”
✅ contractions (“I’m” and “There’s” as opposed to “I am” and “There is”) occur with limited frequency
❌ abbreviations (“television” not “TV”)
✅ occasional use of adverbs (“actually”)
✅ 1 example with a generalization that is based on EITHER:
blending plausible 2nd Person generalizations with 1st Person personal stories (“Think of all the different places in the world you can experience through television! Last night, I watched a program about life in Antarctica, and it was fascinating!”)
OR plausible statistics with specific details and multiple numbers (“People can find out a lot about products from advertising. There’s plenty of evidence that people usually begin the process of making a big purchase by looking at ads and reviews… I’m going to post later about an advertisement that gave me a lot of useful information.”)
(Again, note that the above features appear in the materials currently published. They may or may not appear in future items of this type.)
The total number of words that test-takers are exposed to is probably in the range of 165 to 193.
Section Five: Participants
Section 5 is simply to note that one student is male and one student is female. Images of each student are presented along with their responses.
Section 6: The Test-Taker’s Response
The test-taker’s response is typed in Section 6. A word count is displayed on the screen.
—
If you use these guidelines to create your own content, I will be happy to link to it here. So far, decent questions can be found at:
I think the most frequently asked question about the new TOEFL writing task (“writing for an academic discussion”) is about the timing. Test-takers want to know if the ten minutes provided for the task is for both reading and writing. The answer to that comes from a Tweet by ETS:
The second task requires a limited amount of reading to provide a context for writing and help the writer form a response. The reading is part of the task, so it is included in the 10-minute response time.
So there ya go. You’ve got ten minutes to read the question, read the student responses and write your own response. That will require effective time-management. I recommend quickly skimming the responses so that you can spend most of your time writing your own answer.
I do hope that ETS recognizes that a lot of people want to know this information, and that a Tweet (a reply no less) is probably not the best way to disseminate it!
I was pleased to see Sanaz Toossi win another award for “English,” her play about a group of Iranian students preparing for the TOEFL Test in 2009. She’ll receive an award from the Dramatists Guild next month. Back in February the play won the Obie Award for the best new American play staged off-Broadway. Quite a feat.
The play also enjoyed successful runs in Toronto and Montreal.
I scheduled my next TOEFL “Office Hours.” As usual, this is free drop-in meeting to talk about TOEFL, learning English… and anything else that comes up. Both teachers and students are welcome. This time we’ll continue talking about the upcoming TOEFL revisions and focus more on the new writing task. It is scheduled for April 25. Register below.
I very much enjoyed this interview with Josh MacPherson and Kara McWilliams about the upcoming changes to the TOEFL iBT Test. In addition to getting more details about stuff I already know, I learned a few new things. They are:
The new official prep app will be released “toward the summer.” Its AI scoring will include details about things like fluency, pronunciation and other features. That’s more robust than I expected. Great news.
My Best Scores will be retained. I sort of knew this already, but I’ve actually been asked about it quite a few times. I honestly didn’t realize so many people valued My Best Scores.
The Official Guide will be updated. That’s great. Print books still represent some of the best value when it comes to paid test prep. And since they are widely available in public libraries they make the whole preparation process more equitable. But a tip for ETS: don’t use a scratch-off code for downloadable content. Let me know if you need more details.
In time, score reports may provide more comprehensive information for test-takers about their performance. That’s something I’ve mentioned here before.
Item types will be analyzed and may be revised down the road. My interpretation of this is that the test form isn’t going to be set in stone like it was from 2005 to 2019 and from 2019 to 2023.
Anyway. Do watch the interview. You’ll probably find even more new stuff that I didn’t mention here.
The reading section will be shortened to just two articles, with ten question each. There will be no change to the content of the articles and the question styles.
The listening section will be shortened to just 28 questions in total. There will be no change to the content and question styles.
The speaking section will be exactly the same.
The writing section will have two questions. The first will be the same integrated writing question as before. The second will be an academic discussion question.
There will be no break.
There will be no unscored questions.
The test will take slightly less than two hours to complete.
A few details not mentioned on the ETS website were announced by ETS CEO Amit Sevak in China today. Those include:
Official practice sets with AI scoring will be provided free to test-takers.
The actual date that scores will be reported will be given to test-takers when they finish the test.
Test registration will be faster and easier.
ETS will be “more proactive” when it comes to communication with test-takers.
There are many unanswered questions. Hopefully I can add some answers to these questions in the days ahead.
Questions include…
Will the test be easier or harder? My guess is that the test will be the same difficulty level. A shorter test isn’t necessarily an easier test. ETS always tries to maintain a consistent difficulty across every version of the test.
Will My Best Scores still be included? Yes! ETS has confirmed this.
Here’s the March update to the TOEFL/Duolingo Score Requirement tracker. I didn’t post it here in March since I’m losing interest in it, but I did get a request. The point here is to discover if schools update their Duolingo cut scores in light of update conversation tables (from Duolingo). In March, Rice University increased their requirement by ten points. Also, updated wording on UCLA’s page seems to suggest that they will permanently accept Duolingo scores.
You may have to scroll sideways to see the most recent numbers.
I’ll do an April update towards the end of the month.
Note that after April 15, TOEFL and GRE test-takers in India will no longer be permitted to use their Aadhaar card as their primary form of identification. After that date they must present a valid passport. I hope news of this change is widely communicated, as the Aadhaar card is commonly used by Indian students I interact with. Source.
I spent a few hours fixing up my sample questions for the new TOEFL writing task. I somewhat expanded the discussions and added pictures to each speaker, just like the real test will have. I also revised the recommended template a bit so that it begins with a clear thesis statement. I am also thinking about using short paragraph breaks for fun.
Just published in “Language Testing” are the results of a study into the predictive validity of the Duolingo English Test. It notes:
At the postgraduate level, DET scores were related to academic results. Students with higher DET scores had more academic success. At the undergraduate level, DET scores were not related to academic success. The same is true of TOEFL and IELTS scores.
Students accepted with DET scores enjoyed lower academic success than those accepted with TOEFL and IELTS scores.
Given the above, perhaps higher DET cut scores are needed. Or perhaps students arriving with DET scores may need additional academic support to reach their full potential.
The lower academic success of DET test-takers may have something to do with the test itself. Or it may have nothing to do with the test. There are many things that affect academic success.
I note:
Perhaps early access to the results of this study encouraged Duolingo to adjust its score conversion tables. They did that some months ago and schools that pay attention to that sort of thing have already adjusted their cut scores, as I’ve reported over the past eight or nine months.
The cost of taking the Duolingo English Test has increased to $59. According to customer support, Duolingo is “gradually rolling out this new price.” I can already see the new price in my account. On April 2, the price was $49.
