I’ve recently uploaded a bunch of videos about the new writing task.  Just in case you’ve missed ’em, here they are:

What do test-takers need to perform well on the new TOEFL “writing for an academic discussion” task?

Conjunctions, I guess.

The scoring rubric for this task hints at this when it says that a high-scoring answer contains “effective use of a variety of syntactic structures.”

I started today’s experiments by submitting a response consisting of only simple sentences. It certainly lacked syntactic variety, and received a score of 4.0 from the AI scoring engine:

I like the ideas noted by Claire and Kelly. I feel that the only way to truly solve this problem is to build better schools.  Parents want their children to be educated at the best possible places. It is almost impossible to find great schools in the country. The government currently has money for good things like science laboratories and nice libraries in cities. They ignore rural areas all the time. Rural schools like the one I attended lack even basic educational supplies like computers and sports equipment.  Parents who are concerned about their kids go to bigger places.

I tried to beef it up with some impressive vocabulary but it still scored 4.0:

I respect the ideas noted by Claire and Kelly. I feel that the sole way to truly resolve this dilemma is to construct better schools.  Parents want their children to be educated at the best possible facilities. It is almost impossible to find impressive schools in the country. The government currently has money for valuable things like science laboratories and fantastic libraries in cities. They ignore rural areas all the time. Rural schools like the one I attended lack even basic educational supplies like computers and sports equipment.  Parents who are concerned about their kids go to bigger places.

I wrote a really long version (197 words) but still scored 4.0:

I like the ideas noted by Claire and Kelly. I feel that the only way to truly solve this problem is to build better schools. Schools are the bedrock of all academic systems.  Parents want their children to be educated at the best possible places. It is almost impossible to find great schools in the country. The country only has a limited range of places to learn. The government currently has money for good things like science laboratories and nice libraries in cities. These facilities give adolescents there an advantage. It is clear to see that they learn much more. Their test scores on university entrance exams are the highest in the entire nation. This means they can attend the best universities. The government ignores rural areas all the time. Rural schools like the one I attended lack even basic educational supplies like computers and sports equipment. Students at those schools perform poorly. They struggle to learn advanced concepts. I think they have poor reading comprehension as well. They are unable to go to good universities. It is likely that they don’t enjoy impressive careers either.  Parents who are concerned about their kids go to bigger places.

I’ll spare you the pain of reading it, but a long version with better vocabulary also scored 4.0.

In an effort to insert some syntactic variety, I wrote a version of the original answer with two different coordinating conjunctions (but, but, so). It also got 4.0:

I like the ideas noted by Claire and Kelly, but I feel that the only way to truly solve this problem is to build better schools.  Parents want their children to be educated at the best possible places, but it is almost impossible to find great schools in the country. The government currently has money for good things like science laboratories and nice libraries in cities. They ignore rural areas all the time. Rural schools like the one I attended lack even basic educational supplies like computers and sports equipment, so parents who are concerned about their kids go to bigger places.

And when I used three different coordinating conjunctions (but, yet, so)? It still scored 4.0:

I like the ideas noted by Claire and Kelly, but I feel that the only way to truly solve this problem is to build better schools.  Parents want their children to be educated at the best possible places. It is almost impossible to find great schools in the country. The government currently has money for good things like science laboratories and nice libraries in cities, yet they ignore rural areas all the time. Rural schools like the one I attended lack even basic educational supplies like computers and sports equipment, so parents who are concerned about their kids go to bigger places.

Next, I wrote a version with two different coordinating conjunctions (but, so) and one subordinating conjunction (while) and it scored 5.0:

While I like the ideas noted by Claire and Kelly, I feel that the only way to truly solve this problem is to build better schools.  Parents want their children to be educated at the best possible places, but it is almost impossible to find great schools in the country. The government currently has money for good things like science laboratories and nice libraries in cities. They ignore rural areas all the time. Rural schools like the one I attended lack even basic educational supplies like computers and sports equipment, so parents are concerned about their kids and go to bigger places.

Finally!

It still scored 5.0 when I moved the conjunctions around a bit:

I like the ideas noted by Claire and Kelly, but I feel that the only way to truly solve this problem is to build better schools.  Parents want their children to be educated at the best possible places. It is almost impossible to find great schools in the country. While the government currently has money for good things like science laboratories and nice libraries in cities, they ignore rural areas all the time. Rural schools like the one I attended lack even basic educational supplies like computers and sports equipment, so parents who are concerned about their kids go to bigger places.

And it still scored 5.0 when I used a different subordinating conjunction (when):

I like the ideas noted by Claire and Kelly, but I feel that the only way to truly solve this problem is to build better schools.  Parents want their children to be educated at the best possible places. It is almost impossible to find great schools in the country. The government currently has money for good things like science laboratories and nice libraries in cities, so they should not ignore rural areas all the time. When rural schools like the one I attended lack even basic educational supplies like computers and sports equipment, parents who are concerned about their kids go to bigger places.

So there ya go. If you are planning to take the new TOEFL after July 26, make your writing a bit more sophisticated by including both types of conjunctions. That could certainly show off your ability to effectively “use of a variety of syntactic structures.”

One takeaway of this research is that a possible “magic formula” for the new writing task is: 100+ words, one subordinating conjunction, two coordinating conjunctions.

You can find all of my submissions on this page.

To learn more about the impact of word count on automated scoring of “writing for academic discussion” prompts that will be included on the TOEFL on July 26, I spent some time answering the sample questions provided by ETS. This sort of experimentation is mildly important, I think, as many test-takers (and tutors) hold the idea that only really long responses get high scores on the current TOEFL. This sometimes results in the creation of monster-sized TOEFL essays.

What did I learn?

Happily, the automated scoring system (e-rater) gave me a perfect score of 5.0 for the following 102-word response to sample question one (about how to repopulate the countryside):

“While I appreciate the solutions presented by Claire and Kelly, I feel that the only way to truly solve this problem is to construct better schools.  Parents want their children to be educated at the best possible facilities, but it is almost impossible to find impressive schools in the countryside. Although the government currently provides funding for amenities like science laboratories and lavish libraries in cities, they neglect rural areas. Rural schools, like the one I attended, lack even basic educational supplies like computers and sports equipment.  Consequently, parents who are concerned about their kids head for greener pastures, so to speak.”

The question prompt recommends writing about 100 words, so I’m happy. Students can confidently follow the given instructions, I guess.  

I fiddled about with a series of shorter (but similar) answers and was able to get a score of 4.0 for the following 49-word response:

“While I appreciate your ideas, I think we need better schools.  Parents want their children to utilize excellent facilities, but it’s impossible to find impressive schools in the countryside. Although the government currently funds amenities like libraries in cities, they neglect rural areas.  Consequently, parents head for greener pastures.”

That’s as low as I could go and still get a score of 4.0.

But I could get a score of 3.0 for the following 32-word response:

“While I appreciate your suggestions, we need better schools.  Children need excellent facilities, but the countryside lacks them. Although the government funds academics in cities, they neglect rural areas.  Consequently, parents leave.”

Anything lower than that resulted in a score of 2.0 or less.

A few things are worth mentioning:

1. The automated score will be combined with a human score on the real test.
2. Obviously word count correlates with other features like range of vocabulary and number of grammatical features.
3. The above scores encompass a whole range of scores once they are scaled up. A score of 4.0 from the e-rater could scale up to anything from 21 to 26. My sample would likely be on the low end of this range. It would be nice to get decimals from the ETS website.

You can find a record of everything I submitted along with the e-rater scores over here.

ETS now provides twenty-two samples of the “writing for an academic discussion” questions that will appear on the TOEFL Test starting July 26. Each one includes AI scoring using the same e-rater that is used to (partially) score the real test. More questions will be released in the weeks ahead.

Closer to that day I’ll examine the questions and gather some data related to item design and frequent topics. I’ll also experiment a bit with ETS’s AI scoring.

One thing stands out already, though. Most of these questions are about more challenging topics than appear on the current Independent Writing task (which will be removed July 26). They seem to require more thoughtfulness. Or even some amount of erudition.

Consider Sample 11:

“This week, we will be discussing a shortage of affordable housing that exists in many countries. In these places, housing – both apartments (flats) and houses – are expensive, because populations are growing faster than new housing is being built. Now, think about places in your country that have a housing shortage. In your post, I would like you to indicate the most effective way for the government to address a housing shortage in your country. Please explain why you think so.”

Or Sample 1:

“Let’s think about population Trends in urban and rural areas (villages). Living in urban areas can be expensive; nonetheless, when they have a choice of where to live, people in some countries do not wish to live in rural areas even if the cost of living there is lower. If governments of some countries want to attract more people to live in rural areas or villages, what is the best strategy or approach that governments can use? Why?”

The test-takers might be asked how to solve the housing crisis? How to repopulate the countryside? I like this approach to item design, but I know some people might struggle to come up with ideas.

Compare these to this Independent Writing task from Test 1 in the Official Guide to the TOEFL:

“Some young adults want independence from their parents as soon as possible. Other young adults prefer to live with their families for a longer time. Which of these situations do you think is better?”

Or from Test 2:

“Do you agree or disagree with the following statement? Young people enjoy life more than older people do.”

I know that some of the new sample questions are easier than the ones I quoted above. And some of the independent tasks on the current version of test are harder than what appears in the Official Guide. That said, if you are preparing for the new TOEFL, you should be prepared to THINK. As always, remember that this is not a test of your English skills. It is a test of your ability to use your English skills in an academic context.