Ha ha.  I am a TOEFL essay machine now.  This took about three minutes to create using my fake essay template, and I think it looks pretty decent.

The prompt is:

Do you agree or disagree with the following statement? It is better for children to grow up in the countryside than in a large city. Use specific reasons and examples to develop your essay.

The Essay is:

A lot of people today think that we should live in the city.  However, I strongly believe that it is much better for kids to live in the country for two reasons.  First, it leads to a lot of great job opportunities.  Second, it vastly improves our health and wellbeing, which a lot of people are struggling with nowadays.  To be fair, a lot of older people have the traditional view that cities are the best place for young people to live.  That said, I think this viewpoint is outdated and quite useless in today’s society.

First, life in the countryside can improve our range of job opportunities in the future.  As I implied above, people my parent’s age (and older) think that living in the countryside is actually quite dangerous.  When I was young and they had a lot of influence over my world view, I actually had the same opinion.  At that time, I thought the lack of businesses in the country would actually make it harder for me to get a job, and so I was hostile toward it.  However, after I entered college and my social network broadened, I realized the unique benefits of rural life.  Now I realize that the presence of agriculture can help us find employment in high paying fields.  For example, my young cousin makes a lot of money because he works in a field related to growing organic crops.  His experience changed my perspective, and now I am focusing on farming at university in the hope of achieving the same thing.

Second, life in the countryside has a noticeable effect on our physical health and maybe even our mental health.  I actually read a story about this in the Village Voice Newspaper a few months ago.  It pointed out that if we properly use hiking trails we can avoid the poor health that a lot of people are dealing with nowadays.  The article claimed that 75% of Americans think that the best way of staying fit is making use of rural sports.  Medical experts who reviewed the study results agreed, and suggested that rural lifestyles will have an even greater impact in the future because of the clean air in the countryside.  Consequently, I strongly feel that benefiting from life away from crowded cities is a fantastic way to stay healthy.

In conclusion, I think that it is best for young people to live in the countryside.  This is because it can lead to gainful employment, and because it has a positive impact on our minds and bodies.

Okay, I’m having fun with the Gangnam style TOEFL template I generated yesterday.  This time I tackled the second prompt in my collection.  Obviously it has a lot of overlap, since both deal with the Internet.  Next time I think I will delete the final sentence from the introduction. It lays the template on a bit too thick.  I’ll replace it with nothing, and just jump to the body after the thesis statement.

The prompt is:

Do you agree or disagree with the following statement? It is better to use printed materials such as books and articles to do research than it is to use the internet. Use specific reasons and examples to support your answer.

The “fake essay” is:

A lot of people today think that using online materials for research is a bad idea.  However, I strongly believe that using the Internet for research is wise for two reasons.  First, it leads to a lot of great job opportunities.  Second, it vastly improves our health and wellbeing, which a lot of people are struggling with nowadays.  To be fair, a lot of older people have the traditional view that websites are unreliable.  That said, I think this viewpoint is outdated and quite useless in today’s society.

First, using the Internet for researching topics can improve our range of job opportunities in the future.  As I implied above, people my parent’s age (and older) think that the web is actually quite dangerous.  When I was young and they had a lot of influence over my world view, I actually had the same opinion.  At that time, I thought relying on unreliable online sources would actually make it harder for me to get a job, and so I was hostile toward it.  However, after I entered college and my social network broadened, I realized the unique benefits of cutting edge research that is published online.  Now I realize that learning about the latest academic developments online can help us find employment in high paying fields.  For example, my young cousin makes a lot of money because he works in a field related to crypto-currency.  His experience changed my perspective, and now I am focusing on emerging web-based technologies at university in the hope of achieving the same thing.

Second, medical websites have a noticeable effect on our physical health and maybe even our mental health.  I actually read a story about this in the Village Voice Newspaper a few months ago.  It pointed out that if we properly use websites that report on health trends we can avoid the poor health that a lot of people are dealing with nowadays.  The article claimed that 75% of Americans think that the best way of staying fit is making use of the Internet.  Medical experts who reviewed the study results agreed, and suggested that websites will have an even greater impact in the future because of the number of doctors who are online.  Consequently, I strongly feel that benefiting from online research is a fantastic way to stay healthy.

In conclusion, I think that researching online is beneficial.  This is because it can lead to gainful employment, and because it has a positive impact on our minds and bodies.

There is a wonderful article by Sugene Kim (Nagoya University of Commerce & Business) in this month’s issue of “Assessing Writing” that describes TOEFL writing tutoring in Korea.  I’ve written about how Korean students prepare for the TOEFL a few times (see this article from last year), and Kim’s article touches on some of the same themes.

Kim describes how Korean tutors use templates to prepare students for both of the writing tasks.  But these aren’t templates like the ones on this site.  The templates prepared by Korean tutors are almost complete essays.  The template might come out to 300 or 350 words, and the student simply plugs in 20 or 30 words that match the given topic.  The article includes interviews with students who confirm the effectiveness of this method.  The tutors produce totally personalized templates for each student so they are much harder for ETS to detect.  The article describes how the most successful tutors become stars in education sector.  One is referred to by students as “the Writing sniper.”  Of his work, the interviewee notes:

“He gives lessons based on his own templates, which are famous for ‘mesmerizing’ ETS essay raters.  Most of his former pupils are known to have experiences a dramatic score increase after taking his classes.”

The article doesn’t exaggerate.  I am based in Korea, and one of my close friends in the TOEFL-prep industry is  a star tutor.  His business is booming and he gets results.

A sample template (complete with student annotations) is available in the article.  Personally, I don’t teach this way but if I was going to write a “Korean Style Template” it would look something like this:

Introduction

A lot of people today think that [opposite of thesis].  However, I strongly believe that [thesis] for two reasons.  First, it leads to a lot of great job opportunities.  Second, it vastly improves our health and wellbeing, which a lot of people are struggling with nowadays.  To be fair, a lot of older people have the traditional view that [opposite of thesis].  That said, I think this viewpoint is outdated and quite useless in today’s society.

Body 1

First, [topic] can improve our range of job opportunities in the future.  As I implied above, people my parent’s age (and older) think that [topic] is actually quite dangerous.  When I was young and they had a lot of influence over my world view, I actually had the same opinion.  At that time, I thought [aspect of topic] would actually make it harder for me to get a job, and so I was hostile toward it.  However, after I entered college and my social network broadened, I realized the unique benefits of [aspect of topic].  Now I realize that [specific aspect of topic] can help us find employment in high paying fields.  For example, my young cousin makes a lot of money because he works in a field related to [specific aspect of topic].  His experience changed my perspective, and now I am focusing on [aspect of topic] at university in the hope of achieving the same thing.

Body 2

Second, [topic] has a noticeable effect on our physical health and maybe even our mental health.  I actually read a story about this in the Village Voice Newspaper a few months ago.  It pointed out that if we properly use [topic] we can avoid the poor health that a lot of people are dealing with nowadays.  The article claimed that 75% of Americans think that the best way of staying fit is making use of [aspect of topic].  Medical experts who reviewed the study results agreed, and suggested that [topic] will have an even greater impact in the future because of [aspect of topic].  Consequently, I strongly feel that benefiting from [topic] is a fantastic way to stay healthy.

Conclusion

In conclusion, I think that [thesis].  This is because it can lead to gainful employment, and because it has a positive impact on our minds and bodies.

 

Okay, so I might use this to answer a prompt about the Internet ( Do you agree or disagree with the following statement? Overall, the widespread use of the internet has a mostly positive effect on life in today’s world) thusly:

A lot of people today think that the internet is harmful.  However, I strongly believe that internet access is beneficial for two reasons.  First, it leads to a lot of great job opportunities.  Second, it vastly improves our health and wellbeing, which a lot of people are struggling with nowadays.  To be fair, a lot of older people have the traditional view that the internet is damaging.  That said, I think this viewpoint is outdated and quite useless in today’s society.

First, the internet can improve our range of job opportunities in the future.  As I implied above, people my parent’s age (and older) think that the internet is actually quite dangerous.  When I was young and they had a lot of influence over my world view, I actually had the same opinion.  At that time, I thought websites would actually eliminate jobs, and so I was hostile toward them.  However, after I entered college and my social network broadened, I realized the unique benefits of online communications.  Now I realize that the Internet can help us find employment in high paying fields.  For example, my young cousin makes a lot of money because he works in a field related to online commerce.  His experience changed my perspective, and now I am focusing on information technology at university in the hope of achieving the same thing.

Second, the Internet has a noticeable effect on our physical health and maybe even our mental health.  I actually read a story about this in the Village Voice Newspaper a few months ago.  It pointed out that if we properly use the Internet we can avoid the poor health that a lot of people are dealing with nowadays.  The article claimed that 75% of Americans think that the best way of staying fit is making use of fitness applications.  Medical experts who reviewed the study results agreed, and suggested that the Internet will have an even greater impact in the future because of new developments.  Consequently, I strongly feel that benefiting from the Internet is a fantastic way to stay healthy.

In conclusion, I think that the Internet is beneficial.  This is because it can lead to gainful employment, and because it has a positive impact on our minds and bodies.

But it isn’t as easy as that. A few things are worth mentioning:

  • At first glance, it seems like this template might be useable with only about 10% of the prompts on my sample page.   The star tutor will give students a bunch of templates to cover the whole range of possible prompts.  They will also get them to mix and match body paragraphs covering a range of benefits (jobs, health, quality of life, family bonds, relationships, etc.).
  • I spent about 10 minutes writing this template.  It isn’t great, but I think it demonstrates the point I am trying to convey.  The star tutors work harder to provide much better templates for their students. 
  • As the article points out, students take the test many times.  The system doesn’t always work.
  • As I mentioned in a blog post last year, ETS seems to be penalizing students for fake surveys and research nowadays.  Professor Kim didn’t mention this, but anecdotal evidence I’ve heard seems to corroborate my claim.

What does this all mean?  I don’t know.

Students often ask what they should mention first in each integrated essay body paragraph  – the reading or the lecture?

I generally recommend mentioning the reading first.  However, I decided to do some research to figure out what ETS raters think is best.  I examined a variety of official sources, all of which include sample essays described as “high scoring.” 

I learned that the reading is generally mentioned first, but sometimes the lecture comes first.  This means you can probably do what feels best to you.

Official Guide to the TOEFL

  • Test Two: The reading is not mentioned.  Really.
  • Test Three: The reading is mentioned first.
  • Test Four:  The reading is mentioned first.

I did not include test one in my research as the sample question is terrible, and the sample essay is worse (one giant paragraph).

Official iBT Tests Volume 2

  • Test One: The reading is mentioned first.
  • Test Two: The lecture is mentioned first.
  • Test Three: The reading is mentioned first.
  • Test Four: The reading is mentioned first.
  • Test Five: The reading is mentioned first.

Note that in a few of the essays the body paragraphs begin with something like:  “First, the lecturer says [DETAILS FROM THE READING] are not true.”  I so mark these as the reading being mentioned first.  Note, also, that the Official iBT Tests Volume 1 has no sample essays.  Sucks.

Official Practice Sets

  • Writing Set (voting systems): The lecture is mentioned first

Official Practice Test

  • The reading is mentioned first

Propell Workshop Supplement Dated 2017

  • The reading is mentioned first (a weird essay with two paragraphs in total)

Here’s what I mean when I say that using memorized content is a bad idea.  Check out this bad body paragraph, in response to a question like “Do you prefer to read books or to watch movies based on the books?”

“To begin with, watching movies based on books will let us know if reading the books themselves is a good idea. As we are very busy, it is a real challenge to read even one or two books in a month. If we watch movies, however, we can get an idea of which books we should read in our limited free time. My personal experience is a compelling example of what I mean. When I was young, the book “Harry Potter” seemed interesting to me, but I didn’t read it because it had too many pages. Additionally, I didn’t have enough time to finish it since I practiced basketball almost every day when I was a junior high school student.  I went to the basketball court in my neighborhood every weekend and practiced passing, shooting and dribbling with my friend, Jim, who I had known since I was in elementary school.  We practiced really hard, and, as a result, we were eventually invited to join a local team.  At the end of that year, the team won a local championship.  Had we not practiced every day, we would not have achieved such success. Several years later, a film based on the novel was released. I went to the local cinema and watched the move because it was only ninety minutes long. After I saw the film, I bought a copy of the novel because the movie showed me that it would be a good use of my limited free time.”

 

Note the stuff in bold, which is just stuff the student memorized before going to the test center.  They have inserted it into the paragraph to increase the word count and to add some slightly more complicated sentence structures.  It’s got parenthetical commas, the past perfect tense and a conditional.  Even a transitional phrase!  How nice!  Sadly, it sticks out like a sore thumb.  The grader can tell it is off-topic memorized junk.  This is an essay about books and movies, not an essay about how to win a basketball championship. 

I constantly get paragraphs that have been stretched out with irrelevant digressions like this.  Sometimes you can get away with using them, but generally it is a bad idea.  My impression is that ETS is working harder than ever nowadays to crack down on this junk.

If you see a warning at the test center saying “do not use memorized examples” this is what the warning is about.

 

One of the most frequent questions I get is about the “maximum word count” in the TOEFL essays.  A lot of students think they must not write more than the suggested word count in both essays.  Even some teachers think it is a problem.

Hopefully this blog post will put that rumor to rest.

Here’s what ETS says about the word count for TOEFL writing task 1 (the integrated essay):

(Official Guide to the TOEFL Test, 6th edition, page 192. Published 2021)

And here’s what ETS says about the word count for TOEFL writing task 2 (the independent essay):

(Official Guide to the TOEFL Test, 6th edition, page 199.  Published 2021).

Meanwhile, when ETS is training teachers at their regular “Propell Workshops” they provide sample essays of each score level.  To represent the 5.0 level (the highest possible score) they provide the following 305-word integrated essay:

sample integrated essay

(Propell Workshop for the iBT Test, Page 42.  Published 2019)

And to represent an independent essay scoring 5.0, they provide this 466-word monster:

(TOEFL Teacher Workshop Manual, Page 49)

So there you go.  Write as much as you want.  Don’t worry about being penalized.

The other day, someone asked:

I’ve got twelve months to prepare for the TOEFL, and I need 100 points.  What should I do?

The good news for that student is that they have time to really improve their English fluency instead of just learning TOEFL tricks and strategies.  I know it sounds crazy, but the best way to increase your TOEFL score is to become more fluent in English.

 

Here’s how I responded:

  1.  Get a good grammar book like “English Grammar in Use” (also called “Grammar in Use – Intermediate” in some countries).  I read about a dozen TOEFL essays every day, and I see that most students suffer from grammar and language use problems.   Reduce your error rate and your writing score will go up.
  2. Find someone to practice speaking with.  To improve your score you need to speak fluidly.  You need to eliminate pauses, “umm breaks”, and repetitions.  You need to pronounce vowels and consonants properly.  You need to reduce the effort required to understand what you are saying.  Regular practice will help with this.  You don’t necessarily have t pay big bucks for a special TOEFL teacher to do this.  You can probably find an affordable tutor on a service like italki for this.
  3. Take accurate practice TOEFL tests.  There are 15 official ETS practice tests available (Official Guide x 4, Official iBT Tests x 10, website x 1) plus some PDF junk on the website.  You should work through all of those.  Fortunately, you have time to buy all of the books!  Switch to unofficial material only when you run out.
  4. If you have a year to prepare you can also improve your reading and listening skills in a general sense.  Spend some time reading good non-fiction books and articles (I like Science News, and National Geographic).  Make use of your local library, if they have an English section.  For listening, try Khan Academy, or podcasts like 60 Second Science.
  5. Towards the end of your preparation period take one of the scored practice tests from ETS to gauge your current level and see how to use the last few months most effectively.

 

And, yes, along the way you should devote some time to becoming familiar with the test.  Read the Official Guide cover to cover (a few times).  Read some of the guides on this website and watch some Youtube videos.  Review sample writing and speaking responses.  Just don’t get bogged down in “strategies” if the test is still a year away.

Editor’s Note: This is part three in my series on the e-rater.  You might want to  read part one  and part two.

Les Perelman has a new article in the Journal of Writing Assessment about current problems associated with automatic essay scoring in general, and the ETS e-rater in particular.  This is stuff Perelman has written about before, but I do encourage you to read the article, even if you are familiar with his work.  It is illuminating.

I want to touch on one observation made by Perelman in the article.  He notes:

“Indeed, ETS researchers themselves acknowledge the susceptibility of e-rater to both coaching and gaming when discussing e-rater’s scoring mainland Chinese on average over a half a point higher than human raters (d = 0.60) on the GRE issue essay”

Perelman doesn’t mention it, but the effect is similarly pronounced in the TOEFL , (d = 0.25 on a 5 point scale) according to research from ETS.

Note, also, that Korean students experience a similar benefit, though it is not as large as the one experienced by Chinese students.

How Do Teachers “Game the System”?

In case you are curious, here’s how this kind of TOEFL prep works.

If we assume that almost every prompt can be supported with an argument about health, we teach the student to begin their first body paragraph with a topic sentence like this:

“To begin with, _________ can improve our overall physical condition.”  

They just need to fill in the blank with their choice from the prompt.

Looking at some of my sample questions, this works quite often:

  • To begin with, the widespread use of the Internet can improve our overall physical condition.”
  • To begin with, technologies in the modern world can improve our overall physical condition.”
  • To begin with, living in the country can improve our overall physical condition.”
  • To begin with, modern diets can improve our overall physical condition.”

Of course, not EVERY question can be answered with a comment about health, but if you teach the student five or six different sentences, you can cover most of the prompts.  Like:

  • To begin with, ________ can improve our career prospects.
  • To begin with, ________ can improve our creativity.
  • To begin with, ________ can improve our relationships with our loved ones.

And so on.  These will all sound a bit clunky when used (like the four above do) but they will be grammatically correct and will please the e-rater.

After that, we teach the student a totally memorized sentence to function as “explanatory” content immediately after that sentence.  These just make some generalized sentences about the topic.  Something like:

  • “Most educated people agree that we cannot achieve anything in life without a body that is strong and healthy.”
  • “It is undeniable that our overall quality of life is strongly affected by how much success we enjoy in our career.”
  • “It is undeniable that we will live longer and more prosperous lives if we are imaginative.”
  • “In my culture, everyone feels that maintaining close connections with loved ones is more important than anything else.”

Just one blandly generic sentence isn’t enough to get the student flagged for being off-topic.  The student must depend on their own ability to write the rest of the essay, but if they use this technique in both body paragraphs, 15 or 20 percent of their essay will be written in perfect English.  That’s a nice start.

Some students might use even more memorized content, but of course that increases the risk of being flagged as off-topic.

Does ETS Know?

Yeah.  They say:

“Another possible explanation for the greater discrepancy between human and machine scores for essays from mainland China may be the dominance of coaching schools in mainland China that emphasize memorizing large chunks of text that can be recalled verbatim on the test. Human raters may assign a relatively low score if they recognize this memorized text as being somewhat off topic, though not so far off topic as to generate a score of 0. On the other hand, this grammatical and well-structured memorized text would receive a high score from e-rater. Although automated scoring engines can be trained to identify text that is not at all related to the assigned topic, they may not yet be sensitive enough to recognize this slightly off topic text.”

No shit. What surprises me, as a teacher, is that after saying this they just leave it hanging.  Neither a solution nor a response is really offered.  

They do say that:

“each essay is scored by at least one human plus e-rater. Second, if there is a discrepancy of 1.5 or more points (on the 0–5 score scale) between the human score and the e-rater score, an additional human score is obtained. The item score is then the mean of the three scores (2 human plus e-rater) unless one score is an outlier (more than 1.5 points discrepant), in which case the outlier is discarded and the remaining two scores are averaged.

But this is meaningless, as a 1.5 difference is huge.  That’s a nine point difference on the 30-point scale.  By using memorized content only in the independent task, a student could get a bonus of 2.25 points overall in the writing section without any alarm bells being sounded. 

This is weird, because ETS uses a much better system to prevent this from being a problem in the GRE:

The GRE uses a highly conservative approach in which the machine is used only to flag discrepant human scores to signal the need for a second human rating. Specifically, if the e-rater score rounded to the nearest whole number does not agree exactly with the first human score, then a second human score is obtained; the e-rater score is never averaged with a human score
It is also worth mentioning that ETS has licensed their e-rater (and SpeechRater) tech to the Chinese test prep website KMF which provides very low-cost access to the tech only to Chinese students.  This gives those students the opportunity to experiment with the e-rater and fine-tune the way their write their essays in order to please it.

What about the Plagiarism Warning?

No, you cannot use the exact  sentences mentioned above.  Presumably ETS can write some software that will detect that you’ve copied from this website.  However, wealthy students will just hire a teacher to write personalized content only for them.  That content will not appear anywhere online, and it won’t even be used in anyone else’s tests.  That’s how they can avoid being penalized by automatic plagiarism detection.

This is a Really Bad Thing

It is probably a bad thing that students from some ethnicities do better with the e-rater.  Especially since some ethnicities, particularly African-Americans (on the GRE) and Arabic and Hindi speaking students (on the TOEFL), do worse.

There aren’t just racial implications, but class implications as well.  As Perelman indicates (emphasis is mine):

“It is the following paragraph, however, that contains the most egregious instance of misinformation. “The primary emphasis in scoring the Analytical Writing section is on your critical thinking and analytical writing skills rather than on grammar and mechanics.” (Educational Testing Service, n.d.-a, n.d.-b). E-rater provides half the final score. Yet, e-rater does not emphasize “critical thinking and analytic writing skills.”   Indeed, it is completely oblivious to them. Its closest approximation is its highly reductive feature of development, which is calculated by the number of sentences in each paragraph and the number of paragraphs in the essay. Furthermore, grammar and mechanics compose a significant portion of the features included in e-rater’s calculations. Low-income students will believe these statements and focus on critical thinking and analytic skills. Affluent students who have taken test preparation classes, on the other hand, will be coached to provide e-rater with the proxies that will inflate their scores.

Anyways.  That’s all I have to report. Thank you for listening to my TED Talk.

 

 

I got another question from a student.  They asked:

“I have a question about the independent writing task I hope you can answer. What is the most common type of question that most test takers get in this task?  Is it multiple choice?  Advantages and Disadvantages?  Something else?”

Here’s the answer I gave:

The most common type of independent writing question is the “Agree/Disagree” style, which seems to come up about 50% of the time.  This type presents a single-sentence statement and you must state if you agree or disagree with it.

The “paired choice” question comes up about 25% of the time.  In this one, you must choose between two opposing or related options.  Like whether it is best to study in a group or to study alone.  Sometimes this takes the form of a hypothetical situation, like whether it is better for a company to donate to a museum or to donate to a children’s sports team.

The “multiple choice” question comes up about 25% of the time.  This is a lot like the paired choice, but with three options (sometimes four or five).  It is a bit tricky because with all those options the prompt can take a long time to read.

Finally, in rare cases (or maybe never) you might get a “good idea” prompt, where you are presented with a hypothetical situation, and should state whether you think it is a good idea.  For instance, you might be told that a company will forbid employees from answering emails on the weekend.

I must mention that “advantages and disadvantages” questions are not used on the test.

I heard from a student today who was confused about why their TOEFL writing score was so low.  They got 30 points in the reading section, 29 points in the listening section, 30 points in the speaking section… but just 19 points in the writing section.  I could tell just by reading their e-mail that their grammar was far above the “19 point level,” so clearly that wasn’t the cause of the low score.

So what caused their weird (and disappointing) writing score?  Here’s how I responded:

It is impossible for me to say what caused your low score, since I cannot read the essays you wrote on test day.

However, I have a few ideas:

  1. One or both of your body paragraphs in the independent essay could have been flagged as off-topic. This will result in a major penalty, even if your grammar is perfect.  This is the sort of thing that a score review might help you with.
  2. You might have written really short essays. If you follow the recommended word counts stated on the screen during the test it is hard (but not impossible) to score in the high 20s.
  3. You may have misunderstood some key lecture details in the integrated essay. Again, even with perfect grammar you can get a low score if your details are wrong.
  4. Nowadays ETS seems to be penalizing students who use made-up research or news stories to support their reasons in the independent essay.

 

These are just some of the reasons why a student with an advanced command of the English language might get a low score in the writing section. 

It is important to note that number four is a recent development.  Some students have suggested that they’ve been penalized for this, but obviously this is just speculation on their part.  That said, a recent update to the Official Guide to the TOEFL (August 2020) supports this theory.  Below is that update.

 

Hey, I finally uploaded the 2020 version of my guide to the independent TOEFL essay.  This is the fifth annual installment of this video!  In the next few days, I will update the website version to match the content here.

The new video has a slightly different template, a new sample essay and a few different ideas about grammar.  I also (finally) added an “FAQ” section to the end in order to avoid answering the same questions in the comments again and again!

I’ll create and film a video about the integrated writing task before the end of the month (I hope).