This month I read the May 23 issue of Time Magazine, and found a few useful articles about climate change, both by Aryn Baker.

First up, I read a short essay called “After Visiting Both Ends of the Earth, I Realized How Much Trouble We’re In.”  This description of the impact of climate change is similar in length to what you will read on the TOEFL, and it contains a lot of great vocabulary that you might need to succeed on the test.  I don’t often recommend Time here, but if you skip the political stuff you might find some good scientific content.

Next, I read a longer piece called “Polar Paradox: The Melting Arctic Could Destroy Indigenous Ways of Life While Making Some Alaskans Rich.”  It contains a lot of great on-the-ground reporting that captured my attention.  One of the nice things about Time is that the writing level is fairly similar to what you will see on the test (or in a freshman university textbook).  It contains articles that are challenging for ESL students, but not too challenging.  Consider making a short list of all the words in these two articles that are new to you.  After that you can look up their definitions and play with them a bit.

Moving along, I found a Popular Science article (via Reader’s Digest) about an effort to preserve the sound of a 1727 Stradivarius violin.  This one is a bit more fun that the above two articles, but still includes a lot of useful academic vocabulary.  

Finally, I want to recommend a few things to listen to this month.  Listening is good too, right?  I listened to a trio of Malcom Gladwell podcasts from 2016 about higher education in the USA.  I know that’s something that most readers here have an interest in.  Here are some quick links:

  • Carlos Doesn’t Remember covers the difficult of the American education system to foster the academic development of kids that are smart, but poor.  It’s heartbreaking at times.
  • Food Fight is about the differing ways that American universities use their money.  It’s frustrating.
  • My Little Hundred Million is about how wealthy Americans choose to donate to universities.  It is sometimes inspiring, but often frustrating.

I learned today that ApplyBoard is selling discounted TOEFL vouchers via its website.  If you want, you can go buy one over here.  Or read on for more details.

Basically, you can buy a TOEFL voucher at a 20% discount from Applyboard.  You can then use the voucher when you register for the test.  That’s a nice way to save some money!

Note that vouchers are locked to specific countries and currently can only be purchased for: India, Bangladesh, Pakistan, Sri Lanka, Vietnam, Philippines, Brazil, Canada and Mexico.

Another thing worth noting is that the pricing seems a little bit quirky, so the discount is not always exactly twenty percent.  For your convenience, here’s a list of what you will pay for one voucher in each country:

  • India: $158.57 (17% discount)
  • Bangladesh: $164  (20% discount)
  • Pakistan: $160 (20% discount)
  • Sri Lanka: $148 (20% discount)
  • Vietnam: $160 (20% discount)
  • Brazil: $172 (20% discount)
  • Canada: $151.20 (32% discount)
  • Mexico: $160.00 (20% discount)

The discount in Canada is probably more than 32% since no extra sales taxes are added in the shopping cart.

A few other details are worth mentioning here:

  • You can get even larger discounts if you buy vouchers in bulk. You’ll get a 25% discount if you buy more than 10 vouchers, and a 30% discount if you buy more than 50.
  • You can also buy TOEFL prep materials (e-books and graded TPOs) at discounted prices.
  • You can also buy vouchers for the GRE and the PTE.
  • I don’t know if the vouchers have expiration dates.
  • ETS usually adjusts its prices in August.  I don’t know how this will impact the sale of vouchers.

So there you go.  I hope this helps you save some money on your TOEFL registration. 

 

The British Council recently funded a study comparing the IELTS Academic Test and the Duolingo English Test.  You can read the study here.  The authors of the report suggest that the DET has some weaknesses.  They conclude:

Our analysis demonstrates that, compared to IELTS, DET test tasks under-represent the construct of academic language proficiency as it is commonly understood, i.e., the ability to speak, listen, read, and write in academic contexts. Most of the DET test tasks are heavily weighted towards vocabulary knowledge and syntactic parsing rather than comprehension or production of extended discourse, though the recent addition of Interactive Reading addresses this lack somewhat.

But they do note that:

Scores on the two tests are correlated, which might suggest that DET is a reasonable substitute for IELTS, given its accessibility and low cost. Of course, knowledge of lexis and grammar are essential enabling skills for higher-order cognitive skills, and a test that focuses on these lower-level skills can be useful for making broad distinctions between low, intermediate, and high proficiency learners. However, potential test users should be aware of the limitations of DET in terms of predicting academic success.

The study was done by researchers working out of Georgia State University.

Today I want to pass along a few details from the Virtual Seminar for English Language Teachers hosted by ETS last night.

One of the presenters provided new details about how the speaking questions are scored. He prefaced these details by sharing a sample type 3 speaking question.  Here is the reading part:

And here is a transcript of the listening part:

 

A standard question, right?

Then we were shown the “answer sheet” that is given to raters so they know how to assess topic development.  This is new information.  Here it is:

That’s interesting, right?  My assessment of this is that for an answer to receive a full score for topic development it must explicitly or implicitly reference the term and its definition.  It must also broadly summarize the example.  And then it must include just two of the four main details given in the example.  The last part is new to me.  Generally, I push students to include all of the details.  Perhaps I should reassess my teaching methods.

There you go, teachers.  Some new information about the TOEFL… in 2022.

A few questions remain:

  • Is this always the case?  Will there always be four main details in the example?  Will we always need to include just two of them?  Probably not.  Surely there are cases where more than two details are required.
  • How does this work in lectures which have two totally unique examples?  Often the reading is about a biological feature in animals, while the lecture describes two different animals that have this feature.  Is it okay to ignore one of them?  Probably not.
  • Can any of this learning be applied to TOEFL Speaking question four?  Probably not.
  • Does order matter?  Probably not.

 

 

 

This time I tried to use the SpeechRater implementation at My Speaking Score to produce the highest scoring “short” answer I could come up with.  Once again, my answer is just 74 words in total.  You can hear it below:

 

Here’s a transcript:

Clearly, it’s advantageous to study various subjects while we are at university. This is because it makes it possible to find our true passion. For instance, when I was a freshman, I took courses in chemistry, psychology and history. While I was enamored with all of them, eventually I discovered that I was most fascinated by chemistry. As a result, I ended up majoring in that. Moreover, I gained interdisciplinary insights throughout my journey.

This is almost the same as the last answer.  I’ve improved the vocabulary a little by using slightly less common words.  I added an -ly adverb to the beginning.

The main difference is that I delivered this in a natural speaking voice at a normal pace.  That means I finished after only 30 seconds.  As you will recall, last time I slowed it down to a unnatural pace and finished at 45 seconds.

The result… a score of 3.46, or 27/30.  Wow.  This isn’t a fluke.  I recorded a similar (but slightly different) answer of 80 words and got a score of 3.6.

Fluency

The fluency markers are all fantastic:

But how can that be?  I scored in the 87th percentile for response length, but scored in the 14th percentile for response rate last time… with the same number of words.  My guess is that the SpeechRater compares the number of words in the answer only to responses of the same length.  Do you get what I mean?  My answer was compared only to other 30 second answers and I included more words than most of them.   Again, I want to stress that this isn’t a fluke.  My 80-word answer got a similar response length score.

Perhaps we as teachers we should stress pace rather than word count.

Pronunciation and Vocabulary

Moving on, here are the pronunciation and vocabulary scores:

The pronunciation is better this time, but it still didn’t really like my  rhythm.  That’s a flukier score, I think.  My 80-word answer was in the 98th percentile.  I think the SpeechRater isn’t perfect when it comes to measuring that dimension.  Perhaps I will make 20 unique recordings of the same answer one day and track the scores.

The vocabulary scores are okay.  Perhaps my low word count limited my ability to use a lot of uncommon words in the answer. That makes sense.

Grammar and Coherence

My grammar and coherence scores are as follows

Final Words

So there you have it.  It is possible to get a good SpeechRater score on an independent response with a “short” answer.  This does track with what ETS says about how the human raters check answers.  ETS speakers have said again and again that it is not necessary to speak quickly on the test.  I don’t know if this can be applied to the integrated speaking tasks, but I’ll check my notes about how many actual details need to be included in the answer. At the last presentation I attended it was implied that only half of the details from the lecture of a type three speaking question need to be included.  Maybe thirty seconds is enough.

If any of my faithful readers want to sponsor a few test attempts I’ll march down to the test center and run these experiments in an actual testing environment.

Students often ask me how important it is to speak quickly in the TOEFL speaking section.  Keen students even ask how many words they should include.

I’ve always said that speaking rate is really important.  I’ve urged students to practice speaking quickly, as long as that doesn’t mess with their pronunciation or intonation.  But that’s always just been my gut feeling, stated without solid evidence. In an effort to gather some real data on the issue, I submitted a few of my own practice answers to SpeechRater, the automated scoring software used (together with human raters) by ETS to grade the TOEFL.  I was able to do this by uploading my answers to My Speaking Score, which has licensed SpeechRater.  I encourage both teachers and students to make use of that site.  It is comprehensive and fairly affordable.  A monthly subscription gives you a bunch of credits to upload answers (or if you prefer you can just record them in your browser).  Teacher and student accounts can be linked to facilitate reviews and personal feedback.

I must mention a few disclaimers before I get into the data:

  • The real test uses both a human rater and the SpeechRater.  That means we cannot use SpeechRater alone to completely predict how a given answer would score.
  • While My Speaking Score is meant to be as close to the real TOEFL as possible, it is a third party implementation, so it cannot be perfect.
  • This is a third party blog, not associated with ETS. My interpretation of the numbers below might be totally wrong.

My Sample Answer

Now that I’ve got that out of the way, you should listen to my sample answer:

 

Here’s a transcript:

I think it is much better to study various subjects while we are at university. This is because it helps us to find our true passion. For example, when I was a freshman, I took courses in chemistry, psychology and history. While I loved all of them, eventually I discovered that I was most interested in chemistry. As a result, I ended up majoring in that. Moreover, I gained interdisciplinary insights along the way.

My speaking rate is very low.  It is just 74 words (per the word counter in Google Docs).  I generally recommend about 120 words if students want a perfect score, and about 100 words if they want an “average” score.

However, everything else is pretty good.  My pronunciation and intonation are at a native level.  I’ve included a few fancy words like “passion” and “interdisciplinary” and “insights”.  I’ve also used transitions like “moreover” and “as a result.”  I even included a few conjunctions like “while” and “when.”  There aren’t any “umm” breaks, self-corrections or stutters. 

When I asked experienced TOEFL tutors how this answer would score, I got responses ranging from 24 points to a perfect 30 points.  As you can see, a few of the tutors really liked it!

Meanwhile, SpeechRater gave this one a score of 2.92/4.  That converts to 23 points out of 30.  Good… but not great.

What did SpeechRater Think about my Fluency?

First up, here’s the score for speaking rate.

Not surprisingly, the answer is all the way down in the 14th percentile.  That means I spoke slowly.  There is a penalty for that.  But speaking rate is just one metric.  It can’t account for all the entire seven-point penalty from the SpeechRater.  

Here’s a look at the rest of the fluency metrics:

They aren’t good either.  As you can see, the SpeechRater gave me a poor score for “sustained speech.”  It identified a bunch of disfluencies and I ended up in the 27th percentile.  In this case the disfluencies are silent pauses, but in other answers they might include “uhh” breaks.  It also gave me a fairly poor score for metrics specifically related to pauses, as you can see.  Slowness and pauses usually go hand in hand, as you might expect. 

What did SpeechRater Think of my Pronunciation?

My pronunciation scores were a mixed bag:

Despite my slowness, my rhythm was pretty good.  However, my pronunciation of vowels was merely average.  But how can that be?  I’m a native speaker.  Well, another source of slowness is the way I sometimes draw out my vowel sounds. Notice my pronunciation of “and history” and “all of them” and “most interested”.  The awkwardness is subtle, but noticeable if you are listening for it.  The penalty for doing this is likely small, but I think it added up since I did it multiple times in every sentence.

What did SpeechRater Think of my Vocabulary?

Vocabulary was another mixed bag:

As I mentioned above, I think my answer has a few good words in it.  However, I’m stuck in the 25th percentile for vocabulary depth.  And even though I didn’t really repeat words, my vocabulary diversity score is merely average.  Why?  Well, my guess is that since my total word count is quite low, it is almost impossible for me to include a lot of “uncommon” words.  I mentioned three words I suppose are “uncommon,” but that’s not really enough.  In a more quickly delivered answer I might have had time for seven or eight words, and earned a better score in that domain.  Likewise, a more quickly delivered would have almost automatically included a more diverse vocabulary… and earned a higher score.

A future experiment might involve jamming as many fancy words into an answer of the same length in an attempt to produce the best possible 75 word answer.

What did SpeechRater think of my Grammar?

Ooof.  My grammar score is not good:

I’m all the way down in the 9th percentile.  Again, this is despite the fact that my grammar is flawless.  Again, I think that the brevity of my answer means that I didn’t have the opportunity to use any advanced grammatical structures.  I have a couple of subordinating conjunctions, but that’s about it.  I don’t have any coordinating conjunctions, I don’t have any adverbs and I’m short on adjectives.  There are no conditionals in the answer, either.  Most of the answer is in the past tense. Some people might be able to fit a lot of grammatical conventions into just 75 words, but it isn’t easy. I think my limited use of grammar is common in answers that are delivered slowly.

What did SpeechRater think of my Coherence?

SpeechRater didn’t like my coherence either:

Again, my impression is that my answer was too short to include enough connective devices to please SpeechRater.  There are three obvious transitional phrases in my answer… but an answer with more words overall would naturally have more than that. Likewise, it would probably have a few compound sentences (my answer has none).

Final Words

The point I’m trying to make here is an obvious one, but it is important.  A TOEFL response delivered slowly may draw a low score from the SpeechRater.  In addition to being short overall, it will likely be missing some of the key features the SpeechRater wants to see.  Be careful on test day. 

As I indicated above, a future experiment will be to create the best possible 74 word answer, to see the best-case result for a slow answer.

Hey, it’s my sixth favorite day of the year – the day of the new TOEFL bulletin!  You can download it over here. It will take effect next month.  The new layout makes it difficult to identify changes, but I spotted a few things of note:

  • The “returned payment” fee has increased from $20 to $30.  I honestly don’t know what this fee actually refers to.
  • The refund policy for test takers in Korea has become a bit worse.  Those test takers now must cancel four days before the test to get a 50% refund.  It used to be three.
  • New: Electronic ID may not be used.
  • New: ID must contain your date of birth (and it must match the date of birth used when registering for the test)
  • As I mentioned a few days ago on this blog, sales taxes are now collected for registrations in Vietnam and Singapore (among other countries)

Also, this bulletin excises what seems to be the final official reference to “the ETS.”  Long live “ETS.”