Today I want to write a few words about an interesting new (December, 2019) text from ETS.  “Automated Speaking Assessment” is the first book-length study of SpeechRater, which is the organization’s automated speaking assessment technology.  That makes it an extremely valuable resource for those of us who are interested in the TOEFL and how our students are assessed.  There is little in here that will make someone a better TOEFL teacher, but many readers will appreciate how it demystifies the changes to the TOEFL speaking section that were implemented in August of 2019 (that is, when the SpeechRater was put into use on the test).

I highly recommend that TOEFL teachers dive into chapter five of the book, which discusses the scoring models used in the development of SpeechRater.  Check out chapter four as well, which discusses how recorded input from students is converted into something that can actually be graded.

Chapters six, seven and eight will be the most useful for teachers.  These discuss, in turn:  features measuring fluency and pronunciation, features measuring vocabulary and grammar, and features measuring content and discourse coherence.  Experienced teachers will recognize that these three categories are quite similar to the published scoring rubrics for the TOEFL speaking section. 

In chapter six readers will learn about how the SpeechRater measures the fluency of a student by counting silences and disfluencies.  They will also learn about how it handles speed, chunking and self-corrections.  These are actually things that could influence how they prepare students for this section of the test, though I suspect that most teachers don’t need a book to tell them that silences in the middle of an answer are a bad idea.  There is also a detailed depiction of how the technology judges pronunciation, though that section was a bit to academic for me to grasp.

Chapter seven discusses grammar and vocabulary features that SpeechRater checks for.  Impressively, it just sticks them in a list.  A diligent teacher might create a sort of check list to provide to students. Finally, chapter eight discusses how the software assesses topic development in student answers.

Sadly, this book was finished just before ETS started using automated speaking scoring on high-stakes assessment.  Chapter nine discusses how the technology is used to grade TOEFL practice tests (low-stakes testing), but nothing is mentioned about its use on the actual TOEFL.  I would really love to hear more about that, particularly its ongoing relationship with the human raters who grade the same responses.

I’ve been meaning to write a review of Magoosh TOEFL for ages.  Magoosh has been around forever and is still really popular with students.  But is a good source of information?  Well, I recommend it for reading and listening practice.  It is a poor resource for writing and speaking practice.

Let’s take a closer look…

The Lessons

The lessons aren’t flashy, but they are functional.  They are mostly just the Magoosh teacher talking over a whiteboard style presentation.  Thankfully transcripts are provided, and students can speed up the videos if they find them too slow.  The lessons have all been updated for the 2019 version of the TOEFL, except for a few minor instances which the site’s editors missed.

I like the reading lessons provided by Magoosh quite a lot.  The site provides 19 video lessons, totaling about three hours of video.  Each question type is described in detail with a sample of each one.  Common “answer traps” for each type are described in separate videos.  There aren’t a whole lot of “strategies” other than pacing techniques, but I think that’s probably a fine approach.  As I’ve talked about in my own content, using strategies is probably a bad idea for this section of the test.

The listening lessons are also quite strong.  Honestly, there aren’t any good TOEFL listening lessons online, so I’d enthusiastically recommend this content.  These are presented just like the reading lessons – videos for each section question type and for common traps.  There is more content on basic strategy here, which is probably more appropriate for the listening section.

The speaking lessons are decent.  Speaking question one is described in accurate terms, which is something that most books and websites don’t do.  Magoosh makes sure to mention the “paired choice” style prompt that is mostly ignored by other sources.  It is quite disappointing that the lessons for questions 2 to 4 use the old TOEFL “Quick Prep” sets from ETS, since students can get those for free on their own, but at least it makes the material in the lessons more accurate that most sites.  Heck… that’s what I do for many of my videos.  The templates and timing suggestions for each question (provided in separate videos) are very good.  Sadly, this is all paired with weak practice questions (see below).

The writing lessons are a bit weaker.  There are a few inaccuracies in the videos.  Early in his description of the integrated writing task, the teacher suggests that the details in the reading might be in a different order than the details in the lecture.  That’s not true.  Likewise, the lesson on the independent writing task leaves out the multiple-choice style prompts.  That’s an unfortunate over site. The lessons about constructing the essay, though, are fine.  They will lead to the creation of effective essays.  These are also paired with bad practice questions (see below).

Finally, there are a bunch of grammar videos.  I checked a few of them out and they seem fine.  Honestly, though, there are better places to study grammar.  That’s not why anyone is buying a Magoosh membership.

The Practice

Well, the writing practice is bad.  Of the seven integrated writing questions provided, only three of them are accurate (Globalization, Rococo, Trade).  The rest are badly created with either too many or too few paragraphs, points that don’t match up, or faulty ordering of points.  Sadly the very first practice question (Gone with the Wind) is especially bad.  I wish that one could be moved to the end of the practice section so fewer students would see it.  The independent writing questions are all accurate, but no multiple-choice prompts are provided.  That makes the material seem a bit dated.

The speaking practice is weak.  The practice questions for task one are all accurate… but they don’t include any of the more modern “good idea” style questions.    The sample questions for task two don’t always follow the same structure used by ETS when they create questions.  They seem to emphasize the giving of details about the changes being announced rather than two two reasons for the changes on the real test.  Moreover,  the students in the Magoosh conversations sometimes address details not mentioned in the reading part, which is unlike the real test.  For what it’s worth, I recommend two of the practice questions (vegetarian meals, college radio).  You can skip the rest.  The task three and four practice questions are fine.  I feel that the prompts are a bit too specific and verbose (the real test is more likely to ask the students to more broadly define the term or concept using the examples) the construction of the questions is acceptable here.  Students might not even notice the difference when they take the real test (which is easier).

I like the reading practice a lot.  The passages and questions all look accurate.  The authors of the passages seem to really understand how the test is put together, and they avoid all of the problems that most textbook authors make.  The tests have been updated for the new TOEFL, and the number of questions for each article has been reduced to ten.  I didn’t count the question type distribution, but it seems accurate.  I would wholeheartedly recommend this section of Magoosh.

The listening practice looks good and updated.  I would also recommend it.  It appears quite accurate.  I would also recommend it.  I love that the questions here all include detailed explanations instead of just an answer key.  A lot of work went into those. 

Verdict

If you can afford it, buy this only for listening and reading practice.  There is a ton of good content here to help you prepare for those parts of the test.  Use the speaking and writing content sparingly, and make sure to supplement with some more accurate practice questions (the Official Guide and iBT Tests books, for instance).

 

I was able to sit in on an ETS webinar about the At-Home Edition of the TOEFL.  A few additional details were provided.  They are:

  1.  The test-taker will be recorded for the whole test. 
  2.  Additional countries will be probably be announced later this week.  Update:  the test is available all over the world, except Iran and China.
  3.  Paper score reports are not being sent at this time.
  4.  The score report of the At-Home TOEFL will list “STNRP” followed by the ISO code of the student’s country in the “test center” section.  It will probably also contain the words “Special Home Edition” somewhere.
  5. Notably, the Home Edition of the GRE will not contain any information indicating how the test was taken.
  6. A built-in camera is okay, as long as the computer can be lifted up and rotated to give a view of the whole room.
  7. Food and drink is not allowed.
  8. Religious headgear is okay, but ears must be visible at all times.
  9. AI facial recognition and gaze tracking technology will also be used to prevent cheating.
  10. AI facial recognition is used throughout the test to ensure that the proper person is taking the test.
  11. If ETS discovers that a student is cheating or has broken a rule, the scores will be cancelled.  The student may be banned from taking the test in the future (and other ETS tests).
  12. If the student loses their internet connection during the test the proctor will stop the test and a free retest will be offered (just note that this was TOLD to me, and is not written anywhere as an official policy, so don’t consider it a guarantee).
  13. The free TPO mentioned by ETS is only available in China, and will not be offered to students outside of China.