A new article in British newspaper i is critical of the Duolingo English Test.  It reports that:

“Professor John Heathershaw from the University of Exeter linked the acceptance of ‘things like Duolingo tests’ to lower English language standards and described it as a ‘major issue’ when he appeared before the Foreign Affairs Select Committee last week.”

And:

“Aston University in Birmingham is one of the many UK institutions that adopted Duolingo at the height of the pandemic but has now dropped it. A spokesperson told i that the decision was taken because of concerns about student performance.”

And quoting a spokesman from that university:

“There is no evidence that those students on Duolingo were failing or, indeed, fraudulent, but just not performing at the same level as peers on their course, so we have chosen to remove acceptance for the test.”

I’m going to babble a bit now.  Pardon me.

Older tests like the TOEFL and IELTS are designed around a particular conception of validity. For instance, the makers of the TOEFL would likely argue that we can look at a TOEFL score and make inferences about the test-taker’s ability to do well in university classes because the tasks on the TOEFL resemble (and, more importantly, involve the same knowledge, skills and processes) as the tasks done by students at post-secondary institutions.

Similarly, IDP would tell you that the IELTS General Training Module is a great test for immigration because it includes tasks that resemble what we do in everyday life.  Meanwhile, the TOEIC leans heavily on memos and invoices and emails that people encounter in an office environment.

You can read about this concept of validity in Carol Chapelle’s 2008 doorstopper about the creation of TOEFL iBT.

The Duolingo English Test is a little bit different.  It is certainly a test of one’s English abilities.  That much is obvious.   But is it appropriate for university admissions? While questions of an academic nature have been added to the DET in recent months much of the test score is still determined by “describe this picture” tasks and “fill in the missing letters” tasks that don’t closely resemble things done on a university campus.

Duolingo might argue that such tasks totally suit the purpose of the test and that they really do require the relevant skills and knowledge.  They could be right.  Who knows?

What interests me is that if we reject the idea that validity requires that test items closely resemble tasks performed in real-world contexts we can go ahead and discard all of the older tests and use the Duolingo Test for all possible purposes.  Will the receiving institutions bite?  That remains to be seen.

I’ve spent some time this week trying to help students from Iran affected by score cancellations salvage their academic futures.  To be honest, I haven’t been able to help much.

Here’s something that bugs me: 

A student’s scores were cancelled.  Like many others, she was told:

“As a result of ETS’s rigorous score validation process, we have identified several factors that substantiate this score cancellation, in part because we detected unauthorized recording devices/software that were open during the test session which is a violation of ETS policy”

This could mean anything.  Maybe she set out to cheat and was caught.  Or maybe she forgot to turn off Skype after talking to her grandma.  Who knows?  ETS won’t provide any more information.  But, hey, both are valid reasons to cancel the scores, I suppose.

In any case, she suffered greater consequences than a cancelled score. A very fine university withdrew its offer of admission to grad school and  informed her that (due to the seriousness of the offence) she may never apply again.

I really hope that this institution was given all of the information that was needed to make such a life-changing decision.  I really hope that it knows that sometimes (often?) this particular rule is violated entirely by accident.  To react in such a way they darn well better have been told that the student set out to cheat.  And if this decision was made in error, perhaps the verbiage used to inform them of the violation needs to be adjusted.  Words have meaning and weight, damn it.

In the ancient past, test-takers could call ETS customer service and ask for a “test taker advocate.” I don’t know if they were effective advocates, but the position did exist.  It might be time to bring that position back.  Or to create some sort of ombuds position.  As it stands, students with problems like this often appeal to me for advocacy, and I’m feeling pretty helpless right about now.  

Changes are coming to the GRE subject tests. They are:

  1. After the April 2023 administration, the Chemistry test will be discontinued.
  2. The Physics and Psychology tests will be shortened to two hours.
  3. After April 2023, the Mathematics, Physics and Psychology tests will be offered on computer at test centers, and via at-home testing.

Read about these changes on the ETS website.

The subject tests have really  been whittled down to just the basics.  The Biochem test was eliminated in 2017, while the Biology and Literature tests were eliminated in 2021.

Students often mix up “so” and “so that” in their essays.  Here’s what you should know.

So

“So” is used to indicate an effect or consequence of something.  Like:

I studied hard, so I got a high score.

The effect of studying hard is that I got a high score.

There was a drought, so the people moved west.

The effect of the drought was that people moved west.

I’m tired, so I’m going to bed.

The effect of being tired is that I’m going to bed.

So that

“So that” is used to indicate the purpose or reason for doing something.  Like:

I turned up the heat so that we will feel more comfortable.

My reason for turning up the heat was to make us feel more comfortable.

People moved west so that they could grow crops.

The reason people moved west was to grow crops.

I should go to bed early so that I have plenty of energy tomorrow.

My reason for going to bed early is that I want to have more energy tomorrow.

Some old-fashioned writing uses “in order that” in a similar way.  I don’t recommend using that form in contemporary academic writing.

It was just announced that the GMAT test will be made shorter. Currently, the exam takes 3 hours and 7 minutes to complete. The new version will take just 2 hours and 25 minutes to complete. The exam’s analytical writing assessment will be eliminated. Test-takers will be allowed to “bookmark” a number of questions and return to them later. Reports indicate that the Graduate Management Admission Council hopes to recover market share it has lost in recent years to ETS and its GRE test.

Note that the test was previously shortened (by 30 minutes) back in 2018. It seems like the standardized testing industry is strongly embracing the idea of shorter tests.

The report from Poets and Quants paints a pretty bleak picture of the GMAT’s popularity in their report on the changes. They note that 260,000 people took the GMAT in 2016, while only 156,000 took the test in 2021. Some people I’ve talked to have partially attributed this decline to an online testing experience they call subpar.

Anyway, social media sleuths have uncovered a few additional details through sneaky methods. They suggest:

  1. Sentence correction questions will be eliminated.
  2. More integrated reasoning questions.
  3. No geometry?

Consider these three changes totally unofficial.

Our good friends at EdAgree have 100 TOEFL discount codes to give away before the end of the month.  Update: sorry… the codes have all been claimed.  But stay tuned to the blog for updates.

I believe requests are manually processed, so you might have to wait a day or two to get your code.   You can use the code to register for any test date… but you need to complete the registration before March 31.  The codes can be used for registrations all over the world.

Remember that only 100 codes are available and they are first come first served.  If you want a code you should request one right away.

It is impossible to get a definitive count, but it seems like a significant number of GRE and TOEFL scores from Iran were canceled last week. The scores are from tests taken between November and February. A total of 413 people have signed a petition  regarding the situation, while a Telegram channel serving as a hub for updates now numbers 105 members.

I spoke to an individual with canceled scores (from December) who has already began studies in Canada. Not surprisingly, he is nervous about what happens next.

It seems that in each case, test-takers were notified that the cancellation was due to unauthorized software running during their test administration.

Affected test-takers have reached out to media outlets covering higher education (not me) and to parties in the USA they think will advocate on their behalf (not me). One imagines this story will be reported on in the near future.

Reports are starting to come in that a significant number of TOEFL and GRE scores from at-home tests taken in Iran were canceled last week. I woke up Friday to a handful of emails about the situation. A trusted contact in Iran I spoke to yesterday said there have been “more than thirty” cancellations. Others have mentioned larger numbers.

One of the individuals who emailed me said her scores were successfully reported back in January only to be canceled last week due to unauthorized software being detected during her test. On social media, people have talked about scores from five months ago being canceled.

If any journalists on the higher education beat are reading this, I hope you’ll do a little digging. It looks like a story worth telling. I know it’s difficult to report on Iran in 2023, but I can point you in the right direction if necessary.

I spoke to a fellow last week who had experienced difficulty with an at-home test.  He checked in with the proctor, showed his ID, did a room scan, showed his screen with a mirror… but immediately after the test commenced it was terminated because unauthorized software was detected running on his computer.

The test-provider wouldn’t tell him what software had been detected, and they wouldn’t refund his $215 registration fee.

But, hey, this guy was determined to take the test. He registered to try again the following week.  Sadly, the exact same thing happened.  He found himself down $430.

Picture this: he doesn’t have a score, his deadlines are fast approaching, and he still has no idea what software caused the termination.  No refund has been offered.

Needless to say, the guy decided to take a different test.

For the record, the average monthly wage in his country is $556.

I hear about this sort of thing quite frequently (though most people switch to a new test after the first termination).

I don’t think this guy really set out to cheat.  Some background process was running in Windows, but this dude has no idea what a background process even is.

I recently read that the “Skills for English” test has created a proctoring position called a “Check-in Specialist.”  This individual is responsible for examining the testing environment and checking the test-taker’s ID before the test begins. After these things are done, the rest of the proctoring is handed off to someone else.  I love it.  

Perhaps all testing organizations should consider creating such a position. In addition to the above, the specialist could be trained to detect unauthorized software and quickly shut it down before the test begins.  I mean… the proctors have the ability to remotely control the test-taker’s machine, so why not?

It’s important for the people who design online proctoring services to put themselves in the shoes of the young people taking their tests.  I’ve learned that a lot of young people today don’t really know about the finer details of desktop operating systems.  Many of them don’t know what the Windows Task Manager is.  Some don’t even know what the System Tray is. In 2023 everyone lives and dies by their smartphones, so that level of knowledge is no longer necessary. They think that programs stop running when they exit them.  So… stuff is left running when they take the test. A trained proctor could step in and compensate for  the test-taker’s lack of expertise.

 

Today I learned that the Miller Analogies Test (MAT) will be retired by Pearson in November of this year. The MAT is a graduate admissions test developed by Dr. W. S. Miller of the University of Minnesota in 1926 (decades before the creation of the GRE and GMAT tests). For many years it was offered by The Psychological Corporation, and later by Harcourt and Pearson.

Though obscure, the MAT is still accepted by some schools.

As the name suggests, the test mostly consists of analogies. It also contains a little bit of math, curiously.

While I was in Canada last year, I found a copy of Arco’s “TOEFL” from 1974/75.  This is the oldest TOEFL book I have ever had in my hands.  It was written by Harriet N. Moreno, Edith H. Babin and Carol V Scallon.  Interestingly, the audio files came on vinyl records (which I could not get).

I can’t resist using this opportunity to write a few words about Arco.

Though it is largely forgotten nowadays, Arco was one of the very first publishers of test preparation books in the United States.  Founded in 1937 (two years before Barron’s) by Milton Gladstone, the company eventually branched out into fiction and cheap general reference books of all sorts.  Notably, the publisher attracted the attention of the United States government due to the publication of its “Arco Sophisticates” line of paperback erotica (many written by Jack Woodford). Gladstone was subpoenaed and spoke before the US House of Representatives in 1952.   

In 1978, the publisher was acquired by Simon and Schuster.  Thanks to its long association with test prep books, the “Arco” brand name was valuable enough to be passed through various hands including those of Pearson , IDG, Thomson, Cengage and Peterson’s.  I think Peterson’s still owns the name, but I suspect no one over there even knows that.  The last book to bear the Arco name was published in 2010, as far as I can tell.  It was a guide to the Federal Clerical Exam.  I think Peterson’s retired the brand after that.

Anyhow… the gallery below contains a few pictures of the TOEFL book, including the cover, back cover, preface and “how to be a master test taker” guide.  I took a few more pictures that I won’t share here.  Let me know if you need ’em for some reason.  They include some actual questions.

Rather than

Use “rather than” to state preferences and choices.  Such as:

“He wanted to be a doctor rather than a teacher”

“I prefer to eat pizza rather than salad.”

“I decided to write rather than phone”

“In the end, we decided to go to Toronto on Friday rather than on Sunday”

“He’s probably just lazy, rather than stupid.”

Note how each sentence states a choice or preference.  There is not just a a comparison.

Than

Use “than” to compare two things without making a choice.  As in:

“Doctors make more money than teachers.”

“Waking up early is more beneficial than staying in bed late.”

“Writing is slower than calling.”

“Cats are smaller than dogs”

“Attending university at home is cheaper than going to another country.”

Note how every sentence makes a comparison, but no preference of choice is made.

Pearson just published its annual report. The PTE (Pearson Test of English) was taken 827,000 times in 2022. That’s up from 436,000 times in 2021. I nearly fell out of my chair.

Here are the historic volumes, culled from past reports:

  • 2022: 827,000
  • 2021: 436,000
  • 2020: 350,000
  • 2019: 547,000
  • 2018: 467,000
  • 2017: 359,000

(the last four are estimates based on percentage changes quoted in the annual reports)

It’s worth repeating once again that if a test-maker provides a pleasant experience for test-takers their product will take hold. Even if the test is two hours long, and even if it has a high price tag.  Not every test has to be the Duolingo English Test… 

Think about it!