Gary J Ockey and Evgeny Chukharev-Hudilainen published an interesting article in Applied Linguistics that suggests a few interesting things. It highlights Ockey’s earlier research that suggests that the asynchronous tasks used in the TOEFL iBT speaking section “may not sufficiently assess interactional competence.”

More importantly, it compares the use of a human interviewer (a la IELTS) to a Speech Delivery System (like a chatbot) to elicit spoken English from test-takers. It seems to suggest that “the computer partner condition was found to be more dependable than the human partner condition for assessing interactional competence.” And that both were equal in areas like pronunciation, grammar and vocabulary.

Aha! This information could be used to create a better TOEFL test or a better IELTS test. Someone should let the test makers know.

No need, though, as I read at the end that “this research was funded by the ETS under a Committee of Examiners and the Test of English as a Foreign Language research grant.”

Implement it right away, I say.

I mention this now because the research will be presented tomorrow at an event hosted by the University of Melbourne.

Over the last couple of days I have been playing with ChatGTP to create TOEFL Integrated Writing questions.  I’ve had some success.  My creations aren’t perfect, but in only 30 minutes I can easily put together something that is better than what most major American publishers put in their best-selling books.  That’s remarkable.

Can I do the same with the speaking section?  Yeah, I can. 

Today I will share a couple of AI-generated TOEFL speaking questions.  These are both “type 3” questions, which include a short reading and a brief lecture on the same topic.   In both cases I probably spent about 30 minutes revising them to be more “TOEFL-like”.

Note that eventually I will stick these questions onto pages that students can more easily use to practice for the test.

First up, I generated one about a unique animal feature, which is a fairly common topic on this part of the TOEFL test.

Here’s the reading:

Transparency in Animals

Transparency is the quality of being able to see through an object. While transparency is commonly associated with glass and other transparent materials, it is also found in a number of different animal species. Transparency in animals is typically achieved through the use of specialized cells, tissues, or structures that allow light to pass through their body. This can provide a number of benefits to the animal, such as improved camouflage, enhanced communication, and reduced drag while swimming. Studying the mechanisms of transparency in animals could potentially lead to the development of new materials and technologies that are inspired by the natural world.

And here is me reading the lecture:

 

And here is a transcript of the lecture:

Okay, so I’ve got an example of transparency in the wild. The glass squid is a type of deep-sea squid that is known for its transparent body and long, thin tentacles. Glass squids are found in the deep waters of the ocean… ah… I’d say…ah… they are typically found at depths of 1000 meters or more.

The transparency of the glass squid’s body is thought to be a form of deep-sea camouflage, as it allows the squid to blend in with the surrounding water and avoid being detected by predators. The transparency of the glass squid’s body also helps it to avoid being seen by its prey, allowing it to sneak up on unsuspecting fish that it wants to eat. The ability to use transparency in both of these ways is thought to be extremely important for the survival and success of the glass squid in its challenging deep-sea habitat.

Now, in addition to using transparency for camouflage, the glass squid also uses its transparent body for communication. This is interesting. It has light-emitting organs, called photophores, that are located inside of its body and tentacles. The glass squid uses its photophores to flash patterns of light, which it uses to communicate with other glass squids. This allows it to signal its presence to other members of its species and it also allows it to coordinate its movements with other squids in its group.

Okay, so sometimes the lecture in the type three question relates a sort of anecdote from the speaker’s life.  Can the AI produce one of those?  Yes.

Here is a reading:

Regret Aversion

Regret aversion is a psychological phenomenon in which people are more likely to avoid taking risks or making decisions that may result in regret. This is because people tend to experience negative emotions, such as regret or disappointment, more strongly than positive emotions, such as happiness or satisfaction. As a result, people may avoid taking risks or making decisions that may result in regret, even if those risks or decisions could potentially lead to better outcomes. Regret aversion can affect people’s decision-making in a variety of contexts, including financial decisions, personal relationships, and career choices.

And here is me reading the lecture:

 

And a transcript of the lecture:

Okay, so, I have a perfect example of regret aversion. I had this friend. Alex. Now, Alex was always very cautious when it came to making decisions. He was constantly worried about making the wrong choice and regretting it later on. This tendency became especially pronounced whenever he was faced with a difficult decision.

One time, Alex was considering whether to quit his job and start his own business. He had been working at the same company for several years, but he had always dreamed of being his own boss. The idea of starting his own business was exciting, but it was also risky. If the business failed, Alex could lose a lot of money and damage his reputation. At first, Alex was hesitant to follow through on his plans. He was afraid of the potential consequences if it failed. He was worried that if that happened, he would regret his decision and be disappointed in himself. He was also concerned that he would have to go back to working for someone else, which he didn’t wanna do.

As a result of his regret aversion, Alex decided not to quit his job and start his own business. He continued working at the same company, even though he wasn’t happy there. He missed out on the opportunity to pursue his dream of being his own boss, and he continued to feel unfulfilled and unhappy. Much later, Alex realized that his regret aversion had held him back. He had been so afraid of regretting his decision that he had avoided making a decision altogether. He had missed out on a potentially rewarding opportunity because of his fear of regret.

Okay, everyone is probably sick of reading these… but I’ve got a couple more writing questions I made with ChatGPT.  I’ll post them both here, and then start dumping my creations on YouTube or something.  But I will use the blog to share whatever I can come up with for the other section.  Sorry!

Here’s a reading about the colonization of asteroids:

The idea of colonizing asteroids has long been a topic of fascination and speculation in science fiction and popular culture. In recent years, however, the idea of asteroid colonization has become more realistic and feasible, thanks to advances in technology and space exploration. There are many potential benefits to colonizing asteroids, and these benefits make the pursuit of asteroid colonization a worthwhile endeavor.

One of the most obvious benefits of asteroid colonization is the scientific potential. Asteroids provide a unique environment for scientific research due to their small size, low gravity, and lack of atmosphere. Research conducted in these environments could provide valuable insights related to many different academic fields. Not only that, but asteroids could also serve as stepping stones for future missions to other destinations in the solar system, such as Mars or the moons of Jupiter and Saturn.

Another benefit of asteroid colonization is the potential for economic growth and development. Natural resources mined on asteroids could create new industries and job opportunities, which would generate significant revenue for both governments and private companies. Asteroid mining could also reduce the need for resource extraction on Earth, which could help to preserve our planet’s natural environment.

In addition to economic benefits, asteroid colonization could also have important implications for the long-term survival of humanity. Asteroids could serve as potential refuge for humans in the event of a large-scale disaster on Earth, such as an asteroid impact or a nuclear war. Even if such a disaster doesn’t occur, asteroids could provide valuable information regarding the origins and evolution of the solar system, and how life emerged here in the first place.

 

Here is me reading the corresponding lecture:

 

And here is a transcript of the lecture:

While some people argue that starting colonies on asteroids could have many benefits, this claim is not necessarily supported by evidence. In fact, there are several reasons why starting colonies on asteroids may not provide the advantages that are often claimed.

One reason why asteroid colonization may not provide significant scientific benefits is our lack of knowledge about asteroids. Despite decades of space exploration and study, our knowledge of asteroids is still limited. We don’t know how many asteroids exist, what resources they contain, or what conditions are like on their surfaces. Until we have a better understanding of asteroids, it’s very difficult to predict what scientific benefits colonization might provide.

Next, we can’t exactly predict the economic benefits of settling on asteroids. While asteroids are known to contain valuable resources, such as metals and minerals, it’s not clear how much of these resources exist on asteroids, or how much they would be worth when transported back to Earth. It’s not clear whether mining for resources in space would be more cost-effective or efficient than simply getting them through conventional methods here on earth.

Finally, asteroids are just not suitable for long-term human settlement right now. Most asteroids are small and do not have enough mass to generate significant gravitational pull. This means that any human settlements on asteroids would need to be designed to cope with the challenges of low gravity, such as serious difficulty moving around and possible damage to our bones and muscles. These challenges could make living on asteroids extremely uncomfortable and dangerous in the long run.

Our good friends at EdAgree will host a free Virtual University Fair on December 13 and 15.  Participants will be able to interact with representatives from a slew of universities in the USA, and get prep tips for English proficiency exams.  My presentation from EdAgree’s Test Prep workshop (last week) will be included, and I might send along a new presentation if time permits.

Participants can enter a contest to win a free TOEFL test registration.  Additionally, all participants will receive a coupon code for a discount on their next TOEFL registration.  Click to register.

Well, I’m a little bit obsessed now.  I generated another TOEFL-like integrated writing question using ChatGPT.  I spent about half an hour on this one, generating the passages and revising them to be more TOEFL-like.

(here is my original blog post about this process)

As before, here is the required disclaimer:

“The author generated this text in part with GPT-3, OpenAI’s large-scale language-generation model. Upon generating draft language, the author reviewed, edited, and revised the language to their own liking and takes ultimate responsibility for the content of this publication.”

Here is the reading:

Machu Picchu is a 15th-century Inca citadel situated in the Andean Mountains of Peru. This ancient site has puzzled historians and archaeologists for centuries, with many theories proposed as to its purpose. Archaeologists have put forth three possible reasons for the construction of Machu Picchu.

One possibility is that Machu Picchu was built as a permanent royal estate for the Inca emperor, Pachacuti. As the ruler of a major empire, Pachacuti would have needed a luxurious and impressive residence. Located at the top of a mountain, with breathtaking views of the surrounding landscape, Machu Picchu would have been an ideal location for such a residence. Anyone coming to meet with the emperor would have been impressed by his apparent power and wealth.

Another theory is that Machu Picchu was built as a ceremonial and religious center. The Inca were a deeply religious people, and their empire was filled with temples and shrines dedicated to their gods and goddesses. Machu Picchu, with its impressive stonework and intricate carvings, may have been built as a sacred site for religious ceremonies and rituals. 

A third possibility is that Machu Picchu was built as a defensive fortress. Located at the top of a mountain and surrounded by steep cliffs, the site would have been very difficult to access, but easy to defend. This strategic location, combined with the strong walls and terraces of the citadel, would have made Machu Picchu a perfect military stronghold. Invading armies would have found it a very challenging target.

Here is me badly reading the lecture:

 

Here is a prompt:

Summarize the points made in the lecture, being sure to explain how they oppose specific points made in the reading passage.

Here is a transcript of the lecture:

The author’s ideas about the purpose of Machu Picchu  are certainly interesting, but each of them has a few flaws.

A problem with the idea of Machu Picchu as a royal residence is its remoteness. While the site is located in a beautiful area, it’s far from the center of the Inca empire and would have been difficult to access. This would have made it impractical as a residence for the emperor, who probably needed to be close to the political and administrative centers of the empire. For this reason, the site may have been a temporary vacation home for the emperor, rather than a full-time residence.

Next. While the idea that Machu Picchu was built as a ceremonial and religious center is a popular theory, it is not without its flaws. One challenge to this idea is the lack of evidence of religious activity at the site. While the citadel contains impressive stonework and many elaborate carvings, there are no clear indications that it was used for religious ceremonies or rituals. In contrast, other Inca sites, such as the Temple of the Sun in Cusco, contain clear evidence of religious activity, including altars, offerings, and other religious artifacts.

Lastly, it is difficult to claim that Machu Picchu was built as a defensive fortress.  While the citadel contains strong walls and terraces, there are no clear indications that it was used for military purposes. Very few actual weapons or military tools have been found there. The Inca were known for their elaborate military practices, and the remains of their soldiers and weapons have been found at many other Inca sites. The absence of clear signs of military activity at Machu Picchu suggests that it was not used for defensive purposes.

 

Everyone is talking about ChatGPT this week. I wonder: could this technology be used to create a test of English fluency?

Possibly.  In about twenty minutes I was able to generate a TOEFL integrated writing question that looks okay.  I spent about five minutes generating the reading and lecture script, and about 15 minutes revising them to be slightly more “TOEFL-like.”  I used a few extra minute to record the lecture, though I realize now I should have used an AI to generate that as well.

The result isn’t perfect, but it’s fairly close and can certainly be used as practice material.

(Update: I generated an even better one)

The twenty minutes I spent on this compares favorably to the 200 person-hours we used to spend creating this stuff when I worked for a major publisher.  

Before I share the results, here is the standard disclaimer:

“The author generated this text in part with GPT-3, OpenAI’s large-scale language-generation model. Upon generating draft language, the author reviewed, edited, and revised the language to their own liking and takes ultimate responsibility for the content of this publication.”

Here is the reading.  Of course, I took on a topic related to humans vs automation:

Over the past few decades, robots have become an increasingly important tool in space exploration. These machines offer a number of benefits that make them ideal for use in this field. Overall, there are three key benefits of using robots in space exploration.

First, robots are able to operate in environments that are too dangerous or inhospitable for humans. For example, robots can be sent to explore the surface of Mars, where the temperature, radiation levels, and atmospheric conditions are hostile to human life. Robots can also be used to explore the depths of the ocean or the interior of volcanoes, where humans would be unable to survive. By using robots, we can gather valuable data and samples from these extreme environments without putting human lives at risk.

Second, robots are able to perform tasks that are too difficult or complex for humans to do. For example, robots can be equipped with specialized tools and sensors that allow them to conduct experiments or make measurements that are beyond the capabilities of human astronauts. Robots can also be designed to move in unique ways, such as crawling, swimming, or flying, which allows them to access areas that would be difficult or impossible for humans to reach. This can enable robots to gather data and samples from locations that would otherwise be inaccessible.

Third, using robots is less expensive than sending human astronauts to space. They are able to operate for long periods of time without needing to rest, eat, or drink. This makes them ideal for missions that require sustained exploration over many months or even years. For example, a robot could be sent to a distant planet or moon and do its work without costly supplies being sent from earth.

Here is the lecture audio:

 

Here’s a prompt:

Summarize the points made in the lecture, being sure to explain how they oppose specific points made in the reading passage.

And here’s a transcript of the lecture:

There are a few problems with the claim that robots are always more useful than human astronauts when it comes to space exploration.

First, while it’s true that robots can operate in dangerous environments, their ability to deal with problems is somewhat limited by their design and programming. The fact is, Robots may not be able to respond to dangerous situations that are unexpected or to adapt to conditions they are not specifically designed for. So… if a robot encounters a unique situation on a distant planet it may be unable to continue its mission. In contrast, human astronauts have the ability to think and problem-solve, which allows them to overcome challenges and adapt to new situations.

Second, robots may not be able to provide the same level of scientific data as human astronauts. While robots can be equipped with sensors and instruments that allow them to gather a wide range of data, they can’t make the same connections as humans. Since robots are limited by their programming and technology,  they may not be able to notice or interpret subtle patterns in the data that they collect. In contrast, human astronauts have the ability to think, reason, and make connections, which can lead to amazing breakthroughs and insights.

Lastly, robots are more expensive to develop and operate than you might think. Building a robot that can function in the harsh conditions of space isn’t cheap. It requires special materials and technology, which can drive up the cost. In addition, once a robot is launched into space, it may be difficult or impossible to repair or upgrade it. This means that a tiny malfunction may destroy the whole mission. In contrast, human astronauts can be trained to troubleshoot and repair equipment, which can help to extend the life of a mission.

The cost of taking the TOEFL recently decreased in a few countries. Among the countries I track, the cost decreased in Egypt (-$40), Nigeria (-$80), Germany (-$30), Austria (-$25), Italy (-$35), Spain (-$20), Poland (-$35), and Turkey ($-17).

I’m not sure when the prices dropped as it happened outside of the usual price-adjustment period, but it was sometime between July 1 of this year and today.

This is my first time to notice price drops since I started tracking this stuff in 2020.

I won’t update the tracker until the normal period hits in January/February, but you can see the latest version over here.