Listen and Summarize Oral Evaluation
Wouldn’t it be great if a student could instantly receive a score and feedback, after watching a video and explaining to a classmate what he or she heard? An evaluation like that would be much closer to the way language is actually used in the world. Labodanglais.com makes it possible. Students watch a video and then record themselves summarizing it. The assignment plugin scores their oral automatically. Doesn’t that sound so much better than giving students a listening quiz?
Assigning webcam oral summaries of videos is a social, realistic, and practical alternative to multiple choice listening quizzes. It’s more social because students have to explain what they heard to another person. It’s more realistic than a listening quiz because nobody takes listening quizzes outside of the context of a course, but people tell each other what they just watched on TV or heard on the radio all the time. And webcam oral summary assignments are practical for teachers because Labodanglais.com can evaluate all of your students’ webcam recordings in two minutes, all day, every day, and all semester long without breaking a sweat.
The listening quiz: a pedagogical compromise
The usual way teachers test listening is to show a video to students and have them answer multiple choice questions on paper or online. Students listen, read comprehension questions, and then try to select the correct answers from a short list. Teachers like me believe that listening quizzes are useful in testing comprehension. I use them in all of my courses. On a Moodle system like Labodanglais.com, the answers are automatically checked and scored instantly. That helps teachers like me manage our correction workload.
However, multiple choice listening quizzes don’t evaluate what a student can do with the information they have heard.
Let’s face it. Nobody answers multiple choice questions after watching a video except in the context of a course. Quizzes belong in school and almost nowhere else. In other words, the traditional listening quiz is a pedagogical compromise–not a necessity. Teachers test this way to get students listening attentively and to offer students some feedback on their comprehension quickly, but we shouldn’t think that we have tested a communication skill. For listening to be considered a communication skill in the way that people ordinarily listen to language in their day-to-day lives, students should be able to communicate what they have learned to another person.
“Honey, you won’t believe what I just heard on the radio.”
— Everybody
It is not hard to imagine someone getting into the passenger seat of a car and being told by the driver, “You won’t believe what I just heard on the radio!” Or, a ten-year-old explaining a funny video he or she has seen on YouTube. Or, imagine emerging from the bathroom after brushing your teeth and your favorite someone says, “Honey, there’s a big storm coming. I just heard the forecast.” Each of these is a more plausible example of how people listen to media for comprehension and what they do subsequently with that information. Teachers sometimes forget that the traditional ESL listening quiz lacks the social component of real communication.
Why use automatically scored oral summary assignments?
Education is becoming more social and more skills-focused. Teachers want students to use what they have heard in our courses and communicate it as a meaningful message to someone else to solve a real-world problem, right? Whatever the course, we can certainly do a better job of simulating that experience. To avoid creating an avalanche of correction and grading, we will also want to automate at least some of our evaluations.
Here are two examples of an automatically scored oral summary assignment. The first is for intermediates. The second is for beginners. The steps are the same.
Step 1: show the video
Here’s a video about the social brain, itself a summary of an interesting journal article by Matthew Lieberman, recommended to me by Matt Bury. It’s 7 minutes long. Students are instructed to listen to it with the goal of summarizing it for a classmate. If you watch the video, you will understand how important it is for students to have a social purpose in mind as they learn. If this topic interests you, check out Eric Mazur peer teaching method on YouTube, recommended to me by Avery Rueb.
Step 2: provide a virtual classmate
We can simulate a listener with a video on a loop like this. What you see here is a gif set to loop indefinitely. It gives the impression of a someone listening, nodding, and blinking as people do when they give you their attention.
Step 3: webcam recorder
Labodanglais.com has a plugin that other Moodle websites don’t have. It’s an automatically scored oral assignment plugin that we developed in the summer and tested in the fall semester of 2022. It records webcam videos using a common web recording standard on mobile or a laptop. Students simply click record, stop recording, and save changes.
Step 4: automated evaluation
The automatically scored oral assignment plugin sends the student’s video to Google for transcription, and checks the returned text for word count, grammar, and target structures. These measurements provide assessments of comprehensibility (correctly transcribed and mistranscribed words), fluency (volubility), accuracy (number of grammar errors), and topic-relatedness (number of matches of target vocabulary).
The system is able to perform the evaluation in about 2 minutes and return feedback with video explaining where the oral summary falls short of expectations.
Step 5: revision
In most classrooms, teachers cannot let students redo oral evaluations for a higher score because of time constraints. However, with automated oral summary evaluations like the one described here, the system can evaluate as many of the student’s submissions as the student is willing to record. A tireless teacher’s helper of this gives students the opportunity to improve their score by putting in extra time and effort, linking effort reliably to achievement instead of anxiety levels or health-related absences.
Oral summary assignment for beginners
The example evaluation above is for intermediates or high intermediates. But beginners will benefit from summarizing what they have heard, also. Here’s a 3-minute video for beginners. It describes 4 practice strategies. Instead of the teacher explaining and reexplaining these strategies, it would be much better if students would explain the strategies to each other.
Step 1: show the video
Students watch the video in the lab or for homework. They can watch it as many times as they like.
Step 2: provide a virtual classmate
Here instead of Jackie, our virtual teaching assistant and spokesperson, we can ask students to explain the practice strategies they heard about to Alisha, a hoodie-wearing young college-age woman. She listens attentively, blinking with a neutral expression on her face.
Step 3: webcam recorder
The automatically scored oral assignment plugin records webcam videos using a common web recording standard on mobile or a laptop. Students simply click record, stop recording, and save changes to submit their video for evaluation.
Step 4: automated evaluation
Again, the automatically scored oral assignment plugin sends the student’s video to Google for transcription, and checks the returned text for word count, grammar, and target structures. The system is able to perform the evaluation in about 2 minutes and return feedback with a video explaining where the oral summary falls short of expectations.
The target vocabulary, in the case of this video, include the words phrases, and their synonyms that are related to the oral practice strategies described in the video. The system will generate a score automatically, but teachers can review the videos the students record to makes sure they are making an honest attempt. The teacher can always override the system’s scores if there is any concern that the student hasn’t made an honest attempt to summarize the content of the video.
Step 5: revision
Do-overs are possible and encouraged. Students can record themselves and get a score. If they are happy with it, they can do something else. If they feel that they could do a better job after receiving their score and feedback, they can rerecord themselves a second and a third time. This kind of mastery model scoring makes students who are anxious about their grades very happy. Beginners enjoy a sense of agency, with a level of control over their fate in English class that they have never felt before. In short, effort is rewarded.
Conclusion
As you can probably tell, I’m pretty excited about this type of automatically evaluated listening activity. It is social, realistic, and very practical. Nevertheless, I expect teachers to say, “We don’t have webcams in our language labs.” It is time to explain the value of webcams to your IT department and administration. Really. Webcams are standard issue on laptops and tablets and phones. Your school should try to catch up.
On the other side of the teacher’s desk, I expect students to say, “I don’t want to record myself,” or “I don’t want to show my face,” or “The system is broken–it didn’t transcribe what I said.”
I just spent a semester in Teams and Zoom meetings. Showing your face on video is commonplace. Video recording is the norm on TikTok, Instagram, and other social media websites. Get over it. You don’t want to show your face? See a psychologist.
As for the system being broken because it mis-transcribed what you said, consider this. Your ESL teacher has spent years listening to and making sense of accented, ill-formed English. What your teacher can understand is not an objective standard of comprehensibility. Teachers are pros. We are paid to interact with you. What’s going to happen when you find yourself surrounded by anglophones whose salaries don’t depend on listening to you? Strangers won’t always be so kind. Getting a robot to understand you is good practice.
A robot transcription is an honest transcription. If Google doesn’t understand what you said, maybe you could speak more clearly and work on your pronunciation. A bad transcription is actually implicit feedback on your comprehensibility. Try again for a higher score.
Wanna try it out?
If you would like to give this type of listening activity a try, let me know. Send an email to BokomaruPublications@gmail.com. I will set you up with a demonstration course to play with.