Self-Scoring Text Assignment

Students learn to write better by writing more often. That’s why I have added a new automated evaluation to Labodanglais.com and Labodefrancais.com. It’s a self-scoring writing assignment to evaluate English and French writing. The teacher simply sets thresholds, scores, and feedback messages. Students write, submit, and receive their scores seconds later. So, what’s the effect on teachers and students? Teachers reduce their correction workload. And students get more meaningful writing opportunities with feedback, making more revisions.

In this blog post, I will show you how you can create a self-scoring email writing assignment on a random topic, for example, asking students to summarize a YouTube video on bullying. However, the plugin can be used for any writing task.

By the end of this blog post, you will see how the Self-Scoring Text plugin can revolutionize the way you teach writing in English or French.

What does the self-scoring text assignment do?

In a nutshell, the self-scoring text assignment scores a student’s text in seconds. The student types a text. He clicks submit. A few seconds later, he receives his assignment score and feedback on three aspects of the text: length, accuracy, and the use of target structures.

What does the teacher have to do? As with any writing assignment, the teacher writes a description. Then, the teacher sets the word-count thresholds. For example, fewer than 50 words will trigger a score of 0% and the message “Too short,” but 51 to 100 words triggers a score of 60% with the message “You have written the minimum, but you can get a higher score if you write more.”

The teacher lists topic-related words or grammar structures he expects the student to use, with the expected number he hopes to see in the student’s text. The comment is triggered when fewer than the expected number of the listed targets are found.

Finally, the teacher sets the penalty for each error detected. For example, for advanced students, the penalty could be 20% for each error. For beginners, it could be 5%. Errors are detected by the Virtual Writing Tutor.

Self-Scoring Text versus FastAssignment

In case you were wondering, the Self-Scoring Text assignment plugin is a simplified alternative to the FastAssignment plugin. Both the Self-Scoring Text and FastAssignment plugins are available for all teachers using Labodanglais.com and Labodefrancais.com.

The FastAssignment plugin is more powerful than the Self-Scoring Text plugin described in this post. How? FastAssignment can count paragraphs and check for more features in specific paragraphs or specific positions within a paragraph. For example, and FastAssignment can check the first sentence of a paragraph for topic sentence strength and the last sentence of the first paragraph for thesis statement strength. In this way, FastAssignment can evaluate the structure of a writing assignment. However, this kind of artificial intelligence requires weeks of work and a team of web developers to code each evaluation.

In contrast, this Self-Scoring Text plugin is much simpler to use, allowing teachers to create automated evaluations themselves in minutes.

Example Assignment: write an email summarizing a video

In this blog post, I will show you how the system evaluates hundreds of student submissions, semester after semester on either Labodanglais.com and Labodefrancais.com. All you have to do is set it up. The example I have developed for you involves asking students to write an email to Alan summarizing a video on YouTube. Who is Alan? He’s our old professor robot in the video below. Watch the video to see the specifics of this assignment. Of course, teachers will have access to this exact example in the Sharing Cart repository, which you can use as-is or modify for use with a different video by modifying the required elements.

You can add this video to your course by simply copy pasting this URL in the description box anywhere on Labo: https://labodanglais.com/files/actively_engaged/video/watch-a-video-and-summarize-in-an-email.mp4

Using the new plugin

Turn editing on. Add an activity. Select “Assignment” and click “add.”

There are two plugins to activate. The first you must select is “Self-scoring text” in the “Submission types” section. This will call up the settings form for you to set the thresholds, scores, and feedback messages.

The second plugin to activate is “Self-scoring text” in the “Feedback types” section. This will display the feedback on the student’s assignment page after submitting the assignment for scoring.

Set the system to expect either English or French. If you select English, the system will send the student’s text to the Virtual Writing Tutor for a grammar check. If you select French, the system will send the text to BonPatron.com.

Accuracy is important in summary writing. Grammar errors should be eliminated. A text with fewer errors will merit a higher score. This setting is optional.

The default penalty is 10% per error. For advanced students, you can set the penalty higher. In the example below, I have set it to a 20% penalty for each error. For beginners, you can make the penalty much lower. For example, you could set it to 5%, meaning the student would have to make 20+ errors for the system give 0% for language accuracy.

In the case of a summary of a video or an article, it means explaining the main ideas expressed without the repetition or examples present in the original. Summary writing therefore requires concision. As such, the student should be careful not to write too much or too little. With this plugin, we can reward longer texts and apply a penalty for overly-long texts. We can do that using word-count thresholds.

Also, summary writing involves paraphrase. The exact phrasing of the student’s text may be hard to predict, but topic-related vocabulary can be counted and rewarded accordingly. In the example below, the heading tells the student what is required, The comma-separated target-structure list contains all of the topic-specific words in the video. The “Must match” setting tells the system how many to of the case-insensitive targets must be found in the submitted text to receive 100%.

In the example below, the system will look for 10 of the topic-related words listed. There are 57+ words listed. If the student only uses 9, the system will award only 90% for this dimension of the submitted text and display the comment “I was expecting you to use…”

For our topic bullying, we can find a video on YouTube that is related to that topic, extract the keywords from the transcript, and then tell the system to expect a certain number of those words in the summary. See the steps below.

In the case of an email, there are other required elements that we can expect. For example, we will want a salutation and a closing.

In our example video, Alan says he wants to know specific things about the video. He doesn’t just want a summary of the message. Alan also wants the student to explain who, for whom, what, and where. He also wants the student to comment on anything remarkable about the sights and sounds in the video.

In terms of content, I want you to explain to me, who is in the video, who the video is for, where the action takes place, and anything remarkable about the sights and sounds in the video. Then, I want you to explain to me the key concepts, and the main ideas. What is the takeaway message?

Alan

Required element 3 therefore says, “I want you to explain to me who is in the video…” The feedback hints at the answer but doesn’t provide the exact wording. The student must watch the video again.

The required element section of this evaluation is worth 33% of the total score. The student must use vocabulary linked to the topic for 11%, which he or she can get by watching the video. The student must use email writing conventions by following the instructions for another 11%. Finally, the student must explain who is in the video on YouTube and what the takeaway message is for the last 11%.

Weighting

Word count is the only essential dimension of any Self-Scoring Text assignment. A student cannot get a score without writing something and submitting it. The teacher can, of course, decide how many words will constitute a pass or a fail. If the teacher doesn’t select “Check grammar” or add a “Required element” with target structures, then fluency will account for 100% of the score. These other two are optional.

If the teacher selects “Check grammar” without adding required elements, then fluency is worth 50% and grammar check is worth 50%.

If the teacher adds “Required elements” without adding a grammar check, then fluency is worth 50% and required elements is worth 50%.

If all three are selected, then each dimension of the writing assignment is worth 33% each. In our example on bullying, three sections contribute 33% each. The weightings are as follows:

• Word count 33%
• Grammar check 33%
• Required elements 33%
  • Bullying vocabulary 11%
  • Email conventions 11%
  • Who, what, where, takeaway message 11%

As you can see, the required elements dimension contributes 33% to the total. Within that 33%, each of the three required elements is worth 11%. More elements can be added, but you must save your changes for additional form fields to appear. The number of required elements that you can add is unlimited. Although, three seems to be right number.

Are you ready now to create your own listening summary writing task? If so, I have prepared the steps to follow with tips and tricks to make the task easier and faster. Follow these steps to create your own automatically scored, video summary email writing assignment.

Step 1: Choose a topic

Choose a video. On YouTube, I entered the the search term “bullying.” I found this video and chose it because it has the length and quality I want. Ideally, the teacher should write a summary of the video him or herself to get an idea of what the ideal submission will look like. It will help in preparing the thresholds, scores, comments, and target structure lists for the required elements.

The video I found below is a good one for summary writing because the main point is relatively easy to express. However, the presenter first tells us the usual and wrong way to think about bullying. He talks about dominance behaviour, an imbalance of power, assault and battery, and the alpha male mentality. Students can contrast these concepts with what the speaker believes is the right way to think about bullying. That’s his main point, which he illustrates a number of times by roleplaying and playacting with examples. You’ll notice something noteworthy about the colour, too.

We can add this video to the Self-Scoring summary assignment simply pasting the video URL into the description box. The system embeds the video automatically. See the highlighted URL below.

Step 2: Analyze the vocabulary

With the video embedded, the next step is to analyze the vocabulary by extracting the keywords from the transcript. That’s easily done. On the YouTube page, find the (…) which is a “Show more” menu link next to the download link and select “Show transcript” from the dropdown menu.

The transcript will appear on the right side of the video. Remove the time stamps, copy-paste the transcript into a Word document, and remove all of the unimportant words and phrases. What you will have left is only the most relevant words and phrases from the talk. These words and phrases are so important that it is unlikely that anyone could effectively summarize this video without using 10 or 20 of them. This list will be needed for one of the target elements. We will use it to determine if the student is on or off topic.

Here’s part of my list taken from the video transcript. You will notice that I included “bully, bullies, bullied, bullying.” I have included all four not because I expect students to use all four forms in their summaries, but I want the system to match any of these four if they come up. Similarly, I will have to list “behaviour” and “behavior” or “colour” and “color” if I anticipate both British-Australian-Canadian and American spelling.

alpha male mentality, anger, assault and battery, bounce, bully, bullies, bullying, bullied...

Step 3: Add formal requirements

I want students to write their summary in the form of an email to our virtual teacher Alan. Therefore, I expect students to use the words “Dear Alan” and “Best wishes.” These form a second list of required elements which will be scored. The system will not be able to tell if they have been used in the right place or in the right order, but it will be able to tell if the student has forgotten to use them. Watch the video to hear Alan explain the assignment. These are the targets the system will try to match:

Dear Alan, Best wishes

Step 4: Add context requirements

The next element is the context of the ideas. Since it is in an email to someone, the student should not just launch into the content of the video without first explaining who was in it, who the video was for, where it took place, and anything remarkable about it. This is a normal part of communication. We might say, “I am writing to tell you about a video on YouTube in which Brook…” Talking about the video is not the same as summarizing the message, but some context will help to situate the summary for our reader, our virtual teacher called Alan. These are a few of the targets:

Brook, video, YouTube...

Step 5: Evaluate

The final step is to perform what instructional designers call a “formative evaluation.” In instructional design, we use this phrase differently to the way teachers use it. Instructional designers evaluate the learning intervention–not the student. We want to know how reliably the the self-scoring assignment measures student achievement and what the effect on students has been.

If the system’s scores are too low, we may want to adjust the thresholds so that students don’t get too discouraged. If the scores are too high, we might want to require longer texts, penalize errors more severely, and require more target structure matches. If complete nonsense gets a good score, you might want to plan to spot check your students’ submissions and adjust the nonsense writer’s score accordingly.

All teachers use indicators of quality to determine a score. Psychologists call the rules of thumb and cognitive shortcuts people use for quick judgements “heuristics.” It’s normal. We look for certain features that tell us if the student is on-task, has learned what we taught, and achieved the objectives. We make inferences. While evaluating an essay, we might skim for errors and scan for the thesis statement, topic sentences, and the conclusion, without really reading every word in the student’s submission. We might use rubrics and scales to save time on commenting and explaining scores. It’s a different form of automation and efficiency.

The self-scoring assignment also draws inferences. But as you have probably realized by now, those inferences are drawn from a rather superficial analysis. The robot doesn’t think about the meaning of the message. The system counts words, but it does not check for meaningless repetition of words. The system counts errors, but it penalizes a missing comma and a verb agreement error equally. It counts target elements, but it does not evaluate if these words have been used in a way that makes sense. For most students most of the time, the inferences will be valid. But there will be brilliant non-conformists and wags who will make us question the validity of the system’s inferences. In the future, we’ll use machine learning and broad AI. But for now, this self-scoring assignment is the best we have. And it’s quite good.

Never get discouraged. There will always be mischievous students who will want to make a mockery of your course and self-scoring assignment robot. These scamps mistakenly believe that you don’t have their best interests at heart. Don’t they know that everything we do, as clumsy as it may seem, is actually intended to to help them learn?

Remember that the self-scoring writing assignment system is designed to give students more meaningful practice with feedback without exhausting the teacher. That’s a worthwhile goal.

Conclusion

There you have it. If you want to try automated writing evaluations–whether with video summary writing, article summary writing, storytelling, or email writing–you can. You will have to use Labodanglais.com or Labodefrancais.com, though. These are the only two websites that have this plugin, for now.

We also have a webcam oral evaluation system to automatically score speaking tasks like this “listen and summarize oral evaluation” activity.

Contact me at BokomaruPublications@gmail.com and let me know if you are interested. I will help you get started. If you want, we have textbooks for college ESL students. If you prefer to build your own course, we have a repository of activities that you can select from with the sharing cart.