This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.
About the Author
Cyrus Wong is the senior lecturer of Department of Information Technology (IT) of the Hong Kong Institute of Vocational Education (Lee Wa... and he focuses on teaching public Cloud technologies. He is one of the Microsoft Learn for Educators Ambassador and Microsoft Azure MVP from Hong Kong.
Introduction
Why waste time grading assignments manually when AI can do it in seconds?
It’s impossible to stop students from using AI tools to answer their assignments, and it’s nonsensical to ask them to do something that AI can complete in a second. The key is to ensure that students are working on their assignments on their own and not copying from others, then using AI to rephrase the language. In fact, it’s nonsensical for teachers to score student work manually when all students use AI to prepare their assignments! The ideal solution is to let AI grade it automatically.
At IVE we have created an AI virtual assistant that can help students with their assignments. The assistant can record chats and/or screen activities during tests. However, we are aware of the environmental and economic challenges of using AI, as it consumes a lot of energy and there is a cost of using the AI Service which students should be aware of as part of Responsible AI.
Therefore, we built a solution for our students which not only does the task, but display the cost and token usage for each conversation and limits the daily usage of the system to students to ensure they utilise the services responsibily.
Choosing the right model, parameter and prompt design is essential for students to score well in their assignment. ChatGPT 4 is not always the best option!
Demo
Using Azure Open AI Services virtual assistant to review and assess students' Prompt Engineering skills.
Explain & Demo Automatic Scoring with Azure Open AI Services Live2D Virtual Assistant System by ChatGPT
Architecture of the Solution
The virtual assistant application is built on Azure Static Web Apps and Azure Function, and uses Active Directory Application to authenticate users, with their academic email as their identity.
Each API call is proxied through Azure Function, and before each call, the user's email address is extracted from the request header and checked against the "users" storage table to ensure authorization.
Recording Chat
User messages, Chatgpt responses, model, parameters, cost and tokens usage are stored in the "chatHistory" storage table for future reference.
If the assignment contains multiple questions or tasks, students need to provide a Task ID.
It shows the expenses of the previous conversation and the remaining balance.
Recording Screens
The Role attribute must be set to Teacher to view the teacher share screen panel in users table.
The teacher enters the class ID and selects the “Enable Screen Sharing (3 hours)” button. The system queries the “classes” storage table and retrieves the list of students.
The Azure function “enable-screen-sharing” creates a new entry in the “sessions” storage table and removes the cached screens from the root level.
The student clicks on the “Share Screen” button and their chosen view is uploaded to the Azure function “screens” every 5 seconds. This function stores two copies of the screen images.
The images in the folder are deleted after 7 days and the images in the root are always updated to the latest screen.
The “Start Auto Refresh” button triggers the Azure Function “get-class-screens” to fetch the SAS URL of the latest class screen every 5 seconds. If the cached screens are deleted or the students cannot share their screens, the image will not be displayed.
This feature has helped a lot with online teaching during the pandemic period. Otherwise, most students would just join the MS team meeting and then leave. Having this feature we can track student interaction throught the lecture to ensure they are engaged.
Automatic Grading Assignment
The process mimics how humans work. As a educator/teacher we need to create a marking scheme or use ChatGPT to produce one.
1. The teacher setup filter with date range and an optional Task ID.
2. Define a marking scheme template.
Comments on the following conversations between User and AI in 100 words.
++++++++++++++++++++++++++++++++++++++++++++++++++
{{#conversations}}
User: {{User}}
AI: {{Chatbot}}
{{/conversations}}
++++++++++++++++++++++++++++++++++++++++++++++++++
Total Cost: {{totalCost}}
Total Token: {{totalTokens}}
Give a mark from 1-10 for those conversations.
Rule 1: the effectiveness conversation for maximum 5 marks
Rule 2: the Total Token deduce 0.5 marks for every 1000 tokens. the initial marks is 3 and minimum is 0.
Rule 3: the Total Cost is less than 0.5 gives 2 marks.
Explain the final mark in comments.
Response with JSON in the following format:
{
"marks": 1,
"comments": ""
}
And, merge with the following JSON by mustache.js template engine.
[
{
"etag": "W/\"datetime'2023-04-26T03%3A15%3A44.6113133Z'\"",
"partitionKey": " XXXXXXX@stu.vtc.edu.hk",
"rowKey": "1682478944605",
"timestamp": "2023-04-26T03:15:44.6113133Z",
"taskId": "q1",
"Email": " XXXXXXX@stu.vtc.edu.hk",
"User": "please give me a story about i go to school by bus (funny) at least 250 words",
"Chatbot": "Once upon a time, there was a young girl named Lily. Every morning, she would wake up early and go to school by taking the school bus. Lily found the bus rides boring, so she always tried to make them more fun. \n\nOne day, Lily woke up extra early to prepare for a prank she wanted to pull on her friends. She put on a silly costume and got on the bus, hoping to surprise everyone. As she walked down the aisle, the other students stared at her, wondering what was going on. \n\nLily took a seat at the back of the bus and waited for her friends to get on. When they did, they couldn't believe what they saw. Their friend was dressed as a giant banana, complete with a yellow jumpsuit and a green hat that looked like a stem. \n\nAt first, Lily's friends were confused and asked her why she was dressed like that. Lily just giggled and said she was tired of boring bus rides and wanted to add some excitement to the morning. \n\nAs the bus made its way through the town, people on the streets stared in amazement at the giant banana on the bus. Some even took pictures and waved at Lily, who happily waved back. \n\nWhen the bus arrived at school, Lily's friends couldn't stop laughing. They asked her how she managed to keep a straight face on the bus, but Lily just smiled and said it was all part of the fun. \n\nFrom that day on, the school bus rides were never boring for Lily and her friends. They always looked forward to what kind of silly costume Lily would come up with next. Even the bus driver couldn't help but chuckle at their antics. \n\nAnd so, Lily proved that even the most mundane routines can be made into fun adventures with a little bit of creativity and humor.",
"Model": "gpt-35-turbo",
"CompletionTokens": 371,
"PromptTokens": 40,
"TotalTokens": 411,
"Cost": 0.00022840000000000002,
"max_tokens": 800,
"temperature": 1,
"frequency_penalty": 0,
"presence_penalty": 0,
"top_p": 0.95
},
{
"etag": "W/\"datetime'2023-04-26T03%3A16%3A07.5397668Z'\"",
"partitionKey": " XXXXXXX@stu.vtc.edu.hk",
"rowKey": "1682478967531",
"timestamp": "2023-04-26T03:16:07.5397668Z",
"taskId": "q1",
"Email": " XXXXXXX@stu.vtc.edu.hk",
"User": "please give me a story about i go to school by bus (funny) at least 250 words",
"Chatbot": "Once upon a time in a small town called Funtropolis, there was a girl named Fanny who loved having a good laugh. One day, she woke up feeling particularly excited because she had been told that a new bus route had just started in the town, and that the bus driver was a retired clown. The thought of going to school by bus with the hilarious driver made Fanny's morning extra special.\n\nAs Fanny got ready for school, she prepared her best jokes to share with the driver, and her friends who would be riding the bus with her. She grabbed her backpack and ran outside, waiting eagerly for the bus to arrive. \n\nRight on time, the bus approached, and to Fanny's delight, it was not an ordinary bus. It was painted with colorful stripes and polka dots, and had a large sign on the front that read \"The Laughing Express.\" As the doors swung open, she was greeted by the driver, Mr. Giggles, who had a bright red nose, oversized shoes, and a rainbow wig. \n\n\"Welcome aboard, Fanny!\" Mr. Giggles said with a wide smile as he handed her a balloon animal. She climbed onto the bus and saw that her friends had already taken their seats, grinning with anticipation. The interior of the bus was just as delightful as the outside, with funky patterns on the seats and mini disco balls hanging from the ceiling.\n\nAs the bus began its journey to school, Mr. Giggles regaled the children with his best jokes and hilarious stories about his days in the circus. Fanny and her friends took turns sharing their jokes with Mr. Giggles, who laughed heartily and rewarded them with more balloon animals.\n\nAt one point during the ride, the bus came to a sudden stop, and everyone peered out the window to see what had caused the delay. It turns out that a group of ducks had decided to cross the road right in front of the bus. Mr. Giggles leaned out the window and said, \"Hey, ducks! Why are you crossing the road?\" The ducks quacked in reply, and Mr. Giggles chuckled before exclaiming, \"To get to the other side, of course!\"\n\nEventually, the bus approached the school, and as the children were about to get off, Mr. Giggles surprised them with a farewell confetti shower. Fanny and her friends stepped off the bus, covered in confetti, and couldn't stop laughing as they walked into school.\n\nThat day, Fanny and her friends couldn't stop talking about their incredible bus ride, and Mr. Giggles quickly became a local legend. Every morning, Fanny looked forward to her journey to school, knowing that laughter and joy awaited her aboard The Laughing Express. And, as they say in Funtropolis, they laughed happily ever after.",
"Model": "gpt-4",
"CompletionTokens": 586,
"PromptTokens": 37,
"TotalTokens": 623,
"Cost": 0.019799999999999998,
"max_tokens": 800,
"temperature": 1,
"frequency_penalty": 0,
"presence_penalty": 0,
"top_p": 0.95
},
{
"etag": "W/\"datetime'2023-04-26T03%3A16%3A27.1488001Z'\"",
"partitionKey": " XXXXXXX@stu.vtc.edu.hk",
"rowKey": "1682478987136",
"timestamp": "2023-04-26T03:16:27.1488001Z",
"taskId": "q1",
"Email": " XXXXXXX@stu.vtc.edu.hk",
"User": "please give me a story about i go to school by bus (funny) at least 250 words",
"Chatbot": "Once upon a sunny morning in the quirky little town of Laughterland, a teenage boy named Max was rushing to catch his school bus. Max had a reputation among his friends for having a very unique and amusing imagination. Today, he decided to make his school bus journey as entertaining as possible.\n\nMax quickly put on his favorite outfit - a bright red clown wig, a superhero cape, and mismatched shoes. He grabbed his trusty rubber chicken named Henrietta and dashed out the door just in time to catch the school bus. \n\nAs soon as Max stepped onto the bus, the entire atmosphere changed. A few kids gasped, others chuckled, and some just stared with wide-eyed confusion. Undeterred by their reactions, Max confidently strolled to the middle of the bus and sat down. The wheels of the bus went round and round, and so did Max's imagination.\n\nMax began to tell his fellow students a fantastical tale of the legendary Rubber Chicken Warrior, describing the heroic battles Henrietta had fought against the dreaded Vegetable Monsters. As Max's story unfolded, the students on the bus roared with laughter. Even the bus driver, Mr. Grumpy, couldn't help but crack a smile, his eyes twinkling in the rearview mirror.\n\nWhen the bus reached the school, Max finished his tale with a grand flourish. As the students disembarked, they were still chuckling and chatting animatedly about Max's performance. Max proudly strode into the school, his cape billowing behind him, and gave Henrietta a little squeeze, eliciting a comical squeak that sent a ripple of laughter through the hallway.\n\nFor the rest of the day, Max's friends and classmates couldn't stop talking about his hilarious bus ride adventure. They eagerly anticipated Max's next performance, as his wild imagination and humorous antics had brightened their morning and transformed an ordinary bus ride into an unforgettable memory.\n\nAnd so, Max proved that even a mundane trip to school could be turned into a fantastic adventure with a little bit of imagination and a flair for comedy. Laughterland continued to live up to its name, and Max's joyful spirit spread throughout the school, inspiring others to find the humor in their daily routines.",
"Model": "gpt-4-32k",
"CompletionTokens": 447,
"PromptTokens": 437,
"TotalTokens": 884,
"Cost": 0.07926,
"max_tokens": 800,
"temperature": 1,
"frequency_penalty": 0,
"presence_penalty": 0,
"top_p": 0.95
}
]
3. Click on “Grade with AI” and the generated prompt.
Comments on the following conversations between User and AI in 100 words.
++++++++++++++++++++++++++++++++++++++++++++++++++
User: please give me a story about i go to school by bus (funny) at least 250 words
AI: Once upon a time, there was a young girl named Lily. Every morning, she would wake up early and go to school by taking the school bus. Lily found the bus rides boring, so she always tried to make them more fun.
One day, Lily woke up extra early to prepare for a prank she wanted to pull on her friends. She put on a silly costume and got on the bus, hoping to surprise everyone. As she walked down the aisle, the other students stared at her, wondering what was going on.
Lily took a seat at the back of the bus and waited for her friends to get on. When they did, they couldn't believe what they saw. Their friend was dressed as a giant banana, complete with a yellow jumpsuit and a green hat that looked like a stem.
At first, Lily's friends were confused and asked her why she was dressed like that. Lily just giggled and said she was tired of boring bus rides and wanted to add some excitement to the morning.
As the bus made its way through the town, people on the streets stared in amazement at the giant banana on the bus. Some even took pictures and waved at Lily, who happily waved back.
When the bus arrived at school, Lily's friends couldn't stop laughing. They asked her how she managed to keep a straight face on the bus, but Lily just smiled and said it was all part of the fun.
From that day on, the school bus rides were never boring for Lily and her friends. They always looked forward to what kind of silly costume Lily would come up with next. Even the bus driver couldn't help but chuckle at their antics.
And so, Lily proved that even the most mundane routines can be made into fun adventures with a little bit of creativity and humor.
User: please give me a story about i go to school by bus (funny) at least 250 words
AI: Once upon a time in a small town called Funtropolis, there was a girl named Fanny who loved having a good laugh. One day, she woke up feeling particularly excited because she had been told that a new bus route had just started in the town, and that the bus driver was a retired clown. The thought of going to school by bus with the hilarious driver made Fanny's morning extra special.
As Fanny got ready for school, she prepared her best jokes to share with the driver, and her friends who would be riding the bus with her. She grabbed her backpack and ran outside, waiting eagerly for the bus to arrive.
Right on time, the bus approached, and to Fanny's delight, it was not an ordinary bus. It was painted with colorful stripes and polka dots, and had a large sign on the front that read "The Laughing Express." As the doors swung open, she was greeted by the driver, Mr. Giggles, who had a bright red nose, oversized shoes, and a rainbow wig.
"Welcome aboard, Fanny!" Mr. Giggles said with a wide smile as he handed her a balloon animal. She climbed onto the bus and saw that her friends had already taken their seats, grinning with anticipation. The interior of the bus was just as delightful as the outside, with funky patterns on the seats and mini disco balls hanging from the ceiling.
As the bus began its journey to school, Mr. Giggles regaled the children with his best jokes and hilarious stories about his days in the circus. Fanny and her friends took turns sharing their jokes with Mr. Giggles, who laughed heartily and rewarded them with more balloon animals.
At one point during the ride, the bus came to a sudden stop, and everyone peered out the window to see what had caused the delay. It turns out that a group of ducks had decided to cross the road right in front of the bus. Mr. Giggles leaned out the window and said, "Hey, ducks! Why are you crossing the road?" The ducks quacked in reply, and Mr. Giggles chuckled before exclaiming, "To get to the other side, of course!"
Eventually, the bus approached the school, and as the children were about to get off, Mr. Giggles surprised them with a farewell confetti shower. Fanny and her friends stepped off the bus, covered in confetti, and couldn't stop laughing as they walked into school.
That day, Fanny and her friends couldn't stop talking about their incredible bus ride, and Mr. Giggles quickly became a local legend. Every morning, Fanny looked forward to her journey to school, knowing that laughter and joy awaited her aboard The Laughing Express. And, as they say in Funtropolis, they laughed happily ever after.
User: please give me a story about i go to school by bus (funny) at least 250 words
AI: Once upon a sunny morning in the quirky little town of Laughterland, a teenage boy named Max was rushing to catch his school bus. Max had a reputation among his friends for having a very unique and amusing imagination. Today, he decided to make his school bus journey as entertaining as possible.
Max quickly put on his favorite outfit - a bright red clown wig, a superhero cape, and mismatched shoes. He grabbed his trusty rubber chicken named Henrietta and dashed out the door just in time to catch the school bus.
As soon as Max stepped onto the bus, the entire atmosphere changed. A few kids gasped, others chuckled, and some just stared with wide-eyed confusion. Undeterred by their reactions, Max confidently strolled to the middle of the bus and sat down. The wheels of the bus went round and round, and so did Max's imagination.
Max began to tell his fellow students a fantastical tale of the legendary Rubber Chicken Warrior, describing the heroic battles Henrietta had fought against the dreaded Vegetable Monsters. As Max's story unfolded, the students on the bus roared with laughter. Even the bus driver, Mr. Grumpy, couldn't help but crack a smile, his eyes twinkling in the rearview mirror.
When the bus reached the school, Max finished his tale with a grand flourish. As the students disembarked, they were still chuckling and chatting animatedly about Max's performance. Max proudly strode into the school, his cape billowing behind him, and gave Henrietta a little squeeze, eliciting a comical squeak that sent a ripple of laughter through the hallway.
For the rest of the day, Max's friends and classmates couldn't stop talking about his hilarious bus ride adventure. They eagerly anticipated Max's next performance, as his wild imagination and humorous antics had brightened their morning and transformed an ordinary bus ride into an unforgettable memory.
And so, Max proved that even a mundane trip to school could be turned into a fantastic adventure with a little bit of imagination and a flair for comedy. Laughterland continued to live up to its name, and Max's joyful spirit spread throughout the school, inspiring others to find the humor in their daily routines.
++++++++++++++++++++++++++++++++++++++++++++++++++
Total Cost: 0.0992884
Total Token: 1918
Give a mark from 1-10 for those conversations.
Rule 1: the effectiveness conversation for maximum 5 marks
Rule 2: the Total Token deduce 0.5 marks for every 1000 tokens. the initial marks is 3 and minimum is 0.
Rule 3: the Total Cost is less than 0.5 gives 2 marks.
Explain the final mark in comments.
Response with JSON in the following format:
{
"marks": 1,
"comments": ""
}
4. “ChatGPT Grading” Response
{
"marks": 4,
"comments": "The AI successfully generated three unique and funny stories about going to
school by bus as requested by the user. The stories are engaging and have good pacing.
However, the AI could have been more creative and added more unexpected twists to the
stories to make them even funnier. The Total Token is 0.458, which deducts 1 mark from the
initial mark of 3. The Total Cost is below 0.5, which adds 2 marks. Therefore, the final
mark is 4 out of 10."
}
5. Repeat steps 1 – 4 for several students until the marks and comments are satisfactory, which becomes the final marking scheme.
6. To mark all students in the class automatically, click on the “Grade this class” button.
7. To download the mark report, click on “Mark Report” button.
Deployment
Please refer to my previous post - Maximizing Your Academic Potential with Azure OpenAI Services ChatGPT Virtual Assistant.
Technology Behind the scenes of deployment
To help students save cost by choosing a suitable model, we enable all Azure Open AI Services LLM models by default.
this.openAiCognitiveDeployments = [];
const models = [
{ name: "text-ada-001", version: "1" }, { name: "text-curie-001", version: "1" },
{ name: "text-davinci-002", version: "1" }, { name: "text-davinci-003", version: "1" },
{ name: "code-davinci-002", version: "1" },
{ name: "gpt-35-turbo", version: "0301" }, { name: "gpt-4", version: "0314" }, { name: "gpt-4-32k", version: "0314" },
{ name: "text-similarity-ada-001", version: "1" },
{ name: "text-similarity-curie-001", version: "1" },
{ name: "text-embedding-ada-002", version: "2" },
]
for (const model of models) {
const openaModel = new CognitiveDeployment(this, "openAiCognitiveDeployment" + model.name + model.version, {
name: model.name,
cognitiveAccountId: this.openAiCognitiveAccount.id,
model: {
name: model.name,
format: "OpenAI",
version: model.version,
},
scale: {
type: "Standard"
}
});
this.openAiCognitiveDeployments.push(openaModel);
}
Conclusion
This update allows us to train and evaluate students’ prompt engineering skills to solve problems. We cannot prevent students from using AI and we should enable them to complete assignments with the help of AI but under a fair and supervised environment.
The main issue is fairness - the problem arises when some students use AI while others do it manually, but they are graded with the same criteria.
When students use AI to complete the assignment, the educator/teacher should also use AI to assess their work. AI can help educators/teachers save time and focus on other meaningful tasks that improve their teaching quality.
Some educators/teachers may argue that students do not work hard for their assignments and learn nothing after using AI, especially in Asia. However, many assignments are designed to search for answers in notes or Google information, and then summarize them. Almost all assignments are testing the reading and writing ability. As a result, students who are good at language will always have an advantage and the others may sometimes lose motivation to study. I agree that language courses should not use AI as AI can make anyone a good writer easily. However, with the help of AI, students who are weak in language can regain their interest and confidence to learn as they can complete assignments with AI assistance. In fact, I can set more challenging assignments now as I assume everyone has AI support and set some tasks that were impossible before.
My non-degree courses actually focuses on skills that employers require students to use AI for their internship and graduate job. In the past, some companies would not hire my students because they needed good English skills for documentation and writing. Now, they are willing to recruit students who can solve problems with AI.
AI is essential! Our solution can train and test students effectively with AI and solve problems at a low cost. Lower cost also means less power consumption and more environmental friendliness.
Project collaborators include Shing Seto, Stanley Leung, Ka Ka Leung, XU YUAN and Hang Ming (Leo) Kwok from the IT114115 Higher Diploma in Cloud and Data Centre Administration.
