ICMS AI Pilot Project: Students’ Perspectives

The use of Artificial Intelligence (AI) in college level assessments was evaluated in an ICMS Pilot Project recently.

The verdict by students: AI is useful and here to stay but it cannot replace human intelligence and critical thinking.

ICMS initiated a Pilot Project in Trimester 3, 2023, aimed at evaluating the incorporation of AI tools, such as Chat GPT, into assessments.

ChatGPT is an AI language model developed by OpenAI that generates dialogue and responses based on large datasets of text. It is trained with reinforcement learning through human feedback and reward models to improve its responses.

In considering the increasing automatisation of modern societies, students will likely use AI tools in their professional life. Therefore, they should be given opportunities to learn these skills during their education in an appropriate way.

ICMS has previously reflected on this and implemented policies around the safeguarding of academic integrity with regards to the use of AI and specifically Generative Pre-trained Transformer by students.

Master of Event Management lecturer Ruth Friedmann implemented the Pilot Project that had undergone rigorous evaluation by ICMS academics, and led by DVC (Learning and Teaching) Dr Heidi Le Sueur.

The Pilot Project was conducted under the AIED (Artificial Intelligence in Education) Framework, which provides guidance on understanding, using and responding to AI tools at the institution for students, lecturers and academic leaders at ICMS.

The assessment, using Chat GPT, had to be shown to balance the transformative potential of generative AI technology in improving teaching and learning experiences with the upholding of responsible and ethical practices.

ICMS Pilot Project: Can AI be effectively used in assessments without compromising academic integrity?

The ICMS postgraduate subject: Impacts And Strategic Planning of International Events (EVT603) was used as a Case Study.

Seventeen students were enrolled in the course, the majority of whom were already familiar with generative AI tools, such as Chat GPT developed by Open AI.

The assessment was and individual submission, 3 000 words, and delivered in a report format by the last week of the trimester (Week 11).

It was the third and final assessment of the subject. For the purposes of this assessment tasks, the only tool permitted was ChatGPT.

For the assessment, students were tasked to:

• Develop an event proposal using generative AI (Chat GPT)
• Analyse and critically evaluate ideas generated
• Ensure fit with international event destination and strategies based on existing theory

“The key words here are ‘develop’, where students should use AI to generate a concept or concepts; ‘analyse and evaluate’, where students must base their evidence in theory and industry; and ‘ensure fit’, meaning that students, again, must justify and base in evidence, from theory and industry, on why they have reached their conclusions in their final report,” Friedmann said.

‘This shows that the incorporation of generative AI was only a small component of the entire assessment task. It is a ‘tool’ that assists, not a machine that completes the report on their behalf.”

Insights from Students

Amplifying human capabilities

Recognising that a basic understanding of AI is critical for future career opportunities, students embraced the opportunity to use AI in assessments.

What the majority discovered was that AI was a useful tool that could help them, but not do their work for them.

“In the beginning I was scared, because this was another step into the unknown,” Event Management student Anamaria Mihaescu said.

“However, I came to realise, during the assessment, that AI is not here to replace us; it is here to amplify our capabilities. We could see it as an extension of human potential.”

AI ‘not the answer to everything’

Undergraduate event management lecturer Janene Wardrop, who took the EVT603 course as a student, came to appreciate how AI as a tool can help with ideas and concepts to build upon.

“ChatGPT is a tool which is now being widely used within the events industry, so I was keen to see how it could be used creatively within this subject,” she said.

“I came away from the Pilot Project with a greater understanding of how AI can build the bridge between academic knowledge and practical application, but also how limiting and biased AI can be  in certain areas. It is not the answer to everything.”

AI: A time-saving tool

Event management student Olatunji Omole recognised the speed of automation which AI offers and plans to use these time-saving measures in his future career.

“Automation has reduced some aspects of human functions in every service industry.

“As an event manager, AI will enable me to improve on my planning strategies by providing me with data driven insights to produce tailor made events for my clients, automation of repetitive and time-consuming task, monitoring of marketing and social media engagement, enhanced communication channels through use of chatbots and gathering of post event analysis feedback,” he said.

“The inclusion of AI in my assessment opened my eyes to endless possibilities which can be achieved during the event production process, enabling me to be better prepared for the job market.”

However, for areas where critical thinking is required, he found that AI fell short.

“For me, tools like ChatGPT simply gives you what you put in.”

“The better you are at your prompts the better the outcomes ChatGPT provides you with as a powerful tool which contains millions of data (both verified and unverified).

“This is where critical thinking comes in; the more knowledgeable you are about the subject the more effective you will find the tool to be.”

Goals of Pilot Project

The AIED Framework provided the map to follow regarding what to expect students to gain from the use of AI in the Pilot Project assessment.

By the submission of their final report, students should have gained an understanding of the abilities and limitations of AI and be able to use AI tools responsibly and ethically.

The process should have led to improved collaboration with others, enhanced their ability to critically evaluate AI results, and they should be able to apply AI in real-world situations.

Pilot Project part of ICMS’ ongoing response to AI

ICMS has been proactive in anticipating the implications of generative AI in higher education and has actively worked on developing a response on safeguarding academic integrity since 2022.

Policies specific to the use of AI by students and faculty have been implemented in Academic Integrity procedures and training has been provided to students and staff. The policies are in line with guidelines issued by the Tertiary Education Quality and Standards Agency (TEQSA) for the higher education sector in general.

The Pilot Project discussed here is a considered response to the growing use of AI in higher education, and a practical example of the new reality academics and students must confidently master.

Assessing the assessment: Rigorous Rubric

The marking guide for the Pilot Project assessment deviated from standard assessment rubrics owing to the need to assess mastery of the appropriate use of AI.

Weighting of 60% was given to theory and academic rigour, as would be expected in a postgraduate level assessment.

Because part of the assessment required the use of AI, this was marked using the following parameters: Proficiency and Mastery; Ability to filter and refine options; Demonstrate analytical skills; and Provide rationale for selection.

Delivery: Scaffolded Approach

Considering the ‘newness’ of the assessment protocol, as a Pilot Project, care was taken to give students enough time to fully digest the brief, rubric and tool (Chat GPT).

Over a period of three weeks, students were guided in signing up to Chat GPT, were educated on the benefits, risks and disadvantages of generative AI, and were compelled to sign the Student Submission and AI Use Declaration Form.

Group activities and workshops were held to teach and talk about the ethical and responsible use of AI tools with students  as part of the objective to develop ‘AI literacy’.

AI literacy is understood as “a set of competencies that enables individuals to critically evaluate AI technologies, communicate and collaborate effectively with AI, and use AI as a tool online, at home, and in the workplace” (Long, Blunt and Magerko; 2021, p. 2).

Recommendations from implementing lecturer

Friedmann considered the Pilot Project a success in evaluating how AI can be used in assessments. She noted that students were open to the concept of using AI in assessments, but needed to be guided in how to use the tools ethically, critically, and without academic misconduct.

She found that students came to the realisation that AI would never work as a ‘copy and paste’ solution, but that they could use what AI generated as a starting point for their own academic insights.

The benefits of understanding how to use AI effectively soon became apparent to students, in both their studies and future careers.

Specifically in the events industry, Friedmann believes the human-centred approach to experiential design is still much needed and cannot yet be replaced.

Table: Insights from students: Benefits and Disadvantages of using AI in Assessments

For more information about ICMS, click here.

References

Long, D., Blunt, T., & Magerko, B. (2021). Co-designing AI literacy exhibits for informal learning spaces. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2), 1-35.