Supporting students to develop digital literacy through integration of generative AI into authentic assessment redesign

Research note

By Ruth Friedmann, ICMS Lecturer

Introduction

The early adoption of technology to develop digital literacy of students by incorporating artificial intelligence (AI) into assessments has been argued in the literature as being critical in equipping graduates with the skills needed to thrive in a rapidly changing world. As an industry-focused higher education provider, ICMS promotes the development of AI literacy by fostering the integration of AI tools into learning and teaching. This is through human-centred design, critical thinking, risk management and ethical decision-making.

This research note provides an overview of the redesign process of an assessment, and associated learning and teaching methods, in EVT603 Impacts and Strategic Planning of International Events, a specialisation subject in the Master of Event Management course at ICMS, incorporating the use of generative AI (GenAI). It highlights the innovation introduced to learning and teaching, drawing relevance to professional practice, and builds on the evolving pedagogical approaches to the use of AI in higher education [1],[2],[3],[4].

Integrating GenAI into the learning and teaching process was a core focus of this study, in line with the ICMS’s graduate capabilities and the Artificial Intelligence in Education (AIED) Framework [5]. Specifically, the redesign emphasised developing the ability of students to use GenAI tools effectively and apply critical evaluation, whilst maintaining focus on ethical considerations for responsible and practical applications in their careers.

Rationale for redesign

The events industry is undergoing a significant transformation driven by the integration of AI technologies. Pervasive adoption of AI impacts various aspects of event planning, from initial stages of concept generation to attendee experience optimisation, with research identifying a growing skills gap requiring higher education institutions to adopt a future-focused approach in preparing job-ready graduates [6],[7],[8],[9].

The importance of developing students’ uniquely human skill of evaluative judgement is urgent, with particular emphasis needed on integrating strategies for assessing the quality of AI outputs and processes [1]. Because of this, opportunities in learning and teaching are required to develop students’ AI literacy skills to be able to test the rigour of AI-generated outputs by applying ethics and critical analysis and making recommendations based on evidence and key theory [10].

Process of redesign

The assessment redesign process in this study addressed key principles drawn from the literature [4],[2] as outlined below:

1. Refinement of the assessment task to closely match the subject learning outcomes.
2. Creation of comprehensive instructions and requirements to ensure clarity and consistency.
3. Development of a template for delivery to address the lack of existing exemplars.
4. Design of teaching materials (teaching plans, presentations, activity worksheets and supporting resources) to provide a scaffolded learning experience.
5. Development of ten formative tasks to support delivery.

Innovation and integration of GenAI

Supporting students in developing digital navigation capabilities through authentic engagement with GenAI into the classroom experience, while using critical thinking and judgement was central to the assessment redesign process [1],[5]. In this assessment, students were required to develop and justify an event concept for recommendation. Emphasis was placed on integrating AI meaningfully to achieve the learning outcomes with a focus on critical analysis and ethical decision-making.

The events industry remains human focused with GenAI viewed as a powerful tool now available to enhance delivery.

To address the challenge of differentiating human-created work from AI-generated content, the redesign incorporated stages where students gathered evidence of their critical engagement with AI and learning journey towards final submission. Students were encouraged to critically evaluate GenAI output by utilising theories and models covered in the subject to support their rationale. The redesign also facilitated collaborative work between students and AI by outlining specific acceptable ways for this interaction. Students were encouraged to articulate and reflect on the role of AI in their work including considerations of biases, limitations, and potential inaccuracies in the output.

Recommendations

It remains essential that students gain proficiency in using AI tools and develop strong critical thinking skills, ethical awareness, and the ability to evaluate AI technologies [5]. To achieve this, the following strategies can be incorporated into learning and teaching practices:

1. Ongoing industry liaison: Continuous industry consultation to ensure academic currency and understanding of AI use, expectations and requirements for graduate outcomes.
2. Integration of case studies and real-world applications: Case studies involving use of AI in the hospitality, tourism, and events industries can help students critically analyse various impacts of AI and identify ethical considerations and challenges.
3. Guest lectures from industry experts: Professionals can be invited to share with students their experiences, insights, and potential ethical dilemmas they have encountered, along with current trends and usage of AI.
4. Graduate capabilities focus: Incorporate scholarly research, data analysis, critical evaluation, AI ethics and its application in professional practice within learning and teaching design with a focus on achievement of graduate capabilities.

While most students may possess a user experience advantage regarding specific AI tools, their understanding of critical reflection, evaluation of limitations, accuracy, biases, and ethical concerns often lags. Therefore, AI literacy for academic staff remains critical to ensure continued relevance and engaging delivery, with co-creation through ongoing student feedback and lecturer observations helping to equip students for their future.

Conclusion

This study underscores the importance of embedding a continuous improvement approach to the learning and teaching experience through early adoption of emerging technologies and supporting students to become industry ready with a future-forward thinking approach. With the continued integration of AI tools and human-centred design, critical thinking, risk management and ethics into assessments and the learning and teaching process, students will be able to build and maintain the necessary skills for successful navigation of an increasingly an AI-driven environment and society.

Acknowledgements

This project was funded by the International College of Management Sydney (ICMS) Artificial Intelligence in Education (AIED) Scholarship Grant between January and May 2024.

References:

[1] Bearman, M., Tai, J., Dawson, P., Boud, D., & Ajjawi, R. (2024). Developing evaluative judgement for a time of generative artificial intelligence. Assessment & Evaluation in Higher Education, 1–13. https://doi.org/10.1080/02602938.2024.2335321

[2] Hillier, M. (2023, March 30). A proposed AI literacy framework. TECHE. https://teche.mq.edu.au/2023/03/a-proposed-ai-literacy-framework/

[3] Long, D., Blunt, T., & Magerko, B. (2021). Co-designing AI literacy exhibits for informal learning spaces. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2), 1-35. https://doi.org/10.1145/3476034

[4] Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021). Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2, 100041. https://doi.org/10.1016/j.caeai.2021.100041

[5] International College of Management Sydney. (2024b). Artificial intelligence in education (AIED) framework. International College of Management Sydney. https://policies.icms.edu.au/artificial-intelligence-in-education-aied-framework/

[6] Melbourne Convention and Exhibition Centre. (2023). The exciting role of AI in transforming event planning. https://www.mcec.com.au/stories-and-ideas/ai-in-event-planning

[7] Gurchiek, K. (2024). Employers want new grads with AI experience-knowledge. SHRM. https://www.shrm.org/topics-tools/news/technology/employers-want-new-grads-with-ai-experience–knowledge.

[8] Jones, T. (2019). New research shows growing technology skills gap in events industry. https://www.eventsforce.com/news/new-research-shows-growing-technology-skills-gap-in-events-industry/.

[9] Tech Council of Australia. (2024). Meeting the AI Skills Boom 2024. https://techcouncil.com.au/wp-content/uploads/Meeting-the-AI-Skills-Boom-2024.v2.pdf

[10] Long, P., & Magerko, B. (2020). Digital literacy: Preparing students for the workplace. Educause Review, 55(2), 1-12. https://doi.org/10.1145/3313831.3376727