ChatGPT and the future of educational assessment

This post by Chris Ireland looks at how ChatGPT is being viewed in education and argues that we need to embrace it rather than fight it.

Twenty years ago we were talking about the scourge of copying from the Internet and pasting it into essays. We were told that we had to change our assessment practices. A few did but most just carried on with the help of plagiarism detection tools such as Turnitin. Then came the age of the ghostwriter. Companies were openly advertising, many still do, bespoke essay writing services via the Internet, on campuses and even via advertising media (Bomford, 2016). We were again told to change our assessment practices. A few more did but many again trusted the plagiarism detection industry to deal with the issue. While they should be expected to mop up the 'off the shelf' ghosting with ease, there is something about 'bespoke ghost writing' that suggests plenty have slipped and will continue to slip through the net.

This brings us to the present day and the latest threat to educational assessment, Artificial Intelligence (AI) or more specifically Chat GPT, which first appeared in late November 2022 and which within 5 days or release had received its millionth visitor (Gleason, 2022). In short, "Chat GPT is capable of generating content across a range of subjects that is both informative and reads as if it has been written by a human" (Safari Digital, 2023). In fact, repetition of the same query generally produces responses that vary in content, while small adjustments to the query can easily change the writing style, meaning a few repetitions of the same query can generate a considerable amount of relevant information with content that is similar; often using very different phraseology. You can't say you hadn't been warned; the previous version of ChatGPT wrote an opinion piece for the Guardian in 2020 (GPT-3, 2020). 

Support for those unwilling to change is rapidly being developed. GPTZero has been quickly produced and trialed with the claim that it can tell the difference between human and AI writing (Bowman, 2023). However, the feedback concerning the success of the app from users has been mixed (Cassidy, 2023b).  Turnitin is working on tools that are able to detect the use of AI (Caren, 2022). There is even talk on social media that the AI output can be supported by an invisible watermark designed to reveal if AI was used in the production of a piece of writing (Devaney, 2023). While this is all very useful, there is a sense of always playing catch up developing here. Isn't it time for those who have not changed to bite the bullet? Education can take the easy option and move backwards. So, we have the return of the exam hall with pen and paper (Cassidy, 2023a). There are schools prohibiting access to ChatGPT (Yang, 2023). It has even been claimed that "[s]ometimes, old-fashioned methods are the best" with the call for an end to online assessment and a return entirely to in-person assessment (Sokol, 2022).

The problem with all these solutions to the advent of ChatGPT is they either want to continue with the status quo or move back in time, seeing those old-fashioned assessments through rose tinted glasses. The answer is not a return to pen and paper, a prohibition of the technology or a return to old-fashioned methods. We need to develop assessment fit for the 21st century. Simply banning "ChatGPT is a mistake" (Lipman & Distler, 2023) and not embracing it as part of the tools of the future professional is depriving students of the skills development they need. I'm old enough to remember assessment before the Internet and even before computers were involved. The skills typical assessments were often thought to be developing and the skills that many students used in those assessments were not the same; the rote learner of yesteryear is ready to make a return if we continue in that direction.

The first thing we need to do is accept that ChatGPT is here, here to stay and only going to get more sophisticated. Indeed, some academics are already thinking about how to use it for positive change (Gray, 2023). Those who have rethought assessment practices following the rise of copy and paste and ghostwriting plagiarism will probably adopt the new technology with ease, incorporate it into their assessment design and focus on how it reflects student learning. Those still looking for solutions to fight or resist AI technology will still be doing so for years to come.

So, how should educational assessment adapt to the advent of ChatGPT? I decided to ask ChatGPT. ChatGPT's first response was:

There are a few ways educational assessment could adapt to the advent of Chat GPT and other AI language models:

1. Developing new types of test items that are more difficult for AI models to complete, such as open-ended tasks that require creativity or critical thinking.

2. Using AI models as a tool for scoring test items, but also incorporating other forms of assessment such as interviews or performance tasks to provide a more well-rounded evaluation of a student's abilities.

3. Using AI models to provide real-time feedback to students during the learning process, rather than just as a tool for evaluating their performance on an assessment.

4. Incorporating ethics and AI literacy into the curriculum, so that students understand the limitations of AI models and can critically evaluate the information they are presented with.

5. Using AI models to personalize learning experiences, allowing students to learn at their own pace and in a way that is tailored to their individual needs.

It is important to note that AI models are not a replacement for human teachers and educators, but rather they can be used as a tool to enhance the educational experience.

This contains some of what I was thinking of but without concrete examples and has further ideas that had not occurred to me.

I regenerated the response to the question a further three times and it wrote a paragraph on each occasion:

One way educational assessment could adapt to the advent of Chat GPT is by incorporating natural language processing (NLP) techniques into the assessment process. For example, NLP could be used to automatically grade written responses to open-ended questions, allowing for more efficient and objective evaluation of student understanding. Additionally, Chat GPT could be used to generate sample questions or responses, providing a rich set of data for teachers to use when designing assessments. However, it's also important to keep in mind that AI-powered assessments should be used as a supplement to traditional methods, and not as a replacement for human judgement.

Educational assessment can potentially be adapted to the advent of Chat GPT by utilizing the model for tasks such as automated essay scoring and language proficiency testing. However, it is important to consider the limitations of the technology, such as its inability to fully understand context or intent, and to ensure that the assessments are fair and valid. Additionally, it may be necessary to develop new evaluation metrics to assess the performance of the model in educational settings. It is also important to consider the ethical and legal implications of using AI in educational assessment.

Educational assessment can potentially benefit from the use of Chat GPT in several ways. One potential use could be to incorporate it as a tool for automated essay scoring, as it is able to understand and generate written text. Another use could be to utilize it as a conversational agent to administer and score oral exams, such as language proficiency tests. Additionally, Chat GPT can be used to generate practice test questions and prompts to help students prepare for assessments. However, it's important to note that any implementation of Chat GPT in educational assessment would require thorough testing and validation to ensure the technology is fair and accurate.

I then asked, "If I set my students an essay to do, won't they just ask ChatGPT to write it for them?"

The response was quite balanced and contained what I was hoping to see. While the essay should not necessarily be declared dead, it needs a serious overhaul. The following are two of six paragraphs produced.

... design essay prompts that are difficult for language models to complete. For example, you could ask students to provide their own personal reflections or experiences, or to conduct original research on a topic. These types of prompts are more likely to elicit unique, student-generated content that cannot be easily replicated by a language model.

Ultimately, it is important to remember that the goal of education is to help students learn, and not just to measure their performance. Therefore, it is important to design assessments that align with the learning objectives and that are also meaningful and engaging for students.

What is of course missing from any of these responses are sources. I therefore asked ChatGPT, "What are some academic articles about this?"

This is what it responded with:

There are several academic articles that discuss the use of language models like ChatGPT in educational assessment. Here are a few examples:

1. "Automated Writing Evaluation: An Overview" by Mark D. Shermis and Jill Burstein, published in the Journal of Writing Research in 2013. This article provides an overview of automated writing evaluation (AWE) technology and its potential applications in educational assessment.

2. "Assessing the capabilities of AI-based writing evaluation tools" by Liu Yang, published in Computers & Education in 2020. This article evaluates the capabilities of AI-based writing evaluation tools and discusses the implications for their use in educational assessment.

3. "Natural Language Processing in Educational Assessment" by David A. Wiley, published in Educational Researcher in 2018. This article discusses the potential uses of natural language processing in educational assessment and the challenges that need to be addressed to make it a viable assessment tool.

4. "Exploring the Use of AI-based Automated Essay Scoring in Higher Education" by J. Scott Jordan, published in Journal of Educational Technology Development and Exchange (JETDE) in 2020. This article discusses the use of AI-based automated essay scoring in higher education, including the benefits and limitations of this technology.

You can find these articles by searching for the titles on a database such as JSTOR, or by searching for the authors' names on Google Scholar.

The first entry is very similar to a number of items written by the same authors but does not appear to exist itself. Nothing resembling the second entry could be found. A similar title was found for the third but with different authors and while a researcher named David Wiley publishes in the area of education, there is nothing resembling this in his Google Scholar entry or Educational Researcher. The final entry is similar to the third. Similar article titles, the author, and the journal all exist. The author has some publications which are in a similar area as does the journal but there is no overlap between the three. There is clearly an issue of 'fake' citations with ChatGPT (Zhao, 2022). Indeed, I regenerated the response and a further four suggestions, three of which were purported to be from JETDE, appeared and all were fake.

Notwithstanding this issue of fake sources, academics need to ensure that ChatGPT is taken into account when designing assessments; they need to do it now. As Gleason (2022) says, it is likely that assessments are being submitted by students right now that have been written by AI. Students will encounter AI writing when they enter graduate employment so should be encouraged to engage with it now (Gleason, 2022).

References

Bomford, A. (2016, May 16). The man who helps students to cheat. BBC News. https://www.bbc.co.uk/news/magazine-36276324

Bowman, F. (2023, January 9). A college student created an app that can tell whether AI wrote an essay. NPR.  https://www.npr.org/2023/01/09/1147549845/gptzero-ai-chatgpt-edward-tian-plagiarism

Caren, C. (2022 December 15). AI writing: The challenge and opportunity in front of education now. Turnitin. https://www.turnitin.com/blog/ai-writing-the-challenge-and-opportunity-in-front-of-education-now

Cassidy, C. (2023a, January 10) Australian universities to return to ‘pen and paper’ exams after students caught using AI to write essays. The Guardian. https://www.theguardian.com/australia-news/2023/jan/10/universities-to-return-to-pen-and-paper-exams-after-students-caught-using-ai-to-write-essays

Cassidy, C. (2023b, January 11) College student claims app can detect essays written by chatbot ChatGPT. The Guardian. https://www.theguardian.com/technology/2023/jan/12/college-student-claims-app-can-detect-essays-written-by-chatbot-chatgpt

Devaney, P. (2023, January 1) What is the OpenAI ChatGPT watermark? Ghacks.net. https://www.ghacks.net/2023/01/01/openai-chatgpt-watermark/

Gleason, N. (2022, December 9). Chat GPT and the rise of AI writers: how should higher education respond? THE Campus. https://www.timeshighereducation.com/campus/chatgpt-and-rise-ai-writers-how-should-higher-education-respond

GPT-3 (2020, September 8) A robot wrote this entire article. Are you scared yet, human TheGuardian. https://www.theguardian.com/commentisfree/2020/sep/08/robot-wrote-this-article-gpt-3

Gray, J. (2023, January 11). ChatGPT: How schools and universities are preparing for the artificial intelligence software. ITV.  https://www.itv.com/news/2023-01-10/how-are-schools-and-universities-preparing-for-chatgpt-artificial-intelligence

Lipman, J. & Distler, R. (2023, January 11) Schools shouldn't ban access to ChatGPT. Time. https://time.com/6246574/schools-shouldnt-ban-access-to-chatgpt/

Safari Digital, (2023). Chat GPT and SEO: An Intro to AI Content in SEO for 2023. Safari Digital. https://www.safaridigital.com.au/blog/chat-gpt-and-seo-guide/

Sokol, D. (2022, December 31) Our universities have a cheating problem – it’s time to bin online exams. The Independent.   https://www.independent.co.uk/voices/universities-online-exams-covid-cheating-b2254027.html

Yang, M. (2023, January 6) New York City Schools ban AI chatbot that writes essays and answers prompts.  The Guardian. https://www.theguardian.com/us-news/2023/jan/06/new-york-city-schools-ban-ai-chatbot-chatgpt

Zhao, A. (2022, December 22) Can AI write scholarly articles for us? – An exploratory journey with ChatGPT. Lee Shau Kee Library: Research Bridge. https://library.hkust.edu.hk/sc/chatgpt-write-articles/