GRADERS OF THE FUTURE: COMPARING THE CONSISTENCY AND ACCURACY OF GPT4 AND PRE-SERVICE TEACHERS IN PHYSICS ESSAY QUESTION ASSESSMENTS

Yubin Xu; Lin Liu; Jianwen Xiong; Guangtian Zhu

Error message

Strict warning: Declaration of csl_names::render() should be compatible with csl_rendering_element::render($data, $mode = NULL) in module_load_include() (line 330 of /home/scientia/domains/scientiasocialis.lt/public_html/jbse/includes/module.inc).

Strict warning: Declaration of csl_date::render() should be compatible with csl_rendering_element::render($data, $mode = NULL) in module_load_include() (line 330 of /home/scientia/domains/scientiasocialis.lt/public_html/jbse/includes/module.inc).

Strict warning: Declaration of csl_date_part::render() should be compatible with csl_rendering_element::render($data, $mode = NULL) in module_load_include() (line 330 of /home/scientia/domains/scientiasocialis.lt/public_html/jbse/includes/module.inc).

Strict warning: Declaration of csl_number::render() should be compatible with csl_rendering_element::render($data, $mode = NULL) in module_load_include() (line 330 of /home/scientia/domains/scientiasocialis.lt/public_html/jbse/includes/module.inc).

Strict warning: Declaration of csl_group::render() should be compatible with csl_rendering_element::render($data, $mode = NULL) in module_load_include() (line 330 of /home/scientia/domains/scientiasocialis.lt/public_html/jbse/includes/module.inc).

Strict warning: Declaration of csl_layout::render() should be compatible with csl_rendering_element::render($data, $mode = NULL) in module_load_include() (line 330 of /home/scientia/domains/scientiasocialis.lt/public_html/jbse/includes/module.inc).

Strict warning: Only variables should be passed by reference in theme_biblio_tabular() (line 276 of /home/scientia/domains/scientiasocialis.lt/public_html/jbse/modules/biblio/includes/biblio_theme.inc).

GRADERS OF THE FUTURE: COMPARING THE CONSISTENCY AND ACCURACY OF GPT4 AND PRE-SERVICE TEACHERS IN PHYSICS ESSAY QUESTION ASSESSMENTS

Title	GRADERS OF THE FUTURE: COMPARING THE CONSISTENCY AND ACCURACY OF GPT4 AND PRE-SERVICE TEACHERS IN PHYSICS ESSAY QUESTION ASSESSMENTS
Publication Type	Journal Article
Year of Publication	2025
Authors	Xu, Y, Liu, L, Xiong, J, Zhu, G
Journal	Journal of Baltic Science Education
Volume	24
Issue	1
Start Page	187-207
Pagination	Continuous
Date Published	February/2025
Type of Article	Original article
ISSN	1648-3898
Other Numbers	E-2538-7138
Keywords	AI grader, Human graders, Physics essay question assessment
Abstract	As the development and application of large language models (LLMs) in physics education progress, the well-known AI-based chatbot ChatGPT4 has presented numerous opportunities for educational assessment. Investigating the potential of AI tools in practical educational assessment carries profound significance. This study explored the comparative performance of ChatGPT4 and human graders in scoring upper-secondary physics essay questions. Eighty upper-secondary students’ responses to two essay questions were evaluated by 30 pre-service teachers and ChatGPT4. The analysis highlighted their scoring consistency and accuracy, including intra-human comparisons, GPT grading at different times, human-GPT comparisons, and grading variations across cognitive categories. The intraclass correlation coefficient (ICC) was used to assess consistency, while accuracy was illustrated through Pearson correlation coefficient analysis with expert scores. The findings reveal that while ChatGPT4 demonstrated higher consistency in scoring, human scorers showed superior accuracy in most instances. These results underscore the strengths and limitations of using LLMs in educational assessments. The high consistency of LLMs can be valuable in standardizing assessments across diverse educational contexts, while the nuanced understanding and flexibility of human graders are irreplaceable in handling complex subjective evaluations.
URL	https://oaji.net/articles/2023/987-1740921781.pdf
DOI	10.33225/jbse/25.24.187
Refereed Designation	Refereed
Full Text