Can artificial intelligence create simulation scenarios on par with humans? a comparative analysis of scenarios written by humans and AI, using the case of «Fire in the operating room»

A blind comparative pilot study was conducted to evaluate the quality of simulation scenarios generated by generative artificial intelligence (AI) models compared to human-generated scenarios. The objectives of the study were to test the feasibility of generating scenarios by AI models, compare their quality with human-generated scenarios, and assess the ability of experts to determine authorship. Three models (Grok, ChatGPT and DeepSeek) were used to generate scenarios based on a standardized prompt, and the ROSOMED competition case served as the reference standard. Five independent experts evaluated four scenarios using the original scale of assessment of a simulation scenario (SASS). The ChatGPT scenario (average score of 3.4) outperformed the human scenario (2.9). The AI evaluator generally ranked the scenarios similarly to the human experts, but it showed bias towards its own work when self-evaluating.

1. Al-Elq A. H. Simulation-based medical teaching and learning. J Family Community Med. 2010 Jan; Vol. 17 (1). P. 35–40. DOI: 10.4103/1319-1683.68787. PMID: 22022669; PMCID: PMC3195067

2. Issenberg S. B., McGaghie W. C., Petrusa E. R., Lee Gordon D., Scalese R. J. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review // Med Teach. 2005 Jan. Vol. 27 (1). P. 10–28. DOI: 10.1080/01421590500046924. PMID: 16147767.

3. Kasneci E., Sessler K., Küchemann S., Bannert M., Dementieva D., Fischer F., Gasser U., Groh G., Günnemann S., Hüllermeier E., Krusche S., Kutyniok G., Michaeli T., Nerdel C., Pfeffer J., Poquet O., Michael Sailer M., Schmidt A., Seidel T., Stadler M., Kasneci G. ChatGPT for good? On opportunities and challenges of large language models for education // Learning and Individual Differences. 2023. Vol. 103. № 102274. DOI: 10.1016/j.lindif.2023.102274

4.