Comparative analysis of Serbian-to-English translation: a study of LLM-based system vs university students’ performance

Апстракт

This paper examines the quality of Serbian-to-English translations produced by university students and by a large language model, represented by ChatGPT. The study is situated within the broader context of the increasing use of artificial intelligence and large language models in translation practice and language education. Although AI-assisted translation systems show considerable progress in grammatical accuracy and fluency, their ability to interpret cultural references, pragmatic meaning, idiomatic expressions, and stylistic nuance remains open to further investigation. The main aim of the study was to compare the translation performance of ChatGPT and second-year university students in an academic setting. More specifically, the research sought to identify the strengths and limitations of each translation approach across formal, linguistic, pragmatic, and stylistic dimensions. The study was based on identical Serbian-to-English translation tasks assigned to ChatGPT and to students from the Department of English Language at Singidunum University. A corpus of 50 student translations and one AI-generated translation was analyzed using a Translation Quality Index. This index combined four weighted components: Quantitative Score, Linguistic Score, Pragmatic Score, and Stylistic Score. The evaluation included both quantitative error-based analysis and qualitative assessment of meaning accuracy, contextual appropriateness, cultural localization, fluency, and readability. The results showed that overall translation quality was comparable, with students achieving a final TQI score of 85.08 and ChatGPT achieving 84.77. ChatGPT performed better in formal and linguistic accuracy, particularly in grammar, consistency, and sentence-level correctness. However, students outperformed ChatGPT in pragmatic and stylistic categories, especially in culturally appropriate choices, idiomatic expression, register, and naturalness. These findings suggest that AI systems can effectively support translation practice, but they should not be viewed as replacements for human translators. The study supports a hybrid model in which AI contributes formal precision and efficiency, while human translators provide cultural awareness, interpretive judgment, and stylistic sensitivity.