How Egyptian is AI? A comparative assessment of ChatGPT and Google Gemini v Human English Translations of Egyptian cultural references

Abstract

This study evaluates the effectiveness of AI-generated translation for literary Arabic to English texts. It focuses on the translations generated by Chat GPT (GPT) and Google Gemini (GG) for 100 Arabic cultural references (CRs) extracted from the Egyptian award-winning novel  بيت الديب by Ezzat El Kamhawy (2010) and rendered into English by Nancy Roberts as House of the Wolf (2013) as the reference translation (RT). The translatability of CRs has been a major problem for human translators, let alone AI translation tools. In this study, the AI-generated translations were assessed first using BLEU, METEOR, and chrF metrics vis-à-vis the human translation (HT), then both AI-translations and HT were analysed using Pedersen’s (2011) taxonomy of strategies for assessing the translation of CRs. The scores by the three metrics indicate that BLEU produces the lowest results, while METEOR and chrF scores are significantly higher, and that the scores for GG are slightly better than GPT. The analysis of the strategies indicates that both GPT and GG relied heavily on source language-oriented strategies (SO), while HT adopted target language-oriented strategies (TO) more. GPT used TO strategies 24 times for GG’s 17 times only, giving its translations an edge over GG. Further, while GPT and GG generated 9 mistranslations between them, HT produced zero.  The scores yielded by the three metrics tend to overall confirm the statistics for the strategies.