Estimating the Quality of Translated Texts using Back Translation and Resource Description Framework

Abstract

Natural language translation is an AI-complete problem meaning that if a machine can translat as well as any human being, the machine could be said to be as intelligent as any human being. Once the translation is done, an assessment of the translated text is required for evaluating the translation quality. The goal is to have automatic metrics which can measure semantic equivalence of translated text with the original text having high correlation with human judgment score to save the time and effort in the evaulation process. Existing metrics predominantly focus on syntax, which often fails to capture the intended meaning of the sentences. Our work aims to combine both syntactic and semantic information to better capture the meaning of sentences. In our research we are treating translation as a black box, and focusing exclusively on the quality of the translation once it is completed. After the translation is completed, how can we effectively estimate the quality of translated texts, where back-translation is usually available and/or recommended for sensitive documents. This work proposes a novel metric, GATE11 , for translation quality estimation task, leveraging the Resource Description Framework (RDF) to encode both semantic and syntactical information of the original and back-translated sentences into RDF graphs. The distance between these graphs is measured to get the semantic similarity score to assess the quality of the translation. Unlike traditional metrics like BLEU and METEOR, our approach is reference-less, capturing both semantic and syntactical information for a comprehensive assessment of translation quality. Our results correlate better with human judgment, giving a better Pearson correlation (0.357) as compared to BLEU (0.200), thereby showing ~70% improvement over BLEU. Our research shows that, in the field of translation evaluation, existing resources like back-translation and RDF could be useful. We also propose novel approach of bi-directional entailment among others for measuring the faithfulness of translated texts. Using these approaches, we are able to achieve considerable accuracy on our corpus. To the best of our knowledge, this is the first effort to use entailment classification and RDF schema representation on back-translated texts to automatically assess the quality of professionally translated texts.