Sentence Alignment Using Local and Global Information

Publication date: Available online 1 April 2016
Source:Computer Speech & Language
Author(s): Hamed Zamani, Heshaam Faili, Azadeh Shakery
Parallel corpora are essential resources for statistical machine translation (SMT) and cross language information retrieval (CLIR) systems. Creating parallel corpora is highly expensive in terms of both time and cost. In this paper, we propose a novel approach to automatically extract parallel sentences from aligned documents. To do so, we first train a Maximum Entropy binary classifier to compute the local similarity between each two sentences in different languages. To consider global information (e.g., the position of sentence pairs in the aligned documents), we define an objective function to penalize the cross alignments and then propose an integer linear programming approach to optimize the objective function. In our experiments, we focus on English and Persian Wikipedia articles. The experimental results on manually aligned test data indicate that the proposed method outperforms the baselines, significantly. Furthermore, the extrinsic evaluations of the corpus extracted from Wikipedia on both SMT and CLIR systems demonstrate the quality of the extracted parallel sentences. In addition, Experiments on the English-German language pair demonstrate that the proposed ILP method is a language-independent sentence alignment approach. The extracted English-Persian parallel corpus is freely available for research purposes.



from Speech via a.lsfakia on Inoreader http://ift.tt/1q9EdaT
via IFTTT