To the best of our knowledge, this is the first ever large scale study on Bengali-English machine translation. We release the segmenter, parallel corpus, and the evaluation set, thus elevating Bengali from its low-resource status.
We also evaluate on a new test set of 1000 pairs made with extensive quality control.
Training on neural models, we achieve an improvement of more than 9 BLEU score over previous approaches to Bengali-English machine translation. With the segmenter and the two methods combined, we compile a high-quality Bengali-English parallel corpus comprising of 2.75 million sentence pairs, more than 2 million of which were not available before. In this work, we build a customized sentence segmenter for Bengali and propose two novel methods for parallel corpus creation on low-resource setups: aligner ensembling and batch filtering. Most publicly available parallel corpora for Bengali are not large enough and have rather poor quality, mostly because of incorrect sentence alignments resulting from erroneous sentence segmentation, and also because of a high volume of noise present in them. Publisher = "Association for Computational Linguistics",Ībstract = "Despite being the seventh most widely spoken language in the world, Bengali has received much less attention in machine translation literature due to being low in resources. = "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for nglish Machine Translation",īooktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
0 kommentar(er)
