Neural Machine Translation for Indian Sign Language

Abstract

Sign languages being the primary language of the deaf community, researchers from many elds have been working in this domain from the past two decades. Until now, the majority of the work was in Sign Language Recognition. And only recently, few methods on Sign Language Translation have been developed, but even today, there does not exist any work on Indian Sign Language Translation. This work aims to translate Indian sign language videos to their corresponding spoken Indian English sentences. In this work, we are publicly releasing the rst of its kind Indian Sign Language Translation dataset, namely, the ISI-ISL-DDNEWS-2020T that we collected and annotated. Our dataset has >3 Million sign language frames, which translate to >93 Thousand words made out of >6 Thousand vocabulary words in spoken Indian English language. We also formalize a neural machine translation system trainable end-to-end for Indian Sign Language and benchmark on the said dataset. The model jointly learns the spatial & temporal relationship, underlying language model, and the sign & spoken language alignment. This baseline model gives the translation a BLEU-4 score of 4.02.