Air-Writing Recognition

Abstract

Air-writing is the act of tracing characters or words in free space with a fingertip, recorded by a camera, giving a touch-free input modality for smart displays, augmented and virtual reality, and assistive interfaces. It is difficult because the finger never lifts: connecting strokes join adjacent letters with no pen-up signal to mark boundaries, and the same word varies widely in scale, position, and slant across writers. The WiTA benchmark of Kim et al. provides a large, person-disjoint dataset and a baseline that treats each clip as RGB video, recognised by a spatio-temporal 3D residual network trained with a CTC objective, reaching a character error rate (CER) of 0.292 on the English subset. The main goal of this dissertation was to improve on this error rate, which we achieve: we replace raw video with an explicit fingertip-trajectory sequence extracted from hand landmarks, fed to a Conformer encoder with a joint CTC/attention head. The resulting system attains a test CER of 0.219, improving on the published 0.292 of Kim et al. and 0.299 of Tan et al. by 15–27% relative.