Using Eye-Gaze to Evaluate Neural Attention

Abstract

The ability to selectively concentrate on areas of interest while ignoring the rest is termed as attention in human beings. This ability has played a key role in survival as well as information processing. Neural Attention is said to be an e ort to bring similar action of selectively concentrating areas of relevance in deep neural networks. This simple yet powerful concept has attracted a lot of research in recent years, yielding breakthrough results in Natural Language Processing (NLP) problems and main stream Computer Vision problems such as Image Caption Generation, Neural Machine Translation (NMT), Visual Question Answering (VQA), Action Recognition, Image Segmentation, etc. Only few works has been there articulating the relation between human attention and machine attention. Some recent e orts suggest that automatically learned attention maps can capture informative parts of an input signal and highlight human sensible regions of interest. Also, as the neural attention gets better, so is the performance of the network. However, there are no formal way of bench marking how good the learned attention is in a network. This seems necessary since visualizing attention as a means for logical correctness of the network is common. Since eye-gaze can better capture human visual reasoning, With this work, we are investigating how well neural attention compares with the visual grounding given by human cognitive modality on VQA tasks.