On Automatic Identification of Retail Products in Images of Racks in the Supermarkets

Abstract

An image of a rack in a supermarket displays a number of retail products. The identification and localiza- tion of these individual products from the images of racks is a challenge for any machine vision system. In this thesis, we aim to address this problem and suggest a set of computer vision based solutions for automatic identification of these retail products. We design a novel classifier that differentiates the sim- ilarly looking yet non-identical (fine-grained) products for improving the performance of our machine vision system. The proposed fine-grained classifier simultaneously captures both object-level and part- level (image of an object consists of multiple parts or image patches) cues of the products for accurately distinguishing the fine-grained products. A graph-based non-maximal suppression strategy is proposed that selects a winner region proposal among a group of proposals representing a product. This solves an important bottleneck of conventional greedy non-maximal suppression algorithm for disambiguation of overlapping region proposals generated in an intermediate step of our proposed system. We initiate the solution of the problem of automatic product identification by developing an end-to-end annotation-free machine vision system for recognition and localization of products on the rack. The proposed system in- troduces a novel exemplar-driven region proposal strategy that overcomes the shortcomings of traditional exemplar-independent region proposal schemes like selective window search. Finally, we find the empty spaces (or gaps between products) in each shelf of any rack by creating a graph of superpixels for the rack. We extract the visual features of superpixels from our graph convolutional and Siamese networks. Subsequently, we send the graph along with the features of superpixels to a structural support vector machine for discovering the empty spaces of the shelves. The efficacy of the proposed approaches are established through various experiments on our In-house dataset and three publicly available benchmark datasets: Grozi-120 [Merler et al., IEEE CVPR 2007, 1-8], Grocery Products [George et al., Springer ECCV 2014, 440-455], and WebMarket [Zhang et al., Springer ACCV 2007, 800-810].