Computer Vision Based Planogram Compliance Evaluation
Source abstract
Abstract
Arranging products in stores according to planograms, optimized product arrangement maps, is an important sales enabler and necessary for keeping up with the highly competitive modern retail market. Key benefits of planograms include increased efficiency, maximized retail store space, increased customer satisfaction, visual appeal, and increased revenue. The planograms are realized into product arrangements by humans, a process that is prone to mistakes. Therefore, for optimal merchandising performance, the planogram compliance of the arrangements needs to be evaluated from time to time. We investigate utilizing a computer vision problem setting—retail product detection—to automate planogram compliance evaluation. Retail product detection comprises product detection and classification. The detected and classified products can be compared to the planogram in order to evaluate compliance. In this paper, we propose a novel retail product detection pipeline combining a Gaussian layer network product proposal generator and domain invariant hierarchical embedding (DIHE) classifier. We utilize the detection pipeline with RANSAC pose estimation for planogram compliance evaluation. As the existing metrics for evaluating the planogram compliance evaluation performance assume unrealistically that the test image matches the planogram, we propose a novel metric, called normalized planogram compliance error (EPC), for benchmarking real-world setups. We evaluate the performance of our method with two datasets: the only open-source dataset with planogram evaluation data, GP-180, and our own dataset collected from a large Nordic retailer. Based on the evaluation, our method provides an improved planogram compliance evaluation pipeline, with accurate product location estimation when using real-life images that include entire shelves, unlike previous research that has only used images with few products. Our analysis also demonstrates that our method requires less processing time than the state-of-the-art compliance evaluation methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Industrial Vision Systems and Defect Detection · Image and Object Detection Techniques · Visual Attention and Saliency Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
RELEX Solutions, 00230 Helsinki, Finland
Department of Computer Science, University of Helsinki, 00014 Helsinki, Finland
Funding
Research Council of Finland’s Flagship program: Finnish Center for Artificial Intelligence FCAI · University of Helsinki
References (39)
Show source-provided references
1. Using image analytics to monitor retail store shelves10.1147/JRD.2015.2394513 ↗
2. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press. Available online: http://www.deeplearningbook.org.
3. Davies, E., and Turk, M.A. (2022). Advanced Methods and Deep Learning in Computer Vision, Academic Press.
4. Lin, T.Y., Dollár, P., Girshick, R.B., He, K., Hariharan, B., and Belongie, S.J. (2017, January 21–26). Feature Pyramid Networks for Object Detection. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
5. Girshick, R.B., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
6. Redmon, J., Divvala, S.K., Girshick, R.B., and Farhadi, A. (2016, January 27–30). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
7. ImageNet Large Scale Visual Recognition Challenge10.1007/s11263-015-0816-y ↗
8. Microsoft COCO: Common Objects in Context
9. A comprehensive survey on computer vision based approaches for automatic identification of products in retail store10.1016/j.imavis.2019.03.005 ↗
10. Merler, M., Galleguillos, C., and Belongie, S.J. (2007, January 18–23). Recognizing Groceries in situ Using in vitro Training Data. Proceedings of the 2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007), Minneapolis, MI, USA.10.1109/CVPR.2007.383486 ↗
11. Tonioni, A., Serra, E., and di Stefano, L. (2018, January 12–14). A deep learning pipeline for product recognition on store shelves. Proceedings of the IEEE International Conference on Image Processing, Applications and Systems, IPAS 2018, Sophia Antipolis, France.10.1109/IPAS.2018.8708890 ↗
12. Grocery product detection and recognition10.1016/j.eswa.2017.02.050 ↗
13. Product Recognition in Store Shelves as a Sub-Graph Isomorphism Problem
14. U-PC: Unsupervised Planogram Compliance
15. Karlinsky, L., Shtok, J., Tzur, Y., and Tzadok, A. (2017, January 21–26). Fine-Grained Recognition of Thousands of Object Categories with Single-Example Training. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA.10.1109/CVPR.2017.109 ↗
16. Boll, S., Lee, K.M., Luo, J., Zhu, W., Byun, H., Chen, C.W., Lienhart, R., and Mei, T. (2018, January 22–26). Fine-Grained Grocery Product Recognition by One-Shot Learning. Proceedings of the 2018 ACM Multimedia Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea.
17. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
18. Graph-based non-maximal suppression for detecting products on the rack10.1016/j.patrec.2020.09.023 ↗
19. Qiao, S., Shen, W., Qiu, W., Liu, C., and Yuille, A.L. (2017, January 22–29). ScaleNet: Guiding Object Proposal Generation in Supermarkets and Beyond. Proceedings of the IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy.10.1109/ICCV.2017.199 ↗
20. Goldman, E., Herzig, R., Eisenschtat, A., Goldberger, J., and Hassner, T. (2019, January 16–20). Precise Detection in Densely Packed Scenes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA.10.1109/CVPR.2019.00537 ↗
21. Kant, S. (2020, January 7–10). Learning Gaussian Maps for Dense Object Detection. Proceedings of the 31st British Machine Vision Conference Virtual Event, UK.
22. Pan, X., Ren, Y., Sheng, K., Dong, W., Yuan, H., Guo, X., Ma, C., and Xu, C. (2020, January 13–19). Dynamic Refinement Network for Oriented and Densely Packed Object Detection. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA.10.1109/CVPR42600.2020.01122 ↗
23. Domain invariant hierarchical embedding for grocery products recognition10.1016/j.cviu.2019.03.005 ↗
24. Saran, A., Hassan, E., and Maurya, A.K. (2015, January 18–22). Robust visual analysis for planogram compliance problem. Proceedings of the 14th IAPR International Conference on Machine Vision Applications, MVA 2015, Miraikan, Tokyo, Japan.10.1109/MVA.2015.7153257 ↗
25. SSD: Single Shot MultiBox Detector
26. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography10.1145/358669.358692 ↗
27. Deep Learning for Generic Object Detection: A Survey10.1007/s11263-019-01247-4 ↗
28. Recognizing Products: A Per-exemplar Multi-label Image Classification Approach
29. Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E.B., and Garnett, R. (2019, January 8–14). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, Vancouver, BC, Canada.
30. Focal Loss for Dense Object Detection10.1109/TPAMI.2018.2858826 ↗
31. Bengio, Y., and LeCun, Y. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA. Conference Track Proceedings.
32. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
33. Bengio, Y., and LeCun, Y. (2016, January 2–4). Particular object retrieval with integral max-pooling of CNN activations. Proceedings of the 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico. Conference Track Proceedings.
34. Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., and Wu, Y. (2014, January 23–28). Learning Fine-Grained Image Similarity with Deep Ranking. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA.10.1109/CVPR.2014.180 ↗
35. Briechle, K., and Hanebeck, U.D. (2001, January 19). Template matching using fast normalized cross correlation. Proceedings of the Optical Pattern Recognition XII. International Society for Optics and Photonics, Orlando, FL, USA.
36. The Pascal Visual Object Classes (VOC) Challenge10.1007/s11263-009-0275-4 ↗
37. The OpenCV Library
38. Benchmark for Generic Product Detection: A Low Data Baseline for Dense Object Detection
39. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022.