journal article·2025-06-05Open access

Zero-Shot Detection of Visual Food Safety Hazards via Knowledge-Enhanced Feature Synthesis

Lanting Guo, Xiaoyu Hu, Wenhe Liu, Yang Liu

Applied Sciences· Volume 15· Issue 11

Abstract

Automated food safety inspection systems rely heavily on the visual detection of contamination, spoilage, and foreign objects in food products. Current approaches typically require extensive labeled training data for each specific hazard type, limiting generalizability to novel or rare safety issues. We propose a zero-shot detection framework for visual food safety hazards that enables the identification of previously unseen contamination types without requiring explicit training examples. Our approach adapts and extends the Knowledge-Enhanced Feature Synthesizer (KEFS) methodology to the food safety domain by constructing a specialized knowledge graph that encodes visual safety attributes and their correlations with food categories. We introduce a Food Safety Knowledge Graph (FSKG) that models the relationships between 26 food categories and 48 visual safety attributes (e.g., discoloration, mold patterns, foreign material characteristics) extracted from food safety databases and expert knowledge. Using this graph as the prior knowledge, our system synthesizes discriminative visual features for unseen hazard classes through a multi-source graph fusion module and region feature diffusion model. Experiments on our newly constructed Food Safety Visual Hazards (FSVH) dataset demonstrate that our approach achieves 63.7% mAP in zero-shot hazard detection, outperforming state-of-the-art general zero-shot detection methods by 6.9%. Furthermore, our framework demonstrates robust generalization to fine-grained novel hazard categories while maintaining high detection performance (59.8% harmonic mean) in generalized zero-shot scenarios where both seen and unseen hazards may occur simultaneously. This work represents a significant advancement toward automated, generalizable food safety inspection systems capable of adapting to emerging visual hazards without a costly retraining process.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Spectroscopy and Chemometric Analyses

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

The Department of Food Science and Human Nutrition, University of Illinois Urbana-Champaign, Champaign, IL 61801, USA

Department of Chemical Engineering and Materials Science, Stevens Institute of Technology, Hoboken, NJ 07030, USA

School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213, USA

Department of Computer Science, Worcester Polytechnic Institute, Worcester, MA 01609, USA

References (61)

Show source-provided references
2. A systematic review of consumer perceptions of food fraud and authenticity: A European perspective10.1016/j.tifs.2019.10.005 ↗
3. Zhang, J., Huang, H., Song, G., Huang, K., Luo, Y., Liu, Q., He, X., and Cheng, N. (2022). Intelligent biosensing strategies for rapid detection in food safety: A review. Biosens. Bioelectron., 202.10.1016/j.bios.2022.114003 ↗
4. Hyperspectral imaging technology for rapid detection of various microbial contaminants in agricultural and food products10.1016/j.tifs.2015.08.001 ↗
5. Applications of computer vision for assessing quality of agri-food products: A review of recent research advances10.1080/10408398.2013.873885 ↗
6. Applications of machine learning techniques for enhancing nondestructive food quality and safety detection10.1080/10408398.2022.2131725 ↗
7. Zero-shot learning—A comprehensive evaluation of the good, the bad and the ugly10.1109/TPAMI.2018.2857768 ↗
8. Bansal, A., Sikka, K., Sharma, G., Chellappa, R., and Divakaran, A. (2018, January 8–14). Zero-shot object detection. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01246-5_24 ↗
9. Zhu, P., Wang, H., and Saligrama, V. (2020, January 14–19). Don’t even look once: Synthesizing features for zero-shot detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01171 ↗
10. Rahman, S., Khan, S., and Barnes, N. (2020, January 7–12). Improved visual-semantic alignment for zero-shot object detection. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6868 ↗
11. A Comprehensive Review of Advanced Deep Learning Approaches for Food Freshness Detection10.1007/s12393-024-09385-3 ↗
12. Synthesizing knowledge-enhanced features for real-world zero-shot food detection10.1109/TIP.2024.3360899 ↗
13. Automated ribotyping for the identification and characterization of foodborne clostridia10.4315/0362-028X-69.12.2970 ↗
14. The review of food safety inspection system based on artificial intelligence, image processing, and robotic10.1590/fst.35421 ↗
15. Sun, D.W. (2016). Computer Vision Technology for Food Quality Evaluation, Academic Press.
16. Combining optical spectroscopy and machine learning to improve food classification10.1016/j.foodcont.2021.108342 ↗
17. Multispectral detection of organic residues on poultry processing plant equipment based on hyperspectral reflectance imaging technique10.1016/j.compag.2007.03.008 ↗
18. Food image recognition and food safety detection method based on deep learning10.1155/2021/1268453 ↗
19. Son, G.J., Kwak, D.H., Park, M.K., Kim, Y.D., and Jung, H.C. (2021). U-Net-based foreign object detection method using effective image acquisition system: A case of almond and green onion flake food process. Sustainability, 13.10.3390/su132413834 ↗
20. Algorithm of strawberry disease recognition based on deep convolutional neural network10.1155/2021/6683255 ↗
21. Leveraging artificial intelligence and advanced food processing techniques for enhanced food safety, quality, and security: A comprehensive review10.1007/s42452-025-06472-w ↗
22. Attribute-based classification for zero-shot visual object categorization10.1109/TPAMI.2013.140 ↗
23. Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, T. (2013). Devise: A deep visual-semantic embedding model. Adv. Neural Inf. Process. Syst., 26.
24. Transductive multi-view zero-shot learning10.1109/TPAMI.2015.2408354 ↗
25. Xian, Y., Lorenz, T., Schiele, B., and Akata, Z. (2018, January 18–23). Feature generating networks for zero-shot learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00581 ↗
26. Schonfeld, E., Ebrahimi, S., Sinha, S., Darrell, T., and Akata, Z. (2019, January 15–20). Generalized zero-and few-shot learning via aligned variational autoencoders. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00844 ↗
27. Rahman, S., Khan, S., and Porikli, F. (2018, January 2–6). Zero-shot object detection: Learning to simultaneously recognize and localize novel concepts. Proceedings of the Asian Conference on Computer Vision, Perth, Australia.10.1007/978-3-030-20887-5_34 ↗
28. Hayat, N., Hayat, M., Rahman, S., Khan, S., Zamir, S.W., and Khan, F.S. (December, January 30). Synthesizing the unseen for zero-shot object detection. Proceedings of the Asian Conference on Computer Vision, Kyoto, Japan.
29. Huang, P., Han, J., Cheng, D., and Zhang, D. (2022, January 18–24). Robust region feature synthesizer for zero-shot object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00747 ↗
30. Dong, X., Huang, J., Yang, Y., and Yan, S. (2017, January 21–26). More is less: A more complicated network with less inference complexity. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.205 ↗
31. Gu, J., Zhao, H., Lin, Z., Li, S., Cai, J., and Ling, M. (2019, January 15–20). Scene graph generation with external knowledge and image reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00207 ↗
32. Wang, X., Ye, Y., and Gupta, A. (2018, January 18–23). Zero-shot recognition via semantic embeddings and knowledge graphs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00717 ↗
33. Kampffmeyer, M., Chen, Y., Liang, X., Wang, H., Zhang, Y., and Xing, E.P. (2019, January 15–19). Rethinking knowledge graph propagation for zero-shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01175 ↗
34. Haussmann, S., Seneviratne, O., Chen, Y., Ne’eman, Y., Codella, J., Chen, C.H., McGuinness, D.L., and Zaki, M.J. (2019, January 26–30). FoodKG: A semantics-driven knowledge graph for food recommendation. Proceedings of the Semantic Web–ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand. Proceedings, Part II 18.10.1007/978-3-030-30796-7_10 ↗
35. A survey on food computing10.1145/3329168 ↗
36. Felix, R., Reid, I., and Carneiro, G. (2018, January 8–14). Multi-modal cycle-consistent generalized zero-shot learning. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01231-1_2 ↗
37. Huang, H., Wang, C., Yu, P.S., and Wang, C.D. (2019, January 15–19). Generative dual adversarial network for generalized zero-shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00089 ↗
38. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Adv. Neural Inf. Process. Syst., 27.
39. Kingma, D.P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv.
40. Denoising diffusion probabilistic models
41. Diffusion models beat gans on image synthesis
42. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics.
43. Bacteriological analytical manual (BAM) chapter 5: Salmonella
44. McNAMARA, A., MAGEAU, R., and GREEN, S. (1998). Microbiology Laboratory Guidebook.
45. Kipf, T.N., and Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv.
46. Arjovsky, M., Chintala, S., and Bottou, L. (2017, January 6–11). Wasserstein generative adversarial networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
47. Bergmann, P., Fauser, M., Sattlegger, D., and Steger, C. (2019, January 15–19). MVTec AD–A comprehensive real-world dataset for unsupervised anomaly detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00982 ↗
48. Bossard, L., Guillaumin, M., and Van Gool, L. (2014, January 6–12). Food-101–mining discriminative components with random forests. Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland. Proceedings, Part VI 13.10.1007/978-3-319-10599-4_29 ↗
49. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale10.1007/s11263-020-01316-z ↗
50. Food code
51. Singla, A., Yuan, L., and Ebrahimi, T. (2016, January 15–19). Food/non-food image classification and food categorization using pre-trained googlenet model. Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management, Amsterdam, The Netherlands.10.1145/2986035.2986039 ↗
52. Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst., 28.
53. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
54. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
55. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
56. Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., Corrado, G.S., and Dean, J. (2013). Zero-shot learning by convex combination of semantic embeddings. arXiv.
57. Changpinyo, S., Chao, W.L., Gong, B., and Sha, F. (2016, January 27–30). Synthesized classifiers for zero-shot learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.575 ↗
58. Li, Z., Yao, L., Zhang, X., Wang, X., Kanhere, S., and Zhang, H. (2019, January 29–31). Zero-shot object detection with textual descriptions. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.10.1609/aaai.v33i01.33018690 ↗
59. Visualizing data using t-SNE
60. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, Virtual.
61. Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., and Duerig, T. (2021, January 18–24). Scaling up visual and vision-language representation learning with noisy text supervision. Proceedings of the International Conference on Machine Learning, Virtual.