Image-Based Nutritional Advisory System: Employing Multimodal Deep Learning for Food Classification and Nutritional Analysis
Source abstract
Abstract
Accurate dietary assessment is essential for effective health management and disease prevention. However, conventional methods that rely on manual food logging and nutritional lookup are often time consuming and error prone. This study proposes an image-based nutritional advisory system that integrates multimodal deep learning to automate food classification, volume estimation, and dietary recommendation to address these limitations. The system employs a fine-tuned CLIP model for zero-shot food recognition, achieving high accuracy across diverse food categories, including unseen items. For volume measurement, a learning-based multi-view stereo (MVS) approach eliminates the need for specialized hardware, yielding reliable estimations with a mean absolute percentage error (MAPE) of 23.5% across standard food categories. Nutritional values are then calculated by referencing verified food composition databases. Furthermore, the system leverages a large language model (Llama 3) to generate personalized dietary advice tailored to individual health goals. The experimental results show that the system attains a top 1 classification accuracy of 91% on CNFOOD-241 and 80% on Food 101 and delivers high-quality recommendation texts with a BLEU-4 score of 45.13. These findings demonstrate the system’s potential as a practical and scalable tool for automated dietary management, offering improved precision, convenience, and user experience.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Nutritional Studies and Diet
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan 701, Taiwan
References (60)
Show source-provided references
1. Food Image Classification and Data Extraction Using Convolutional Neural Network and Web Crawlers10.1016/j.procs.2022.12.410 ↗
2. Development of Korean Food Image Classification Model Using Public Food Image Dataset and Deep Learning Methods10.1109/ACCESS.2022.3227796 ↗
3. He, Y., and Yin, S. (2023, January 12–14). Food Images Classification Based on Improved Convolutional Neural Network. Proceedings of the 2023 4th International Conference on Electronic Communication and Artificial Intelligence (ICECAI), Guangzhou, China.10.1109/ICECAI58670.2023.10176811 ↗
4. Islam, M.T., Siddique, B.M.N.K., Rahman, S., and Jabid, T. (2018, January 21–24). Food Image Classification with Convolutional Neural Network. Proceedings of the 2018 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS), Bangkok, Thailand.10.1109/ICIIBMS.2018.8550005 ↗
5. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, PMLR, Virtual.
6. Thames, Q., Karpur, A., Norris, W., Xia, F., Panait, L., Weyand, T., and Sim, J. (2021, January 20–25). Nutrition5k: Towards automatic nutritional understanding of generic food. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00879 ↗
7. Yang, T.-H. (2023). E-Volume: Learning-based Multi-View Stereo Volume Estimator—A Case Study on Food on the Dining Table. [Master’s Thesis, Department of Computer Science and Information Engineering].
8. Lo, F.P.-W., Sun, Y., Qiu, J., and Lo, B. (2018). Food Volume Estimation Based on Deep Learning View Synthesis from a Single Depth Map. Nutrients, 10.10.3390/nu10122005 ↗
9. Pfisterer, K.J., Amelard, R., Chung, A.G., Syrnyk, B., MacLean, A., Keller, H.H., and Wong, A. (2022). Automated food intake tracking requires depth-refined semantic segmentation to rectify visual-volume discordance in long-term care homes. Sci. Rep., 12.10.1038/s41598-021-03972-8 ↗
10. Team, M.L. (2024, May 20). Introducing Meta Llama 3: The Most Capable Openly Available LLM to Date. Available online: https://ai.meta.com/blog/meta-llama-3/.
11. Xian, Y., Schiele, B., and Akata, Z. (2017, January 21–26). Zero-shot learning-the good, the bad and the ugly. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.328 ↗
12. Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly10.1109/TPAMI.2018.2857768 ↗
13. Self-supervised embedding for generalized zero-shot learning in remote sensing scene classification10.1117/1.JRS.17.032405 ↗
14. Ge, Y., Ren, J., Gallagher, A., Wang, Y., Yang, M.H., Adam, H., and Zhao, J. (2023, January 17–24). Improving zero-shot generalization and robustness of multi-modal models. Proceedings of the IEEE/CVF Conference on Computer Vision And pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01067 ↗
15. Self-supervised learning: Generative or contrastive
16. ImageNet classification with deep convolutional neural networks
17. Simonyan, K., and Zisserman, A. (2014). Intense convolutional networks for large-scale image recognition. arXiv.
18. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
19. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
20. Tan, M., and Le, Q. (2019, January 9–15). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
21. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
22. Attention is all you need
23. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 12–18). A simple framework for contrastive learning of visual representations. Proceedings of the International Conference on Machine Learning, PMLR, Vienna, Austria.
24. Iter, D., Guu, K., Lansing, L., and Jurafsky, D. (2020). Pretraining with contrastive sentence objectives improves discourse performance of language models. arXiv.10.18653/v1/2020.acl-main.439 ↗
25. A contrastive framework for neural text generation
26. Bootstrap your own latent-a new approach to self-supervised learning
27. Ando, Y., Ege, T., Cho, J., and Yanai, K. (2019, January 21). Depthcaloriecam: A mobile application for volume-based foodcalorie estimation using depth cameras. Proceedings of the 5th International Workshop on Multimedia Assisted Dietary Management, New York, NY, USA.10.1145/3347448.3357172 ↗
28. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. (2006, January 17–22). A comparison and evaluation of multi-view stereo reconstruction algorithms. Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), New York, NY, USA.
29. Goesele, M., Snavely, N., Curless, B., Hoppe, H., and Seitz, S.M. (2007, January 14–21). Multi-View Stereo for Community Photo Collections. Proceedings of the 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil.10.1109/ICCV.2007.4408933 ↗
30. Agarwal, S., Snavely, N., Simon, I., Seitz, S.M., and Szeliski, R. (October, January 29). Building Rome in a day. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan.10.1109/ICCV.2009.5459148 ↗
31. Ji, M., Gall, J., Zheng, H., Liu, Y., and Fang, L. (2017, January 22–29). Surfacenet: An end-to-end 3d neural network for multiview stereopsis. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.253 ↗
32. Learning a multi-view stereo machine
33. Yao, Y., Luo, Z., Li, S., Fang, T., and Quan, L. (2018, January 8–14). Mvsnet: Depth inference for unstructured multi-view stereo. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01237-3_47 ↗
34. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
35. Gao, J., and Lin, C.-Y. (2004). Introduction to the Special Issue on Statistical Language Modeling, ACM. [3rd ed.].10.1145/1034780.1034781 ↗
36. Recurrent Neural Network Based Language Modeling in Meeting Recognition10.21437/Interspeech.2011-720 ↗
37. Class-based n-gram models of natural language
38. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., and Brockman, G. (2021). Evaluating large language models trained on code. arXiv.
39. ChatGPT for good? On opportunities and challenges of large language models for education10.1016/j.lindif.2023.102274 ↗
40. Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., and Hou, Y. (2023). A survey of large language models. arXiv.
41. GPT-3: Its nature, scope, limits, and consequences10.1007/s11023-020-09548-1 ↗
42. OpenAI (2023). GPT-4 Technical Report. arXiv.
43. Training language models to follow instructions with human feedback
44. Language models are few-shot learners
45. Deep reinforcement learning from human preferences
46. Ziegler, D.M., Stiennon, N., Wu, J., Brown, T.B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019). Fine-tuning language models from human preferences. arXiv.
47. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network10.1016/j.physd.2019.132306 ↗
48. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., and Huang, F. (2023). Qwen technical report. arXiv.
49. Le Scao, T., Fan, A., Wolf, T., Gugger, S., Minixhofer, B., Singh, R., Rasmussen, A., Muennighoff, N., Szlam, A., and Scialom, T. (2024, June 12). Bloom: A 176B-Parameter Open-Access Multilingual Language Model. Available online: https://huggingface.co/bigscience/bloom.
50. Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Scao, T.L., Bari, M.S., Shen, S., Yong, Z.X., and Schoelkopf, H. (2022). Crosslingual generalization through multitask finetuning. arXiv.10.18653/v1/2023.acl-long.891 ↗
51. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv.
52. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv.
53. Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., and Hu, G. (2020). Revisiting pre-trained models for Chinese natural language processing. arXiv.10.18653/v1/2020.findings-emnlp.58 ↗
54. Pre-training with whole word masking for chinese bert10.1109/TASLP.2021.3124365 ↗
55. Semantic cosine similarity
56. Tai, C.E.A., Keller, M., Kerrigan, M., Chen, Y., Nair, S., Xi, P., and Wong, A. (2023). Nutritionverse-3d: A 3d food model dataset for nutritional intake estimation. arXiv.
57. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv.
58. Bossard, L., Guillaumin, M., and Van Gool, L. (2014). Food-101–mining discriminative components with random forests. Lecture Notes in Computer Science, Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, 6–12 September 2014, Springer. Proceedings, Part VI 13.10.1007/978-3-319-10599-4_29 ↗
59. Fan, B.-K., and CNFOOD-241 (2024, June 12). Mendeley Data 2022, V1. Available online: https://data.mendeley.com/datasets/fspyss5zbb/1.
60. Yang, T.-L. (2020). Taiwanese-Food-101: A Dataset for Recognizing Taiwanese Cuisines and its Application. [Master’s Thesis, Department of Electronic Engineering, National Taipei University of Technology].