A Smart System for Text-Lifelog Generation from Wearable Cameras in Smart Environment Using Concept-Augmented Image Captioning with Modified Beam Search Strategy
Source abstract
Abstract
During a lifetime, a person can have many wonderful and memorable moments that he/she wants to keep. With the development of technology, people now can store a massive amount of lifelog information via images, videos or texts. Inspired by this, we develop a system to automatically generate caption from lifelog pictures taken from wearable cameras. Following up on our previous method introduced at the SoICT 2018 conference, we propose two improvements in our captioning method. We trained and tested the model on the baseline MSCOCO datasets and evaluated on different metrics. The results show better performance compared to our previous model and to some other image captioning methods. Our system also shows effectiveness in retrieving relevant data from captions and achieve high rank in ImageCLEF 2018 retrieval challenge.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Video Analysis and Summarization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Information Technology and Software Engineering Lab, VNUHCM—University of Science, Ho Chi Minh 800010, Vietnam
References (32)
Show source-provided references
1. Nguyen, V.T., Le, K.D., Tran, M.T., and Fjeld, M. (2016, January 12–15). NowAndThen: A Social Network-based Photo Recommendation Tool Supporting Reminiscence. Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia, Rovaniemi, Finland.10.1145/3012709.3012738 ↗
2. Dang-Nguyen, D.T., Piras, L., Riegler, M., Zhou, L., Lux, M., and Gurrin, C. (2018, January 22–23). Overview of ImageCLEFlifelog 2018: Daily Living Understanding and Lifelog Moment Retrieval. Proceedings of the 5th Italian Workshop on Artificial Intelligence and Robotics (AIRO 2018), Trento, Italy.
3. LifeLogging: Personal Big Data10.1561/1500000033 ↗
4. UbiqLog: A generic mobile phone-based life-log framework10.1007/s00779-012-0511-8 ↗
5. He, K., Gkioxari, G., Dollar, P., and Girshick, R. (2017, January 22–29). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.322 ↗
6. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
7. Minh-Triet, T., Thanh-Dat, T., Tung, D.D., Viet-Khoa, V.H., Quoc-An, L., and Vinh-Tiep, N. (2018, January 10–14). Lifelog Moment Retrieval with Visual Concept Fusion and Text-based Query Expansion. Proceedings of the CLEF 2018 Working Notes, Avignon, France.
8. Farhadi, A., Hejrati, M., Sadeghi, M.A., Young, P., Rashtchian, C., Hockenmaier, J., and Forsyth, D. (2010, January 5–11). Every Picture Tells a Story: Generating Sentences from Images. Proceedings of the 11th European Conference on Computer Vision: Part IV, Crete, Greece.10.1007/978-3-642-15561-1_2 ↗
9. Kulkarni, G., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., and Berg, T.L. (2011, January 20–25). Baby Talk: Understanding and Generating Simple Image Descriptions. Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995466 ↗
10. Yatskar, M., Galley, M., Vanderwende, L., and Zettlemoyer, L. (2014, January 23–24). See No Evil, Say No Evil: Description Generation from Densely Labeled Images. Proceedings of the Third Joint Conference on Lexical and Computational Semantics (*SEM 2014), 2014, Dublin, Ireland.10.3115/v1/S14-1015 ↗
11. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and Tell: A Neural Image Caption Generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
12. Karpathy, A., and Fei-Fei, L. (2015, January 7–12). Deep Visual-Semantic Alignments for Generating Image Descriptions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298932 ↗
13. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
14. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
15. You, Q., Jin, H., Wang, Z., Fang, C., and Luo, J. (2016, January 27–30). Image Captioning With Semantic Attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.503 ↗
16. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18–22). Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, Salt Lake City, UT, USA.10.1109/CVPR.2018.00636 ↗
17. Viet-Khoa, V.H., Quoc-An, L., Duy-Tam, N., Mai-Khiem, T., and Minh-Triet, T. (2018, January 6–7). Personal Diary Generation from Wearable Cameras with Concept Augmented Image Captioning and Wide Trail Strategy. Proceedings of the Ninth International Symposium on Information and CommunicationTechnology (SoICT 2018), DaNang City, Viet Nam.
18. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
19. Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv.
20. Redmon, J., and Farhadi, A. (2016). YOLO9000: Better, Faster, Stronger. arXiv.10.1109/CVPR.2017.690 ↗
21. Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems (NIPS), Curran Associates, Inc.
22. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, January 11–14). SSD: Single Shot MultiBox Detector. Proceedings of the ECCV, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_2 ↗
23. Lin, T., Maire, M., Belongie, S.J., Bourdev, L.D., Girshick, R.B., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014). Microsoft COCO: Common Objects in Context. arXiv.10.1007/978-3-319-10602-1_48 ↗
24. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv.
25. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., and Macherey, K. (2016). Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv.
26. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
27. Lavie, A., and Agarwal, A. (2007, January 23). Meteor: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments. Proceedings of the Second Workshop on Statistical Machine Translation, Prague, Czech Republic.10.3115/1626355.1626389 ↗
28. Lin, C.Y. (2004, January 25–26). ROUGE: A Package for Automatic Evaluation of Summaries. Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), Barcelona, Spain.
29. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. (2015, January 7–12). CIDEr: Consensus-Based Image Description Evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7299087 ↗
30. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., and Goel, V. (2016). Self-critical Sequence Training for Image Captioning. arXiv.10.1109/CVPR.2017.131 ↗
31. Mun, J., Cho, M., and Han, B. (2016). Text-guided Attention Model for Image Captioning. arXiv.10.1609/aaai.v31i1.11237 ↗
32. Devlin, J., Gupta, S., Girshick, R.B., Mitchell, M., and Zitnick, C.L. (2015). Exploring Nearest Neighbor Approaches for Image Captioning. arXiv.