Recommendations for Different Tasks Based on the Uniform Multimodal Joint Representation
Source abstract
Abstract
Content curation social networks (CCSNs), such as Pinterest and Huaban, are interest driven and content centric. On CCSNs, user interests are represented by a set of boards, and a board is composed of various pins. A pin is an image with a description. All entities, such as users, boards, and categories, can be represented as a set of pins. Therefore, it is possible to implement entity representation and the corresponding recommendations on a uniform representation space from pins. Furthermore, lots of pins are re-pinned from others and the pin’s re-pin sequences are recorded on CCSNs. In this paper, a framework which can learn the multimodal joint representation of pins, including text representation, image representation, and multimodal fusion, is proposed. Image representations are extracted from a multilabel convolutional neural network. The multiple labels of pins are automatically obtained by the category distributions in the re-pin sequences, which benefits from the network architecture. Text representations are obtained with the word2vec tool. Two modalities are fused with a multimodal deep Boltzmann machine. On the basis of the pin representation, different recommendation tasks are implemented, including recommending pins or boards to users, recommending thumbnails to boards, and recommending categories to boards. Experimental results on a dataset from Huaban demonstrate that the multimodal joint representation of pins contains the information of user interests. Furthermore, the proposed multimodal joint representation outperformed unimodal representation in different recommendation tasks. Experiments were also performed to validate the effectiveness of the proposed recommendation methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image and Video Retrieval Techniques · Image Retrieval and Classification Techniques · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
College of Artificial Intelligence, North China University of Science and Technology, Tangshan 063009, China
References (39)
Show source-provided references
1. Gilbert, E., Bakhshi, S., Chang, S., and Terveen, L. (May, January 27). “I Need to Try This!”: A Statistical Overview of Pinterest. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France.
2. Bernardini, C., Silverston, T., and Festor, O. (2014, January 4–8). A Pin is Worth a Thousand Words: Characterization of Publications in Pinterest. Proceedings of the 2014 International Wireless Communications and Mobile Computing Conference (IWCMC), Nicosia, Cyprus.10.1109/IWCMC.2014.6906377 ↗
3. Copy Content, Copy Friends: Studies of Content Curation and Social Bootstrapping on Pinterest10.1145/2641730.2641734 ↗
4. Gelley, B., and John, A. (2015, January 14–18). Do I Need To Follow You?: Examining the Utility of The Pinterest Follow Mechanism. Proceedings of the 18th Acm Conference on Computer Supported Cooperative Work & Social Computing, Vancouver, BC, Canada.10.1145/2675133.2675209 ↗
5. Zhong, C., Kourtellis, N., and Sastry, N. (2016, January 17–20). Pinning Alone? A Study of the Role of Social Ties on Pinterest. Proceedings of the International AAAI Conference on Web and Social Media, Cologne, Germany.
6. Liakos, P., Papakonstantinopoulou, K., Sioutis, M., Tsakalozos, K., and Delis, A. (2016, January 9–10). Pinpointing Influence in Pinterest. Proceedings of the 2nd International Workshop on Social Influence Analysis Co-Located with 25th International Joint Conference on Artificial Intelligence (IJCAI 2016), New York, NY, USA.
7. Venkatadri, G., Goga, O., Zhong, C., Viswanath, B., Gummadi, K.P., and Sastry, N. (2016, January 12). Strengthening Weak Identities Through Inter-Domain Trust Transfer. Proceedings of the 25th International Conference on World Wide Web, Montreal, QB, Canada.10.1145/2872427.2883015 ↗
8. Mao, J., Jiajing, X., Jing, Y., and Yuille, A. (2016). Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images. Advances in Neural Information Processing Systems 29, NIPS Proceedings.
9. Jing, Y., Liu, D., Kislyuk, D., Zhai, A., Xu, J., Donahue, J., and Tavel, S. (2015, January 8–9). Visual Search at Pinterest. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, Australia.10.1145/2783258.2788621 ↗
10. Geng, X., Zhang, H., Song, Z., Yang, Y., Luan, H., and Chua, T.S. (2014, January 3–7). One of a Kind: User Profiling by Social Curation. Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, USA.10.1145/2647868.2654950 ↗
11. Cinar, Y.G., Zoghbi, S., and Moens, M.F. (2015, January 14–17). Inferring User Interests on Social Media From Text and Images. Proceedings of the 2015 IEEE International Conference on Data Mining Workshop (ICDMW), Atlantic City, NJ, USA.10.1109/ICDMW.2015.208 ↗
12. A picture tells a thousand words—About you! User interest profiling from user generated visual content10.1016/j.sigpro.2015.10.032 ↗
13. Kamath, K.Y., Popescu, A.M., and Caverlee, J. (2013, January 10–14). Board Recommendation in Pinterest. Proceedings of the Late-Breaking Results, Project Papers and Workshop Proceedings of the 21st Conference on User Modeling Adaptation, and Personalization, Rome, Italy.
14. Zhong, C., Karamshuk, D., and Sastry, N. (2015, January 18–22). Predicting Pinterest: Automating a Distributed Human Computation. Proceedings of the 24th International Conference on World Wide Web, Geneva, Switzerland.10.1145/2736277.2741671 ↗
15. Jing, Y., Wu, L., Zhang, X., and Wang, D. (July, January 12). Recommending Users to Follow based on User Taste Homophily for Content Curation Social Networks. Proceedings of the 6th International Workshop on Social Recommender Systems (SRS 2015), Sydney, Australia.
16. Yang, X., Li, Y., and Luo, J. (2015, January 26–30). Pinterest Board Recommendation for Twitter Users. Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia.10.1145/2733373.2806375 ↗
17. Li, Y., Cong, Y., Mei, T., and Luo, J. (November, January 29). User-Curated Image Collections: Modeling and Recommendation. Proceedings of the 2015 IEEE International Conference on Big Data (Big Data), Santa Clara, CA, USA.10.1109/BigData.2015.7363803 ↗
18. Geng, X., Zhang, H., Bian, J., and Chua, T.S. (2015, January 7–13). Learning Image and User Features for Recommendation in Social Networks. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.486 ↗
19. Wu, L., Wang, D., Guo, C., Zhang, J., and Chen, C.W. (2016, January 4–6). User Profiling by Combining Topic Modeling and Pointwise Mutual Information (TM-PMI). Proceedings of the International Conference on Multimedia Modeling, Miami, FL, USA.10.1007/978-3-319-27674-8_14 ↗
20. Wu, L., Zhang, D., Zhang, X., Jing, Y., Liu, H., and Chen, C.W. (2016, January 19–21). Recommending Followees Based on Content Weighted User Interest Homophily. Proceedings of the International Conference on Internet Multimedia Computing and Service, Xi’an, China.10.1145/3007669.3007692 ↗
21. Zhai, A., Kislyuk, D., Jing, Y., Feng, M., Tzeng, E., Donahue, J., Du, Y.L., and Darrell, T. (2017, January 3). Visual Discovery at Pinterest. Proceedings of the 26th International Conference on World Wide Web Companion, Geneva, Switzerland.10.1145/3041021.3054201 ↗
22. Liu, D.C., Rogers, S., Shiau, R., Kislyuk, D., Ma, K.C., Zhong, Z., Liu, J., and Jing, Y. (2017, January 3). Related Pins at Pinterest: The Evolution of a Real-World Recommender System. Proceedings of the 26th International Conference on World Wide Web Companion, Geneva, Switzerland.10.1145/3041021.3054202 ↗
23. MLLDA: Multi-level LDA for modelling users on content curation social networks10.1016/j.neucom.2016.08.114 ↗
24. Wu, L., Zhang, L., Jian, M., Zhang, D., and Liu, H. (2017, January 23–25). Image Recommendation on Content-based Bipartite Graph. Proceedings of the Internet Multimedia Computing and Servic, Qingdao, China.
25. Multi-perspective User2Vec: Exploiting re-pin activity for user representation learning in content curation social network10.1016/j.sigpro.2017.07.002 ↗
26. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013, January 2–4). Efficient Estimation of Word Representations in Vector Space. Proceedings of the 2013 International Conference on Learning Representations Workshop, Scottsdale, AZ, USA.
27. Lev, G., Klein, B., and Wolf, L. (2015, January 17–19). In Defense of Word Embedding for Generic Text Representation. Proceedings of the Natural Language Processing and Information Systems—20th International Conference on Applications of Natural Language to Information Systems, Passau, Germany.10.1007/978-3-319-19581-0_3 ↗
28. Ma, L., Lu, Z., Shang, L., and Li, H. (2015, January 7–13). Multimodal Convolutional Neural Networks for Matching Image and Sentence. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV) (2015), Santiago, Chile.10.1109/ICCV.2015.301 ↗
29. Qian, S., Zhang, T., and Xu, C. (2016, January 15–19). Multi-view Topic-opinion Mining for Social Event Analysis. Proceedings of the 2016 ACM on Multimedia Conference, Amsterdam, The Netherlands.10.1145/2964284.2964294 ↗
30. Jia, X., Wang, A., Li, X., Xun, G., and Zhang, A. (November, January 29). Multi-modal learning for video recommendation based on mobile application usage. Proceedings of the IEEE International Conference on Big Data, Santa Clara, CA, USA.
31. Ngiam, J., Khosla, A., and Kim, M. (July, January 28). Multimodal Deep Learning. Proceedings of the 28th International Conference on Machine Learning, Washington, DC, USA.
32. Multimodal Learning with Deep Boltzmann Machines
33. Zhang, H., Yang, Y., Luan, H., Yan, S., and Chua, T.S. (2014, January 3–7). Start from Scratch: Towards Automatically Identifying, Modeling, and Naming Visual Attributes. Proceedings of the 22Nd ACM International Conference on Multimedia, Orlando, FL, USA.10.1145/2647868.2654915 ↗
34. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems 25, NIPS Proceedings.
35. ImageNet Large Scale Visual Recognition Challenge10.1007/s11263-015-0816-y ↗
36. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. Advances in Neural Information Processing Systems 26, NIPS Proceedings.
37. Perronnin, F., and Dance, C. (2007, January 17–22). Fisher Kernels on Visual Vocabularies for Image Categorization. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.10.1109/CVPR.2007.383266 ↗
38. Perronnin, F., Sánchez, J., and Mensink, T. (2010, January 5–11). Improving the Fisher Kernel for Large-Scale Image Classification. Proceedings of the 11th European Conference on Computer Vision: Part IV, Heraklion, Greece.10.1007/978-3-642-15561-1_11 ↗
39. Image Classification with the Fisher Vector: Theory and Practice10.1007/s11263-013-0636-x ↗