SimCDL: A Simple Framework for Contrastive Dictionary Learning
Source abstract
Abstract
In this paper, we propose a novel approach to the dictionary learning (DL) initialization problem, leveraging the SimCLR framework from deep learning in a self-supervised manner. Dictionary learning seeks to represent signals as sparse combinations of dictionary atoms, but effective initialization remains challenging. By applying contrastive learning, we encourage similar representations for augmented versions of the same sample while distinguishing between different samples. This results in a more diverse and incoherent set of atoms, which enhances the performance of DL applications in classification and anomaly detection tasks. Our experiments across several benchmark datasets demonstrate the effectiveness of our method for improving dictionary learning initialization and its subsequent impact on performance in various applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Lexicography and Language Studies · Translation Studies and Practices
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Automatic Control and Computers, National University of Science and Technology Politehnica Bucharest, Splaiul Independentei nr. 313, Sector 6, 060042 București, Romania
Funding
Ministry of Research, Innovation and Digitization, CNCS—UEFISCDI
References (27)
Show source-provided references
1. Pati, Y., Rezaiifar, R., and Krishnaprasad, P. (1993, January 1–3). Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. Proceedings of the 27th Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, USA.
2. Dumitrescu, B., and Irofti, P. (2018). Dictionary Learning Algorithms and Applications, Springer.10.1007/978-3-319-78674-2 ↗
3. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation10.1109/TSP.2006.881199 ↗
4. Rubinstein, R., Zibulevsky, M., and Elad, M. (2008). Efficient Implementation of the K-SVD Algorithm Using Batch Orthogonal Matching Pursuit, Computer Science Department—Technion. Technical report.
5. Hadsell, R., Chopra, S., and LeCun, Y. (2006, January 17–22). Dimensionality reduction by learning an invariant mapping. Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), New York, NY, USA.
6. Gutmann, M., and Hyvärinen, A. (2010, January 13–15). Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy. JMLR Workshop and Conference Proceedings.
7. Self-supervised representation learning: Introduction, advances, and challenges10.1109/MSP.2021.3134634 ↗
8. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020, January 14–19). Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00975 ↗
9. Misra, I., and Maaten, L. (2020, January 14–19). Self-supervised learning of pretext-invariant representations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00674 ↗
10. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 13–18). A simple framework for contrastive learning of visual representations. Proceedings of the International Conference on Machine Learning, Virtual Event.
11. Ilie-Ablachim, D., and Dumitrescu, B. (2023, January 17–20). Classification With Dictionary Learning and A Distance Barrier Promoting Incoherence. Proceedings of the 2023 IEEE 33rd International Workshop on Machine Learning for Signal Processing (MLSP), Rome, Italy.10.1109/MLSP55844.2023.10285866 ↗
12. Wang, T., and Isola, P. (2020, January 13–18). Understanding contrastive representation learning through alignment and uniformity on the hypersphere. Proceedings of the International Conference on Machine Learning, Virtual Event.
13. Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., and Tucker, G. (2019, January 9–15). On variational bounds of mutual information. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
14. On implementing 2D rectangular assignment algorithms10.1109/TAES.2016.140952 ↗
15. Jan (2024, September 01). Rotation Matrix. Available online: https://www.mathworks.com/matlabcentral/fileexchange/66446-rotation-matrix.
16. Ramirez, I., Sprechmann, P., and Sapiro, G. (2010, January 13–18). Classification and clustering via dictionary learning with structured incoherence and shared features. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.10.1109/CVPR.2010.5539964 ↗
17. Incoherent frames design and dictionary learning using a distance barrier10.1016/j.sigpro.2023.109019 ↗
18. Schroff, F., Kalenichenko, D., and Philbin, J. (2015, January 7–12). Facenet: A unified embedding for face recognition and clustering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298682 ↗
19. From few to many: Illumination cone models for face recognition under variable lighting and pose10.1109/34.927464 ↗
20. Martinez, A. (2024, September 01). The AR Face Database: CVC Technical Report 24. Available online: https://portalrecerca.uab.cat/en/publications/the-ar-face-database-cvc-technical-report-24.
21. Sim, T., Baker, S., and Bsat, M. (2002, January 20–21). The CMU pose, illumination, and expression (PIE) database. Proceedings of the Fifth IEEE International Conference on Automatic Face Gesture Recognition, Washinton, DC, USA.
22. Fei-Fei, L., Fergus, R., and Perona, P. (July, January 27). Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories. Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition, Washington, DC, USA.
23. Sadanand, S., and Corso, J. (2012, January 16–21). Action bank: A high-level representation of activity in video. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6247806 ↗
24. Visualizing data using t-SNE
25. Pyod: A python toolbox for scalable outlier detection
26. Adbench: Anomaly detection benchmark
27. Label consistent K-SVD: Learning a discriminative dictionary for recognition10.1109/TPAMI.2013.88 ↗