Concatenated Attention: A Novel Method for Regulating Information Structure Based on Sensors
Source abstract
Abstract
This paper addresses the challenges of limited training data and suboptimal environmental conditions in image processing tasks, such as underwater imaging with poor lighting and distortion. Neural networks, including Convolutional Neural Networks (CNNs) and Transformers, have advanced image analysis but remain constrained by computational demands and insufficient data. To overcome these limitations, we propose a novel split-and-concatenate method for self-attention mechanisms. By splitting Query and Key matrices into submatrices, performing cross-multiplications, and applying weighted summation, the method optimizes intermediate variables without increasing computational costs. Experiments on a real-world crack dataset demonstrate its effectiveness in improving network performance.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Privacy-Preserving Technologies in Data · Infrastructure Maintenance and Monitoring · Adversarial Robustness in Machine Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Division of Electrical Engineering and Computer Science, Kanazawa University, Kanazawa 9201192, Japan
Faculty of Electrical, Information and Communication Engineering, Kanazawa University, Kanazawa 9201192, Japan
References (53)
Show source-provided references
1. Deep learning for computer vision: A brief review10.1155/2018/7068349 ↗
2. A review of convolutional neural networks in computer vision10.1007/s10462-024-10721-6 ↗
3. A survey on underwater computer vision
4. Artificial intelligence-based sensors for next generation IoT applications: A review10.1109/JSEN.2021.3055618 ↗
5. Machine learning/artificial intelligence for sensor data fusion–opportunities and challenges10.1109/MAES.2020.3049030 ↗
6. Fabre, W., Haroun, K., Lorrain, V., Lepecq, M., and Sicard, G. (2024). From Near-Sensor to In-Sensor: A State-of-the-Art Review of Embedded AI Vision Systems. Sensors, 24.10.3390/s24165446 ↗
7. Sustainable ai: Environmental implications, challenges and opportunities
8. A systematic review of Green AI10.1002/widm.1507 ↗
9. A fully automatic AI system for tooth and alveolar bone segmentation from cone-beam CT images10.1038/s41467-022-29637-2 ↗
10. ChatGPT for shaping the future of dentistry: The potential of multi-modal large language model10.1038/s41368-023-00239-y ↗
11. A review of traffic congestion prediction using artificial intelligence10.1155/2021/8878011 ↗
12. Pan, Z., Sharma, A., Hu, J.Y.C., Liu, Z., Li, A., Liu, H., Huang, M., and Geng, T. (2023, January 8–10). Ising-traffic: Using ising machine learning to predict traffic congestion under uncertainty. Proceedings of the AAAI Conference on Artificial Intelligence, Montréal, QC, Canada.10.1609/aaai.v37i8.26121 ↗
13. Reporting of artificial intelligence prediction models10.1016/S0140-6736(19)30037-6 ↗
14. Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI10.1136/bmj-2022-070904 ↗
15. Tuberculosis (TB) classification in chest radiographs using deep convolutional neural networks
16. Devnath, L., Fan, Z., Luo, S., Summons, P., and Wang, D. (2022). Detection and visualisation of pneumoconiosis using an ensemble of multi-dimensional deep features learned from Chest X-rays. Int. J. Environ. Res. Public Health, 19.10.3390/ijerph191811193 ↗
17. Internet of Healthcare Things: A contemporary survey10.1016/j.jnca.2021.103179 ↗
18. An intelligent fault detection approach based on reinforcement learning system in wireless sensor network10.1007/s11227-021-04001-1 ↗
19. Wei, K., Fu, Y., Yang, J., and Huang, H. (2020, January 13–19). A physics-based noise formation model for extreme low-light raw denoising. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00283 ↗
20. Rhodopsin, light-sensor of vision10.1016/j.preteyeres.2022.101116 ↗
21. Ai, S., and Kwon, J. (2020). Extreme low-light image enhancement for surveillance cameras using attention U-Net. Sensors, 20.10.3390/s20020495 ↗
22. Jung, M., and Cho, J. (2024). Enhancing Detection of Pedestrians in Low-Light Conditions by Accentuating Gaussian–Sobel Edge Features from Depth Maps. Appl. Sci., 14.10.3390/app14188326 ↗
23. Multimodal machine learning in precision health: A scoping review10.1038/s41746-022-00712-8 ↗
24. The shaky foundations of large language models and foundation models for electronic health records10.1038/s41746-023-00879-8 ↗
25. Marton, S., Lüdtke, S., and Bartelt, C. (2022). Explanations for neural networks by neural networks. Appl. Sci., 12.10.3390/app12030980 ↗
26. Le, A.T., Shakiba, M., Ardekani, I., and Abdulla, W.H. (2024). Optimizing Plant Disease Classification with Hybrid Convolutional Neural Network–Recurrent Neural Network and Liquid Time-Constant Network. Appl. Sci., 14.10.3390/app14199118 ↗
27. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
28. Tan, M. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv.
29. Chiruzzo, L., Jiménez-Zafra, S.M., and Rangel, F. (2024, January 24). Overview of IberLEF 2024: Natural Language Processing Challenges for Spanish and other Iberian Languages. Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2024), co-located with the 40th Conference of the Spanish Society for Natural Language Processing (SEPLN 2024), Valladolid, Spain.
30. Multi-task learning in natural language processing: An overview10.1145/3679013 ↗
31. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
32. Li, B., Hu, Y., Nie, X., Han, C., Jiang, X., Guo, T., and Liu, L. (2023, January 17–24). Dropkey for vision transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.02174 ↗
33. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00986 ↗
34. A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends10.1109/TPAMI.2024.3415112 ↗
35. Performance comparison of pretrained convolutional neural networks on crack detection in buildings
36. Krizhevsky, A. (2025, January 05). Learning Multiple Layers of Features from Tiny Images. Available online: https://www.cs.utoronto.ca/~kriz/learning-features-2009-TR.pdf.
37. K-means clustering algorithms: A comprehensive review, variants analysis, and advances in the era of big data10.1016/j.ins.2022.11.139 ↗
38. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. (2011). The Caltech-Ucsd Birds-200-2011 Dataset, California Institute of Technology.
39. A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities10.1145/3582688 ↗
40. Nilsback, M.E., and Zisserman, A. (2008, January 16–19). Automated flower classification over a large number of classes. Proceedings of the 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, Bhubaneswar, India.10.1109/ICVGIP.2008.47 ↗
41. From show to tell: A survey on deep learning-based image captioning10.1109/TPAMI.2022.3148210 ↗
42. Parkhi, O.M., Vedaldi, A., Zisserman, A., and Jawahar, C. (2012, January 16–21). Cats and dogs. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248092 ↗
43. Kaya, M., and Bilge, H.Ş. (2019). Deep metric learning: A survey. Symmetry, 11.10.3390/sym11091066 ↗
44. Systematic literature review: Quantum machine learning and its applications10.1016/j.cosrev.2024.100619 ↗
45. Bossard, L., Guillaumin, M., and Van Gool, L. (2014, January 6–12). Food-101–mining discriminative components with random forests. Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland. Proceedings, Part VI 13.10.1007/978-3-319-10599-4_29 ↗
46. Khattak, M.U., Rasheed, H., Maaz, M., Khan, S., and Khan, F.S. (2023, January 17–24). Maple: Multi-modal prompt learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01832 ↗
47. Bishop, C.M., and Nasrabadi, N.M. (2006). Pattern Recognition and Machine Learning, Springer.
48. Zhang, L., Yang, F., Zhang, Y.D., and Zhu, Y.J. (2016, January 25–28). Road crack detection using deep convolutional neural network. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.10.1109/ICIP.2016.7533052 ↗
49. Yang, F., Zhang, L., Yu, S., Prokhorov, D., Mei, X., and Ling, H. (2019). Feature Pyramid and Hierarchical Boosting Network for Pavement Crack Detection. arXiv.10.1109/TITS.2019.2910595 ↗
50. Eisenbach, M., Stricker, R., Seichter, D., Amende, K., Debes, K., Sesselmann, M., Ebersbach, D., Stoeckert, U., and Gross, H.M. (2017, January 14–19). How to Get Pavement Distress Detection Ready for Deep Learning? A Systematic Approach. Proceedings of the International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA.10.1109/IJCNN.2017.7966101 ↗
51. Automatic road crack detection using random structured forests10.1109/TITS.2016.2552248 ↗
52. Automatic Crack Detection on Two-Dimensional Pavement Images: An Algorithm Based on Minimal Path Selection10.1109/TITS.2015.2477675 ↗
53. CrackTree: Automatic crack detection from pavement images10.1016/j.patrec.2011.11.004 ↗