A Universal Activation Function for Deep Learning
Source abstract
Abstract
Recently, deep learning has achieved remarkable results in fields that require human cognitive ability, learning ability, and reasoning ability. Activation functions are very important because they provide the ability of artificial neural networks to learn complex patterns through nonlinearity. Various activation functions are being studied to solve problems such as vanishing gradients and dying nodes that may occur in the deep learning process. However, it takes a lot of time and effort for researchers to use the existing activation function in their research. Therefore, in this paper, we propose a universal activation function (UA) so that researchers can easily create and apply various activation functions and improve the performance of neural networks. UA can generate new types of activation functions as well as functions like traditional activation functions by properly adjusting three hyperparameters. The famous Convolutional Neural Network (CNN) and benchmark dataset were used to evaluate the experimental performance of the UA proposed in this study. We compared the performance of the artificial neural network to which the traditional activation function is applied and the artificial neural network to which the UA is applied. In addition, we evaluated the performance of the new activation function generated by adjusting the hyperparameters of the UA. The experimental performance evaluation results showed that the classification performance of CNNs improved by up to 5% through the UA, although most of them showed similar performance to the traditional activation function.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Neural Networks and Applications · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (34)
Show source-provided references
1. Imagenet classification with deep convolutional neural networks
2. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2014. [Online]. Available: https://arxiv.org/abs/1409.1556
3. Going deeper with convolutions
4. Deep residual learning for image recognition
5. Deep learning approach based on residual neural network and SVM classifier for driver’s distraction detection10.3390/app12136626 ↗
6. R. Girshick, J. Donahue, T. Darrell and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” 2014. [Online]. Available: https://arxiv.org/abs/1311.252410.1109/CVPR.2014.81 ↗
7. You only look once: Unified, real-time object detection
8. Fast R-CNN
9. A. Bochkovskiy, C. Y. Wang and H. Y. M. Liao, “YOLOv4: Optimal speed and accuracy of object detection,” 2020. [Online]. Available: https://arxiv.org/abs/2004.10934
10. ReID-DeePNet: A hybrid deep learning system for person re-identification10.3390/math10193530 ↗
11. Comparing the performance of pre-trained deep learning models in object detection and recognition
12. Simple online and realtime tracking
13. Simple online and realtime tracking with a deep association metric
14. A. V. D. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals et al., “WaveNet: A generative model for raw audio,” 2016. [Online]. Available: https://arxiv.org/abs/1609.03499
15. Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao et al., “FastSpeech 2: Fast and high-quality end-to-end text to speech,” 2020. [Online]. Available: https://arxiv.org/abs/2006.04558
16. Natural TTS synthesis by conditioning WaveNet on Mel spectrogram predictions
17. BERT: Pre-training of deep bidirectional transformers for language understanding
18. A. Adhikari, A. Ram, R. Tang and J. Lin, “DocBERT: BERT for document classification,” 2019. [Online]. Available: http://arxiv.org/abs/1904.08398
19. SciBERT: A pretrained language model for scientific text
20. A deep network-based trade and trend analysis system to observe entry and exit points in the forex market10.3390/math10193632 ↗
21. On the approximation of the step function by some sigmoid functions10.1016/j.matcom.2015.11.005 ↗
22. On general form of the tanh method and its application to nonlinear partial differential equations10.3934/naco.2016007 ↗
23. Rectified linear units improve restricted boltzmann machines
24. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
25. B. Xu, N. Wang, T. Chen and M. Li, “Empirical evaluation of rectified activations in convolutional network,” 2015. [Online]. Available: https://arxiv.org/abs/1505.00853
26. Lung pattern classification for interstitial lung diseases using a deep convolutional neural network
27. Complex-valued neural networks with nonparametric activation functions10.1109/TETCI.2018.2872600 ↗
28. D. A. Clevert, T. Unterthiner and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (ELUs),” 2015. [Online]. Available: https://arxiv.org/abs/1511.07289
29. RMAF: Relu-memristor-like activation function for deep learning10.1109/ACCESS.2020.2987829 ↗
30. Improved convolutional neural network based on fast exponentially linear unit activation function10.1109/ACCESS.2019.2948112 ↗
31. A novel softplus linear unit for deep convolutional neural networks10.1007/s10489-017-1028-7 ↗
32. Dual rectified linear units (DReLUs): A replacement for tanh activation functions in quasi-recurrent neural networks10.1016/j.patrec.2018.09.006 ↗
33. A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” 2009. [Online]. Available: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.222.9220&rep=rep1&type=pdf
34. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2014. [Online]. Available: https://arxiv.org/abs/1412.6980