Developing Language-Specific Models Using a Neural Architecture Search
Source abstract
Abstract
This paper applies the neural architecture search (NAS) method to Korean and English grammaticality judgment tasks. Based on the previous research, which only discusses the application of NAS on a Korean dataset, we extend the method to English grammatical tasks and compare the resulting two architectures from Korean and English. Since complex syntactic operations exist beneath the word order that is computed, the two different resulting architectures out of the automated NAS language modeling provide an interesting testbed for future research. To the extent of our knowledge, the methodology adopted here has not been tested in the literature. Crucially, the resulting structure of the NAS application shows an unexpected design for human experts. Furthermore, NAS has generated different models for Korean and English, which have different syntactic operations.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Neural Networks and Applications · Natural Language Processing Techniques · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of English Literature, College of Humanities, Jeonbuk National University, Jeonju-si 54896, Korea
Department of Elctronic Engineering, Korea National University of Transportation, Chungju 27469, Chungcheongbuk-do, Korea
References (28)
Show source-provided references
1. Park, K., Shin, D., and Chi, S. (2019). Variable Chromosome Genetic Algorithm for Structure Learning in Neural Networks to Imitate Human Brain. Appl. Sci., 9.10.3390/app9153176 ↗
2. Park, K., Shin, D., and Yoo, Y. (2020). Evolutionary Neural Architecture Search (NAS) Using Chromosome Non-Disjunction for Korean Grammaticality Tasks. Appl. Sci., 10.10.3390/app10103457 ↗
3. Deep learning-based CSI feedback approach for time-varying massive MIMO channels10.1109/LWC.2018.2874264 ↗
4. Visual analytics in deep learning: An interrogative survey for the next frontiers10.1109/TVCG.2018.2843369 ↗
5. Li, A.A., Trappey, A.J., Trappey, C.V., and Fan, C.Y. (2019, January 6–9). E-discover State-of-the-art Research Trends of Deep Learning for Computer Vision. Proceedings of the 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), Bari, Italy.10.1109/SMC.2019.8914555 ↗
6. Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era10.1109/TPAMI.2019.2954885 ↗
7. Lopez, M.M., and Kalita, J. (2017). Deep Learning applied to NLP. arXiv.
8. Recent trends in deep learning based natural language processing10.1109/MCI.2018.2840738 ↗
9. Assessing the ability of LSTMs to learn syntax-sensitive dependencies10.1162/tacl_a_00115 ↗
10. A survey of deep learning: Platforms, applications and emerging research trends10.1109/ACCESS.2018.2830661 ↗
11. Self-Driving Cars: Evaluation of Deep Learning Techniques for Object Detection in Different Driving Conditions
12. Deep learning in agriculture: A survey10.1016/j.compag.2018.02.016 ↗
13. Zoph, B., Vasudevan, V., Shlens, J., and Le, Q.V. (2018, January 18–23). Learning transferable architectures for scalable image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00907 ↗
14. Real, E., Aggarwal, A., Huang, Y., and Le, Q.V. (2019, January 2–9). Regularized evolution for image classifier architecture search. Proceedings of the Aaai Conference on Artificial Intelligence, Honolulu, HI, USA.
15. Zoph, B., Cubuk, E.D., Ghiasi, G., Lin, T.-Y., Shlens, J., and Le, Q.V. (2019). Learning data augmentation strategies for object detection. arXiv.10.1109/CVPR.2019.00020 ↗
16. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8–14). Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_49 ↗
17. Elsken, T., Metzen, J.H., and Hutter, F. (2018). Neural architecture search: A survey. arXiv.10.1007/978-3-030-05318-5_3 ↗
18. Wong, C., Houlsby, N., Lu, Y., and Gesmundo, A. (2018, January 3–8). Transfer learning with neural automl. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
19. Jiang, Y., Hu, C., Xiao, T., Zhang, C., and Zhu, J. (2019, January 3–7). Improved differentiable architecture search for language modeling and named entity recognition. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hongkong, China.10.18653/v1/D19-1367 ↗
20. Optional A-scrambling
21. Schmaltz, A., Kim, Y., Rush, A.M., and Shieber, S.M. (2017). Adapting sequence models for sentence correction. arXiv.10.18653/v1/D17-1298 ↗
22. Saito, M. (1985). Some asymmetries in Japanese and their theoretical implications. [PhD Thesis, MIT].
23. Chomsky, N. (1995). The Minimalist Program, Massachusetts Institute of Technology.
24. NAS-Unet: Neural architecture search for medical image segmentation10.1109/ACCESS.2019.2908991 ↗
25. Back, T. (1996). Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms, Oxford University Press.10.1093/oso/9780195099713.001.0001 ↗
26. Theory of genetic algorithms10.1016/S0304-3975(00)00406-0 ↗
27. Ahn, S.-H. (1991). Korean quantification and Universal Grammar. [Ph.D. Thesis, University of Connecticut].
28. Nam, K.S., and Ko, Y.G. (1993). The Standard Korean Grammar, Top Publishing Company.