Knowing Knowledge: Epistemological Study of Knowledge in Transformers
Source abstract
Abstract
Statistical learners are leading towards auto-epistemic logic, but is it the right way to progress in artificial intelligence (AI)? Ways to discover AI fit the senses and the intellect. The structure of symbols–the operations by which the intellectual solution is realized–and the search for strategic reference points evoke essential issues in the analysis of AI. Studying how knowledge can be represented through methods of theoretical generalization and empirical observation is only the latest step in a long process of evolution. In this paper, we try to outline the origin of knowledge and how modern artificial minds have inherited it.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Language and cultural evolution · Evolutionary Algorithms and Applications · Computability, Logic, AI Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Innovation and Information Engineering, Guglielmo Marconi University, Via Plinio 44, 00193 Rome, Italy
Department of Enterprise Engineering, University of Rome Tor Vergata, Viale del Politecnico, 1, 00133 Rome, Italy
Department of History, Cultural Heritage, Education and Society, University of Rome Tor Vergata, Viale del Politecnico, 1, 00133 Rome, Italy
References (53)
Show source-provided references
1. Chomsky, N. (1965). Aspects of the Theory of Syntax, The MIT Press.10.21236/AD0616323 ↗
2. The faculty of language: What’s special about it?10.1016/j.cognition.2004.08.004 ↗
3. Ranaldi, L., Fallucchi, F., and Zanzotto, F.M. (2022). Dis-Cover AI Minds to Preserve Human Knowledge. Future Internet, 14.10.3390/fi14010010 ↗
4. Chomsky, N. (1957). Syntactic Structures, Cambridge Center for Behavioral Studies (CCBS).10.1515/9783112316009 ↗
5. On certain formal properties of grammars10.1016/S0019-9958(59)90362-6 ↗
6. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv.
7. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language Models are Few-Shot Learners. arXiv.
8. Deep learning10.1038/nature14539 ↗
9. Sinha, K., Sodhani, S., Dong, J., Pineau, J., and Hamilton, W.L. (2019). CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text. arXiv.10.18653/v1/D19-1458 ↗
10. Talmor, A., Elazar, Y., Goldberg, Y., and Berant, J. (2020). oLMpics—On what Language Model Pre-training Captures. arXiv.10.1162/tacl_a_00342 ↗
11. McCoy, T., Pavlick, E., and Linzen, T. (August, January 28). Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics.
12. Ranaldi, L., Ranaldi, F., Fallucchi, F., and Zanzotto, F.M. (2022). Shedding Light on the Dark Web: Authorship Attribution in Radical Forums. Information, 13.10.3390/info13090435 ↗
13. Ranaldi, L., Fallucchi, F., Santilli, A., and Zanzotto, F. (December, January 29). KERMITviz: Visualizing Neural Network Activations on Syntactic Trees. Proceedings of the Metadata and Semantic Research, Virtual Event.
14. Finding structure in time10.1207/s15516709cog1402_1 ↗
15. Quo vadis artificial intelligence?10.1007/s44163-022-00022-8 ↗
16. Zalta, E.N. (2019). Plato’s Timaeus. The Stanford Encyclopedia of Philosophy, Metaphysics Research Lab, Stanford University. [Summer 2019 ed.].
17. Explaining language universals10.1017/S002222670001450X ↗
18. Fodor, J.A. (1983). The Modularity of Mind: An Essay on Faculty Psychology, MIT Press.10.7551/mitpress/4737.001.0001 ↗
19. Jakobovits, L.A., and Miron, M.S. (1967). A Review of B. F. Skinner’s Verbal Behavior. Readings in the Psychology of Language, Prentice-Hall.
20. Second Language Acquisition and Universal Grammar10.1017/S0272263100009049 ↗
21. Logic and language acquisition10.1017/S0140525X00071648 ↗
22. Cristianini, N., and Shawe-Taylor, J. (2000). An Introduction to Support Vector Machines and Other Kernel-based Learning Methods, Cambridge University Press.10.1017/CBO9780511801389 ↗
23. Moschitti, A. (2006, January 3–7). Making Tree Kernels practical for Natural Language Learning. Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics EACL’06, Trento, Italy.
24. Collins, M., and Duffy, N. (2002, January 6–12). New Ranking Algorithms for Parsing and Tagging: Kernels over Discrete Structures, and the Voted Perceptron. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, PA, USA.10.3115/1073083.1073128 ↗
25. Culotta, A., and Sorensen, J. (2004, January 21–26). Dependency Tree Kernels for Relation Extraction. Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, Stroudsburg, PA, USA. Association for Computational Linguistics, ACL’04.10.3115/1218955.1219009 ↗
26. Pighin, D., and Moschitti, A. (2010, January 15–16). On Reverse Feature Engineering of Syntactic Tree Kernels. Proceedings of the Fourteenth Conference on Computational Natural Language Learning, Uppsala, Sweden. Association for Computational Linguistics.
27. Zanzotto, F.M., and Dell’Arciprete, L. (2012). Distributed Tree Kernels. arXiv.
28. Zanzotto, F.M., Santilli, A., Ranaldi, L., Onorati, D., Tommasino, P., and Fallucchi, F. (2020, January 16–20). KERMIT: Complementing Transformer Architectures with Encoders of Explicit Syntactic Interpretations. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics.10.18653/v1/2020.emnlp-main.18 ↗
29. Understanding the performance of knowledge graph embeddings in drug discovery
30. Chen, S., Liu, X., Gao, J., Jiao, J., Zhang, R., and Ji, Y. (2021, January 7–11). HittER: Hierarchical Transformers for Knowledge Graph Embeddings. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Punta Cana, Dominican Republic. Association for Computational Linguistics.10.18653/v1/2021.emnlp-main.812 ↗
31. Lennox, J.G., and Bolton, R. (2010). Definition in Aristotle’s Posterior Analytics. Being, Nature, and Life in Aristotle: Essays in Honor of Allan Gotthelf, Cambridge University Press.10.1017/CBO9780511919275 ↗
32. Zalta, E.N. (2018). Aristotle’s Natural Philosophy. The Stanford Encyclopedia of Philosophy, Metaphysics Research Lab, Stanford University. [Spring 2018 ed.].
33. Zalta, E.N. (2020). Aristotle’s Logic. The Stanford Encyclopedia of Philosophy, Metaphysics Research Lab, Stanford University. [Fall 2020 ed.].
34. Natural language and natural selection10.1017/S0140525X00081061 ↗
35. Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V., and Salakhutdinov, R. (2019). Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. arXiv.10.18653/v1/P19-1285 ↗
36. Hewitt, J., and Manning, C.D. (2019, January 2–7). A Structural Probe for Finding Syntax in Word Representations. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA. Association for Computational Linguistics.
37. Language as shaped by the brain10.1017/S0140525X08004998 ↗
38. Innateness and the instinct to learn10.1590/S0001-37652004000200002 ↗
39. Marcus, G. (2018). Innateness, AlphaZero, and Artificial Intelligence. arXiv.
40. Core knowledge10.1111/j.1467-7687.2007.00569.x ↗
41. Binding at Birth: The Newborn Brain Detects Identity Relations and Sequential Position in Speech10.1162/jocn_a_00157 ↗
42. Children Creating Core Properties of Language: Evidence from an Emerging Sign Language in Nicaragua10.1126/science.1100199 ↗
43. The biological basis of language: Insight from developmental grammatical impairments10.1016/j.tics.2014.07.001 ↗
44. Zhu, M., Zhang, Y., Chen, W., Zhang, M., and Zhu, J. (2013, January 4–9). Fast and Accurate Shift-Reduce Constituent Parsing. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria. Association for Computational Linguistics.
45. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA. Association for Computational Linguistics.
46. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books. arXiv.10.1109/ICCV.2015.11 ↗
47. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., and Funtowicz, M. (2019). HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv.10.18653/v1/2020.emnlp-demos.6 ↗
48. Kingma, D.P., and Ba, J. (2015). Adam: A Method for Stochastic Optimization. arXiv.
49. Neural Network Acceptability Judgments10.1162/tacl_a_00290 ↗
50. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. (2017, January 3–4). SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, BC, Canada. Association for Computational Linguistics.10.18653/v1/S17-2001 ↗
51. Sharma, L., Graesser, L., Nangia, N., and Evci, U. (2019). Natural Language Understanding with the Quora Question Pairs Dataset. arXiv.
52. Levesque, H.J., Davis, E., and Morgenstern, L. (2012, January 10–14). The Winograd Schema Challenge. Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, Rome, Italy. KR’12.
53. Podkorytov, M., Biś, D., and Liu, X. (2021, January 18–22). How Can the [MASK] Know? The Sources and Limitations of Knowledge in BERT. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.10.1109/IJCNN52387.2021.9534299 ↗