Off-the-shelf deep learning is not enough, and requires parsimony, Bayesianity, and causality
Source abstract
Abstract
Abstract Deep neural networks (‘deep learning’) have emerged as a technology of choice to tackle problems in speech recognition, computer vision, finance, etc. However, adoption of deep learning in physical domains brings substantial challenges stemming from the correlative nature of deep learning methods compared to the causal, hypothesis driven nature of modern science. We argue that the broad adoption of Bayesian methods incorporating prior knowledge, development of solutions with incorporated physical constraints and parsimonious structural descriptors and generative models, and ultimately adoption of causal models, offers a path forward for fundamental and applied research.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Topic Modeling · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (52)
Show source-provided references
1. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).10.1038/nature14539 ↗
2. Rosenblatt, F. Principles of Neurodynamics. Perceptrons and the Theory of Brain Mechanisms (Cornell Aeronautical Lab Inc Buffalo NY, 1961).10.21236/AD0256582 ↗
3. Lee, K. K. et al. Using neural networks to construct models of the molecular beam epitaxy process. IEEE Trans. Semicond. Manuf. 13, 34–45 (2000).10.1109/66.827338 ↗
4. Sumpter, B. G. & Noid, D. W. Potential energy surfaces for macromolecules. a neural network technique. Chem. Phys. Lett. 192, 455–462 (1992).10.1016/0009-2614(92)85498-Y ↗
5. Doersch, C. Tutorial on variational autoencoders. Preprint at https://arxiv.org/abs/1606.05908 (2016).
6. Su, J., Vargas, D. V. & Sakurai, K. One pixel attack for fooling deep neural networks. IEEE Trans. Evol. Comp. 23, 828–841 (2019).10.1109/TEVC.2019.2890858 ↗
7. Warde-Farley, D. & Goodfellow, I. In Perturbations, Optimization, and Statistics (eds Hazan, T., Papandreou, G. & Tarlow, D.) Ch. 11 (MIT Press, 2016).
8. Alcorn, M. A. et al. Strike (with) a pose: neural networks are easily fooled by strange poses of familiar objects. Proc. IEEE Conf. Comp. Vis. Patt. Recog. 4845–4854 (IEEE, New York, 2019).10.1109/CVPR.2019.00498 ↗
9. Pearl, J. Causality (Cambridge Univ. Press, 2009).10.1017/CBO9780511803161 ↗
10. O’neil, C. Weapons of Math Destruction: How big data Increases Inequality and Threatens Democracy (Broadway Books, 2016).
11. Wagner, C. H. Simpson’s paradox in real life. Am. Statistician 36, 46–48 (1982).10.1080/00031305.1982.10482778 ↗
12. Hoyer, P. O., Janzing, D., Mooij, J. M., Peters, J. & Schölkopf, B. Nonlinear causal discovery with additive noise models. Adv. Neur. Inf. Proc. Sys. 21, 689–696 (2009).
13. Peters, J., Janzing, D. & Schölkopf, B. Elements of Causal Inference: Foundations and Learning Algorithms (MIT Press, 2017).
14. Ziatdinov, M. et al. Deep learning of atomically resolved scanning transmission electron microscopy images: chemical identification and tracking local transformations. ACS Nano 11, 12742 (2017).10.1021/acsnano.7b07504 ↗
15. Ovchinnikov, O., Jesse, S., Bintacchit, P., Trolier-McKinstry, S. & Kalinin, S. V. Disorder identification in hysteresis data: recognition analysis of the random-bond–random-field ising model. Phys. Rev. Lett. 103, 157203 (2009).10.1103/PhysRevLett.103.157203 ↗
16. Kumar, A. et al. Spatially resolved mapping of disorder type and distribution in random systems using artificial neural network recognition. Phys. Rev. B. 84, 024203 (2011).10.1103/PhysRevB.84.024203 ↗
17. Iten, R., Metger, T., Wilming, H., Del Rio, L. & Renner, R. Discovering physical concepts with neural networks. Phys. Rev. Lett. 124, 010508 (2020).10.1103/PhysRevLett.124.010508 ↗
18. Vlcek, L., Maksov, A. B., Pan, M., Vasudevan, R. K. & Kalinin, S. V. Knowledge extraction from atomically resolved images. ACS Nano 11, 10313–10320 (2017).10.1021/acsnano.7b05036 ↗
19. Koch-Janusz, M. & Ringel, Z. J. N. P. Mutual information, neural networks and the renormalization group.Nat. Phys. 14, 578–582 (2018).10.1038/s41567-018-0081-4 ↗
20. Kingma, D. P. & Welling, M. Auto-encoding variational bayes. Preprint at https://arxiv.org/abs/1312.6114 (2013).
21. Peretto, P. Collective properties of neural networks: a statistical physics approach. Biol. Cybern. 50, 51–62 (1984).10.1007/BF00317939 ↗
22. Arnab, A. et al. Conditional random fields meet deep neural networks for semantic segmentation: combining probabilistic graphical models with deep learning for structured prediction. IEEE Signal Process. Mag. 35, 37–52 (2018).10.1109/MSP.2017.2762355 ↗
23. Xie, T. & Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett. 120, 145301 (2018).10.1103/PhysRevLett.120.145301 ↗
24. Karpatne, A. et al. Theory-guided data science: a new paradigm for scientific discovery from data. IEEE Trans. Knowl. Data Eng. 29, 2318–2331 (2017).10.1109/TKDE.2017.2720168 ↗
25. Silver, D. et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 1140–1144 (2018).10.1126/science.aar6404 ↗
26. Berner, C. et al. Dota 2 with large scale deep reinforcement learning. Preprint at https://arxiv.org/abs/1912.06680 (2019).
27. Popova, M., Isayev, O. & Tropsha, A. Deep reinforcement learning for de novo drug design. Sci. Adv. 4, eaap7885 (2018).10.1126/sciadv.aap7885 ↗
28. Zhou, Z., Kearnes, S., Li, L., Zare, R. N. & Riley, P. Optimization of molecules via deep reinforcement learning. Sci. Rep. 9, 1–10 (2019).10.1038/s41598-018-37186-2 ↗
29. Hecht-Nielsen, R. Neural Networks for Perception (Elsevier, 1992).
30. Gershgorn, D. Quartz. https://qz.com/1034972/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world/ (2017).
31. Mannodi-Kanakkithodi, A. et al. Scoping the polymer genome: a roadmap for rational polymer dielectrics design and beyond. Mater. Tod. 21, 785–796 (2018).10.1016/j.mattod.2017.11.021 ↗
32. Ramakrishna, S. et al. Materials informatics. J. Intell. Manuf. 30, 2307–2326 (2019).10.1007/s10845-018-1392-0 ↗
33. Rickman, J., Lookman, T. & Kalinin, S. Materials informatics: from the atomic-level to the continuum. Acta Mater. 168, 473–510 (2019).10.1016/j.actamat.2019.01.051 ↗
34. Perez, F. & Granger, B. E. Project Jupyter: Computational Narratives as the Engine of Collaborative Data Science. https://blog.jupyter.org/project-jupyter-computational-narratives-as-the-engine-of-collaborative-data-science-2b5fb94c3c58 (2015).
35. Ziatdinov, M., Nelson, C., Vasudevan, R. K., Chen, D. & Kalinin, S. V. Building ferroelectric from the bottom up: the machine learning analysis of the atomic-scale ferroelectric distortions. Appl. Phys. Lett. 115, 052902 (2019).10.1063/1.5109520 ↗
36. Gómez-Bombarelli, R. et al. Automatic chemical design using a data-driven continuous representation of molecules. ACS Cent. Sci. 4, 268–276 (2018).10.1021/acscentsci.7b00572 ↗
37. Gómez-Bombarelli, R. et al. Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach. Nat. Mater. 15, 1120–1127 (2016).10.1038/nmat4717 ↗
38. Sanchez-Lengeling, B. & Aspuru-Guzik, A. J. A. Inverse molecular design using machine learning: generative models for matter engineering. Science 361, 360–365 (2018). c. s.10.1126/science.aat2663 ↗
39. Tabor, D. P. et al. Accelerating the discovery of materials for clean energy in the era of smart automation. Nat. Rev. Mater. 3, 5–20 (2018).10.1038/s41578-018-0005-z ↗
40. Nikolaev, P. et al. Autonomy in materials research: a case study in carbon nanotube growth. NPJ Comp. Mater. 2, 16031 (2016).10.1038/npjcompumats.2016.31 ↗
41. Noack, M. M., Doerk, G. S., Li, R., Fukuto, M. & Yager, K. G. Advances in kriging-based autonomous X-ray scattering experiments. Sci. Rep. 10, 1–17 (2020).10.1038/s41598-019-56847-4 ↗
42. Noack, M. M. et al. A kriging-based approach to autonomous experimentation with applications to X-ray scattering. Sci. Rep. 9, 1–19 (2019).10.1038/s41598-019-48114-3 ↗
43. Kusne, A. G. et al. On-the-fly machine-learning for high-throughput experiments: search for rare-earth-free permanent magnets. Sci. Rep. 4, 1–7 (2014).10.1038/srep06367 ↗
44. Gromski, P. S., Henson, A. B., Granda, J. M. & Cronin, L. How to explore chemical space using algorithms and automation. Nat. Rev. Chem. 3, 119–128 (2019).10.1038/s41570-018-0066-y ↗
45. Steiner, S. et al. Organic synthesis in a modular robotic system driven by a chemical programming language. Science 363, eaav2211 (2019).10.1126/science.aav2211 ↗
46. Henson, A. B., Gromski, P. S. & Cronin, L. Designing algorithms to aid discovery by chemical robots. ACS Cent. Sci. 4, 793–804 (2018).10.1021/acscentsci.8b00176 ↗
47. Campbell, Z. S. & Abolhasani, M. Facile synthesis of anhydrous microparticles using plug-and-play microfluidic reactors. React. Chem. Eng. 5, 1198–1211 (2020).10.1039/D0RE00193G ↗
48. Epps, R. W., Felton, K. C., Coley, C. W. & Abolhasani, M. Automated microfluidic platform for systematic studies of colloidal perovskite nanocrystals: towards continuous nano-manufacturing. Lab a Chip 17, 4040–4047 (2017).10.1039/C7LC00884H ↗
49. Ronneberger, O., Fischer, P. and Brox, T. U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234–241).10.1007/978-3-319-24574-4_28 ↗
50. Ziatdinov, M. AtomAI. https://github.com/ziatdinovmax/atomai (2020).
51. Dablander, F. An introduction to causal inference. Preprint at https://psyarxiv.com/b3fkw/ (2020).10.31234/osf.io/b3fkw ↗
52. Vasudevan, R. K., Ziatdinov, M., Jesse, S. & Kalinin, S. V. Phases and interfaces from real space atomically resolved data: physics-based deep data image analysis. Nano Lett. 16, 5574–5581 (2016).10.1021/acs.nanolett.6b02130 ↗