Analog Resistive Switching Devices for Training Deep Neural Networks with the Novel Tiki-Taka Algorithm
Source abstract
Abstract
High Resolution Image Download MS PowerPoint Slide A critical bottleneck for the training of large neural networks (NNs) is communication with off-chip memory. A promising mitigation effort consists of integrating crossbar arrays of analogue memories in the Back-End-Of-Line, to store the NN parameters and efficiently perform the required synaptic operations. The “ Tiki-Taka ” algorithm was developed to facilitate NN training in the presence of device nonidealities. However, so far, a resistive switching device exhibiting all the fundamental Tiki-Taka requirements, which are many programmable states, a centered symmetry point, and low programming noise, was not yet demonstrated. Here, a complementary metal-oxide semiconductor (CMOS)-compatible resistive random access memory (RRAM), showing more than 30 programmable states with low noise and a symmetry point with only 5% skew from the center, is presented for the first time. These results enable generalization of Tiki-Taka training from small fully connected networks to larger long-/short-term-memory types of NN.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Advanced Memory and Neural Computing · Ferroelectric and Negative Capacitance Devices · Transition Metal Oxide Nanomaterials
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
IBM Research Europe - Zürich, Rüschlikon, Zürich CH 8803, Switzerland
IBM Research - Yorktown Heights, Yorktown Heights, New York 10598, United States
Funding
Schweizerischer Nationalfonds zur F?rderung der Wissenschaftlichen Forschung · H2020 Marie Sklodowska-Curie Actions · H2020 Industrial Leadership · HORIZON EUROPE Digital, Industry and Space
References (34)
Show source-provided references
1. Amodei, D.; Hernandez, D. AI and compute. OpenAI Blog (16 May 2018); https://openai.com/blog/ai-and-compute/.
2. 10.1109/JPROC.1998.65876210.1109/JPROC.1998.658762 ↗
3. 10.1038/s41565-020-0655-z10.1038/s41565-020-0655-z ↗
4. Horowitz M. · 2014
5. 10.1038/s41928-017-0006-810.1038/s41928-017-0006-8 ↗
6. 10.1038/s41928-018-0092-210.1038/s41928-018-0092-2 ↗
7. 10.1109/TVLSI.2022.322139010.1109/TVLSI.2022.3221390 ↗
8. 10.1109/TCSII.2021.307861410.1109/TCSII.2021.3078614 ↗
9. 10.3389/fnins.2016.0033310.3389/fnins.2016.00333 ↗
10. 10.1088/0268-1242/31/6/06300210.1088/0268-1242/31/6/063002 ↗
11. 10.1109/JETCAS.2016.254771810.1109/JETCAS.2016.2547718 ↗
12. 10.3389/fnins.2020.0010310.3389/fnins.2020.00103 ↗
13. 10.1016/S0893-6080(98)00116-610.1016/S0893-6080(98)00116-6 ↗
14. 10.1038/ncomms1519910.1038/ncomms15199 ↗
15. 10.1038/s41586-020-1942-410.1038/s41586-020-1942-4 ↗
16. Gong, N. Deep learning acceleration in 14nm CMOS compatible ReRAM array: device, material and algorithm co-optimization. Technical Digest - International Electron Devices Meeting, IEDM 2022, 2022; pp 3371–3374.10.1109/IEDM45625.2022.10019569 ↗
17. 10.1063/1.512056510.1063/1.5120565 ↗
18. Kim S.
19. 10.1038/s41467-023-41958-410.1038/s41467-023-41958-4 ↗
20. 10.1002/advs.20230301810.1002/advs.202303018 ↗
21. 10.1038/s41598-023-42214-x10.1038/s41598-023-42214-x ↗
22. 10.1002/aelm.20220044810.1002/aelm.202200448 ↗
23. 10.3389/frai.2021.69914810.3389/frai.2021.699148 ↗
24. 10.3389/fnano.2022.100826610.3389/fnano.2022.1008266 ↗
25. 10.1109/LED.2014.233431110.1109/LED.2014.2334311 ↗
26. 10.1109/TED.2015.241811410.1109/TED.2015.2418114 ↗
27. 10.1039/c2tc00481j10.1039/c2tc00481j ↗
28. 10.1088/0268-1242/12/9/00110.1088/0268-1242/12/9/001 ↗
29. 10.1039/C9MH02033K10.1039/C9MH02033K ↗
30. Rasch M. J.
31. 10.1093/comjnl/7.2.15510.1093/comjnl/7.2.155 ↗
32. Newville, M.; Stensitzki, T.; Allen, D. B.; Rawlik, M.; Ingargiola, A.; Nelson, A. LMFIT: Non-linear least-square minimization and curve-fitting for Python. Astrophysics Source Code Library, 2016, ascl–1606. https://ui.adsabs.harvard.edu/abs/2016ascl.soft06014N/abstract.
33. 10.1109/MSP.2012.221147710.1109/MSP.2012.2211477 ↗
34. Bengio, Y.; Ducharme, R.; Vincent, P. A Neural Probabilistic Language Model. Advances in neural information processing systems, 2000, 13. https://proceedings.neurips.cc/paper_files/paper/2000/file/728f206c2a01bf572b5940d7d9a8fa4c-Paper.pdf.