Quantized artificial neural networks implemented with spintronic stochastic computing
Source abstract
Abstract
Abstract An artificial neural network (ANN) inference involves matrix vector multiplications that require a very large number of multiply and accumulate operations, resulting in high energy cost and large device footprint. Stochastic computing (SC) offers a less resource-intensive ANN implementation with minimal accuracy loss. Random number generators (RNG) are required to implement SC in hardware. These can be realized through stochastic-magnetic tunnel junctions (s-MTJ), where the energy barrier to switch between the ‘up’ and ‘down’ states is designed to be small, enabling thermal noise to generate a random bit stream. While s-MTJs have previously been used to implement SC-ANNs, these studies have been limited to architectures with continuously varying (i.e., analog) weights. In this work, we study the use of SC for matrix vector multiplication with quantized synaptic weights and quantized outputs. We show that a quantized SC-ANN, implemented by using experimentally obtained s-MTJ bitstreams and a limited number of discrete quantized states for both weights and hidden layer nodes in an ANN, can effectively reduce time (latency) and energy consumption in SC compared to an analog implementation, while largely preserving accuracy. We implemented quantization with 5 and 11 quantized states, along with SC configured with stochastic bitstream lengths of 100, 200, 300, 400, and 500 on neural networks with one hidden layer and three hidden layers. Inference was performed on the MNIST dataset for both training with SC and without SC. Training with SC provided better accuracy for all cases. For the shortest bitstream of 100 bits, the highest accuracies were 92% for one hidden layer and over 96% for three hidden layers. The overall system attained its peak accuracy of 96.82% using a 400 bit stochastic bitstream with three hidden layers. Our investigations demonstrate 9× improvement in latency and 2.6× improvement in energy consumption using the quantized SC approach compared to a similar s-MTJ based ANN architecture without quantization.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Neural Networks and Reservoir Computing · Error Correcting Code Techniques · Advanced Memory and Neural Computing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (46)
Show source-provided references
1. The hardware and algorithm co-design for energy-efficient DNN processor on edge/mobile devices10.1109/TCSI.2020.3021397 ↗
2. Targeting DNN inference via efficient utilization of heterogeneous precision DNN accelerators10.1109/TETC.2022.3178730 ↗
3. Fully parallel stochastic computing hardware implementation of convolutional neural networks for edge computing applications10.1109/TNNLS.2022.3166799 ↗
4. sBSNN: stochastic-bits enabled binary spiking neural network with on-chip learning for energy efficient neuromorphic computing at the edge10.1109/TCSI.2020.2979826 ↗
5. Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks10.1145/2897937.2898011 ↗
6. “Towards acceleration of deep convolutional neural networks using stochastic computing10.1109/ASPDAC.2017.7858306 ↗
7. “Probabilistic-based neural network implementation10.1109/IJCNN.2012.6252807 ↗
8. A stochastic computational multi-layer perceptron with backward propagation10.1109/TC.2018.2817237 ↗
9. A hardware implementation of a radial basis function neural network using stochastic logic
10. DSCNN: hardware-oriented optimization for stochastic computing based deep convolutional neural networks
11. An energy-efficient online-learning stochastic computational deep belief network10.1109/JETCAS.2018.2852705 ↗
12. An energy-efficient and noise-tolerant recurrent neural network using stochastic computing10.1109/TVLSI.2019.2920152 ↗
13. A survey of stochastic computing neural networks for machine learning applications10.1109/TNNLS.2020.3009047 ↗
14. A fast fault-tolerant architecture for Sauvola local image thresholding algorithm using stochastic computing10.1109/TVLSI.2015.2415932 ↗
15. Stochastic circuits for real-time image-processing applications
16. A low power fault-tolerant architecture for the kernel density estimation based image segmentation algorithm
17. Review of magnetic tunnel junctions for stochastic computing10.1109/JXCDC.2022.3227062 ↗
18. Integer factorization using stochastic magnetic tunnel junctions10.1038/s41586-019-1557-9 ↗
19. Self-heating phase-change memory-array demonstrator for true random number generation10.1109/TED.2017.2673867 ↗
20. A small area and low power true random number generator using write speed variation of oxidebased RRAM for IoT security application10.1109/ISCAS.2017.8051019 ↗
21. A fast, energy efficient and tunable magnetic tunnel junction based bitstream generator for stochastic computing10.1109/TCSI.2022.3173030 ↗
22. A true random number generator based on parallel STT-MTJs10.23919/DATE.2017.7927058 ↗
23. Binary and ternary true random number generators based on spin orbit torque10.1109/IEDM.2018.8614638 ↗
24. Implementation of artificial neural networks using magnetoresistive random-access memory-based stochastic computing units10.1109/LMAG.2021.3071084 ↗
25. Energy-efficient stochastic computing with superparamagnetic tunnel junctions10.1103/PhysRevApplied.13.034016 ↗
26. Neuromorphic computing using non-volatile memory10.1080/23746149.2016.1259585 ↗
27. Experimental demonstration and tolerancing of a large-scale neural network (165 000 synapses) using phase-change memory as the synaptic weight element10.1109/TED.2015.2439635 ↗
28. Quantized neural networks: training neural networks with low precision weights and activations
29. Energy efficient learning with low resolution stochastic domain wall synapse for deep neural networks10.1109/ACCESS.2022.3196688 ↗
30. Quantized magnetic domain wall synapse for efficient deep neural networks10.1109/TNNLS.2024.3369969 ↗
31. Quantized non-volatile nanomagnetic domain wall synapse based autoencoder for efficient unsupervised network anomaly detection10.1088/2634-4386/ad49ce ↗
32. A domain wall-magnetic tunnel junction artificial synapse with notched geometry for accurate and efficient training of deep neural networks10.1063/5.0046032 ↗
33. Implementation of binarized neural networks immune to device variation and voltage drop employing resistive random access memory bridges and capacitive neurons10.1038/s44172-024-00226-z ↗
34. Stochastic neural computation. I. computational elements10.1109/12.954505 ↗
35. Asymmetry of MTJ switching and its implication to STT-RAM designs10.1109/DATE.2012.6176695 ↗
36. Switching current reduction using perpendicular anisotropy in CoFeB-MgO magnetic tunnel junctions10.1063/1.3567780 ↗
37. Spin torque switching of perpendicular Ta|CoFeB|MgO-based magnetic tunnel junctions10.1063/1.3542842 ↗
38. A perpendicular-anisotropy CoFeB–MgO magnetic tunnel junction10.1038/nmat2804 ↗
39. Probabilistic computing with voltage-controlled dynamics in magnetic tunnel junctions10.1088/1361-6528/acf6c7 ↗
40. Integrated probabilistic computer using voltage-controlled magnetic tunnel junctions as its entropy source
41. Demonstration of nanosecond operation in stochastic magnetic tunnel junctions10.1021/acs.nanolett.0c04652 ↗
42. Approximate de-randomizer for stochastic circuits10.1109/ISOCC.2015.7401667 ↗
43. Multiplexer optimization for adders in stochastic computing10.1145/3611315.3633256 ↗
44. Computing arithmetic functions using stochastic logic by series expansion10.1109/TETC.2016.2618750 ↗
45. Quantization and training of neural networks for efficient integer-arithmetic-only inference
46. Accelerating deep convolutional neural network base on stochastic computing10.1016/j.vlsi.2020.09.008 ↗