Neuromorphic-P2M: processing-in-pixel-in-memory paradigm for neuromorphic image sensors

Neuromorphic-P2M: processing-in-pixel-in-memory paradigm for neuromorphic image sensors. Kaiser, M. A., Datta, G., Wang, Z., Jacob, A. P., Beerel, P. A., & Jaiswal, A. R. Frontiers in Neuroinformatics, 2023.

Paper doi abstract bibtex

Edge devices equipped with computer vision must deal with vast amounts of sensory data with limited computing resources. Hence, researchers have been exploring different energy-efficient solutions such as near-sensor, in-sensor, and in-pixel processing, bringing the computation closer to the sensor. In particular, in-pixel processing embeds the computation capabilities inside the pixel array and achieves high energy efficiency by generating low-level features instead of the raw data stream from CMOS image sensors. Many different in-pixel processing techniques and approaches have been demonstrated on conventional frame-based CMOS imagers; however, the processing-in-pixel approach for neuromorphic vision sensors has not been explored so far. In this work, for the first time, we propose an asynchronous non-von-Neumann analog processing-in-pixel paradigm to perform convolution operations by integrating in-situ multi-bit multi-channel convolution inside the pixel array performing analog multiply and accumulate (MAC) operations that consume significantly less energy than their digital MAC alternative. To make this approach viable, we incorporate the circuit's non-ideality, leakage, and process variations into a novel hardware-algorithm co-design framework that leverages extensive HSpice simulations of our proposed circuit using the GF22nm FD-SOI technology node. We verified our framework on state-of-the-art neuromorphic vision sensor datasets and show that our solution consumes 2× lower backend-processor energy while maintaining almost similar front-end (sensor) energy on the IBM DVS128-Gesture dataset than the state-of-the-art while maintaining a high test accuracy of 88.36%.

@ARTICLE{10.3389/fninf.2023.1144301,
AUTHOR={Kaiser, Md Abdullah-Al and Datta, Gourav and Wang, Zixu and Jacob, Ajey P. and Beerel, Peter A. and Jaiswal, Akhilesh R.},   
TITLE={Neuromorphic-P2M: processing-in-pixel-in-memory paradigm for neuromorphic image sensors},      
JOURNAL={Frontiers in Neuroinformatics},      
VOLUME={17},           
YEAR={2023},      
URL={https://www.frontiersin.org/articles/10.3389/fninf.2023.1144301},       
DOI={10.3389/fninf.2023.1144301},      
ISSN={1662-5196},   
ABSTRACT={Edge devices equipped with computer vision must deal with vast amounts of sensory data with limited computing resources. Hence, researchers have been exploring different energy-efficient solutions such as near-sensor, in-sensor, and in-pixel processing, bringing the computation closer to the sensor. In particular, in-pixel processing embeds the computation capabilities inside the pixel array and achieves high energy efficiency by generating low-level features instead of the raw data stream from CMOS image sensors. Many different in-pixel processing techniques and approaches have been demonstrated on conventional frame-based CMOS imagers; however, the processing-in-pixel approach for neuromorphic vision sensors has not been explored so far. In this work, for the first time, we propose an asynchronous non-von-Neumann analog processing-in-pixel paradigm to perform convolution operations by integrating in-situ multi-bit multi-channel convolution inside the pixel array performing analog multiply and accumulate (MAC) operations that consume significantly less energy than their digital MAC alternative. To make this approach viable, we incorporate the circuit's non-ideality, leakage, and process variations into a novel hardware-algorithm co-design framework that leverages extensive HSpice simulations of our proposed circuit using the GF22nm FD-SOI technology node. We verified our framework on state-of-the-art neuromorphic vision sensor datasets and show that our solution consumes ~2× lower backend-processor energy while maintaining almost similar front-end (sensor) energy on the IBM DVS128-Gesture dataset than the state-of-the-art while maintaining a high test accuracy of 88.36%.}
}

Downloads: 0

{"_id":"S5ukre38YqrHwcirK","bibbaseid":"kaiser-datta-wang-jacob-beerel-jaiswal-neuromorphicp2mprocessinginpixelinmemoryparadigmforneuromorphicimagesensors-2023","author_short":["Kaiser, M. A.","Datta, G.","Wang, Z.","Jacob, A. P.","Beerel, P. A.","Jaiswal, A. R."],"bibdata":{"bibtype":"article","type":"article","author":[{"propositions":[],"lastnames":["Kaiser"],"firstnames":["Md","Abdullah-Al"],"suffixes":[]},{"propositions":[],"lastnames":["Datta"],"firstnames":["Gourav"],"suffixes":[]},{"propositions":[],"lastnames":["Wang"],"firstnames":["Zixu"],"suffixes":[]},{"propositions":[],"lastnames":["Jacob"],"firstnames":["Ajey","P."],"suffixes":[]},{"propositions":[],"lastnames":["Beerel"],"firstnames":["Peter","A."],"suffixes":[]},{"propositions":[],"lastnames":["Jaiswal"],"firstnames":["Akhilesh","R."],"suffixes":[]}],"title":"Neuromorphic-P2M: processing-in-pixel-in-memory paradigm for neuromorphic image sensors","journal":"Frontiers in Neuroinformatics","volume":"17","year":"2023","url":"https://www.frontiersin.org/articles/10.3389/fninf.2023.1144301","doi":"10.3389/fninf.2023.1144301","issn":"1662-5196","abstract":"Edge devices equipped with computer vision must deal with vast amounts of sensory data with limited computing resources. Hence, researchers have been exploring different energy-efficient solutions such as near-sensor, in-sensor, and in-pixel processing, bringing the computation closer to the sensor. In particular, in-pixel processing embeds the computation capabilities inside the pixel array and achieves high energy efficiency by generating low-level features instead of the raw data stream from CMOS image sensors. Many different in-pixel processing techniques and approaches have been demonstrated on conventional frame-based CMOS imagers; however, the processing-in-pixel approach for neuromorphic vision sensors has not been explored so far. In this work, for the first time, we propose an asynchronous non-von-Neumann analog processing-in-pixel paradigm to perform convolution operations by integrating in-situ multi-bit multi-channel convolution inside the pixel array performing analog multiply and accumulate (MAC) operations that consume significantly less energy than their digital MAC alternative. To make this approach viable, we incorporate the circuit's non-ideality, leakage, and process variations into a novel hardware-algorithm co-design framework that leverages extensive HSpice simulations of our proposed circuit using the GF22nm FD-SOI technology node. We verified our framework on state-of-the-art neuromorphic vision sensor datasets and show that our solution consumes 2× lower backend-processor energy while maintaining almost similar front-end (sensor) energy on the IBM DVS128-Gesture dataset than the state-of-the-art while maintaining a high test accuracy of 88.36%.","bibtex":"@ARTICLE{10.3389/fninf.2023.1144301,\r\nAUTHOR={Kaiser, Md Abdullah-Al and Datta, Gourav and Wang, Zixu and Jacob, Ajey P. and Beerel, Peter A. and Jaiswal, Akhilesh R.}, \r\nTITLE={Neuromorphic-P2M: processing-in-pixel-in-memory paradigm for neuromorphic image sensors}, \r\nJOURNAL={Frontiers in Neuroinformatics}, \r\nVOLUME={17}, \r\nYEAR={2023}, \r\nURL={https://www.frontiersin.org/articles/10.3389/fninf.2023.1144301}, \r\nDOI={10.3389/fninf.2023.1144301}, \r\nISSN={1662-5196}, \r\nABSTRACT={Edge devices equipped with computer vision must deal with vast amounts of sensory data with limited computing resources. Hence, researchers have been exploring different energy-efficient solutions such as near-sensor, in-sensor, and in-pixel processing, bringing the computation closer to the sensor. In particular, in-pixel processing embeds the computation capabilities inside the pixel array and achieves high energy efficiency by generating low-level features instead of the raw data stream from CMOS image sensors. Many different in-pixel processing techniques and approaches have been demonstrated on conventional frame-based CMOS imagers; however, the processing-in-pixel approach for neuromorphic vision sensors has not been explored so far. In this work, for the first time, we propose an asynchronous non-von-Neumann analog processing-in-pixel paradigm to perform convolution operations by integrating in-situ multi-bit multi-channel convolution inside the pixel array performing analog multiply and accumulate (MAC) operations that consume significantly less energy than their digital MAC alternative. To make this approach viable, we incorporate the circuit's non-ideality, leakage, and process variations into a novel hardware-algorithm co-design framework that leverages extensive HSpice simulations of our proposed circuit using the GF22nm FD-SOI technology node. We verified our framework on state-of-the-art neuromorphic vision sensor datasets and show that our solution consumes ~2× lower backend-processor energy while maintaining almost similar front-end (sensor) energy on the IBM DVS128-Gesture dataset than the state-of-the-art while maintaining a high test accuracy of 88.36%.}\r\n}\n\n","author_short":["Kaiser, M. A.","Datta, G.","Wang, Z.","Jacob, A. P.","Beerel, P. A.","Jaiswal, A. R."],"bibbaseid":"kaiser-datta-wang-jacob-beerel-jaiswal-neuromorphicp2mprocessinginpixelinmemoryparadigmforneuromorphicimagesensors-2023","role":"author","urls":{"Paper":"https://www.frontiersin.org/articles/10.3389/fninf.2023.1144301"},"metadata":{"authorlinks":{}}},"bibtype":"article","biburl":"https://bibbase.org/f/6NMaaRPNDKskwdzK4/ajey-2023.bib","dataSources":["yCKGz5KC8LnXA68E7"],"keywords":[],"search_terms":["neuromorphic","p2m","processing","pixel","memory","paradigm","neuromorphic","image","sensors","kaiser","datta","wang","jacob","beerel","jaiswal"],"title":"Neuromorphic-P2M: processing-in-pixel-in-memory paradigm for neuromorphic image sensors","year":2023}