Content-aware frame interpolation (CAFI): Deep Learning-based temporal super-resolution for fast bioimaging

Content-aware frame interpolation (CAFI): Deep Learning-based temporal super-resolution for fast bioimaging. Priessner, M., Gaboriau, D. C., Sheridan, A., Lenn, T., Chubb, J. R., Manor, U., Vilar, R., & Laine, R. F. bioRxiv, Cold Spring Harbor Laboratory, 2021.

Paper doi abstract bibtex

The development of high-resolution microscopes has made it possible to investigate cellular processes in 4D (3D over time). However, observing fast cellular dynamics remains challenging as a consequence of photobleaching and phototoxicity. These issues become increasingly problematic with the depth of the volume acquired and the speed of the biological events of interest. Here, we report the implementation of two content-aware frame interpolation (CAFI) deep learning networks, Zooming SlowMo (ZS) and Depth-Aware Video Frame Interpolation (DAIN), based on combinations of recurrent neural networks, that are highly suited for accurately predicting images in between image pairs, therefore improving the temporal resolution of image series as a post-acquisition analysis step. We show that CAFI predictions are capable of understanding the motion context of biological structures to perform better than standard interpolation methods. We benchmark CAFI\textquoterights performance on six different datasets, obtained from three different microscopy modalities (point-scanning confocal, spinning-disk confocal and confocal brightfield microscopy). We demonstrate its capabilities for single-particle tracking methods applied to the study of lysosome trafficking. CAFI therefore allows for reduced light exposure and phototoxicity on the sample and extends the possibility of long-term live-cell imaging. Both DAIN and ZS as well as the training and testing data are made available for use by the wider community via the ZeroCostDL4Mic platform.Competing Interest StatementThe authors have declared no competing interest.

@article {Priessner2021.11.02.466664,
	author = {Martin Priessner and David C.A. Gaboriau and Arlo Sheridan and Tchern Lenn and Jonathan R. Chubb and Uri Manor and Ramon Vilar and Romain F. Laine},
	title = {Content-aware frame interpolation (CAFI): Deep Learning-based temporal super-resolution for fast bioimaging},
	elocation-id = {2021.11.02.466664},
	year = {2021},
	doi = {10.1101/2021.11.02.466664},
	publisher = {Cold Spring Harbor Laboratory},
	abstract = {The development of high-resolution microscopes has made it possible to investigate cellular processes in 4D (3D over time). However, observing fast cellular dynamics remains challenging as a consequence of photobleaching and phototoxicity. These issues become increasingly problematic with the depth of the volume acquired and the speed of the biological events of interest. Here, we report the implementation of two content-aware frame interpolation (CAFI) deep learning networks, Zooming SlowMo (ZS) and Depth-Aware Video Frame Interpolation (DAIN), based on combinations of recurrent neural networks, that are highly suited for accurately predicting images in between image pairs, therefore improving the temporal resolution of image series as a post-acquisition analysis step. We show that CAFI predictions are capable of understanding the motion context of biological structures to perform better than standard interpolation methods. We benchmark CAFI{\textquoteright}s performance on six different datasets, obtained from three different microscopy modalities (point-scanning confocal, spinning-disk confocal and confocal brightfield microscopy). We demonstrate its capabilities for single-particle tracking methods applied to the study of lysosome trafficking. CAFI therefore allows for reduced light exposure and phototoxicity on the sample and extends the possibility of long-term live-cell imaging. Both DAIN and ZS as well as the training and testing data are made available for use by the wider community via the ZeroCostDL4Mic platform.Competing Interest StatementThe authors have declared no competing interest.},
	URL = {https://www.biorxiv.org/content/early/2021/11/03/2021.11.02.466664},
	eprint = {https://www.biorxiv.org/content/early/2021/11/03/2021.11.02.466664.full.pdf},
	journal = {bioRxiv}
}

Downloads: 0

{"_id":"4Zh6fys6egdGePATH","bibbaseid":"priessner-gaboriau-sheridan-lenn-chubb-manor-vilar-laine-contentawareframeinterpolationcafideeplearningbasedtemporalsuperresolutionforfastbioimaging-2021","author_short":["Priessner, M.","Gaboriau, D. C.","Sheridan, A.","Lenn, T.","Chubb, J. R.","Manor, U.","Vilar, R.","Laine, R. F."],"bibdata":{"bibtype":"article","type":"article","author":[{"firstnames":["Martin"],"propositions":[],"lastnames":["Priessner"],"suffixes":[]},{"firstnames":["David","C.A."],"propositions":[],"lastnames":["Gaboriau"],"suffixes":[]},{"firstnames":["Arlo"],"propositions":[],"lastnames":["Sheridan"],"suffixes":[]},{"firstnames":["Tchern"],"propositions":[],"lastnames":["Lenn"],"suffixes":[]},{"firstnames":["Jonathan","R."],"propositions":[],"lastnames":["Chubb"],"suffixes":[]},{"firstnames":["Uri"],"propositions":[],"lastnames":["Manor"],"suffixes":[]},{"firstnames":["Ramon"],"propositions":[],"lastnames":["Vilar"],"suffixes":[]},{"firstnames":["Romain","F."],"propositions":[],"lastnames":["Laine"],"suffixes":[]}],"title":"Content-aware frame interpolation (CAFI): Deep Learning-based temporal super-resolution for fast bioimaging","elocation-id":"2021.11.02.466664","year":"2021","doi":"10.1101/2021.11.02.466664","publisher":"Cold Spring Harbor Laboratory","abstract":"The development of high-resolution microscopes has made it possible to investigate cellular processes in 4D (3D over time). However, observing fast cellular dynamics remains challenging as a consequence of photobleaching and phototoxicity. These issues become increasingly problematic with the depth of the volume acquired and the speed of the biological events of interest. Here, we report the implementation of two content-aware frame interpolation (CAFI) deep learning networks, Zooming SlowMo (ZS) and Depth-Aware Video Frame Interpolation (DAIN), based on combinations of recurrent neural networks, that are highly suited for accurately predicting images in between image pairs, therefore improving the temporal resolution of image series as a post-acquisition analysis step. We show that CAFI predictions are capable of understanding the motion context of biological structures to perform better than standard interpolation methods. We benchmark CAFI\\textquoterights performance on six different datasets, obtained from three different microscopy modalities (point-scanning confocal, spinning-disk confocal and confocal brightfield microscopy). We demonstrate its capabilities for single-particle tracking methods applied to the study of lysosome trafficking. CAFI therefore allows for reduced light exposure and phototoxicity on the sample and extends the possibility of long-term live-cell imaging. Both DAIN and ZS as well as the training and testing data are made available for use by the wider community via the ZeroCostDL4Mic platform.Competing Interest StatementThe authors have declared no competing interest.","url":"https://www.biorxiv.org/content/early/2021/11/03/2021.11.02.466664","eprint":"https://www.biorxiv.org/content/early/2021/11/03/2021.11.02.466664.full.pdf","journal":"bioRxiv","bibtex":"@article {Priessner2021.11.02.466664,\r\n\tauthor = {Martin Priessner and David C.A. Gaboriau and Arlo Sheridan and Tchern Lenn and Jonathan R. Chubb and Uri Manor and Ramon Vilar and Romain F. Laine},\r\n\ttitle = {Content-aware frame interpolation (CAFI): Deep Learning-based temporal super-resolution for fast bioimaging},\r\n\telocation-id = {2021.11.02.466664},\r\n\tyear = {2021},\r\n\tdoi = {10.1101/2021.11.02.466664},\r\n\tpublisher = {Cold Spring Harbor Laboratory},\r\n\tabstract = {The development of high-resolution microscopes has made it possible to investigate cellular processes in 4D (3D over time). However, observing fast cellular dynamics remains challenging as a consequence of photobleaching and phototoxicity. These issues become increasingly problematic with the depth of the volume acquired and the speed of the biological events of interest. Here, we report the implementation of two content-aware frame interpolation (CAFI) deep learning networks, Zooming SlowMo (ZS) and Depth-Aware Video Frame Interpolation (DAIN), based on combinations of recurrent neural networks, that are highly suited for accurately predicting images in between image pairs, therefore improving the temporal resolution of image series as a post-acquisition analysis step. We show that CAFI predictions are capable of understanding the motion context of biological structures to perform better than standard interpolation methods. We benchmark CAFI{\\textquoteright}s performance on six different datasets, obtained from three different microscopy modalities (point-scanning confocal, spinning-disk confocal and confocal brightfield microscopy). We demonstrate its capabilities for single-particle tracking methods applied to the study of lysosome trafficking. CAFI therefore allows for reduced light exposure and phototoxicity on the sample and extends the possibility of long-term live-cell imaging. Both DAIN and ZS as well as the training and testing data are made available for use by the wider community via the ZeroCostDL4Mic platform.Competing Interest StatementThe authors have declared no competing interest.},\r\n\tURL = {https://www.biorxiv.org/content/early/2021/11/03/2021.11.02.466664},\r\n\teprint = {https://www.biorxiv.org/content/early/2021/11/03/2021.11.02.466664.full.pdf},\r\n\tjournal = {bioRxiv}\r\n}\r\n\r\n","author_short":["Priessner, M.","Gaboriau, D. C.","Sheridan, A.","Lenn, T.","Chubb, J. R.","Manor, U.","Vilar, R.","Laine, R. F."],"key":"Priessner2021.11.02.466664","id":"Priessner2021.11.02.466664","bibbaseid":"priessner-gaboriau-sheridan-lenn-chubb-manor-vilar-laine-contentawareframeinterpolationcafideeplearningbasedtemporalsuperresolutionforfastbioimaging-2021","role":"author","urls":{"Paper":"https://www.biorxiv.org/content/early/2021/11/03/2021.11.02.466664"},"metadata":{"authorlinks":{}}},"bibtype":"article","biburl":"https://github.com/salkmanorlab/manor_publications/raw/main/manorlab_pubs.bib","dataSources":["edXsq84pNeYoA6wo6","dr3nCm3wyZkR7nFZf"],"keywords":[],"search_terms":["content","aware","frame","interpolation","cafi","deep","learning","based","temporal","super","resolution","fast","bioimaging","priessner","gaboriau","sheridan","lenn","chubb","manor","vilar","laine"],"title":"Content-aware frame interpolation (CAFI): Deep Learning-based temporal super-resolution for fast bioimaging","year":2021}