Abstract
The proliferation of microscopy methods for live-cell imaging offers many new possibilities for users but can also be challenging to navigate. The prevailing challenge in live-cell fluorescence microscopy is capturing intra-cellular dynamics while preserving cell viability. Computational methods can help to address this challenge and are now shifting the boundaries of what is possible to capture in living systems. In this Review, we discuss these computational methods focusing on artificial intelligence-based approaches that can be layered on top of commonly used existing microscopies as well as hybrid methods that integrate computation and microscope hardware. We specifically discuss how computational approaches can improve the signal-to-noise ratio, spatial resolution, temporal resolution and multi-colour capacity of live-cell imaging.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$189.00 per year
only $15.75 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Ducret, A., Quardokus, E. M. & Brun, Y. V. MicrobeJ, a tool for high throughput bacterial cell detection and quantitative analysis. Nat. Microbiol. 1, 16077 (2016).
Levet, F. et al. SR-Tesseler: a method to segment and quantify localization-based super-resolution microscopy data. Nat. Methods 12, 1065–1071 (2015).
Rizk, A. et al. Segmentation and quantification of subcellular structures in fluorescence microscopy images using Squassh. Nat. Protoc. 9, 586–596 (2014).
Ulman, V. et al. An objective comparison of cell-tracking algorithms. Nat. Methods 14, 1141–1152 (2017).
Axelrod, D., Koppel, D. E., Schlessinger, J., Elson, E. & Webb, W. W. Mobility measurement by analysis of fluorescence photobleaching recovery kinetics. Biophys. J. 16, 1055–1069 (1976).
Elson, E. L. & Magde, D. Fluorescence correlation spectroscopy. I. Conceptual basis and theory. Biopolymers 13, 1–27 (1974).
Gahlmann, A. & Moerner, W. E. Exploring bacterial cell biology with single-molecule tracking and super-resolution imaging. Nat. Rev. Microbiol. 12, 9–22 (2014).
Saxton, M. J. & Jacobson, K. Single-particle tracking: applications to membrane dynamics. Annu. Rev. Biophys. Biomol. Struct. 26, 373–399 (1997).
Moen, E. et al. Deep learning for cellular image analysis. Nat. Methods 16, 1233–1246 (2019).
Van Valen, D. A. et al. Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments. PLoS Comput. Biol. 12, e1005177 (2016).
Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).
Laissue, P. P., Alghamdi, R. A., Tomancak, P., Reynaud, E. G. & Shroff, H. Assessing phototoxicity in live fluorescence imaging. Nat. Methods 14, 657–661 (2017).
Grimm, J. B. et al. A general method to improve fluorophores for live-cell and single-molecule microscopy. Nat. Methods 12, 244–250 (2015).
Lukinavičius, G. et al. A near-infrared fluorophore for live-cell super-resolution microscopy of cellular proteins. Nat. Chem. 5, 132–139 (2013).
Shaner, N. C. et al. Improving the photostability of bright monomeric orange and red fluorescent proteins. Nat. Methods 5, 545–551 (2008).
Gustafsson, M. G. L. et al. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys. J. 94, 4957–4970 (2008).
Hell, S. W. Far-field optical nanoscopy. Science 316, 1153–1158 (2007).
Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645 (2006).
Rust, M. J., Bates, M. & Zhuang, X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3, 793–795 (2006).
Denk, W., Strickler, J. H. & Webb, W. W. Two-photon laser scanning fluorescence microscopy. Science 248, 73–76 (1990).
Huisken, J., Swoger, J., Del Bene, F., Wittbrodt, J. & Stelzer, E. H. K. Optical sectioning deep inside live embryos by selective plane illumination microscopy. Science 305, 1007–1009 (2004).
Möckl, L., Roy, A. R., Petrov, P. N. & Moerner, W. E. Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGnet. Proc. Natl Acad. Sci. USA 117, 60–67 (2020).
Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).
Huang, T. S., Yang, G. J. & Tang, G. Y. A fast two-dimensional median filtering algorithm. IEEE Trans. Signal. Process. 27, 13–18 (1979).
Strong, D. & Chan, T. Edge-preserving and scale-dependent properties of total variation regularization. Inverse Problems 19, S165–S187 (2003).
Buades, A., Coll, B. & Morel, J.M. A non-local algorithm for image denoising. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05). 60-65 (2005).
Huang, X. et al. Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy. Nat. Biotechnol. 36, 451–459 (2018).
Lecun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Chen, J. et al. Three-dimensional residual channel attention networks denoise and sharpen fluorescence microscopy image volumes. Nat. Methods 18, 678–687 (2021).
Fang, L. et al. Deep learning-based point-scanning super-resolution imaging. Nat. Methods 18, 406–416 (2021).
Qiao, C. et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat. Methods 18, 194–202 (2021).
Qiao, C. et al. Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes. Nat. Biotechnol. 41, 367–377 (2023).
Gustafsson, M. G. L. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J. Microsc. 198, 82–87 (2000).
Chen, B. C. et al. Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution. Science 346, 1257998 (2014).
York, A. G. et al. Instant super-resolution imaging in live cells and embryos via analog image processing. Nat. Methods 10, 1122–1130 (2013).
Hoebe, R. A. et al. Controlled light-exposure microscopy reduces photobleaching and phototoxicity in fluorescence live-cell imaging. Nat. Biotechnol. 25, 249–253 (2007).
Chu, K. K., Lim, D. & Mertz, J. Enhanced weak-signal sensitivity in two-photon microscopy by adaptive illumination. Opt. Lett. 32, 2846–2848 (2007).
Li, B., Wu, C., Wang, M., Charan, K. & Xu, C. An adaptive excitation source for high-speed multiphoton microscopy. Nat. Methods 17, 163–166 (2020).
Staudt, T. et al. Far-field optical nanoscopy with reduced number of state transition cycles. Opt. Express 19, 5644–5657 (2011).
Chakrova, N., Canton, A. S., Danelon, C., Stallinga, S. & Rieger, B. Adaptive illumination reduces photobleaching in structured illumination microscopy. Biomed. Opt. Express 7, 4263–4274 (2016).
Göttfert, F. et al. Strong signal increase in STED fluorescence microscopy by imaging regions of subdiffraction extent. Proc. Natl Acad. Sci. USA 114, 2125–2130 (2017).
Heine, J. et al. Adaptive-illumination STED nanoscopy. Proc. Natl Acad. Sci. USA 114, 9797–9802 (2017).
Dreier, J. et al. Smart scanning for low-illumination and fast RESOLFT nanoscopy in vivo.Nat. Commun. 10, 556 (2019).
Vinçon, B., Geisler, C. & Egner, A. Pixel hopping enables fast STED nanoscopy at low light dose. Opt. Express 28, 4516–4528 (2020).
Štefko, M., Ottino, B., Douglass, K. M. & Manley, S. Autonomous illumination control for localization microscopy. Opt. Express 26, 30882–30900 (2018).
Chiron, L. et al. CyberSco.Py an open-source software for event-based, conditional microscopy. Sci. Rep. 12, 11579 (2022).
Almada, P. et al. Automating multimodal microscopy with NanoJ-Fluidics. Nat. Commun. 10, 1223 (2019).
Edelstein, A. D. et al. Advanced methods of microscope control using μManager software. J. Biol. Methods 1, e10 (2014).
Pinkard, H., Stuurman, N., Corbin, K., Vale, R. & Krummel, M. F. Micro-Magellan: open-source, sample-adaptive, acquisition software for optical microscopy. Nat. Methods 13, 807–809 (2016).
Fox, Z. R. et al. Enabling reactive microscopy with MicroMator. Nat. Commun. 13, 2199 (2022).
Casas Moreno, X. et al. An open-source microscopy framework for simultaneous control of image acquisition, reconstruction, and analysis. HardwareX 13, e00400 (2023).
Schermelleh, L. et al. Super-resolution microscopy demystified. Nat. Cell Biol. 21, 72–84 (2019).
Wu, Y. & Shroff, H. Multiscale fluorescence imaging of living samples. Histochem. Cell Biol. 158, 301–323 (2022).
Sarder, P. & Nehorai, A. Deconvolution methods for 3-D fluorescence microscopy images. IEEE Signal. Process. Mag. 23, 32–45 (2006).
Lucy, L. B. An iterative technique for the rectification of observed distributions. Astron. J. 79, 745–754 (1974).
Richardson, W. H. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am. 62, 55–59 (1972).
Sage, D. et al. DeconvolutionLab2: an open-source software for deconvolution microscopy. Methods 115, 28–41 (2017).
Zhao, W. et al. Sparse deconvolution improves the resolution of live-cell super-resolution fluorescence microscopy. Nat. Biotechnol. 40, 606–617 (2022).
Li, Y. et al. Incorporating the image formation process into deep learning improves network performance. Nat. Methods 19, 1427–1437 (2022).
Chhetri, R. K. et al. Whole-animal functional and developmental imaging with isotropic spatial resolution. Nat. Methods 12, 1171–1178 (2015).
Wu, Y. et al. Spatially isotropic four-dimensional imaging with dual-view plane illumination microscopy. Nat. Biotechnol. 31, 1032–1038 (2013).
Wu, Y. et al. Multiview confocal super-resolution microscopy. Nature 600, 279–284 (2021).
Ingaramo, M. et al. Richardson-Lucy deconvolution as a general tool for combining images with complementary strengths. ChemPhysChem 15, 794–800 (2014).
Kazemipour, A. et al. Kilohertz frame-rate two-photon tomography. Nat. Methods 16, 778–786 (2019).
Wang, H. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods 16, 103–110 (2019).
Li, X. et al. Three-dimensional structured illumination microscopy with enhanced axial resolution. Nat. Biotechnol. 41, 1307–1319 (2023).
Weigert, M., Royer, L., Jug, F. & Myers, G. Isotropic reconstruction of 3D fluorescence microscopy images using convolutional neural networks. Preprint at https://doi.org/10.48550/arXiv.1704.01510 (2017).
Hampson, K. M. et al. Adaptive optics for high-resolution imaging. Nat. Rev. Methods Prim. 1, 68 (2021).
Ji, N. Adaptive optical fluorescence microscopy. Nat. Methods 14, 374–380 (2017).
Wang, K. et al. Rapid adaptive optical recovery of optimal resolution over large volumes. Nat. Methods 11, 625–628 (2014).
Liu, T. L. et al. Observing the cell in its native state: imaging subcellular dynamics in multicellular organisms. Science 360, eaaq1392 (2018).
Rodríguez, C. et al. An adaptive optics module for deep tissue multiphoton imaging in vivo. Nat. Methods 18, 1259–1264 (2021).
Streich, L. et al. High-resolution structural and functional deep brain imaging using adaptive optics three-photon microscopy. Nat. Methods 18, 1253–1258 (2021).
Lin, R., Kipreos, E. T., Zhu, J., Khang, C. H. & Kner, P. Subcellular three-dimensional imaging deep through multicellular thick samples by structured illumination microscopy and adaptive optics. Nat. Commun. 12, 3148 (2021).
Zheng, W. et al. Adaptive optics improves multiphoton super-resolution imaging. Nat. Methods 14, 869–872 (2017).
Saha, D. et al. Practical sensorless aberration estimation for 3D microscopy with deep learning. Opt. Express 28, 29044–29053 (2020).
Vinogradova, K. & Myers, E.W. Estimation of optical aberrations in 3D microscopic bioimages. 7th International Conference on Frontiers of Signal Processing, ICFSP. 97-103 (2022).
Feng, B. Y. et al. NeuWS: neural wavefront shaping for guidestar-free imaging through static and dynamic scattering media. Sci. Adv. 9, eadg4671 (2023).
Balzarotti, F. et al. Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes. Science 355, 606–612 (2017).
Deguchi, T. et al. Direct observation of motor protein stepping in living cells using MINFLUX. Science 379, 1010–1015 (2023).
Wolff, J. O. et al. MINFLUX dissects the unimpeded walking of kinesin-1. Science 379, 1004–1010 (2023).
Royer, L. A. et al. Adaptive light-sheet microscopy for long-term, high-resolution imaging in living organisms. Nat. Biotechnol. 34, 1267–1278 (2016).
Pinkard, H. et al. Learned adaptive multiphoton illumination microscopy for large-scale immune response imaging. Nat. Commun. 12, 1916 (2021).
Durand, A. et al. A machine learning approach for online automated optimization of super-resolution optical microscopy. Nat. Commun. 9, 5247 (2018).
Sheppard, C. J. R. Super-resolution in confocal imaging. Optik 80, 53–54 (1988).
Müller, C. B. & Enderlein, J. Image scanning microscopy. Phys. Rev. Lett. 104, 74–83 (2010).
York, A. G. et al. Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy. Nat. Methods 9, 749–754 (2012).
Schulz, O. et al. Resolution doubling in fluorescence microscopy with confocal spinning-disk image scanning microscopy. Proc. Natl Acad. Sci. USA 110, 21000–21005 (2013).
De Luca, G. M. R. et al. Re-scan confocal microscopy: scanning twice for better resolution. Biomed. Opt. Express 4, 2644–2656 (2013).
Roth, S., Sheppard, C. J. R., Wicker, K. & Heintzmann, R. Optical photon reassignment microscopy (OPRA). Opt. Nanoscopy 2, 5 (2013).
Azuma, T. & Kei, T. Super-resolution spinning-disk confocal microscopy using optical photon reassignment. Opt. Express 23, 15003–15011 (2015).
Hofmann, M., Eggeling, C., Jakobs, S. & Hell, S. W. Breaking the diffraction barrier in fluorescence microscopy at low light intensities by using reversibly photoswitchable proteins. Proc. Natl Acad. Sci. USA 102, 17565–17569 (2005).
Chmyrov, A. et al. Nanoscopy with more than 100,000 ‘doughnuts’. Nat. Methods 10, 737–740 (2013).
Bodén, A. et al. Volumetric live cell imaging with three-dimensional parallelized RESOLFT microscopy. Nat. Biotechnol. 39, 609–618 (2021).
Lelek, M. et al. Single-molecule localization microscopy. Nat. Rev. Methods Prim. 1, 39 (2021).
Speiser, A. et al. Deep learning enables fast and dense single-molecule localization with high accuracy. Nat. Methods 18, 1082–1090 (2021).
Mahecic, D. et al. Event-driven acquisition for content-enriched microscopy. Nat. Methods 19, 1262–1267 (2022).
Alvelid, J., Damenti, M., Sgattoni, C. & Testa, I. Event-triggered STED imaging. Nat. Methods 19, 1268–1275 (2022).
Tsien, R. Y. The green fluorescent protein. Annu. Rev. Biochem. 67, 509–544 (1998).
Lambert, T. J. FPbase: a community-editable fluorescent protein database. Nat. Methods 16, 277–278 (2019).
Cheng, S. et al. Single-cell cytometry via multiplexed fluorescence prediction by label-free reflectance microscopy. Sci. Adv. 7, eabe0431 (2021).
Christiansen, E. M. et al. In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173, 792–803.e19 (2018).
McRae, T. D., Oleksyn, D., Miller, J. & Gao, Y.-R. Robust blind spectral unmixing for fluorescence microscopy using unsupervised learning. PLoS One 14, e0225410 (2019).
Seo, J. et al. PICASSO allows ultra-multiplexed fluorescence imaging of spatially overlapping proteins without reference spectra measurements. Nat. Commun. 13, 2475 (2022).
Digman, M. A., Caiolfa, V. R., Zamai, M. & Gratton, E. The phasor approach to fluorescence lifetime imaging analysis. Biophys. J. 94, L14–L16 (2008).
Lanzanò, L. et al. Encoding and decoding spatio-temporal information for super-resolution microscopy. Nat. Commun. 6, 6701 (2015).
Scipioni, L., Rossetta, A., Tedeschi, G. & Gratton, E. Phasor S-FLIM: a new paradigm for fast and robust spectral fluorescence lifetime imaging. Nat. Methods 18, 542–550 (2021).
Smith, J. T., Ochoa, M. & Intes, X. UNMIX-ME: spectral and lifetime fluorescence unmixing via deep learning. Biomed. Opt. Express 11, 3857–3874 (2020).
Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).
Saguy, A. et al. DBlink: dynamic localization microscopy in super spatiotemporal resolution via deep learning. Nat. Methods 20, 1939–1948 (2023).
Milias-Argeitis, A. et al. In silico feedback for in vivo regulation of a gene expression circuit. Nat. Biotechnol. 29, 1114–1116 (2011).
Emiliani, V., Cohen, A. E., Deisseroth, K. & Häusser, M. All-optical interrogation of neural circuits. J. Neurosci. 35, 13917–13926 (2015).
Schmidt, M. & Lipson, H. Distilling free-form natural laws from experimental data. Science 324, 81–85 (2009).
Colón-Ramos, D. A., La Riviere, P., Shroff, H. & Oldenbourg, R. Transforming the development and dissemination of cutting-edge microscopy and computation. Nat. Methods 16, 667–669 (2019).
Conrad, C. et al. Micropilot: automation of fluorescence microscopy-based imaging for systems biology. Nat. Methods 8, 246–249 (2011).
Ouyang, W. et al. BioImage model zoo: a community-driven resource for accessible deep learning in bioimage analysis. Preprint at bioRxiv https://doi.org/10.1101/2022.06.07.495102 (2022).
Katona, G. et al. Fast two-photon in vivo imaging with three-dimensional random-access scanning in large tissue volumes. Nat. Methods 9, 201–208 (2012).
Krull, A., Buchholz, T. O. & Jug, F. Noise2void – learning denoising from single noisy images. Preprint at https://doi.org/10.48550/arXiv.1811.10980 (2019).
Ouyang, W., Aristov, A., Lelek, M., Hao, X. & Zimmer, C. Deep learning massively accelerates super-resolution localization microscopy. Nat. Biotechnol. 36, 460–468 (2018).
Greener, J. G., Kandathil, S. M., Moffat, L. & Jones, D. T. A guide to machine learning for biologists. Nat. Rev. Mol. Cell Biol. 23, 40–55 (2022).
Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).
Wang, Z., Simoncelli, E.P. & Bovik, A.C. Multiscale structural similarity for image quality assessment. In 37th Asilomar Conference on Signals, Systems & Computers, 2003 Vol. 2, 1398–1402 (2003).
Zhang, R., Isola, P., Efros, A.A., Shechtman, E. & Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 586–595 (2018).
Lehtinen, J. et al. Noise2Noise: learning image restoration without clean data. In 35th International Conference on Machine Learning, ICML 2018 4620–4631 (2018).
Kefer, P. et al. Performance of deep learning restoration methods for the extraction of particle dynamics in noisy microscopy image sequences. Mol. Biol. Cell 32, 903–914 (2021).
Wu, Y. et al. Reflective imaging improves spatiotemporal resolution and collection efficiency in light sheet microscopy. Nat. Commun. 8, 1452 (2017).
Shaevitz, J. W. & Fletcher, D. A. Enhanced three-dimensional deconvolution microscopy using a measured depth-varying point-spread function. J. Opt. Soc. Am. A 24, 2622–2627 (2007).
Yanny, K., Monakhova, K., Shuai, R. W. & Waller, L. Deep learning for fast spatially varying deconvolution. Optica 9, 96–99 (2022).
Guo, M. et al. Rapid image deconvolution and multiview fusion for optical microscopy. Nat. Biotechnol. 38, 1337–1346 (2020).
Acknowledgements
We thank Grant Kroeschell and Richard Ikegami (Shroff lab) as well as Jiji Chen (NIH Advanced Imaging and Microscopy Resource) for help with figure preparation. This work was supported by the Howard Hughes Medical Institute (HHMI) and the École Polytechnique Fédérale de Lausanne (EPFL).
Author information
Authors and Affiliations
Contributions
The authors contributed equally to all aspects of the article.
Corresponding author
Ethics declarations
Competing interests
H.S. is co-inventor on US patent 9,696,534, owned by NIH and licensed to VisiTech International and Yokogawa Electric Corporation, describing multi-focal and analogue implementations of structured illumination microscopy (SIM), including the instant SIM mentioned here. H.S. has also filed invention disclosures on four-beam SIM and multi-view confocal microscopy, both of which rely on the deep learning strategies mentioned here.
Peer review
Peer review information
Nature Reviews Molecular Cell Biology thanks Ricardo Henriques, who co-reviewed with Estibaliz Gómez de Mariscal, Lothar Schermelleh and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Related links
CSBDeep: https://csbdeep.bioimagecomputing.com
Figshare for ref. 103: https://doi.org/10.6084/m9.figshare.c.4537607
FIJI n2v: https://imagej.net/plugins/n2v
FIJI nlm: https://imagej.net/plugins/non-local-means-denoise/
GitHub 3D-RCAN: https://github.com/AiviaCommunity/3D-RCAN
GitHub autopilot: https://microscopeautopilot.github.io/
GitHub CSBDeep: https://github.com/CSBDeep/CSBDeep
GitHub DL-SR: https://github.com/qc17-THU/DL-SR
GitHub etSTED: https://github.com/jonatanalvelid/etSTED-widget-base
GitHub JiLabAO: https://github.com/JiLabUCBerkeley/JiLabAO
GitHub Knerlab: https://github.com/Knerlab
GitHub n2v: https://github.com/juglab/n2v
GitHub phasenet: https://github.com/mpicbg-csbd/phasenet
GitHub prevedel-lab/AO: https://github.com/prevedel-lab/AO.git
GitHub PSSR: https://github.com/BPHO-Salk/PSSR
GitHub pycudasirecon: https://github.com/tlambert03/pycudasirecon
GitHub rDL-SIM: https://github.com/qc17-THU/rDL-SIM
GitHub Richardson-Lucy-Net: https://github.com/MeatyPlus/Richardson-Lucy-Net
GitHub SIMreconProject: https://github.com/eexuesong/SIMreconProject
GitHub Sparse-SIM: https://github.com/WeisongZhao/Sparse-SIM
GitHub Testa Lab: https://github.com/TestaLab
GitHub UNMIX-ME: https://github.com/jasontsmith2718/UNMIX-ME
Google Code msim: http://code.google.com/p/msim/
Micro-manager EDA plugin: https://pypi.org/project/eda-plugin/
Napari n2v: https://www.piwheels.org/project/napari-n2v/
Napari nlm: https://www.napari-hub.org/plugins/napari-nlm
Skimage nlm: https://scikit-image.org/docs/stable/auto_examples/filters/plot_nonlocal_means.html
Zenodo Henry Pinkard (2020): https://doi.org/10.5281/zenodo.4314107
Glossary
- Aberrations
-
Distortions in images generated by an optical system due to deviations in the properties of real optical components, such as lenses, mirrors and filters, as compared with theoretical models or due to refractive index variations in the sample.
- Artificial neural network
-
(ANN). A computational model composed of interconnected nodes and layers, designed to loosely mimic the structure and function of the brain.
- Binning
-
The denoising process of combining data from adjacent pixels in an image, which results in fewer pixels.
- Bleed-through
-
Overlap in the emission spectra of two distinct fluorophores, leading to detection of both at the same wavelength.
- Centroid
-
The centre position of an object, corresponding to the weighted mean of pixel intensity values.
- Continuity
-
Property of an object consisting of containing no resolved gaps in space (spatial continuity) or time (temporal continuity).
- Crosstalk
-
Undesired mixing of signals. For example, overlap in the excitation spectra of two distinct fluorophores, leading to excitation of both by the same wavelength of light. Or, in multi-focal microscopy, fluorescence from one excitation spot contributing to the signal of neighbouring regions.
- Deconvolution
-
Since an image is blurred by the convolution of the fluorescently labelled object with the point spread function of the microscope, this image processing method attempts to computationally reverse this effect.
- Deep learning
-
A class of machine learning algorithms based on artificial neural networks containing multiple data processing layers.
- Depletion doughnut
-
A doughnut-shaped illumination light used to turn off the fluorescence in the periphery of the focal spot.
- Downsampling
-
Reducing the sampling rate, for example, spatially or temporally.
- Dwell time
-
The time a focused laser beam is applied to each location in the specimen being imaged.
- Fluorescence lifetime
-
The time a fluorophore spends in the excited state before emitting a photon and returning to the ground state.
- Gaussian readout noise
-
Noise that follows a Gaussian distribution and is independent of pixel intensity values, for example, noise generated by a camera chip when it converts charge into voltage.
- Ground truth
-
The target of a deep learning model, for example, a label against which the predictions of a model are compared during training.
- Hallucinations
-
Neural network outputs that look plausible but have no basis in the input data.
- Lattice light-sheet
-
Light-sheet generated by scanning a 2D lattice of structured light known as Bessel beams.
- Light-sheet microscopy
-
Method in which a thin slice of a specimen is illuminated perpendicular to the imaging orientation.
- Linearity
-
The property of two quantities (for example, intensities) being linearly proportional, such that their values are related by a multiplicative constant.
- Low-pass filter
-
An operation that passes frequencies below a cut-off value, which corresponds to retaining lower-resolution features in an image.
- Mean absolute error
-
(MAE). The mean of the absolute value of the difference between measured and predicted values, for example, ground truth pixel intensity values and those output by a network. Used as a metric of how well a model captures the data.
- Mean squared error
-
(MSE). The mean squared difference between measured and predicted values, for example, ground truth pixel intensity values and those output by a network. Used as a metric of how well a model captures the data.
- Median filtering
-
An operation that replaces the intensity value in a pixel with the median value of its neighbours.
- Multiphoton microscopy
-
Method in which multiple photons must be simultaneously absorbed by a single fluorophore to bring it into its excited state.
- Mutual information
-
A measure of the extent to which two quantities depend on one another, related to how precisely one quantity can be predicted based on the value of the other.
- Nyquist–Shannon sampling theorem
-
Principle that defines the maximum spacing between measurements that will be sufficient to determine a given frequency component within a signal; for example, to resolve dynamics at a timescale of T seconds, the time between images should be less than T/2.
- Peak signal-to-noise ratio
-
(PSNR). The ratio between the squared maximum possible signal in an image and the mean squared error. This is reported in units of decibels, so the logarithm of the ratio is taken and multiplied by 10. Used as a metric of how well a model captures the data.
- Photon budget
-
Fluorescence signal detected from an object of interest during an experiment, typically finite due to photobleaching.
- Point spread function
-
The intensity distribution of a point-like source when imaged through a microscope.
- Poisson noise
-
Noise (that is, shot noise) that follows a Poisson distribution, for example, arising from measuring photons because they are discrete particles.
- Pyramid of frustration
-
Concept illustrating the tradeoffs in fluorescence microscopy, where each axis defines one measurement property such as signal-to-noise ratio or spatial or temporal resolution. The fixed photon budget implies that improving along one dimension leads to degradation along another.
- Reference datasets
-
Data used within a field to compare the performance of algorithms, in benchmarking comparisons.
- Reversible saturable optical fluorescence transitions
-
(RESOLFT). A super-resolution technique suitable for live-cell imaging and based on reversibly switching fluorescent probes and patterned illumination.
- Sampling
-
Recording a signal in a discontinuous manner, at specific locations or times.
- Signal-to-noise ratio
-
(SNR). The ratio between signal and noise that can be estimated on a per-pixel basis as the mean intensity value divided by the standard deviation of the intensity.
- Single-molecule localization microscopy
-
(SMLM). A class of super-resolution microscopy techniques based on imaging single molecules whose signals have been isolated, then combining their sub-pixel locations to form a composite image.
- Spatial frequencies
-
Just as a function can be decomposed into a sum of sines and cosines (compared with Fourier transform), an image can be decomposed into a sum of waves with different spatial frequencies. These represent the image at different resolution levels, with higher spatial frequencies describing finer image details and lower spatial frequencies describing coarser details.
- Spatial resolution
-
The smallest distance at which two features can be distinguished.
- Spectrum
-
Response of a fluorophore as a function of wavelength of light. The excitation spectrum reflects the capacity of light absorbed at different wavelengths to generate fluorescence of a particular wavelength, while the emission spectrum reflects the emission of light at different wavelengths following excitation at a particular wavelength.
- Spinning disk confocal microscopy
-
Method in which an array of focused excitation laser beams is produced by an array of microlenses on a disk that spins to scan the specimen. Out-of-focus emission light is rejected by a confocal array of pinholes.
- Statistical distance
-
An objective score that summarizes statistical differences between two objects, for example, between a prediction and the training set in machine learning. Possibilities include ‘total variational distance’ and ‘Kullback–Liebler divergence’.
- Stimulated emission depletion
-
(STED) microscopy. A super-resolution technique based on patterned illumination, typically doughnut shaped, which is used to deplete the fluorescence of commonly used fluorescent probes.
- Structural similarity index measure
-
(SSIM). A measure of how similar two images are based on distortions that humans tend to perceive: the weighted product of luminance (average brightness), contrast (standard deviation of pixel intensity values) and structure (cross-covariance).
- Structured illumination microscopy
-
(SIM). A class of super-resolution microscopy techniques that use patterned excitation light combined with optical or digital image processing to recover information below the diffraction limit.
- Super-resolution
-
Imaging techniques that achieve spatial resolutions surpassing the diffraction limit of light.
- Synthetic data
-
Data generated computationally using a model. Can be combined with real data to create semi-synthetic data.
- Temporal resolution
-
The time between consecutive images of the same part of the specimen.
- Total internal reflection fluorescence
-
Method in which a specimen is illuminated at the coverslip-media interface by an evanescent field generated by a laser at an incident angle sufficient to cause total internal reflection.
- Training data
-
Data used to train a machine learning algorithm to make predictions based on supervised learning. The quality and quantity of training data are key determinants of the performance of artificial neural networks.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Shroff, H., Testa, I., Jug, F. et al. Live-cell imaging powered by computation. Nat Rev Mol Cell Biol (2024). https://doi.org/10.1038/s41580-024-00702-6
Accepted:
Published:
DOI: https://doi.org/10.1038/s41580-024-00702-6