Oliver Bimber* and David C. Schedl
Faculty of Engineering and Natural Sciences, Johannes Kepler University Linz, 4040, Austria
Light-field microcopy (LFM) supports single-shot volumetric recording and instant 3D excitation. It is an emerging technology for fast wide-field 3D imaging of neuronal activity, and has the potential to enable scanless 3D photostimulation for applications in optogenetics. This article introduces the basics of light-field technology, reviews the LFM principles and current implementations, summarizes first applications for optical imaging and photostimulation of neural activity, and outlines alternatives that go beyond LFM.
Light-field imaging is a technique that records four-dimensional slices of the five-dimensional plenoptic function by applying an appropriate ray-space parametrization. In many cases, microlens arrays (MLAs) are used in the intermediate image plane of optical instruments to multiplex 2D spatial and 2D directional information on the same image sensor (Figure 1). These 4D light-field representations can then be integrated computationally to achieve, for instance, digital refocussing, focal stack computation, depth estimation, and perspective variations after a single recording.
Figure 1: Recording light fields with microlens arrays: (a) MLAs are used in the intermediate image plane to multiplex spatial (s, t) and directional (u, v) information on a single image sensor. When a light field is recorded, the 5D plenoptic function, P(x, y, z, φ, θ ), is reduced by ray-space parametrization to 4D, L(s, t, u, v). Example light-field recordings of a dog flea (Ctenocephalides canis): (b) the MLA structure is clearly visible in the sensor recordings; (c) the perspective can be changed after a single-shot recording by computational integration.
Aside from MLAs, other optical modulators, such as micromirror arrays (MMAs), camera arrays (CAs), amplitude modulators (AMs), or phase modulators (PMs) can be employed for ray-space multiplexing, and various optical microscopes have used this idea for light-field imaging and illumination. One major advantage of light-field microscopes (LFMs) is their support for single-shot volumetric recording and instant 3D excitation—with potential applications in areas such as optogenetics1. A main limitation is their reduced lateral resolution that results from ray-space multiplexing. This, however, might be overcome by future high-resolution microoptics, spatial light modulators, and image sensors.
The first LFM prototype2 recorded light fields through an MLA in the intermediate image plane of a standard microscope’s imaging path, as illustrated in Figure 1. The probe could be viewed from different directions (which corresponds to tilting the stage of a classical microscope), a 3D focal stack could be computed (which requires scanning when using classical microscopes), and a 3D volume could be reconstructed by deconvolution. All of this was enabled computationally with a single light-field recording. Today, slightly modified consumer light-field cameras in combination with standard microscopes provide low-cost entry into LFM3. LFM technology can also be applied to miniature microscopes4 used on freely moving animals. Furthermore, it provides a platform for optical depth estimation of microscopic probes5. In all cases, the focal length and pitch of the MLA depends on the numerical aperture (NA) and magnification of the objective. Thus, conventional MLAs must be chosen to suit the microscope objectives to be used. Elastic MLAs6, in contrast, can change their focal length dynamically and are therefore applicable in combination with multiple objectives.
In addition to single-shot volumetric recordings, instant generation of 3D illumination patterns is another requirement of modern microscopy applications, such as optogenetics. By placing an additional MLA in the illumination path of an optical microscope, light-field illumination can be achieved when a high-resolution spatial light modulator (SLM), such as a DMD or LCoS chip, is employed. This was first demonstrated for manually defined light-field patterns that mimic simple dark-field and oblique illuminations7. A more recent approach is to derive light-field illumination patterns dynamically from light-field recordings to support controlled generation of structured volumetric illumination patterns in the probe (e.g., fluorescence particles or neuronal cells)8,9. Figure 2 illustrates the principle of an LFM that utilizes light fields for imaging and illumination.
Figure 2: (a) Schematic optics of an imaging and illumination LFM for fluorescence applications: The illumination pattern is generated by an SLM (yellow; showing an example illumination light-field pattern), focused on micro lens array MLA2 by relay lens R2, and projected onto the probe via tube lens T2 and the objective lens (OBJ) after passing through an excitation filter (EX) and a dichroic mirror (BS). The illuminated probe particles fluoresce while the entire volume is recorded by the imaging path of the LFM. Light from the samples is focused on imaging micro lens array MLA1 via OBJ and tube lens T1 by passing BS and the emission filter EM. The imaging light field is then recorded by the camera (CAM; purple; showing an example imaging light field) via relay lens R1, which is focused on the back-focal plane of microlense array MLA1. (b) Volumetric light-field excitation (VLE)8 supports the excitation of desired regions in the probe while avoiding excitation of others: Sample of two occluding microspheres i and ii; (c) only i is excited, while ii is to remain unexcited; (d) excitation of ii while avoiding excitation of i.
Application of MLAs in the imaging and illumination paths of a single-shot LFM (Figure 3a), however, reduces the spatial resolution of the recordings and of the light pattern—which is one of the main limitations of light-field microscopy. An n × n MLA together with an image sensor or spatial light modulator of resolution m × m, for instance, reduces the spatial resolution in the field plane from m × m to n × n, while supporting a directional resolution of (m/n) × (m/n). The spatial resolution can be increased by downscaling the microlens’ pitch (leading to a higher resolution MLA). The directional resolution can be increased by using a high resolution image sensor or light modulator (large m), where the minimum sensor, pixel, or mirror size is limited by the Sparrow resolution 0.47λ M/NA (M is the magnification of the microscope and λ is the wavelength of light). Furthermore, slight optical misalignment and manufacturing imprecisions will reduce the achievable resolution. After recording, the focus within a probe can be synthetically changed within an axial range of approximately ((2 + (m/n)2)λη)/(2NA2), where η is the refractive index of the imaging medium.
Overcoming or avoiding these limitations to resolution is a main goal of current LFM development and research. Improved spatial resolutions can be achieved, for example, by shifting the MLA10,11 or shifting the stage12—both require temporal scanning. Applying 3D deconvolution13 or placing additional phase masks in the optical path14 also enhances spatial resolution. Using a camera array15 with multiple imaging sensors instead of an MLA and a single image sensor preserves the sensor’s original resolution in the light-field recording. By focusing the MLA on the intermediate image plane (instead of placing the MLA there), as shown in Figure 3b, the spatial resolution can be increased at the cost of a more complex and error-prone image registration for reconstruction16. Furthermore, aperture-mask coding is enabled with an SLM positioned at the aperture plane of the imaging path and supports full-sensor-resolution light-field recording by scanning17 (Figure 3c). Sequential random illumination patterns for LFM support enhanced resolution, but also rely on scanning18.
Instead of using a single MLA, two MMAs aligned in series (placed at the intermediate image plane and the aperture plane) can be applied to generate illumination light fields (Figure 3d)19. These illumination light fields, however, are constrained when compared to full 4D light-field control, as a spatial pattern (on MMA1) will be projected in all active directions (controlled by MMA2). This means that the excitation pattern will be the same for every direction. Patterns that differ in each direction can in this case be achieved only by scanning. In comparison to 4D LFM, however, the spatial resolution is increased because the full light field need not be multiplexed on a single SLM.
Figure 3: Common LFM designs: (a) MLAs placed at intermediate image plane in imaging and illumination paths of a single-shot LFM support volumetric excitations and recordings. (b) Focusing the MLA on the intermediate image plane increases the spatial resolution but requires probe-dependent image registration. (c) An SLM at the aperture plane supports full-sensor-resolution light-field recording by scanning. (d) MMAs at the intermediate image plane (MMA1) and at the aperture plane (MMA2) can be used for light-field illumination. Unlike with (a), not all light-field patterns can be generated without scanning.
Fast readouts are important for animal observation, and LFM is one of the few methods that supports instant (i.e., non-scanning) imaging and illumination of large volumes.
LFM imaging has been used in various microscopic applications for observing neural activity in animals, such as C. elegans, larval zebrafish, flies, and mice20-27,4. Optical recordings of neuronal activity are achieved by organic fluorescence dyes that are calcium- or voltage-sensitive and can be genetically encoded in neurons (i.e., optogenetics)28. In most studies, the objective is mounted on the animal while the animal is fixed to avoid the need for tracking. Recently, however, LFM imaging and tracking of neural activity of freely moving animals (i.e., zebrafish larvae) has been shown25. Furthermore, scattering, which is a limiting factor in various types of tissue (e.g., mammalian brains), is encoded in light-field recordings and can be utilized by techniques that rely on the computational decomposition of scattered fluorescence sources24,9,26,27. LFM imaging in highly scattering tissue, such as the mammalian cortex, at depths of up to 380 µm has recently been demonstrated27. This principle was applied successfully to miniature microscopes4 mounted on freely moving mice.
LFM can be used not only for imaging, but also for precise volumetric illumination. For applications in optogenetics, genetically modified neurons (expressing light-sensitive opsins) can be photostimulated by concentrated light pulses. In volumetric light-field excitation (VLE)8,9, light is concentrated simultaneously at multiple volumetric positions by means of a 4D illumination light field. For a transparent non-scattering probe a defocus-free volume can be computed from a single light-field recording by 3D deconvolution. Given a selection of points within this volume, a 4D light-field pattern is then computed that concentrates light at desired volumetric positions and avoids light concentration at others8. For scattering probes, however, this approach has limitations: First, precise optical calibration is required to map light-field rays to volumetric positions. Second, it ignores scattering in media. Third, deconvolution is ill-posed and relies on heavy parameter tuning, leading to reconstruction errors. By avoiding deconvolution and calibration, compressive light-field excitation (CVLE)9 takes scattering into account. It relies on a fast adaptive light-transport sampling followed by light-field factorization. The measured light transport represents the interaction of illumination and imaging light rays with the probe (including the impact of dispersion). By assuming isotropy, a non-negative matrix factorization of the light transport leads to de-correlated imaging and illumination light-field footprints of individual particles (i.e., fluorescence microspheres or neuronal cells). For stationary probes, instantaneous (i.e., one-emission / one-shot) excitation and imaging of multiple particles of interest is possible (Figure 4). For moving probes, light-transport sampling and factorization must be repeated.
Figure 4: Compressive volumetric light-field excitation (CVLE)9: (a) imaging light field, where colors correspond to 30 computationally decomposed fluorescence microspheres. (b) Volumetric reconstruction of imaging light field after exciting only the 30 microspheres of interest with the illumination light field. The CVLE LFM prototype9 in this experiment was equipped with a 60 /1.2NA objective, a 250 µm pitch MLA, and a 4 megapixel sensor, thus achieving a lateral resolution of n = 56 and a directional resolution of m/n = 35 pixels for imaging a volume of 100 µm axial, and 234 µm lateral size. The samples were 10 µm to 20 µm Fluorescent Green Polyethylene microspheres (peak excitation 470 nm; peak emission 505 nm) mixed with silicone elastomer.
The CVLE LFM prototype9 in this experiment was equipped with a 60 /1.2NA objective, a 250 µm pitch MLA, and a 4 megapixel sensor, thus achieving a lateral resolution of n = 56 and a directional resolution of m/n = 35 pixels for imaging a volume of 100 µm axial, and 234 µm lateral size. The samples were 10 µm to 20 µm Fluorescent Green Polyethylene microspheres (peak excitation 470 nm; peak emission 505 nm) mixed with silicone elastomer.
LFM imaging is an emerging technology for wide-field volumetric non-scanning imaging of neuronal activity (see recent reviews29,30). An alternative that enables instant volumetric imaging is the simultaneous recording of multiple focal planes with either a single or with multiple cameras31. Compared to LFM, this has the advantage of avoiding any computational reconstruction. However, the optical complexity of such setups imposes limits on the number of focal planes achievable.
Photostimulation with LFM illumination relies on one-photon (1p) excitation. Due to aberrations and scattering, however, 1p illumination degrades considerably in brain tissue. Thus, current LFM illumination techniques might only be applicable to relatively thin or transparent probes. Furthermore, LFM illumination has, to our knowledge, not been shown in any in-vivo experiments.
Two-photon (2p) or multi-photon techniques might be more suitable for photostimulation as they offer deeper tissue penetration due to the use of longer wavelengths. Out-of-focus excitations that can occur with 1p illumination are also greatly reduced. Common opsins, however, require photocurrents that were difficult to achieve with early 2p scanning techniques28.
Therefore, scanless techniques using computer-generated holography (CGH) and temporal focusing (TF) are applied for two-photon photostimulation in volumes. Recent advances in opsin design and photostimulation with CGH and TF (called 3D-SHOT) support simultaneous 3D photostimulations of multiple neuronal cells in volumes32,33.
2p LFM illumination has not yet been demonstrated, but is a promising future direction. Compared to 1p excitation, deeper tissue penetration would be enabled and out-of-focus excitation avoided. Compared to CGH, some disadvantages of digital holography, such as limitations imposed on possible excitation patterns34, zero-order diffraction spots, and speckles, could be avoided. However, 2p LFM illumination requires efficient optics, strong lasers, and sensitive opsins.