Revolutionizing Vision: Neuromorphic Imaging Transforms Speed, Efficiency, and Precision

Imagine a super-fast camera that continuously captures images from pixels and detects only changes in light brightness. Each pixel works independently and constantly, with microsecond detection. This is the basis for event-based or neuromorphic imaging.

A 2022 McKinsey report identified neuromorphic computing as one of the top ten technology trends with the potential to reshape several markets and industries in the coming decades.

Rapidly capturing change

In event-based imaging, each pixel on the sensor has a set threshold, and when that threshold is exceeded, the pixel activates, capturing only what has changed from the previous threshold. This approach transmits only relevant data,  rather than an entire image, creating an “extreme region of interest” that reduces computing time and processing resources.

Since only the pixels detecting movement are activated, the resulting image has minimal latency due to the small amount of data being updated. Additionally, less power is required to capture the smaller image.  

Dynamic range redefined

Another advantage of neuromorphic pixels is their capability to capture more than 120 dB of dynamic range.

Sensors based on these pixels, like a human retina, can locally adapt to vast changes in brightness, detect edges, signal temporal changes, and sense motion. Their photoreceptors continuously monitor intensity and adjust to the local image over time and space, maximizing dynamic range.

Caltech, MIT, and John Hopkins have been researching this topic since the 1990s. This work, spurred by Northrup Grumman, Rockwell International, and DARPA, evolved under the leadership of researcher Christof Koch.

Christof Koch was a leading professor of computation and neural systems at Caltech, and he was recognized for his work on neuromorphic systems. He later became president of the Allen Institute for Brain Science.

Diverse applications of neuromorphic imaging

Applications run the gamut:

• Autos: ADAS and cabin monitoring

• Medical: Fluidics and cell tracking

• Sports: Player tracking, swing analysis

• Defense: Rocket, bullet, and bomb analysis

• Industrial: High-speed counting and vibration analysis

Combining event-based and color sensors

New camera systems can integrate event-based sensors with traditional color sensors to produce images that are both more accurate and visually appealing. By combining these technologies, the resulting images can be fast and precise.

Industrial applications are plentiful due to their speed and low data requirements.

Currently, Sony has partnered with Prophesse, a French company that has commercialized neuromorphic sensors for integration with existing camera modules. According to Prophesse, its sensors produce up to 1000 times less data than a conventional sensor while achieving a higher equivalent temporal resolution of more than 10,000 frames per second.

Based in Zurich, iniVation is a Swiss company also pioneering the field with its Aeveon sensors, which feature 346 x 260 pixels.