Video monitoring is an expanding industry; security and surveillance cameras have become ubiquitous and are used in a wide variety of contexts. One of the limitations of current camera solutions is that image quality deteriorates drastically in low-lighting and harsh weather conditions (fog, snow, rain, etc.). By integrating a hyperspectral filter on top of a gated CMOS imager sensor, the European-funded I-ALLOW project resulted in a prototype solution for competitively-priced video surveillance that ensures high visibility in any situation and enables distance targeting as well as integrated video content analysis.
Tackling the limitations of outdoor video surveillance
It’s crucial that we can rely on security and surveillance cameras 24/7, regardless of any external circumstances. Even if something happens in the middle of a dark and foggy night, we still need to be able to monitor incidents closely. Yet, most current video surveillance solutions are unable to provide high-quality images in this kind of situation. And the few high-end professional cameras that can do this are often too expensive or too big to be widely implemented.
The I-ALLOW project, funded by the EU under the H2020 program, has tackled this problem. The result is a prototype camera that uses a novel hyperspectral filter combining both conventional color (RGB) and near-infrared (NIR) vision, which allows you to virtually see through smoke, fog, rain, etc. Moreover, because of the gated imager sensor, the camera enables distance targeting, which means that you can choose to focus solely on objects that are further away without being distracted by details nearby.
The prototype was developed within an international consortium, bringing together the expertise of four partners, i.e. Aitek (Italy), BrightWayVision (Israel), Proprs Ltd (UK) and imec (Belgium). The collaboration resulted in a demonstrator camera that is ready to be fine-tuned and commercialized. The technology readiness level of the prototype is estimated at TRL level 5 to 6, meaning that it has been tested in a relevant environment and performed close to the expected level. The final prototype was tested in three diverse contexts, with the support of the Maritime Office in Gdynia (Poland), Tecnositaf (Italy), and SNCF (France).
Combining cutting-edge technology: a gated CMOS imager sensor with hyperspectral filter
The solution developed within the I-ALLOW project is based on an innovative CMOS imager sensor, developed by BrightWay Vision. This sensor combines two advanced features. First, the camera sends out a pulsed near-infrared (NIR) signal, invisible to the human eye. These light waves reflect against the objects they encounter so the signal returns in the direction of the camera. The sensor then captures the returning light waves and, based on the known speed of light, automatically determines how far away an object is (time-of-flight). This is combined with a second innovative feature, i.e. the gated principle. The gated sensor uses the pulsed NIR light to focus solely on objects at a predefined distance.
This is useful in cases where it’s important not to clutter images with irrelevant information closer to the camera. For instance, in traffic, the headlights of an oncoming car may obscure the camera’s view. By adapting the camera’s exposure time in such a way that you ignore the first light wave signals that are sent back and only start capturing the signals after, for example, 3 milliseconds, the headlights will not be part of your image, thus rendering an uncluttered view of oncoming traffic. In the same way, the gated principle can be used to ‘see’ through smog, fog, rain, snow, etc.
Using infrared pulses would normally require a dedicated infrared sensor, but in this project imec developed a novel hyperspectral filter that combines both NIR and conventional color (RGB) filters on a single chip. Leveraging their background in CMOS scaling, the imec researchers were able to deposit and pattern these filters directly on top of the CMOS sensor pixels at wafer level. This specific application was extra challenging as it required the researchers to combine imec technology with external components. The color filter in this application, for instance, came from an external foundry and was combined with imec’s NIR filter. To make this work, imec researchers needed to adapt the design of the filters to make sure they matched.
Combining NIR and RGB filters also implies that a trade-off needed to be found between the number of NIR and RGB tiles in the pattern that is repeated on the imager sensor. BrightWay Vision designed two potential patterns: the first one appears to be the most interesting one in this context as it provides slightly more infrared information. They also developed an advanced demosaicing algorithm to make sure that both the RGB and the NIR image can be rendered in the same resolution.
Imec deposits the RGB and NIR filter on top of BrightWay Vision’s gated CMOS imager sensor at wafer level.
A third partner, Aitec, was then responsible for building the camera around this advanced imager sensor. They integrated all the components and developed advanced software for video analysis, specifically targeted at security camera applications.
Advanced testing in three use cases
One of the strengths of the I-ALLOW project is that the prototype has been tested in three diverse contexts in close cooperation with international industry partners.
The first context in which the camera was successfully tested is in the Polish harbor of Gdynia. The camera was installed on top of a surveillance tower that provides navigational instructions to ships entering the harbor and on a ship that was making its way through a canal. The advantage of the I-ALLOW solution is that clear vision is guaranteed even at night or in harsh weather conditions where regular cameras fall short.
In a second testing environment, the camera was placed inside an Italian road tunnel. The advantages brought by the I-ALLOW system in this case are two-fold. The first benefit is that the system offers automatic video content analysis. Thus, the control room and other drivers can be alerted automatically in case an accident occurs inside the tunnel. The gated imager ensures good video quality, even in situations where regular cameras fail, e.g. when there is a lot of smoke development.
Finally, the camera was also tested in French railway stations. In this context, the system can be used for a variety of purposes. It can be mounted on top of a train to provide a clearer image in difficult weather conditions, but it could also be placed somewhere along the tracks to locate potential trespassers, or it could be used for surveilling the railway station itself, e.g. for tracking suspicious individuals. In this case, the NIR filter is important to ensure good visibility - even at night or on rainy, foggy days. The color filter can then be used to provide additional details.
In all of these situations, regular cameras are already in use today. The problem is that in harsh weather conditions their video quality becomes so poor that it no longer provides an added value. To a certain extent, current hyperspectral cameras could also offer a solution here, but of course these are very expensive and – due to their size – rather cumbersome to use. The I-ALLOW solution on the other hand could be produced at a competitive price and is about the same size as a regular surveillance camera, making it easy to integrate it in a wide variety of contexts.
The road ahead: customized hyperspectral filters
The practical tests have proven that the security camera demonstrator developed within the I-ALLOW project offers commercial opportunities. Additionally, the project allowed each individual partner to further fine-tune their own expertise.
Imec, for instance, already had a lot of experience with more generic hyperspectral camera systems, offered as off-the-shelf cameras. In the framework of the I-ALLOW project, this know-how was used to develop a hyperspectral filter adapted to a specific security camera application, customizing it to work with BrightWay Vision’s gated CMOS sensor. The expertise imec acquired in this project creates opportunities for other use cases and application domains.
Want to know more?
- Learn more on imec’s expertise in hyperspectral imaging on this page.
- Read this paper for more information on the technology that imec and BrightWay Vision developed together.
- The I-ALLOW project is funded by the EU under the H2020 program (contract number: 645262). For more information, check the I-ALLOW project website.