Image sensors and vision systemsimec GhentR&D with multiple partners

5 min

In-line software closes the quality gap between lens and imager

In a number of application areas, the quality of digital image sensors is outperforming the quality of optical lens systems. The sensors thus also capture color aberrations and other unwanted effects in the final images. Now, software comes in for an effective quality compensation at affordable cost.

Scroll

The chain is as strong as its weakest link. In vision systems, image processors have evolved at such rapid pace that lens technologies have not been able to catch up. As in: it is not always possible to produce high-end lenses within the cost- and other requirements for certain applications. That’s why researcher Ljubomir Jovanov and his colleagues at IPI – imec’s Image Processing and Interpretation research group at the University of Ghent (UGent) – have now developed a dedicated software solution to compensate for unwanted artefacts and color aberrations which result from this quality gap. With their software, they initially target the markets of security cameras and high-end broadcasting cameras.

Downsides of the race for more and smaller pixels 

We are witnessing dramatic increase of the resolution of imaging sensors in the last few decades. While only fifteen years ago typical consumer grade camera had a resolution of 2 Megapixels, current camera models on the market easily reach 25 Megapixel resolution for static images. A similar trend is visible for video cameras, which now move to 33 Megapixel resolution. While high resolution is a very desirable characteristic of the camera, it brings a number of challenges for camera constructors. Some of the challenges brought by the increased resolution, like higher memory and processing power requirements, can be solved by using more recent and powerful hardware.
However, due to the further reduction of the pixel size, lens imperfections become more visible. Due to the lateral chromatic aberration, color components are not reaching the same pixel location, causing color artefacts in the affected regions. 

exist project

Left: Lateral chromatic aberration is an effect whereby color components are not reaching the same focal location and – in the case of digital imagers – cause color artefacts in the affected regions. Right (picture taken from Wikipedia): color filter arrays separate incoming light into its color components, which is the common way to enable digital color capture.

Moreover, most cameras manufactured today use one imaging sensor employing color filter arrays, like the RGB-based Bayer pattern. Such filters separate the incoming light in its color components and are the only way to enable post-correction of chromatic aberration. As a result, demosaicing becomes a necessary step to reconstruct a homogeneous color in the final image. In this process, chromatic aberration shifts the location of color components and makes the artefacts even more pronounced, especially at the edges of the image or screen.

Software comes in where lenses cannot be optimized

While chromatic aberration can be diminished by careful lens design, such solution is not always possible. Lenses that would meet the quality requirements may become highly expensive and too large to fit the specified physical camera format. 
test image

test image2

Top: Raw image without software correction. The more to the left edge of the image, the more pronounced the color artefacts. Bottom: Image after software-based correction, eliminating all artefacts.

That’s why IPI, an imec research group at UGent, has now developed integrated video enhancement solution that deals with a multitude of unwanted image distortions in one go. The computational imaging-based method simultaneously performs noise reduction, chromatic aberration correction and demosaicing of the color components. Performing individual processing steps for correction of the mentioned artefacts would have been easier to program, but would lead to much too slow processing.

In-line correction and highly flexible towards new settings

So far, lens distortion correction algorithms were only available for offline processing, for example in a software like Photoshop. The imec solution performs these corrections in real time. As an additional improvement, the algorithm is implemented on a PC platform with a GPU. It is capable of processing UltraHD 4k video stream at 30 fps. When the lens zooms in or out, the software adapts its processing parameters in order to compensate for the modified distortion characteristics.

Typically, video enhancement algorithms for broadcast cameras are implemented in FPGA (Field Programmable Gate Array), which makes the solution difficult to adapt to a new type of lens, imaging sensor or a new FPGA chip.

The GPU-based solution is highly flexible and allows easier modification of the parameters of the algorithm, better portability to new hardware and easier adaptation to a new lens, higher resolution etc.

The performance improvement was made possible by using the in-house developed Quasar platform, a development environment that optimizes algorithms for heterogeneous systems.

Demonstrated in real-life setting

A large part of the development was performed in the framework of the EU-funded EXIST program, which ran from 2015 to end 2018. And it has been demonstrated in the application areas of the industrial partners, Grass Valley and Adimec. Grass Valley is a global player in content and media technology. Adimec makes reliable industrial cameras for demanding applications at global OEMs in Machine Vision, Healthcare and Global Security industries. Imec welcomes camera manufacturers, security companies and professionals involved in still or moving image correction to investigate technology transfer in to their solutions.

 

Want to know more?

About Ljubomir Jovanov

Ljubomir Jovanov is researcher at IPI, imec’s Image Processing and Interpretation research group at the University of Ghent. He was born in Pančevo, Yugoslavia, on July 13, 1975. He received dipl.ing. degree in Electrical Engineering and mr. ing. degree in Telecommunications, in 2000 and 2005 respectively, both from the University of Novi Sad, Serbia. In 2006 he joined the Department of Telecommunications and Information Processing of Ghent University, Belgium as a PhD student. His main research interests are image, video and 3D restoration, image analysis and processing of multimedia data. In 2011 he defended his PhD thesis entitled "Advanced Restoration Techniques for 3D Images and Video".

This website uses cookies for analytics purposes only without any commercial intent. Find out more here. Our privacy statement can be found here. Some content (videos, iframes, forms,...) on this website will only appear when you have accepted the cookies.

Accept cookies