Until recently, qualitatively converting images from standard to high-dynamic range (SDR to HDR) was only possible manually. But now IPI, imec’s Image Processing and Interpretation research group at the University of Ghent, has developed a unique set of algorithms to enable the high-quality, automatic conversion of images. In this article, Jan Aelterman, postdoctoral researcher, and Hiep Luong, project manager, tell us more about this groundbreaking technology.
About nits, HDR and SDR
200, 1,000 or 10,000 nits: anyone buying a new television set will shortly be faced with making a choice about the number of nits. The nit value indicates the maximum brightness, hence the brightest white on the screen. The nit value is expressed in cd/m2 (candela per square meter). The ‘dynamic range’ of a TV is the difference between the darkest black and the brightest white that the set can display at the same time. The more nits, the greater the ‘dynamic range’ and the closer the picture on the TV screen is able to approach reality. From 1,000 nits upward, we talk about “high dynamic range” or HDR. Or in the persuasive marketing terms of the shop display: “Ultra HD Premium”. HDR represents a significant quality improvement compared with the usual SDR television sets in our living rooms at the moment. The most advanced HDR prototypes currently have a range of 10,000 nits.
Jan Aelterman: “According to the trade press and media companies such as Netflix, the gain in quality going from SDR to HDR is far more important than the race for more pixels and higher resolution. HDR pictures are so close to reality that viewers even spontaneously experience depth, even though they are not watching a 3D program. In fact, HDR has nothing to do with depth.”
According to the trade press and media companies such as Netflix, the gain in quality going from SDR to HDR is far more important than the race for more pixels and higher resolution.
A thousand years’ worth of low-quality film archive
But to enjoy HDR, you need more than just an HDR television. You also need the content to be of HDR quality. Exactly the same thing applies with resolution: watching low-resolution pictures on a hi-res TV does not enhance your viewing experience. Hiep Luong: “Enjoying SDR is only possible if the right cameras, storage media and transmission technology are used at every stage of the process, from recording the images right through to displaying them. The technology to do this is now available, but lots of new content is still being produced in SDR so that it can reach non-HDR consumers. This applies both for professional recordings and for the clips we film on our smartphones.”
And then there are picture archives. For example, the content listed in the online film encyclopedia, imdb.com, already accounts for more than a thousand years of film archives in SDR. Let alone the gigantic quantities of archived images stored in SDR by public and private broadcasters.
Manual conversion is not viable
Converting SDR images to HDR is not straightforward. The solution providing the highest quality consists of manual conversion, image by image. This involves an enormous amount of work that may be of value in exceptional cases, but which isn’t usually financially viable. Jan Aelterman: “The makers of Star Trek invested millions of dollars in a manual conversion from low to high resolution, only to discover later in the sales numbers that their investment didn’t generate enough return.”
Is it possible to convert automatically to HDR? Until now there have been four major obstacles standing in the way: avoiding ‘artifacts’ (such as ‘ghost’ or ‘halo’ images that appear unwanted in the new image); recognizing light objects in dark settings; converting light and dark settings in a visually distinct way; and carrying out all of the complex calculations needed in real-time on a standard graphic processor. All of the existing software solutions find themselves struggling with one or more of these problems and hence are unable to achieve the required quality in the end result.
Automatically enjoying higher quality
The software developed by IPI performs extremely well on all fronts.
For example, it automatically removes any artifacts that may become visible during the conversion process. Digital image compression involves selecting which bits are relevant for the end result. With SDR, these choices do not result in a visible loss of picture quality. But when converting to HDR, they do become clearly visible in the form of visual artifacts, such as poor-quality color gradients or blurred edges. The imec software stops these artifacts from forming. By extracting knowledge from the surrounding bits, it provides a realistic interpretation for the missing information.
How visual artifacts occur during traditional conversion to HDR (left – what should be a smooth color gradient becomes ‘pixelated’) and how the imec software stops it from happening (right).
The software is also ingenious in the way it recognizes light sources and light objects in darker settings. Standard software solutions only look at the brightness of the individual pixels, not at the object level. As a result, they make all luminous objects look more or less equally bright and also peak (saturate) at the same brightness. For example, a candle flame looks just as bright as the flame from a welding torch in the same setting, which is a phenomenon that is normal in SDR. But in HDR, a distinction between objects such as the welder flame and the candle flame can be made convincingly. Same goes for the recognition of specular reflections (whereby incident light is not being diffused, but being reflected into a single outgoing direction). The imec software is smart enough to recognize such objects and to make more distinction between them, based on their brightness.
Example of light saturation with standard conversion (left), in which the streetlamp peaks at a lower brightness. On the right the conversion by imec software: thanks to image recognition, the streetlamp is given a more pronounced contour and brightness.
Also important is the ability of the imec software to handle light scenes differently from dark ones. Algorithms for machine-learning recognize the darker settings and leave them undisturbed, while standard software often makes them light when it is not wanted. By contrast, the imec software makes the light settings additionally bright, whereas standard software leaves them unnecessarily dark.
The imec software leaves darker settings basically undisturbed (above) in the conversion from SDR (left) to HDR (right) and makes light settings extra bright (below).
Real-time conversion
Possibly the main benefit of the imec software is that it performs all of these complex calculations so efficiently that real-time conversion becomes possible on a relatively simple graphic processor.
Thanks to the in-house development of the Quasar platform for software development, the relatively complex code can still work efficiently on various common hardware platforms. The software functions on a central high-performance server used, for example, by a television station, but just as well on the much simpler processor in your television, smartphone or set-top box. As just one example: thanks to a collaboration with TPVision, imec SDR to HDR algorithms are currently part of the image processing in the latest range of Philips 803 and 903 OLED televisions.
Software building blocks allow wider application
And to make the story complete, all of the benefits of the IPI solution can also be used separately. They can be applied in more areas than simply converting images from SDR to HDR. Hiep Luong:
“Our software helps the Belgian public broadcaster VRT to remove artifacts from its archive of digital images.
On modern TV sets, lots of unwanted visual artifacts tend to appear, particularly in digitalized recordings from the 1980s and 1990s made using Betacam technology. The imec software upgrades the quality of digitalized archive images to broadcast quality.”
Left: detail of a digitalized Betacam archive image. Right: result using imec (super-resolution) software.
A brighter future
Development of the software took place partly during the imec.icon HD2R project, which ran from 2015 to 2017 with business partners such as VRT, Barco, the post-production company Grid-VFX and others. Since then the software has demonstrated its value on multiple occasions. Hiep Luong: “The most relevant companies in the media sector now know where to find us. And we have a pipeline of exciting collaboration opportunities. We are still looking for partners who want to use our software to lift the quality of their picture material to a higher level.”
Want to know more?
- Article in Tweakers concerning the imec software (in Dutch).
- Homepage of IPI, the imec research group at Ghent University.
- Project page of the imec.icon HD2R project.
- Homepage of Quasar, the platform developed by IPI for fast development of GPU code
