From JPEG to Bruegel
The move from analog to digital images – which started some three decades ago – was exciting but also brought along a number of challenges. How to store and share the images efficiently? How to support different media, bandwidths, or display qualities? How to annotate and secure images? How to allow for amateur and professional uses alike? And how to make best use of the ever-expanding possibilities of modern image sensors?
These questions were tackled by the Joint Photographic Experts Group (JPEG), the standardization committee that laid down the ways to encode, compress, process, and share digital images. Over the years, their work has been immensely influential and has resulted in a series of successful and widely adopted coding algorithms and file formats such as JPEG and its successor JPEG 2000, and more recently also JPEG XR (Extended Range, with support for e.g. high dynamic range images), JPSearch (standardizing image search and retrieval), JPEG XT (an extension implementing amongst others high dynamic range imagery while guaranteeing backward compatibility with legacy JPEG implementations) and JPEG XS (offering low latency, low complexity near-lossless coding).
Imec’s ETRO research group has had a key role in that effort. Today, Peter Schelkens is chairman of the JPEG Coding, Test & Quality subgroup, the group that is responsible for the specification of the coding standards, validation and quality assessment procedures. In line with that work, ETRO has built an extensive experience in user quality testing, helping to formulate standards, creating test suites, and setting up suitable laboratory infrastructure.
One recent and very visible result is the way Frederik Temmermans and the ETRO lab helped assemble high-resolution images of the paintings of Jan Van Eyck (c. 1390 – 1441 CE) and Pieter Bruegel (1525–1569 CE) and make them available online.
The Bruegel paintings belong to the Kunsthistorisches Museum of Vienna. They were photographed and scientifically analyzed, ready for the high-profile Bruegel exhibition running this winter in Vienna, the first time ever that such a major part of Bruegel’s work – 75% of all surviving works – is brought together. The images were made in the visible, infrared and X-ray domain. They were stitched together from many smaller high-resolution pictures, an endeavor in which ETRO was key. Finally, the images were put online by Universum Digitalis, an ETRO spinoff co-founded by Frederik Temmermans. On the websites, the users may zoom in down to the painting’s individual brush strokes.
Imagining new ways of imaging
The Van Eyck and Bruegel pictures are a culmination of what digital photography and JPEG file formats do so well: capturing scenes in a 2D representation, at a specific instance and with a fixed perspective and focus.
But today, technology is emerging to harvest much more information from a scene, such as depth, time-lapse, angle and/or hyperspectral information. And at the same time, researchers are looking to build displays that can render that extra information. Displays that show a much richer representation of the world, as if we would be looking at the real thing, and even walking around it.
So in 2014 the JPEG committee started work on these new imaging possibilities. The effort was named JPEG Pleno and is to become a framework to represent and exchange new imaging and rendering modalities. It will include new tools for improved compression, and functionality for image manipulation, metadata, image access and interaction, and privacy and security.
JEPG Pleno is primarily concerned with point cloud, light field, and holographic imaging.
But today, technology is emerging to harvest much more information from a scene, such as depth, time-lapse, angle and/or hyperspectral information. And at the same time, researchers are looking to build displays that can render that extra information. Displays that show a much richer representation of the world, as if we would be looking at the real thing, and even walking around it.
A point cloud image consists of a collection of data points usually acquired with a 3D scanner or LIDAR. It is used to generate and represent a surface or scene in 3D, like in a highly-detailed terrain map, a quality assessment of robot-assembled components, or a traffic situation around a self-driving vehicle.
A light field image – also known as plenoptic image, hence the name JPEG Pleno – records the amount of light at every point in a scene, seen from every direction. This amount of light can be captured by either an array of cameras or by a dedicated light field camera that uses microlenses to sample each individual ray of light that contributes to the final image. Displayed on a light-field display, a different view of the scene is presented each time the viewer changes his position (his viewing angle) before the display.
Holography, widely seen as the holy grail of imaging. A hologram records not only the intensity of the light as in a conventional picture, but also its phase. When illuminated with the same coherent laser light with which it was created, the hologram will reveal objects with real depth, changing aspect when viewed from other angles. In contrast to light-field images, holograms are continuous: when a person moves in front of them, they change continuously, not discretely. Holograms, in other words, allow recreating the original scene as closely as possible.
Driving new applications
These innovative ways of imaging will add layers of richness to existing functionality. They allow for whole new functions with better and more intense user experiences, overcoming the shortcomings of traditional photography, where the imaging parameters such as focus and viewing angle are all set and fixed at the time of acquisition.
In contrast, they will make it possible to manipulate images after they have been captured, changing for example in real-time the focus, field of depth, or viewer perspective.
From a light-field image, for example, a whole series of conventional pictures can be derived, equivalent to what a photographer would get when he were to walk around the scene and make a snapshot every meter or so. But it doesn’t end there: from a light-field capture, it is also possible to deduce depth and thus create 3D images. Or recognize and manipulate shapes. And from there, it’s only a short way to AR/VR applications.
Holographic data representations carry even more information than light field images, and thus even more possibilities for interactive content consultation and manipulation. One emerging application is lensless microscopy, where a sample may be captured by illuminating it with a laser beam. As such microscopes don’t require bulky and expensive optics, this allows for microsized, even on-chip microscopes. Also with traditional microscopes, the field of view gets smaller with magnification. So to capture a wider scene, labs need a translation stage, taking a series of pictures that are then stitched together. Not so with lensless holography, which can capture a large field of view in one go and in exquisite detail. But considering how the underlying technologies are maturing, holography also promises to go macroscopic to take up a considerable part of the imaging space.
PLENO: a container to include it all
While working with these modalities and weaving them into applications, the ETRO team soon came to realize that application chains don’t have to be restricted to just one modality. In the ETRO lab, for example, researchers start with a point cloud to generate holograms. But as there are no displays available yet to show high-resolution holograms, they convert them into a light field representation, which they then display on a high-quality light field display. A real eye-opener for the researchers working on optical holography, because they’ve never seen their holograms displayed on such a large format.
Many future applications will be like that: working with the various modalities and with many levels of resolution and quality. Therefore, JPEG Pleno will be set up as a container, holding all the information that allows describing, processing, efficiently coding and rendering the various image modalities. And to be able to do that, the Pleno container will e.g. also have to contain all the parameters and calibration information of the imaging step. This information will be kept until the visualization step, where it will help determine how the image can be rendered at high quality.
Co-developing JPEG Pleno with display technology
Concurrent with the development of JPEG Pleno, there is also the effort to build displays that can render the new viewing experience. Imec researchers are e.g. working on thin-film displays to represent high-resolution light-field images and holograms.
The creation of a light-field display is a multidisciplinary collaboration between various imec research groups. The engineers do not work on the display technology only, but also on the best ways and formats to bring the data into the display. This involves e.g. looking at various in-display decompression technologies. The result is, in effect, an intimate co-development between the display technology, compression/decompression software, and the emerging JPEG Pleno standard.
The development of a hardware platform for holography is quite a challenge. It requires, amongst others, a patterning resolution substantially better than half the wavelength (typically even better than 100 nm), memory architectures with a high efficiency and density, close to 2Tbit/s internal data rates, and high-quality waveguides integrated on top of CMOS and accurate controllable light modulators integrated into the waveguide or close to it.
For all of these challenges, imec is working on solutions, which makes the research center an ideal place to develop both a hardware platform for holography and its enabling algorithmic layer. The effort is led by Xavier Rottenberg and subsumes two threads that have been awarded a prestigious ERC grant (European Research Council): one headed by Peter Schelkens and concerned with the digital representation of holograms, and another headed by Jan Genoe, working on the material stack for light modulators controlled by electrical fields. These developments are also at the heart of a venture, set up under xPAND and imec.invest and involved with co-developing standards, compression software, and display resulting in a most efficient interface.
Want to know (and see) more?
- To see the Van Eyck paintings in exquisite detail, go to http://closertovaneyck.kikirpa.be
- To zoom in on the Bruegel paintings of the Kunsthistorisches Museum, visit http://insidebruegel.net
