Hyperspectral (HS) imaging sees more than your eyes. That’s why it has been used in numerous applications. In this article we highlight three recent use cases that clearly highlight the invaluable role of imec’s HS technology: Da Vinci’s Last Supper, pythons and precision agriculture. For all cases, imec designed portable and robust cameras that can be deployed on location. At the same time, the new shortwave infrared sensors were put to the test.
What do paintings, snakes and crops have in common? They are all applications where hyperspectral (HS) imaging can make a difference. HS cameras divide the light reflected by an object into many narrow spectral bands, which they register and process separately. This way, they capture a spectral signature for each pixel in an image. Imec makes HS single-chip imagers that can be mass produced and are extremely compact. For many applications, imagers can be built at a price much closer to a normal camera, than to a research instrument, thanks to the technology used to make ordinary computer chips. New since last year are the shortwave infrared sensors, for which the first camera prototypes are now being built.
The cameras designed around these chips are light and compact, making them suitable for a wide range of applications in, for example, precision agriculture or medical instrumentation. The imaging devices themselves can be flexibly designed to meet the demands of the application by selecting the spectral bands and resolution that best suit that application.
"Of course, we are not experts in brain surgery, skin cancer detection, potato sorting and painting. What we do is calibrate the technology and make it portable and robust so that it can be used in all these applications.
It's very enriching for us to interact with these different experts, to see how we can help them do their job," says Andy Lambrechts, group leader of the Integrated Imaging group at imec. Together with colleagues from imec Florida, the group shows the versatility of HS imaging in some recent applications.
HS imaging sensors.
The Da Vinci of the Low Countries
Exactly 500 years ago Leonardo Da Vinci died. His famous fresco 'The Last Supper' can still be admired in Milan. The American art historian Jean-Pierre Isbouts is convinced that the master also had a hand in a copy in the abbey of Tongerlo in Belgium. To confirm his theory, he contacted imec. Imec's HS cameras see more than ordinary cameras. "We visually divide the work into different layers per wavelength", explains Wouter Charle, team leader of the HSI Software team. "This resulted in a set of 150 images, on which researchers can then perform their analysis.
Different types of information can emerge. A restorer, for example, will ensure that any added paint remains invisible to the naked eye. With the HS camera, on the other hand, you can see whether the paint has a different chemical composition. That way you can ignore the more recent finishings and focus on the oldest layers of paint. In addition – because the HS images are combined with detailed images from a traditional camera – you can also study the fine technique of painting and derive characteristics such as the way in which the brush was held. Using infrared or deeper spectral bands, you can, for example, also look at deeper, underlying layers or cracks that may have been repaired. Based on this information, the team of Isbouts will now continue laying the pieces of the puzzle.
Images being registered with the HS camera.
With the HS camera in hand...
A big advantage of imec's HS cameras is that they are portable and compact, making them perfect for use on location. As a result, the right images were available for analysis in just a few hours. Everything started with good preparation. "We explored the location once in advance to see the size of the painting and how much light was present. This allowed us to decide which lenses and lighting were needed and to return the second time with the right equipment," says Imaging Systems engineer Robin Lefrant.
"Professor Isbouts had already discussed with us which parts of the painting he wanted to image in greater detail. After all, he was not interested in a widescreen view, but rather in the heads of the figures depicted. His theory was that if Da Vinci had worked on the painting, he would probably have painted the details of the most important figures. With our camera you can record an image in just a few minutes, but the challenge is the right calibration, the right set-up and focus. And, compared to existing scientific HS cameras - which are much larger - this went very smoothly."
The compact HS camera at work on location.
It was also the first time that the new shortwave infrared sensors were used in the field. Several scientific HS cameras already exist in near and shortwave infrared, also for art inspection. "This is a new domain for us, but at the same time a perfect application for our HS cameras. The traditional scientific cameras for those wavelengths, which are normally used to inspect paintings, are large and difficult to set up. Very often a painting like this has to be transferred to a special location, which makes the whole process very expensive. Our cameras, on the other hand, are small and easy to transport, so that we can take high-quality shots on the spot without too much effort," concludes Wouter Charle.
There's a catch to catching pythons
The next application takes us to Florida, where exotic pythons threaten the native fauna of the Everglades. The animals have no natural enemies there and have caused a 90% decline in the mammalian population. The snakes are tracked down with great effort, but so far with limited results. Pythons are very well camouflaged and almost invisible in the vegetation with traditional cameras. Moreover, they are cold-blooded animals, so thermal cameras are also not a solution. "Ronald Driggers, professor at the University of Central Florida (UCF) and imec scientist, brought this problem to our attention and proposed a collaboration between imec, UCF, Extended Reality Systems (ERS) and the University of Florida. ERS is a small, local company that was actively looking for a solution to detect pythons in the Everglades with optical sensors," says Orges Furxhi, group leader of the Camera Systems & Computational Imaging group in imec Florida.
Python image at 850 nm (inverted and saturated) in a mixed background.
Orges Furxhi: "At that moment we also called on the Leuven team and their HS cameras. They trained us in the use of the cameras, helped us with the first measurements and taught us the best measurement practices with the HS imagers. We then measured the spectral fingerprint of the pythons – first in captivity, but also in the wild with the vegetation in the background. These measurements were then used to design a single-band camera and algorithms for detecting pythons. Since then, we have shown that the camera and algorithms are effective in detecting pythons. The team from Leuven also provided us with multispectral cameras with real-time image capture (the snapshot cameras) to record additional images of the snakes and the backgrounds. This data was used to build up a deep learning network for automatic python detection."
Precision agriculture with Flemish technology
Andy Lambrechts: "This year the Hyperspectral Team is also participating in a number of projects with Flemish and European players on precision agriculture. For one of them, several Flemish research centers have set up a joint testing ground to work on 'smart crop protection' in potato and fruit cultivation. By installing HS cameras on drones and tractors, plant diseases such as fire blight (a bacterial plant infection, especially in pears) can be detected more quickly and in a more targeted manner, and can therefore be tackled with precision. These are often multi-disciplinary projects that bring together experts from different fields. On imec’s side, Field Support Engineer Roeland Vandebriel, is focusing on the camera technology, while other groups specialize in mounting our cameras on drones or tractors, in fine-tuning disease detection algorithms or in sharing their agricultural know-how. By bringing all this knowledge together, these kinds of multidisciplinary projects can lead up to real prototypes that link everything together and validate the application."
Aerial images made with a hyperspectral camera, developed by VITO Remote Sensing and incorporating an imec sensor. Left are four images taken on different days over a winter wheat experimental plot partially treated with fungicides and hosting 40 different varieties (data acquired in collaboration with CRA-W, Gembloux (B)) with ‘normalized difference vegetation index’ (NVDI) overlay (= an indicator for live green vegetation). As a function of time certain varieties become infected and loose their greenness On the right, image data of experimental strawberry fields are shown in which different plots have undergone different treatments (data acquired in collaboration with PCfruit, Sint-Truiden (B)): at the top, an RGB picture and at the bottom, a chlorophyll index map showing the level of chlorophyll (an indication for the plants’ general health). The maps illustrate the different growth response of the strawberry plants as function of their environment (irrigation and fertilization) All this perfectly illustrates that the spectral fingerprints that the camera records as raw data can be transformed – through spectral image processing – into useful and actionable maps for the farmer and researcher. (Courtesy of VITO Remote Sensing)
In addition to the HS cameras themselves, a great deal of attention is paid to calibration so that high-quality data can be supplied to researchers. "Our software is to a certain extent application-agnostic," explains Vincent Radelet, Hyperspectral software engineer. "In first instance, we do not work on the recognition of fire blight, the software’s primary goal is to ensure the quality of the data produced by our camera. It reads out individual images and combines them into 1 HS image. At the same time, the sensitivities of the sensor are removed and the difference in lighting is eliminated. By compensating for all possible perturbations, spectroscopy is brought to image level. Our software controls the cameras in a user-friendly way and presents the acquired data to the researchers in the best possible condition. This allows the experts to perform their analyses on a pure HS image, without the side effects that camera components can have on the image recording.”
The right HS camera for the job
For all applications, the first choice was a generic camera. Andy Lambrechts: "Especially if you don't know in advance what you're looking for, it's advisable to use the camera with the highest number of spectral bands. That is the best starting point to determine where the information is hidden and which spectral bands you need. Also, if you start with the richest dataset, you don't have to go back on location, but you can simulate in software what the image with less spectral bands would have looked like. Then you can also develop an application-specific sensor afterwards, with only the bands of interest."
A snapscan camera setup.
For still images, the snapscan cameras are used. The sensors in these cameras scan and generate very high-resolution images. And that means for the HS camera not only high resolution based on form information (the number of pixels in an image) as with an ordinary photo, but also in number of spectral bands: for cameras that record visible light and near infrared light, there are 150 bands and in shortwave infrared 100. The snapscans are not the preferred solution when the object moves too much. In this case, other sensors are used that alternate between 16 different spectral bands on individual pixels of the image sensor. This allows you to make a direct video recording at a lower resolution.
More cameras, more applications
"Last year, for the first time, we launched a shortwave infrared sensor. We are now building the various scanning and video-rate cameras for this and we are developing the right corrections. We have known for several years that there were many applications of shortwave infrared – even more so than for visible and near infrared – but because these sensors are so complex, it also takes more time to develop them.
The first prototypes have already generated a lot of interest. Our unique way of making compact and mass-producible HS cameras for that area is very sought-after,”
concludes Andy Lambrechts.
A snapscan camera for visible and near infrared imaging.
Want to know more?
- More information about the HS activities in imec can be found on the website.
- Read the press release on HS imaging for the Da Vinci fresco (Dutch only).
- The python research was published in 2 papers: 1) On reflectivity of pythons by Ronald Driggers, 2) On camera design and algorithms by Gonzalo Vaca-Castano. Both papers can be requested via the contact form.
- Our HS cameras in precision agriculture: read about it in the press release (Dutch only).
