ILLUMINATE

ILLUMINATE had several specific goals. The first was to create hyper-realistic datasets and a movie staging a compelling story. Secondly, it aimed to develop a multimedia data representation that would be optimal for processing and streaming immersive experiences while offering high-quality and interactive visualization, something which is currently nonexistent. It also sought to create a streaming protocol that can handle low-latency viewpoint changes and requests, along with rendering and visualization proof-of-concepts for its new representations. Last but not least, the project investigated protocols for assessing the quality of experience (QoE) of interactive immersive experiences.

THE OUTCOMES

1. End-to-end proof-of-concept demonstrators

The work conducted in the ILLUMINATE project resulted in an end-to-end proof-of-concept demonstrating the full experience including streaming, along with seven other proofs-of-concept each focusing on a particular aspect of the end-to-end delivery chain. As yet, no commercial solution delivers immersive video streaming with the same degrees of freedom and hyper-realism as those demonstrated in the ILLUMINATE proofs-of-concept. In particular, the project delivered: - specific guidelines for storytelling in 6 DoF experiences - improved depth estimation for light-field cameras - AI-based view synthesis for free viewpoint generation - novel real-time depth-image-based rendering techniques - multi-plane images for view-dependent 6 DoF - a novel learning-based representation format for 6 DoF content that allows for real-time light-field rendering

2. Tools for assessing the quality of immersive experiences

In addition, the project created a new Quality of Experience model and procedure to assess the quality of both the immersion and storytelling. This protocol provides insights into which dimensions contribute most to the QoE and which system, context or user factors lead to significant fluctuations in these dimensions and, thus, the general perception of quality.

3. Resources for further research

ILLUMINATE also led to numerous research publications and new insights in the domain. These publications included eight journal papers and eleven conference papers. The project also generated datasets for the research community as well as Intellectual Property Rights (IPR) – in particular, a patent was filed. regarding a novel way to represent and interactively render content with 6 degrees of freedom.