NimbleAI is on: The Horizon Europe project aimed to push the boundaries of neuromorphic vision

On 23rd and 24th of November 2022 the 19 project partners from across Europe gathered in Arrasate- Mondragón (Spain) to officially kick-start NimbleAI  – the 3-year and 10 M€ research project which will improve energy-efficiency and performance of next-generation neuromorphic chips that sustain event-based vision. Their goal is to create an integral neuromorphic sensing-processing 3D silicon stacked architecture to efficiently run accurate and diverse computer vision algorithms in resource- and area-constrained chips destined to endpoint devices.

As seen in recent technological trends, biological systems are a “golden benchmark” for electronic systems as well as a source of inspiration. NimbleAI thus leverages the power of nature’s computing: the project’s global system architecture is based on and motivated by the biological eye-brain system. These organic vision capabilities are honed by natural selection and apply the fundamental energy-saving principle of capturing, processing, and storing data only when necessary. Hence, eyes continuously sense and encode the changing surrounding environment in a way that is manageable for the brain.

Following bio-inspired principles, NimbleAI will deal with the conscious and unconscious rationale behind the decision-making process on what visual stimuli to process – and what stimuli to discard without processing. In NimbleAI, a frugal always-on sensing stage will build basic understanding of the visual scene and drive a multi- tiered collection of highly specialized event-driven processing kernels and neural networks to perform visual inference of selected stimuli using the bare minimum amount of energy.

NimbleAI envisions a highly integrated 3D stacked silicon architecture where sensing, memory, communications, and processing are physically fused and accuracy, energy, resources, and time are dynamically traded off to enhance the overall perception, that is, maximize the amount of valuable visual information that can be captured and timely processed. As it occurs in biological visual systems, sensing and processing components will be adjusted at runtime to match each other, and operate jointly at the optimal temporal and data resolution scale across image regions.

NimbleAI expects to achieve 100x energy-efficiency improvement and 50x latency reduction (with regard to. CPU/GPU) by relying on the technologies, components, and techniques listed below:

• Light-field-enabled dynamic vision sensing.
• Event-based inference and processing.
• Specialized processing with in-memory computing and programmable logic (eFPGA).
• Embedded ReRAM-based storage.
• 3D integration of circuit layers (TSV-based inter-layer data movement).
• Mutual adaptation of sensing and processing to operate at optimal DVFS point.
• Dedicated software tools

NimbleAI will design EDA tools to customize and integrate the technologies and components listed above and deliver a prototypic FPGA implementation of the 3D silicon stacked sensing-processing architecture, as well as the corresponding programming tools to develop and run computer vision applications on it. The prototype will be used as a research vehicle to test novel concepts, computer vision algorithms, and runtime optimizations. NimbleAI results will lead to practical implementations in the next generations of commercially available neuromorphic chips. In fact, the project will manufacture a few testchips to validate some key concepts in silicon.

The consortium is formed by the following organizations: Ikerlan S. Coop. (ES), Barcelona Supercomputing Center (ES), Menta SAS (FR), Universiteit Leiden (NL), Codasip s.r.o. (CZ), GrAI Matter Labs B.V. (NL), University Of Manchester (UK), Consejo Superior De Investigaciones Científicas (ES), Universitat Politècnica De València (ES), Monozukuri S.P.A. (ITA), Politecnico Di Milano (IT), Commisssariat A L’energie Atomique Et Aux Energies Alternatives (FR), IMEC – Interuniversity Microelectronics Centre (BE), Raytrix Gmbh (DE), AVL List Gmbh (AT), ULMA Medical Technologies S. Coop. (ES), Viewpointsystem GmbH (AT), Queen Mary University Of London (UK) and Technische Universitat Wien (AT).

NimbleAI has received funding from the European Union’s Horizon Europe Research and Innovation Action programme (RIA) under Grant Agreement 101070679 and from UK Research and Innovation’s Horizon Europe guarantee scheme. This funds UK entities who have applied successfully to Horizon Europe but are unable to sign a grant agreement with the EU prior to formal association of the UK to the programme.