Generative AI for smarter CCAM: enhancing perception, decision-making, and validation (CCAM Partnership)

Expected Outcome

Project results are expected to contribute to all the following expected outcomes:

• Availability and integration of advanced, trustworthy, energy-efficient perception systems, exploiting technological advancements of Generative AI (GenAI) to enhance situational awareness and support safe decision-making;
• Enhanced Vulnerable Road User (VRU) safety, based on elevated, more temper-proof perception and understanding of their behaviour and intention predictions;
• Enhanced robustness of CCAM systems - both on-board and on the infrastructure side - in critical situations due to their training, virtual testing and validation in scenarios generated by GenAI, complementing existing scenario databases for the testing and validation of CCAM systems;
• Enhanced understanding of the relevance and limitations of using GenAI for CCAM;
• Tools and harmonised approaches for the use of GenAI in mobility technology development, training and validation, as well as for systemic applications such as traffic management and remote control, integrating them into existing approaches.

Scope

Pilots and demonstrations using Level 3 and 4 vehicle services face major challenges in perception and decision making, highlighting the necessity for low-latency solutions that enhance responsiveness and situational awareness in real-time operating conditions. This is especially relevant for driving in more complex environments like urban areas, where environmental variance is higher and where new scenarios can be regularly encountered. Furthermore, there is the need to limit the latency, bandwidth and energy use for on-board calculations, as well as the need to enhance the security, privacy and reliability (e.g. scene understanding and prediction of near-future scenario development). For rapid decision-making in interactions with VRUs, this is essential for implementing CCAM-enabled solutions and ensuring scalability.

At the same time, developments of sector-agnostic technologies show advancements -such as GenAI- that can be beneficial for CCAM. First exploratory steps can be expected from a project funded under HORIZON-CL5-2023-D6-01-02[1] regarding the potential in the virtual generation of edge cases, which could be used for the development, training, virtual testing and validation of CCAM systems.

Further advancements in GenAI applications specifically for the CCAM domain need to be developed, trained and validated[2]. Thus, proposed actions shall include approaches to exploit further technological advancements for CCAM. Major steps are needed to advance to highly advanced, ultra-safe, trustworthy and energy efficient real-time perception and decision-making systems for automated vehicles, specifically focusing on scalable solutions and the exploitation of GenAI. These advancements should leverage low latency systems or distributed computing resources to facilitate real-time processing, thereby improving system responsiveness and safety. This topic will thus contribute to the AI Continent Action Plan[3] by fostering AI development and adoption in the automotive sector.

Proposed actions are expected to address all the following aspects:

• Development of tools and approaches for robust environment perception and decision making (at the edge, on-board, at infrastructure or back-office). These approaches shall aim at accelerating and advancing the reasoning of decision making, increasing the level of efficiency, (cyber)-security and reliability of the applications, with path planning as initial use case. This is to support amongst others the perception of VRUs, the prediction of their behaviour and their intentions, and includes data sharing approaches for CCAM solutions to create a larger time window for actions in near accident scenarios. The use of advanced GenAI, including Large Language Models (LLMs), Vision Language Models (VLMs) or Vision Language Action (VLAs) can significantly enhance these capabilities by leveraging their advanced contextual reasoning and pattern recognition. Furthermore, GenAI can complement existing perception systems by improving sensory input interpretation and providing enriched environmental contexts, which enhance decision-making and adaptability.
• Scenario generation of interactions of CCAM enabled vehicles with other road users, which is essential for advances in validation and testing, extending existing datasets and scenarios as GenAI can, based on existing data, deliver variations of scenarios (e.g. cultural differences of road users and infrastructure variability.)
• Integration of GenAI technologies into existing approaches (development, training and validation) for their further enrichment. Understanding the limits of using GenAI technologies as well as the benefits and develop guidelines for valid approaches for this integration (including consideration of gender biases and fairness to ensure AI systems are transparent and accountable) and providing an outlook on the uptake of the tools and approaches developed can be done for a variety of CCAM components and technologies, as well as for systemic applications such as traffic management and remote control.
• Encouraging collaboration with the European Software-defined Vehicle (SDV) initiative by adopting existing interfaces and building blocks, and proposing new ones developed within the project for potential inclusion in the SDV framework.

Proposed actions should include measures to ensure close coordination with the European Connected and Autonomous Vehicle Alliance (ECAVA) announced in the European Automotive Action Plan.

This topic implements the co-programmed European Partnership on ‘Connected, Cooperative and Automated Mobility’ (CCAM). As such, projects resulting from this topic will be expected to report on results to the European Partnership ‘Connected, Cooperative and Automated Mobility’ (CCAM) in support of the monitoring of its KPIs.

Projects funded under this topic are expected to liaise with the ADRA Partnership[4] in order to explore and leverage complementarities between their respective activities and findings.

Projects resulting from this topic are expected to apply the European Common Evaluation Methodology (EU-CEM) for CCAM[5].

Technology Readiness Level - Technology readiness level expected from completed projects

Activities are expected to achieve TRL 5-6 by the end of the project – see General Annex B.

[1] https://cordis.europa.eu/project/id/101146542

[2] This topic also encourages alignment with national projects and initiatives to allow building up of existing efforts and solutions.

[3] https://digital-strategy.ec.europa.eu/en/library/ai-continent-action-plan

[4] AI, Data and Robotics Association: https://adr-association.eu/

[5] See the evaluation methodology here.