The arTIco project: Bridging AI and Vehicle Safety

Mission and Innovation: Transforming Virtual Model Validation

The project aims to develop an expert knowledge-based machine learning approach for improved evaluation of virtual models, with a focus on crash test scenarios. The application example of crash test dummies for passive vehicle safety should demonstrate the methodology's performance. The current correlation method (ISO 18571) is used as a reference while the new approaches are intended to enhance the established ones with expert knowledge, make them more advanced, and in the future, be an alternative to current certification and validation procedures.

Strategic Objectives: Advancing Digital Twin Technology

• Develop a method to conserve subjective expert knowledge in evaluating virtual models, particularly focusing on the WorldSID 50M dummy model used in crash simulations.
• Create an approach that adds value to existing objective state-of-the-art methods, such as ISO 18571, for assessing the correlation between virtual models and hardware tests.
• Establish a correlation evaluation system based on the overall behaviour of virtual models, moving beyond simple corridor checks or isolated metric comparisons.
• Design a rating catalogue and criteria that capture the nuanced expertise of human evaluators in assessing simulation results.
• Implement machine learning techniques to automate and replicate expert assessment of virtual model performance.

Project’s Outcomes: The Power of Hybrid AI

The arTIco project yielded substantial and promising results in its quest to enhance virtual model evaluation for crash test simulations. At the heart of the project's achievements lies the development of a Hybrid Neural Network algorithm. This supervised learning model combines convolutional blocks for automatic feature extraction along with engineered features. The algorithm processes complex simulation data alongside additional key features to predict expert ratings with good accuracy.

To train this advanced model, the team generated a comprehensive database of 1000 Crash-Test-Simulation variants (samples), each rated by three experts. This labelled dataset formed the bedrock for training and validating the machine learning model. Complementing this data-driven approach, the project established a nuanced four-tier rating system —Good, Acceptable, Marginal, and Poor— supported by detailed evaluation criteria for various aspects of the WorldSID 50M dummy model, including individual sensors and overall performance, see Figure 1.

Figure 1: Showcase illustration of methodology

The project's innovation extended to the creation of the arTIco metric, a custom evaluation metric ranging from -1 to +1, designed to provide a qualitative assessment of model performance. The final model demonstrated good results, achieving a score of 0.9 (+/- 2%) on validation data and 0.8 (+/- 2.5%) on test data. When compared to existing methods, the arTIco approach showcased its ability to identify subtle nuances in model performance that might be overlooked by traditional corridor checks or ISO 18571 scores. A case study illustrated this capability: the arTIco AI predicted a “Marginal” rating for a model that failed the corridor check but paradoxically scored high on the ISO scale.

Through this process, the project provided valuable insights into the complexities of conserving expert knowledge, highlighting challenges such as managing inconsistent ratings and the importance of considering multiple signals and their interrelationships. Looking forward, the team identified potential enhancements to the system, including the incorporation of kinematic information, improved explainability through techniques like Shapley values, and the extension of the method to other dummy types and more complex load cases.

In essence, the arTIco project successfully developed a novel, more nuanced, and comprehensive assessment method for crash test simulations.

By Felix Stocker, AI Data Scientist department CAE, IDIADA Fahrzeugtechnik GmbH

Patrick Fischer, Leader Simulation & Innovation, IDIADA Fahrzeugtechnik GmbH