The societal benefits of autonomous vehicles (AVs) — those that operate fully without a driver — have never been clearer than they are right now. The delivery of essential items, such as medical supplies and food, with limited human contact, would serve us well in the current Coronavirus climate.
While the development of AV performance is key, we must not neglect the assurance of their safety.
Why safety standards are not enough
Current standards for the safety of road vehicles (e.g. ISO/PAS 21448 and ISO 26262) do not map well to AVs (SAE level 3 and above) because they assume the presence of a human driver to take over in an emergency.
There are emerging standards that don’t assume human drivers, with perhaps the most prominent being UL 4600. That standard focuses on developing a valid safety case for an autonomous product. But it is not, and cannot be, fully prescriptive — we still need to do bespoke work to build the safety case for the AV ourselves.
Our approach to the safety assurance of AVs
As UL-4600 outlines, a valid safety case for an AV must be based on evidence from a combination of the following techniques:
- Formal analysis
- Functional testing (black box testing)
- Structural testing (white box testing)
- Closed course testing
- Public road testing
Our current research as part of the EU project Safer Autonomous Systems (SAS) focuses on the simulation-based testing for verification and validation (V&V) of AVs.
We are testing the AV in simulation, considering the AV as a black-box or in some cases as a grey-box. We will test the AV with challenging situations (edge cases included) with a feedback mechanism/metric called “situation coverage”.
Situation coverage is a coverage criterion that will let our automated test generation algorithm know which situations have been covered in our testing and which have yet to be tested. It will also highlight those situations that would cause the AV to fail: these will be given a higher score so that similar situations are searched more by our automated test generation algorithm. We are also looking into using methods such as fault injection in conjunction with situation coverage-based testing.
Challenges in our proposed situation coverage approach
One of the most challenging tasks with this approach is to design the situation hyperspace i.e. the simulated world around the AV. It is from this hyperspace that our automated test generation algorithm systematically selects challenging situations that a real AV would face.
For this task, we are examining existing AV world ontologies closely. These include the works of Krzysztof Czarnecki in designing operational world models for defining the Operational Design Domain (ODD) and the National Highway Traffic Safety Administration ODD for AV.
Our situation hyperspace is different from these ontologies as it has to be structured in a way that our automated test generation algorithm can easily understand and navigate to find and implement valid real-world situations.
These situations challenge the AV, possibly causing it to fail. The algorithm gives a situation coverage metric and a pass or fail score. The algorithm continues to navigate the situation hyperspace, finding more challenging situations for the AV to test it thoroughly.
In the end, some confidence measure of our testing can be provided to the public: a situation coverage metric along with the pass/fail rate of the AV under test.
Practical guidance
The safety assurance of AVs is the key to unlocking their benefits. To develop the safety case that a standard such as UL-4600 requires, we must be confident in every element of assurance, from road tests and simulation-based V&V to hardware verification and micro-chip level verification.
Our proposed situation coverage-based testing for V&V of AVs is a part of this safety assurance. As our research continues and we obtain confidence measures that can be used with the public, regulators, and others, we’ll provide practical guidance in the AAIP Body of Knowledge for all to access and use.
About the Author: Zaid Tahir
Zaid did his bachelor’s in Avionics Engineering from the National University of Sciences and Technology (NUST) Pakistan in 2015. During his bachelor’s, he worked on many projects relating to robotics, especially those that involved autonomous vehicles and unmanned aerial vehicles. This passion in robotics led him to enroll and complete a master’s degree in Robotics and Intelligent Machine Engineering from NUST Pakistan in 2018. Zaid did a technical job in an engineering firm for three years while doing his masters. As soon as his masters was completed, his dedication and devotion to robotics pulled him in applying for a PhD that was specifically related to autonomous vehicles which had been his passion all along and oddly enough, currently, he is handling the testing of autonomous cars as ESR7 in the MSCA ETN-SAS project.
