Testing Tomorrow's Self Driving Cars Today

According to a report issued by the World Health Organisation in September 2016, about 1.25 million people die each year as a result of road traffic accidents. Even more disturbing is the finding that road traffic injuries are the leading cause of death among young people, aged 15–29 years – people in the prime of their life.

And in August 2016 the Obama administration delivered what it called an “unprecedented call to action” as it released a final analysis of traffic fatality data from 2015, indicating that road deaths jumped by 7.2 percent over those recorded in 2014 to 35,092; the highest since 2008.

To counter the rising death-toll the September 2015 United Nations Sustainable Development Summit set an ambitious road safety target of halving the global number of road fatalities and injuries by 2020. Considering the ever increasing traffic on the roads this may seem like a bridge too far!

Fortunately, Original Equipment Manufacturers have been whittling away at the cause of many of these fatalities – the human driver. Through the introduction of Advanced Driver Assist Systems manufacturers have slowly pried away several key safety functions from the driver and handed them over to the car. So, functions such as Adaptive Cruise Control and Autonomous Emergency Braking have been handed to the machine in an attempt to make our roads safer: An April 2016 Business Insider report quoted findings from KPMG that in the UK alone, self-driving cars will lead to 2,500 fewer deaths between 2014 and 2030.

But man’s obsession with the machine doesn’t end at ADAS and semi automated Level 3 and 4 (as defined by the National Highway Traffic Safety Administration) vehicles, but rather the promise of a fully self-driving car controled and taught by artificial intelligence.

The immediate problem with this is that, being an immature technology, engineers, legislators and the public at large need assurance as to how safe this new tech is; and the only way to do this is through testing… extensive testing!

Why double up real world evaluation with virtual testing?

The question, though, is how many kilometers of driving would it take to make statistically significant safety comparisons between autonomous vehicles and those with human drivers? Considering that current human-piloted vehicles have a history spanning more than a century and have covered billions of kilometers, autonomous vehicles would have to be test-driven hundreds of billions of kilometers within the next few years to confidently demonstrate their safety.

"The highest autonomous distance any autonomous-vehicle developer has covered is about 2.4 million kilometers, and that took several years. The information gathered is important, but it does not come close to the level of driving that is needed to calculate safety rates," said Susan M. Paddock, senior statistician  and co-author of the 2016 Rand Corporation study into self-driving transportation. "Even if autonomous vehicle fleets are driven 10 million kilometers, one still would not be able to draw statistical conclusions about safety and reliability."

For example, assuming a self-driving vehicle fleet had a 20 percent lower fatality rate than that for human drivers: Substantiating this with 95 percent confidence would necessitate driving 8 billion kilometers. This would be equivalent to driving every road in Texas nearly 16,000 times over. It would take a fleet of 100 vehicles, driving 24/7, around 225 years to deliver these results!

To address the challenges, Motor manufacturers have embarked on a two-pronged approach to the problem: 

• On-road driving to log miles under real driving conditions and verify the results obtained through virtual testing
• Intensive virtual testing to evaluate various scenarios; simulating traffic and road conditions, as well as predicting vehicle response to system changes before updating test vehicles

The automotive industry has used road testing to verify everything from performance and economy to cooling system performance, so it’s natural that this would form an important part of verifying the performance and safety of ADAS and self-driving technology.

To facilitate the introduction of the new technology the state of California was the first in North America to grant permits to test autonomous vehicles on public roads.

The value of road testing autonomous vehicles

To date the California Department of Motor Vehicles has issued permits for 11 manufacturers to test self-driving cars on public roads.

As can be expected Google leads the pack with 73 cars and 230 approved human drivers; that’s nearly as many as the 77 cars with permits from the other 10 companies combined. Google also has 70 more human drivers qualified to test drive self-driving cars than its rivals have altogether. Tesla comes in second for the most cars with test-driving permits: As of March 1, it had eight registered vehicles and five permitted drivers.

But why the fascination with California? 

Well for starters it’s home to Google’s self-driving project located in Mountain View, where over the past six years the tech company has racked up over 2.4 Million kilometers. Secondly the weather is generally good; good visibility removes one of the many variables associated with road testing self-driving vehicles.

Testing autonomous vehicles in adverse conditions

But in the real world, vehicles do not always operate under favorable conditions and as self-driving technology matures manufacturers will have to test in adverse conditions as well, to ensure the functionality and safety of the systems.

The ability of self-driving vehicles to function in inclement weather is critical, consequently road test programs have already been expanded to Austin Texas and Kirkland Washington, which records significant rainfall throughout the year.

LiDAR and camera sensors’ performance is significantly impaired with reduced visibility so, although there are several software programs available that can simulate these conditions to virtually test the sensors, vehicles will eventually need to be evaluated under varying conditions from snow and ice, through to fog, night driving and bright sunshine.

In a dramatic test of autonomous technology’s low light performance, Ford recently navigated a Ford Fusion Hybrid autonomous research vehicle with no headlights along a twisty stretch of desert road at night, guided only by LiDAR. Although the test was arranged for the benefit of the press, it did demonstrate the level of evaluation that is required to validate the performance of autonomous technology.

And it’s not only weather and visibility that impact on ADAS and self-driving systems, adverse environmental conditions also affect performance. According to Jennifer Haroon, head of business operations for the Google Self-Driving Car project, testing in the Phoenix area with its vast expanse of desert will help autonomous vehicle manufacturers better understand how the sensors and cars handle extreme temperatures and dust.

And with self-driving technology set to replace conventional vehicles across the globe it’s important that testing is stepped up outside of the US.

Self-driving cars appear in tests across Europe

In Europe where ADAS has already achieved a high level of acceptance in series production cars, the first test drive of self-driving cars took place on the A2 motorway connecting Amsterdam with the Belgian border in March 2016. A fleet of fifty semi-autonomous cars drove down the road in columns of six, autonomously adjusting their speeds and following distances. 

Never before have semi- autonomous cars driven amongst normal traffic on a public road in Europe. The test was an initiative of insurance broker and risk consultant Aon, Business Lease Nederland, Royal HaskoningDHV and Prodrive Training.

This was a tangible sign that the Dutch national government aims to take the lead in testing self-driving vehicles in Europe. To achieve this legislation has already been amended by the Ministry of Infrastructure and the Environment (I&M) in 2015 in preparation for the implementation of large-scale tests with self-driving passenger cars and trucks on public roads. 

In collaboration with the Rijkswaterstaat (the Road Authority) and the RDW (Dutch Vehicle Authority) the Ministry of I&M has been exploring safe ways to conduct tests on the public roads. 

Under the amended legislation, the RDW has the option of issuing an exemption for self-driving vehicles under specific conditions: Companies that wish to test self-driving vehicles must first convince and demonstrate that the tests will be conducted in a safe manner, before submitting an application for admission. 

In an effort to draw attention to this progressive approach more than a dozen self-driving trucks made by six of Europe’s largest manufacturers were driven to Rotterdam’s port in so-called “truck platoons” in April 2016.

Notwithstanding the success of the truck platoon demonstration, the EU’s legislation regarding testing of self-driving cars on public roads is restrictive and complex, with a U.N. regulation even limiting cars without human drivers from going faster than 10 km per hour. Negotiations to amend this ruling won’t be concluded before the second half of 2017. 

To circumvent many of these rules a start-up company, Self-Driving Track Days, has been formed with the sole purpose, of promoting the testing of fully autonomous cars across Europe. The testing will be private, and is designed to allow any team that develops autonomous vehicles to test its accomplishments in a controlled environment. 

The first Self-Driving Track Day event will be held this November, at the UK’s Bruntingthorpe Proving Ground. There, researchers will be able to experiment with the settings, sensors, and software of their autonomous vehicles. The first event of this kind in Europe has access to a 1.6-kilometer long multi-lane motorway straight, which will be used for the first driverless tests of independent projects.

Unlike events reserved for automakers, the Self-Driving Track Days operation is designed to encourage inventors, innovators, and researchers to explore diverse solutions, in the hope of speeding up time-to-market. Instead of charging massive levies to allow the rental of a track or a research institute, the event will offer access to specialist training from industry professionals without the enormous costs usually assosciated with the development of new technologies.

Automotive companies, start-ups, and students will be urged to join and form multi-partner teams at the networking meet-up events organized in various venues, such as the Google Campus in London and AutoWorld Brussels.

[inlinead]

However blanket EU restrictions have not completely stopped road testing of autonomous vehicles. In Finland versions of a driverless bus, Easymile EZ-10, have been roaming suburban Helsinki this summer as part of a month-long trial. Similar vehicles have already been tried out in the Netherlands. 

While in Gothenburg, Volvo will begin testing 100 cars in 2017. The testing environment will be limited: certified roads only, weather limitations, no traffic lights, no pedestrians and no bicycles allowed and the maximum speed will be restricted to 70 Kmh.

Virtual testing speeds up the development of autonomous vehicles

While most self-driving car development teams acknowledge that actual road testing is crucial, virtual simulation is equally important.

Commenting on the efficacy of road testing Mark Hakim, managing director at Qosmotec commented: “It all depends how you are testing and whether every possible situation has been covered by the tests.” Qosmotec develops test systems that allow virtual-drive tests in the lab before the deployment on the road for fully connected vehicles.

“Field tests leave a high degree of uncertainty. You need to construct and reproduce traffic situations to ensure they are actually tested”, says Hakim. Even if all cars were driving autonomously and communicating with each other, problems concerning radio transmission could still occur. Intended to evaluate the impact of these disruptions, Qosmotec’s virtual test system allows engineers to detect potential problems and emulate real-life traffic scenarios in the lab.

Typically, before any modifications are made to an autonomous system they would first be tested in a virtual environment using thousands of variations, before integrating the update into the road-test vehicles.

A virtual environment allows engineers to assess new software for autonomous systems in a realistic virtual environment, testing how tweaks to their behavior might play out in reality. 

For example, a new emergency braking system was virtually tested by Google for over 16,000 kilometers to evaluate how often it was engaged and how it performed in emergency situations. And in another test, 50 virtual self-driving cars were dropped into a digital version of Mountain View to see how they interacted with each other, computer-generated pedestrians and other simulated drivers.

Test drivers and engineers can acquire considerable amounts of data from these simulations. "Each time a test driver takes over from the self-driving car, we're able to play back the exact situation and predict via simulation what could have happened if the car had been left to drive itself. If the simulation shows that the driving needs to be improved, our engineers can make refinements to the software, and run those changes in simulation to test the fixes," says Google spokeswoman Katelin Jabbari.

As a result, these test vehicles drive a simulated six million kilometers each day, or the equivalent of circling the equator five times every hour. 

Because virtual testing is focused on rare and potentially hazardous scenarios, these tests equate to many generations of human driving. "These are not straight and empty roads but six million challenging kilometers that actually teach us to modify the way the self-driving car should react to unexpected or safety critical situations" said Jabbari.

Expanding the horizons of virtual testing

However driving culture and norms vary considerably around the world. Therefore self-driving companies need to collect a lot more infrastructure and digital data about other global regions before the technology can be rolled out with confidence.

To solve this problem agent-based modeling such as the Ricardo Agent Drive simulation software can be used to create real-world driving scenarios to test complex driving situations for autonomous vehicles. These scenarios revolve around the agents and are therefore independent of specific geographical locations.

This form of testing places agents (vehicles, people or infrastructure) with specific driving features (such as selfish, aggressive, defensive) in a defined environment (cities, freeways or test tracks) to understand the complex interactions that occur in real-world situations. 

Employing this approach OEMs and their suppliers are able to cut product development cycles, reduce costs related to test-vehicle damage and lower the risk of harming vehicle occupants under test conditions.

However, when considering regional differences it is also important to ensure ADAS systems also comply with standards and regulations specific to the country. And it’s not just compliance with regulations that manufacturers have to contend with: every country has its own driving conditions influenced by the region’s driving culture.

Adaptive cruise control illustrates this point perfectly: 

The ACC system is dependent on driver behavior, which is often unique to the country, and must therefore be adjusted accordingly. Research has shown that drivers in the USA tend to brake and accelerate smoothly and slowly, whereas German drivers brake and accelerate far more aggressively. 

Testing these apparently small differences in braking and acceleration is extremely time-consuming and can lead to a significant increase in costs attributed to testing. 

To ensure repeatability and save time, a brake simulator is used to carry out the simulation with the integrated ACC control unit. The simulation includes probability checks, such as dynamic check sums and rolling counters, to ensure the pre-defined braking parameters are met, thereby validating the system’s performance.

Notwithstanding the importance of these virtual simulation tests, the injection of the imperfect human into the equation can never be fully replicated: by nature humans have varying and inconsistent reactions to situations, and therefore reliable testing has to include the human driver.

With level four automated vehicles expected on public roads as early as 2020, there are several concerns surrounding the driver or the human operator within these highly automated or semi-automated driving systems.

Issues related to transition of control from automated mode to manual mode, overreliance on automated systems and the resulting consequences of automated driving system errors are very difficult to test in a purely virtual environment.