Component failures caused by software discrepancies - part I

In March 2016 J.D. Power launched SafetyIQ, a new online application designed to help the automotive industry analyze vehicle safety data more efficiently. The application integrates National Highway Traffic Safety Administration (NHTSA) with J.D. Power automotive data. Upon the launch Vice-President of JD Power Renee Stephens said, “During the past 20 years, more than 428 million vehicles have been affected by safety recall decisions in the United States, with more than 51 million vehicles impacted in 2015 alone, more than in any previous year.”

According to the ‘Automotive Warranty and Recall Report 2016’ by financial advisors Stout Risius Ross, software-related recalls had grown from 5% in 2011 to 15% of all recalls in 2015. This is stark evidence that while the growth in E/E systems in our vehicles can improve safety, comfort and convenience, it has also resulted in an increase in related safety issues.

This becomes an ever more pressing matter as the auto industry continues to progress with sophisticated Advanced Driver Assistance Systems, and pushes further down the road towards autonomous vehicles. Google - one of the leaders in autonomous vehicle testing in the USA - released figures in January 2016 pertaining to the safety of its self-driving cars. Between September 2014 and November 2015 Google’s autonomous cars experienced failures on 272 occasions, 13 of which would have resulted in a crash had the occupant of the vehicle not intervened and taken control. To provide context, Google’s data shows that 49 autonomous cars drove 424,000 miles during the 14-month period and suffered 341 disengagements - where the car unexpectedly handed control back to the driver, or the driver took control of their own accord. 272 of these disengagements were found to be cases where the car had detected technology failures and alerted the driver to the need to take control.

Stout Risius Ross, in the ‘Automotive Warranty and Recall Report 2016’, point to the volume of software code in today’s vehicles, comparing up to 100 million lines of code in a car of today with the 9 million lines of code in an F-35 Fighter Jet. Research conducted by J.D. Power via its safety IQ application reveals that there have been 189 distinct software recalls issued over five years, affecting some 13 million vehicles. Furthermore, 141 of those presented a higher risk of crashing.

[inlinead]

This is an issue that affects manufacturers across the industry, not only in terms of the immediate costs and damage to reputation, but also in the quest to convince the public that autonomous vehicles are safer than human drivers. Here we’ll take a snapshot of some of the recalls and problems that have been faced over the last few years.

Toyota Unintended Acceleration Case

One of the most high-profile software-related faults of recent years was the unintended acceleration issue that resulted in Toyota agreeing a $1.2 billion settlement in 2014. The bill, which was the largest penalty ever imposed on a car manufacturer, brought to an end a four-year federal criminal investigation into whether Toyota properly reported safety complaints about sudden acceleration.

The issue was first highlighted in 2009 after a fatal crash involving a Toyota-built Lexus near San Diego. The crash was thought to have been caused by floor mats jamming the accelerator, and Toyota subsequently recalled millions of vehicles to replace floor mats and fix a mechanical issue that may also have caused the problem. However, the later court ruling led to Toyota admitting that it had deceived regulators and consumers about the two defects related to the sudden acceleration. Yet another offshoot of this story involved a Minnesota resident who had already spent two years in prison for vehicular manslaughter before evidence came to light that he may have been a victim of the acceleration problem. He was released and charges against him were dropped in 2010.

The potential cause of the problem was highlighted in research carried out by the Barr Group, in 2013. The Barr Group acted as a primary witness for the plaintiffs, and conducted in-depth research into the software design and development involving safety-critical systems.

Toyota’s electronic throttle control system (ETCS) source code is of unreasonable quality. Barr’s conclusions after the research were as follows:

• Toyota’s source code is defective and contains bugs, including bugs that can cause unintended acceleration (UA)
• Code-quality metrics predict presence of additional bugs
• Toyota’s fail safes are defective and inadequate (referring to them as a “house of cards” safety architecture)
• Misbehaviors of Toyota’s ETCS are a cause of UA

The ECM software was at the core of the technical investigation, and some of the key findings were listed as follows:

Mirroring (where key data is written to redundant variables) was not always done. The research highlighted the significance of this in light of stack overflow. Toyota had claimed that just 41% of the allocated stack space was in use, but Barr’s findings suggested this figure was closer to 94%. In addition, stack-killing, MISRA-C rule violating recursion was found in the code, and the CPU didn’t have incorporated memory protection to guard against stack overflow.

Two key items were not mirrored: the RTOS’ critical internal data structures, and most importantly, the TargetThrottleAngle global variable. Toyota had performed a stack analysis, but the Barr group concluded that it had made various mistakes. Toyota had, according to Barr, missed some of the calls made via pointer, missed stack usage by library and assembly functions (around 350 in total), and missed RTOS use during task switching. Toyota also failed to perform run-time stack monitoring.

Toyota’s ETCS used a version of OSEK, which is an automotive standard RTOS API. However, the CPU vendorsupplied version was not certified compliant. Unintentional RTOS shutdown was also investigated as a potential source of unintended acceleration. With single bits of memory controlling each task, corruption due to software or hardware failure will suspend needed tasks or start unwanted tasks. Vehicle testing confirmed that one particular dead task would result in loss of throttle control, and that a driver may even have to fully remove their foot from the brake pedal during unintended acceleration in order to end the unwanted acceleration.

A New Approach

The Toyota incident resulted in the largest fine for an automaker ever recorded, but it was far from an isolated problem. Almost every car manufacturer has been affected by software related recalls for one reason or another, and the increase in the number and frequency of these callbacks has driven new approaches to safety and to the way in which recalls are dealt with.

Read part II of this article here