Fending Off Cyber Attacks – Hardening ECUs by Fuzz Testing

In designing vehicle communication networks, security test procedures play an important role in the development
process. Fuzz testing, which originated in the traditional IT field, offers automatable tests which are used to efficiently
check an ECU’s security-related capabilities. But how can test cases be generated using familiar exchange formats?

Today’s motor vehicles are increasingly being integrated effects of a successful attack. Although development pro-
into the Internet of Things (IoT). Vehicles are connecting to cesses for vehicle security have become better and are
devices, other vehicles, infrastructure and cloud services via using increasingly more detailed threat models, there is still
new interfaces. On the one hand, this enables new func- a gap in the development tools regarding the identification
tions and business models. However, this added connectiv- and prevention of security vulnerabilities in existing soft-
ity also opens more potential areas of attack related to ware.
cyber security threats. The rapid introduction of new and In ECU tests (Figure 1), the focus is on validating functional
complex functions harbors the risk that development requirements and error states within a defined scope –
teams will hardly be able to keep up with the new attack which is limited by the boundaries of the selected parame-
vectors that result and safeguard against them. ters. As soon as all functional requirements have been
General guidelines for developing secure vehicle systems tested sufficiently, it is also necessary to work in the securi-
are described in the Cybersecurity Guidebook for ty area with system inputs which are not part of the
Cyber-Physical Vehicle Systems (J3061). It contains strate- “expected” parameters. Vulnerabilities lurk in this comple-
gies and techniques for preventing and detecting security mentary set of inputs, and they are just waiting to be
breaches, and it lists measures to mitigate the undesirable exploited. Due to time constraints, it is not possible to test

01
 Technical Article / August 2018

all possible input values, especially when they depend on Compared with other security test methods such as pene-
different states and conditions. Therefore, the goal of fuzz tration tests, this approach offers a good cost-benefit ratio.
testing is not to test all the input parameters of a system Fuzzing engines can be categorized according to three
completely, but instead to programmatically generate criteria:
security tests that are more likely to identify implementa- >>Generative engines and mutation engines: Generative
tion errors. engines generate test data programmatically, while
mutation engines generate test data by mutating
What is Fuzz Testing and What Benefits Does It Offer? known valid data.
The purpose of fuzz testing is to detect unknown vulnera- >>“Simple” and “intelligent” fuzzing engines: In contrast to
bilities by stressing a system with queries which are in part “simple” engines, intelligent engines are familiar with the
random and quite inappropriate. This concept is not only structure of the data. However, one engine is not neces-
applied in security tests, but in robustness tests in general. sarily better than the other. An entirely “simple” engine,
Fuzz tests go far beyond the typical tests developers per- for example, can send fully random data to a system
form to study all reasonably predictable manipulations. without requiring any knowledge of the system states or
They involve generating additional test cases which ideally system parameters, like a data checksum in a message.
come up with fully unexpected data. Randomly generated An entirely “intelligent” engine would follow the protocol
input data that is applied over many test runs attains a and system specifications 100%, so it could potentially
level of test coverage that is hardly attainable by manually never detect errors. A good middle path is to use a
created tests. system that intelligently chooses which system parame-
Since fuzz testing is the technique preferred by hackers for ters and logical approaches to subject to fuzz testing.
revealing vulnerabilities, developers need an easy-to-use >>The third classification criterion is knowledge about the
extension for test tools which they can use to meet the system states and program structure. Here, the fuzz
challenges of attackers head-on. Ideally, fuzz testing will tests check either a black box, white box or gray box
support a method of test case generation that is largely system. For some system properties, a certain level of
automated and can indicate problems “relatively” quickly. knowledge of the system state is essential. This applies

Asset Definition

Security Validation
Threat and Risk
Assessment
Penetration Testing
Derivation of
Security Goals
Fuzz Testing
Security Architecture
Design & Analysis
Functional Security
Testing
Security Mechanisms
Design & Analysis

Secure Implementation of Nominal

Functions and Security Mechanisms

Figure 1: Fuzz Testing as part of the V-model for developing security-related ECUs

02
 Technical Article / August 2018

to testing of diagnostic messages, for example. In UDS >>A successful fuzz test solution must keep records of the
(Unified Diagnostic Services), many functions and sub- test cases that are executed along with all relevant
functions are protected by the 0x27 service, and possibly parameters. This lets testing engineers replicate the
even by the knowledge of which session the diagnostic tests to compare results based on clearly defined test
server is in. Without knowledge of this state machine, a cases and parameters.
fuzzing engine would be unable to penetrate deeply into >>The test tool should deliver reports that lend themselves
the ECU’s code base. to implementing corrective actions as well as relevant
summaries that present additional insights to test engi-
Many fuzz test engines are designed for black box tests. neers.
The Vector approach, on the other hand, aims to use proj-
ect configuration data supplied by the tool chain so that it Currently, there are some promising smaller vehicle-specific
can offer white box tests with knowledge of the network prototypes and IT-specific fuzzing solutions available.
protocol and the specified device behavior. Furthermore, However, there is no standard automotive fuzzer for vehi-
when software changes are made, developers can scale the cle tests which fulfills all the mentioned requirements. The
generated test cases to adapt them to the scope of the signals in vehicle networks pose a real challenge in the
changes. This significantly reduces the time required for development of fuzz tests. In generating inputs that can be
tests in an incremental, continuous integration process. processed by ECUs, fuzzing tools which are popular in the
IT sector cannot always fulfill requirements such as float-
Requirements for Successful Fuzz Testing Solutions ing-point numbers, signed and unsigned N-bit integers,
Based on own experiences and discussions with developers signal multiplexing and changing byte orders in the signal.
of vehicle tests and security service providers, Vector has
defined key requirements for successful fuzz testing solu- Concept for Automatically Generating and Executing
tions in the automotive industry. Fuzz Tests
>>Automotive OEMs and suppliers generally have full How can the described requirements be fulfilled? The core
access to databases in which the communication of the idea is to use vehicle communication databases to auto-
entire vehicle network or of a subsystem is described. matically generate and execute the fuzz tests (Figure 2).
Defined in these databases are regular messages, The overall concept consists of a system under test (SUT),
expected value ranges, lengths, timing information and a communication database, a vehicle test framework and
data types. These descriptions exist in standard data an engine for creating fuzz tests. The SUT might be a
exchange formats such as DBC, LDF, FIBEX and AUTO- remaining bus simulation, an individual ECU, a complete
SAR System Extract. The descriptions are an excellent subsystem of multiple ECUs or a complete vehicle system.
source of information for configuring fuzz test genera-
tors, which automatically generate test cases.

Automotive Testing Framework

Monitoring
Network and
Data
Monitoring Communication Communication
Visualization Database
System
under Test Logging Signal Selection

Fuzz Testing
Reporting Config. Generation
Configuration

Sending / Receiving

Fuzz Testing Engine

Fuzzed Messages Generate Fuzzed

Signal Data
Communication Channel
Figure 2: Concept for automatically generated fuzz tests

03
 Technical Article / August 2018

CANoe vTESTstudio
Monitoring
Network and
Data
Monitoring Communication Automotive Fuzz Test Configuration
Visualization Communication
System
Database
under Test Logging Signal Selection
(SUT) Fuzz Test Generator
Reporting Sending/Receiving

Fuzz Test Unit Fuzz Tests

Fuzzed Signals

Communication Channel

Figure 3: Fuzz testing with the standard CANoe tool

A combination of simulated and real elements is also possi- prepares as a message that it sends to the SUT over the
ble. The SUT is connected to the vehicle test framework via vehicle bus. Fuzz test execution is logged in such a way that
one or more communication channels. all the test messages can be examined. Because the fuzz
Vector uses the communication database as the central tests are integrated in CANoe, that tool’s internal logging
supplier of input data for the vehicle test framework. function can be used to record messages received by the
Among other things, the test framework can visualize the SUT and those sent to it. All test cases exhibiting unexpected
message contents, definitions of the network nodes and behavior are highlighted in the test window (Figure 4). The
parameterization of the communication channels allocated created test cases are formulated in CANoe’s own CAPL
to them. The user can now select the signals to be subjected programming language, and this makes it possible to view
to the fuzz test and store them in a configuration. Subse- them and edit them if necessary.
quently, the test framework of the fuzz test engine supplies
the interfaces (APIs) for bus access based on its knowledge Challenges
of the protocol, data types and conversion rules that are It must always be remembered that fuzz testing is not a
used. comprehensive security solution; rather it is a brick that is
The fuzz test engine generates the test values for the SUT used as part of the solution. Although fuzzing engines are
based on the configuration created using the vehicle test continually becoming more refined, and there are new
framework. The strategies for generating the test vectors monitoring approaches, this technology as a whole still
depend on the specific capabilities of the fuzz test engine. faces some challenges. In fuzz tests, errors can go unno-
The generated test data is then passed to the vehicle test ticed if they do not cause a complete program crash. Code
framework, which packs the data for the relevant bus or should be developed so that assertions are implemented
communications channel and sends it to the SUT.

Implementation Using Standard Software

The described fuzz test framework was created by using
standard software and extending it (Figure 3). Serving as
the test framework is the testing and development soft-
ware CANoe, which is used by many ECU testers and offers
user-friendly test flow control. Moreover, it fulfills require-
ments for processing vehicle communication protocols and
communication database formats with its libraries, APIs
and interfaces. Based on the imported database, CANoe
visualizes the network and suggests a list of signals the
user can select from. It then generates the configuration of
the fuzz test engine according to the user’s selections. This
is based on the vTESTstudio test design tool.
Figure 4: CANoe logs clearly which executed test cases were
The engine generates the signals – based on the configura- passed and which failed.
tion and its own fuzzing algorithms – which CANoe then

04
 Technical Article / August 2018

correctly in all software components, and they support

investigation into the causes in case of a failure. The analy-
sis of test cases leading to a crash is may be difficult.
Depending on the available monitoring, and in software
with complex inputs it takes much more effort to create
fuzz tests that provide sufficient code coverage.

Advantages of Fuzz Testing

Fuzz testing offers far-reaching automation that can
reveal critical system defects which would remain hidden in
manual security audits and software audits. Fuzz testing
also provides an overall picture of software robustness.
This helps to identify security vulnerabilities and to correct
them before they can be exploited for attacks in the field.
When fuzz testing is used early in an ECU’s development
cycle, security gaps can be revealed early on when it is less
costly or time-consuming to correct them. In combination
with existing tools and test frameworks from the automo-
tive industry, fuzz testing can make an effective contribu-
tion toward validating vehicle software. The option of pro-
grammatically generating fuzz tests from existing data-
bases makes it possible to test signals as well as
state-dependent protocols.

Holger Heinemann (Dipl.-Ing. (FH))

is manager in the R&D department at Vector Informatik GmbH.
In this role, he is responsible for Cyber Security Test Tools.

Translation of a German publication in Hanser automotive,

special edition “Safety & Security” August 2018

Image rights: Vector Informatik GmbH

05