10 January 2012

In-target test automation set to boost embedded software quality

Releasing a product with bugs is potentially very expensive, especially when the cost of field upgrades, recalls and repairs are considered. Less quantifiable – but at least as important – is the damage done to reputation and consequent loss of customer goodwill.

Even so, many embedded products are released without the necessary and/or desirable level of testing. Consequently, software quality is receiving increasing focus in embedded application development.

Most existing software testing tools can only execute the tests on a pc, but the actual system will have numerous differences, including hardware interfaces, timing issues and memory constraints. Therefore, it is important to run as many software tests as possible on the actual embedded hardware to minimise interplatform differences.

The challenge is to find automated in-target software testing tools that can be integrated into developers' embedded development environments. These tools should be able to analyse the application source code, autogenerate a test suite and run it automatically on the target board. However, a test report showing 100% success may not be sufficient as the test procedure itself may not have been applied to more than a small fraction of the code. Understanding the quality of test procedures thus becomes critical in judging whether a product is ready for release.

Unit tests
A key element of the software testing process is 'unit testing'. Unit tests call up C programming functions with different combinations of input parameter values to drive the code through different execution paths. Other than for trivial functions, writing unit tests by hand is laborious, costly and tedious, leaving the possibility that many possible execution paths remain untested. A detailed analysis of how input parameters drive the code is therefore required, so as many of the important execution paths throughout a function as possible are exercised.

Another challenge is maintaining the synchronisation of unit tests with code under development. This is often exacerbated when delivery schedules become compressed. If source code and unit tests become out of sync, the tests become less useful – at the time they are needed most.

Unit test tools available to pc developers are less useful to embedded developers as they rarely manage compilation, downloading and execution of the test suites on embedded boards. Tools that create unit tests automatically, build them into the embedded application and allow them to run on the target via a convenient pathway – such as a Jtag debugger – will result in optimal usage of time and maximum productivity.

An effective approach is to use an embedded test automation system that is integrated into the C/C++ development integrated development environment (IDE), so that synchronisation is easy to maintain and ancillary tools, such as Jtag debug probes, can be used to full effect.

This new breed of professional embedded test automation tools can analyse the source code and generate and execute suitable test suites automatically on the target board. An example is Atollic's TrueVERIFIER, which autogenerates a test suite implemented in C and makes many function calls with different combinations of input parameters, thus driving a large number of different execution paths.

Figure 1 shows how a trivial C function can be tested, by calling it many times with different input parameters which have been selected and generated automatically by the testing tool.



Once the test suite has been generated, it must be compiled, linked and executed on the target system. Many low cost unit test tools exist that only run unit tests on a Windows based pc. The inability of these tools to integrate with the embedded tool development environment results means test cannot be executed on the target board. However, TrueVERIFIER integrates into a professional embedded IDE, enabling synchronisation of test with code development, and takes advantage of integration with tools such as Jtag debug probes.

Measuring test quality
Once code has been developed and tested, the focus shifts to understanding what actually happened during testing. Code coverage measurement, performed using dynamic execution flow analysis, is commonly used to study which parts of the code have been tested. Some code coverage analysis types are:

• Statement or block coverage. This measures how many C statements or code blocks have been executed during a test session, but does not measure how branches affect which C statements or code blocks are executed.
• Function coverage. This measures which or how many of the C functions have been called during a test session, but not which, or how many, function calls in a code section are executed or the quality of the testing of the function itself.
• Function call coverage. Measures which or how many of the function calls in a code section have been called during a test session.
• Branch coverage. Measures whether all code blocks and alternate branch paths have been executed in a code section – such the if and the else part in an if-else statement. Branch coverage typically requires a code section to be executed a sufficient number of times, so that all alternative branch directions will be tested. As all branch paths must be executed, all corresponding code blocks are executed as well.
• Modified Condition/Decision Coverage (MC/DC). An advanced type of code coverage analysis which is applied to applications of which the highest reliability is expected. It extends branch coverage with the additional requirement that all sub expressions in complex decisions – such as in a complex if-statement – must drive the branch decision independently of the other sub-expressions.

The few test quality measurement tools that have been available have been limited by such problems as: testing using weak types of code coverage analysis only; testing in pc environments and not on the embedded target; expense; difficult to use; and lacking integration.

An example of an integrated tool that supports a range of code coverage analysis types is TrueANALYZER. Connecting to the target via a Jtag probe, the embedded application runs directly on the target board with the code coverage information recorded for later inspection and quantitative analysis.

Figure 2 shows a trivial code section that contains three blocks: a red code block, which is always executed; a green block, sometimes executed depending on the branch decision made in the if-statement; and a blue block, again, always executed.



The branch decision in the if-statement drives the selection of one of the two possible execution paths: either directly from the red to the blue code block; or passing from red to blue via the green code block.

Figure 3 shows how TrueANALYZER tool can detect the different types of code coverage, including statement/block coverage, function coverage, function call coverage, branch coverage and MC/DC.



TrueVERIFIER and TrueANALYZER are integrated in Atollic TrueSTUDIO C/C++ IDE. These embedded test tools will help software developers to be confident of delivering high quality software and make a significant contribution to new products that work first time.

Magnus Unemyr, vp sales and marketing, and Mark Moran, US general manager, are with Atollic.

Author
Magnus Unemyr and Mark Moran

Supporting Information

Downloads
39397\P25-26.pdf

Websites
http://www.atollic.com

This material is protected by Findlay Media copyright
See Terms and Conditions.
One-off usage is permitted but bulk copying is not.
For multiple copies contact the sales team.

Do you have any comments about this article?


Add your comments

Name
 
Email
 
Comments
 

Your comments/feedback may be edited prior to publishing. Not all entries will be published.
Please view our Terms and Conditions before leaving a comment.

Related Articles

Green light for FinFET flow

EDA software vendor Synopsys has announced that its Galaxy design flow has been ...

Four IDEs for Kinetis support

Freescale has named four tools as featured integrated development environments ...

ARM Compilers based on LLVM

ARM's future compilers will be built on the LLVM open source framework, the ...

Software tools: Cost vs. value

Depending upon who you talk to and the scale of the design, software ...

ECAD and MCAD crossover

At an industry event late in 2013, the speaker asked all mechanical engineers ...

Safe start to software test

Embedded software testing is a discipline that is both easier and more ...

KeyStone software

This whitepaper explores how Texas Instruments' KeyStone multicore SoCs offload ...

Using Linux in medical devices

This whitepaper explores the issues that software developers and medical device ...

High speed board design

Istvan Nagy, electronics design engineer at Blue Chip Technology, a leading UK ...

Embedded software solution

Digia announced a major upgrade to its Qt Enterprise Embedded solution at ...

C-RUN analysis tool

IAR Systems used Embedded World to launch C-RUN - its fully integrated runtime ...

OpenCL development kit

Altera has announced a new software development kit for OpenCL, a move which ...

Device Developers' Conference

20th May 2014, Holiday Inn, Bristol

Device Developers' Conference

22nd May 2014, Menzies Hotel, Cambridge

Device Developers' Conference

3rd June 2014, Cheadle House, Manchester

LightCrafter software upgrade

This video explains the steps needed to perform an upgrade of the LightCrafter ...

Altium design secret 19

Output Job Files enable repeatable, pre-configured generation of fabrication, ...

WEBENCH Filter Designer

Bonnie demonstrates how to design, optimise and simulate active filter ...

The Heartbleed glitch

Last week we were awakened to the news of a new software 'glitch' called ...

Embedded engineering needs

Software is a fundamental part of developing today's electronic devices. It's ...

The eco cloudy system thing...

One of the bothersome aspects of coming to grips with new, popular shifts in ...

Andreas Pabinger, Wind River

Three years ago, Wind River Systems – developer of the VxWorks real time ...

Cyrille Comar, AdaCore Europe

Cyrille Comar, co founder and managing director of AdaCore Europe, speaks to ...

Martin Harris, Altium

Chris Shaw asks Martin Harris about the latest developments at Altium