Test-Driven Hardware Development: True or False?

Few generalizations about hardware design are more widely accepted than this: it is better to find errors early. And yet the traditional design-then-verify flow gives errors ample time to embed themselves in the design before even starting to look for them. It need not be so.

A technique borrowed from the software world—Test-Driven Development (TDD)—seeks to eliminate the problem by, in effect, reversing a portion of the design flow (Figure 1). In TDD, you develop the test harness for each unit of code before you write the code. It may sound like a rather pointless change in scheduling. But increasing numbers of design teams are finding that TDD makes a big difference: shortening design time, improving quality, and preparing design teams for emerging issues such as security and functional safety.

Figure 1. TDD shortens design cycles and improves quality by getting things a little reversed.

TDD Defined

So what is TDD? It is an iterative design methodology based on the concepts of unit testing. In unit test, rather than attempting to test an entire functional block of code at once, you partition the block into units and test each unit in isolation before integrating it into the block. So what is a unit? The partitioning process is more art than algorithm. You want a unit to have a small number of inputs and outputs, a clear function that is independent of the inputs—and especially independent of the state of other units—and minimal internal state. In other words, you define your units so they are easy to test, without letting the number of units get out of hand.

The innovation of TDD is that you define the units from the requirements, before the code exists (Figure 2). Then, before you write the code for a unit, you write the test harness for it—again, working directly from the requirements and the software architecture. You check the test by applying it to the existing code for the overall block, if there already is any. The block should of course fail the new test, since you haven’t written the new code yet.

.Figure 2. The TDD iteration starts with the requirements.

Now—finally, I hear you say—you write the code for the unit, apply the test harness, and iterate until the unit passes with no errors. Because you defined the unit for testability, run time should be short and—critically—the time to diagnose an error should be far shorter than if you had discovered the error at the block or system level. There’s just not that much going on inside a properly defined unit. After the code unit passes its test, you add it to the block and start the process over, writing the test harness for the next unit of requirements.

But why write the tests before there is any code to test? Is that just an affectation, one of those weird methodological quirks that software people are subject to from time to time? Experience says it is not. Writing the test harness first overcomes one of the major issues in testing: the tendency of developers to make the same errors in writing the test that they made in writing (or reviewing) the code. “TDD forces you to think like a user, not like a coder,” says Neil Johnson, chief technologist at XtremeEDA and a long-time advocate of TDD for hardware developers. You end up testing against the original requirements instead of testing against your understanding of your own code structure.

Additionally, if less obviously, TDD can induce coders to focus on the structure of the requirements and on the overall software architecture of the block before they dive into coding. This in itself has benefits.

But there is a larger issue behind the test-first approach, observes functional-verification expert Brian Bailey. TDD creates a second, independent interpretation of the design requirements: one that can be compared against the implementation code by simply running the unit tests. This two-track process can catch not only coding errors, but misinterpretations of the requirements and even ambiguities in the requirements documents—problems that can prove devilishly hard to diagnose later in the design flow.

But Does It Do Hardware?

Fine, I hear you say, for software folks. But we are doing hardware design. What does TDD have for us?

Caution is reasonable, in a world where design automation has often over-promised. Bailey says, “If you just move software concepts to hardware, you can expect problems. Software is essentially done when you’ve debugged it. A hardware design still has to go through synthesis, place and route, test insertion, layout … you have many chances to break it still.”

But within the confines of proving the functional correctness of the design, TDD is directly applicable. “Some software techniques do get mangled when you use them for hardware,” Johnson agrees. “But TDD works exactly the same whether you are writing a program in C or describing behavior in System Verilog, or even describing a structure in Verilog.”

At the functional level, TDD concepts are not new to hardware design. “Assertions are in a way a form of TDD,” Bailey points out. If you are following best practices, you are writing your assertions from the requirements, just as you would in a TDD flow. But one of the challenges in exploiting TDD in hardware design is that there can be several very different levels of abstraction hiding under the label of functional design.

Many teams begin by interpreting the requirements as a behavioral model in C, or even in a more modern language like Python or Java. In a TDD flow, this model would require test harnesses in its own language. Other teams may go directly from requirements into System Verilog, in which case they would probably develop the unit tests within the Universal Verification Methodology (UVM) framework. “But there are many designers who have never written behavioral code,” Johnson says. “They go directly from the requirements documents to RTL. So we are seeing some designers using TDD with Verilog.”

Generating behavioral tests for a hardware-description language does require some care, Johnson says. After all, there will be far more detail in the Verilog code than would be necessary to simply describe the functional behavior of the unit. Consciously raising the level of abstraction in the unit tests you are writing, for instance by thinking of the unit as a set of functions rather than a lump of in-line code, can help. “You can create your own test API,” Johnson says.

Even if the design starts in C, it will pass through register transfer level (RTL) eventually. It would be very valuable if the unit tests could be applied at every level of abstraction, rather than originating in one language and then getting manually translated into something else once or twice. That is the goal of Accellera’s Portable Stimulus (PS) Specification working group: to define a language for specifying behavior once, and using that specification to stimulate the behavioral models and even the implementation at a variety of levels, clear through physical prototyping (Figure 3). PS is not dependent on TDD, but it will certainly help TDD.

Figure 3. Portable Stimulus would allow a single capture of behavioral intent to stimulate many different levels of abstraction.

 

The Skeptic’s Corner

With all those promises, TDD can generate a lot of questions, especially in the hardware world, where design and verification are traditionally separate professions, applied in a strictly serial way. For instance, is TDD just an attempt by verification engineers to muscle into design territory? No, says Bailey. “TDD and PS both aim at creating two parallel, equally important views of the requirements: one defining the behavior and the other defining the implementation.” The test design and logic design efforts are unlikely to make the same errors simply because they are doing very different things. In this way TDD can be almost as effective as two independent design teams working in parallel: an approach that is known to have advantages, but that is too expensive for most projects. Here, there is no redundant work, since both the design and its tests have to be created anyway, but there is much of the improved quality that comes from redundancy.

But even when there is no separate verification team, TDD’s test-first approach tends to focus designers on the requirements before they are drawn into the complexities of implementation. The surest way to find out whether you really understand a requirement, Johnson says, is to try to write a test for it.

But doesn’t all this partitioning—critics would say fragmenting— of the design, test-writing, version control, tracking, and building become an administrative nightmare? Yes, it can be, without automation. Consequently, a number of platforms have appeared to automate the TDD process. Even so, isn’t all of this stuff going to destroy the project schedule? No, and maybe. Both software teams with long TDD experience and hardware teams using TDD claim that it generally shortens the overall project, sometimes dramatically. This is due to the larger portion of errors getting caught in unit tests, where they are easier to isolate and diagnose. If just a few such errors get through to system integration, they can be ruinous to the system verification schedule.

There is one undeniable problem, though. The test-first approach puts a number of non-coding activities before the point at which you start cranking out code, and by interleaving test development and coding it slows down the rate at which you are producing test-ready code. If you have an old-school manager who is gauging progress by lines of code per week or by degree of functional coverage, or some such metric, your manager may seriously misunderstand the rate of progress on the project. This can cause, shall we say, unnecessary friction in the design team.

A more serious objection is that unlike many software projects, most hardware designs include large amounts of reused intellectual property (IP)—either licensed from third parties, picked up as open-source, or re-used from previous projects. Since this IP is supposedly already verified, this would appear to limit the use of TDD to the functional design of the relatively small portion of the system that will be newly-written.

But there is an important role for TDD in IP reuse as well, Baily and Johnson concur. One of the most frequent sources of failure in IP reuse is misappropriation: when you use the IP block in a way for which it was never tested, or maybe never intended. By starting with your system requirements, you can generate tests that will verify that the IP block actually does what you assume it does. This may be possible even if you don’t have access to a behavioral model of the IP in order to create unit tests. You can write tests based on units of your requirements, and then black-box test an entire encrypted model of the IP rather than individual units of model code.

There are some interesting extensions to this idea. Bailey points out that one crucial but often-ignored step in functional verification is to prove that the IP blocks are not performing unintended functions. “The units of your requirements often don’t exactly match the structure of the IP you are using,” Bailey explains. “This can lead to functions you don’t use, or don’t even know about, still sitting in the IP. If they get inadvertently activated, they can cause havoc.” Using TDD unit tests in combination with activity monitoring, you can identify regions of the IP code that are dead in your use case, or units that appear to get activated at inappropriate times, and remove or disable them. This could be a crucial step toward getting functional-safety certification for a design.

Another idea carries this concept even further. Some of the IP in your system may be in the form of chips rather than design files. Often the chips come with marginally informative or simply incorrect data sheets, and without executable models. Using TDD to generate tests and a development kit of hardware test fixture to perform the tests, you can go a long way toward verifying your understanding of the chip’s operation, especially during initialization, sleep/wake, and other corner sequences.

One final question has no good answer today. What about timing? You set timing constraints before synthesis, very much in the way TDD creates test harnesses before coding. Isn’t there an analogy?

The quick answer is no. TDD works in the functional domain, which is inherently untimed. But still … The problem with making an analogy, Bailey says, is that while TDD derives functional tests from system requirements, there is no known way to derive timing constraints from system performance requirements. The latter are stated in terms of minimum data rates and function latencies. But CPU-based systems are so non-deterministic that there is little relationship between when a function will complete and the path delays inside its hardware blocks. Even with deterministic hardware the relationship can be complex. So timing remains beyond the pale.

Experience of a range of hardware design teams argues that TDD is indeed an important methodological approach. It requires a lot of personal reorientation, benefits greatly from use of a specialized platform, and may not feel familiar to most designers the first time through. But the ability to focus test on the system requirements, and to isolate errors early in the design flow, make it an idea whose time is very much now.


CATEGORIES : All/ AUTHOR : Ron Wilson

13 comments to “Test-Driven Hardware Development: True or False?”

You can leave a reply or Trackback this post.
  1. You may look at Logic – CMake, SystemVerilog and SystemC utilities for creating, building and testing RTL projects for FPGA: https://github.com/tymonx/logic

    Currently it supports unit testing (TDD) Verilog/SystemVerilog modules in three ways: SVUnit (SystemVerilog), GoogleTest + SystemC and UVM-SystemC. But it is matter of time to support more (VUnit? no problem).

  2. on fig.1 can be seen an error, namely the gearwheels will not spin at all. some of them at least. and not only because of their square teeth, say that is for simplicity, but because of the number of teeth.
    and the point: if the teeth represent a unit each, then [the] blocage is an INTER unit issue.
    and can’t be tested or even noticed using test units.
    i shall stop here.
    regards, s kasabov

    • you’re right… unit testing only is a flawed approach to verifying an fpga. no doubt. carrying on with the analogy, the limits of unit testing would be to verify the characteristics of the individual gears, the shape of the teeth, diameter, number of teeth, colour, etc. complementary to the unit tests would be some form of integration testing that pairs well with the size of the sub system you’re dealing with.

  3. Alexei Fetissov says: -#1

    “A technique borrowed from the software world” …? LOL

  4. Nikola Trajic says: -#1

    REPLY to A Fetissov
    Chip Verification is strictly a software discipline, and it is legitimate to borrow TDD methodology from the software world. Hardware development has its own specifics due to real-time signals i.e. time constraints, nevertheless some aspects of it can be captured by simulations and various software techniques, for e.g. Bertrand Meyer’s “Design by Contract”, encapsulation, and Assertions (which are inherent in System Verilog).
    Neil Johnson of Extreme EDA, and Bob Bailey bring the essence: “You/Engineer end up testing against the original requirements”. && “TDD creates a second, independent interpretation of the design requirements: one that can be compared against the implementation code by simply running the unit tests.”
    TDD enhances robustness of the new code in production, ensuring it behaves according to the requirements. TDD enhances IP reuse by determining whether the IP block actually does what you assume it does, through generated tests, based on units of your requirements. Discovery what IP actually does, can be done by black-box testing of encrypted model IP, and does not require knowledge of behavioural model. TDD works in the functional domain, which is inherently untimed.
    In hardware development the goal is to synthesize tangible implementations from behavioural models. Accellera’s Portable stimulus (PS) Specification helps in capturing the behaviour, in defining the accuracy of behavioural models under various stimuli, and in stimulating physical implementation at various levels. PS is independent of TDD, but it helps TDD by allowing unit tests to check further levels of abstraction.
    There is no known way in TDD to derive timing constraints from system performance requirements, although some functional snapshots how system behaves can be obtained through unit tests.

  5. Nikola Trajic says: -#1

    When statically compiled languages are used (for e.g. Verilog, System Verilog, VHDL, C), verification engineer plays the role of a software engineer, designing software against system requirements and implementing fully structured programs, which role is to test each chip subsystem. Applying TDD methodology makes these new programs more robust, and force programmer to more accurately satisfy the system requirements. Dynamic scripting languages go further (e.g. Python), by allowing verification engineer to check in the interpreter results of various executions: single line, or a single function call execution, or to immediately check the result of execution of several lines of code. This is another degree of freedom, which increases productivity of the verification engineer. The verification engineer still plays a role of the software engineer – by writing the fully structured Python un-synthesizable programs intended for compilation, and by applying TDD by using Python Unit test geared libraries. It is beneficial for the verification engineer i.e. software engineer, to understand object orientation, encapsulation, inheritance, design by contract, and all nitty-gritties of Python language, especially regular expressions. aka software assertions.
    FOR SCEPTICS:
    There are some cases when the scripting language is specifically designed for verification of hardware.
    Such a case is the E-language (Specman). E programs do not have to satisfy the specific program structure. Verification engineer is right-out writing test cases, which are based on aspects, and AOP (Aspect Oriented Programming principles) – and are directly addressing the design requirements. Additional features like: expediting parallel debugging, reverse debugging from a break point, etc are extremely increasing productivity of the E verification engineer. Releasing a verification engineer from knowing many skills of the software engineer, E-lang allows hardware design engineer to be productive in the verification task.

  6. These two quotes taken together — with some thought and design/debug experience — tell me that the TDD approach is good but not do-able given the current tools and design approach.

    ” TDD works in the functional domain, which is inherently untimed.”

    “But within the confines of proving the functional correctness of the design, TDD is directly applicable. “Some software techniques do get mangled when you use them for hardware,” Johnson agrees. “But TDD works exactly the same whether you are writing a program in C or describing behavior in System Verilog, or even describing a structure in Verilog.””

    1) There is no notion of anything in hardware being untimed.
    2) The only way to observe anything is to create a test bench, compile(which includes place and route) then analyze the wave form. (if you are able to remember what the function was that you were interested in to begin with.)

    No, the sky is not falling! By the way Python is not the only modern language. I use C# classes for hardware registers and memory, Arithmetic expressions to manipulate data, and Boolean Algebra to define function.

    I start by initializing internal values, apply inputs, evaluate control and arithmetic nets, and single step or set break points to run tests and see everything that happens. Probably taking less time than to create a Verilog test bench.

  7. Karl Stevens says: -#1

    Nikola Trajic says:
    ” Additional features like: expediting parallel debugging, reverse debugging from a break point, etc are extremely increasing productivity of the E verification engineer. Releasing a verification engineer from knowing many skills of the software engineer, E-lang allows hardware design engineer to be productive in the verification task.”

    Design task is to turn requirements into implementable function.
    Verification task is to ensure that implemented function satisfies requirements.

    E-lang allows design engineer to be productive in verification task verification task

    Design engineer has other things to do such as close timing and synthesize hardware.

    A way of describing function(requirements) that is useful for both design and verification is needed.

    It is important that design and verification be separate so that any misunderstanding of requirements in design are not propagated into verification.

    And the design engineer does not need to be bothered to learn and use a new scripting language. Some software engineer needs to develop a tool that is intuitively usable by the designer. Preferably it would also be usable for the verification engineer.

    Before Verilog hardware was dataflow that included arithmetic and logic operators, and Boolean nets that combined inputs with internal states to control the data flow. No software engineers were needed.

    Then Verilog cam along and because it could be used for simulation it was mistakenly assumed that it was also a design language. VHDL was also developed because some though there was a better way. So they based VHDL on ADA which was “a modern programming language” Ever since then tool vendors had to support both only because some users preferred Verilog, some preferred VHDL, and both require the design to be “compiled” and a test bench has to be designed to produce a meaningful wave form from simulation.

    Probably the successful products relied on proprietary tools.

  8. Karl Stevens says: -#1

    There are relatively few things that define hardware:
    inputs, outputs, flip-flops, registers, memory, Boolean networks, and data flow expressions.
    Then to run functional test the flip-flops and registers have to be initialized/reset.
    Events assign values at specified times or at a delay interval after a condition or event occurs either to inputs, flip-flops, or registers.
    Here is a first pass input format to describe flip-flops, registers, buses, and Boolean nets:(memories TBD).
    name : number // initialize at t = 0
    name : number1 @ number2 // value = number1 at t = number2
    name : number1 @ number2 : number3 // value = number1 at t = number2 repeats each
    //number3 interval
    name : number name ? exp name : number name = exp
    Boolean net : net1 ? net2 // bool net node value equal to bool net1 if bool net2 is true
    //else false
    Boolean net : exp ? net // bool net node value equal to expression if bool net2 is true
    // else false
    Boolean net : exp1 ? exp2 // bool net node value equals expression1 if bool
    // expression2 is true else false
    Boolean net ? exp // bool net node value true if bool expression evaluates to
    // true else false
    bus name = exp // bus value equals exp evaluation
    bus name1 = bus2 ? net // bus1 value equals bus2 value if net is true else 0
    bus name = exp ? net // bus equals exp value if net is true else 0
    +name ? net // set ff nam equal true if net is true
    +name ? exp // reset ff nam equal false if exp is true
    !name ? net // set ff nam equal true if net is true
    !name ? exp // reset ff nam equal false if exp is true
    This illustrates pretty much what happens and has always happened in hardware. The above text can be parsed to define the hardware blocks and the events/conditions that provide the function.

    WHAT NO NEW PROGRAMMING LANGUAGE?

  9. Lars Asplund says: -#1

    “Few generalizations about hardware design are more widely accepted than this: it is better to find errors early”

    Most people would also agree that a day of coding is likely to result in several bugs. With that in mind we should run our unit tests several times a day to find the defects early. That’s what experienced unit testers do but something that most people new to unit testing feel is “too much”. The key is to add support and remove practices that prevents such an approach

    1. You need a test framework providing test automation. It should be practical to run and develop (TDD style or not) the first test case on the first day of coding. It should be practical to run and develop new tests several times a day. Every day.
    2. Unit tests are created and run by the code developer. Leaving unit testing to an external verification team will break the short feedback loop.
    3. Unit tests must be written in a language that the developer knows well and that’s usually the language used for production code. This is why there is a unit testing framework for almost every HDL and software language you can think of (https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks) rather than a few frameworks using special verification languages.

    Unit testing should not be seen as a separate activity but rather an integral part of developing code. It’s a tool that developers use just like googling for solutions to a problem or sketching ideas on a whiteboard. It’s not something you make special plans for but rather something that’s part of the normal code developing routine. This is what you do to build confidence that your code does what you intended it to do.

    An important aspect of unit testing is the effect it has on your design. You’re forced into designing small units with a well-defined and testable function. In other words, you’re forced into a modular design. Many unit testers would say that this is the greatest benefit of unit testing.

  10. Alas. TDD was an ideal that has not been met in practice in the software world. In principle it failed because the software needed to be designed to the point that test cases could be expressed in terms of invariants, pre-conditions, post conditions etc.

    Of course, design to the level required was an anti-pattern in agile/open source world so there was a failure mode built into TDD, being it was attached to (so-called) agile mentalities.

    Mature and deep (enough) design was necessary to attain the write test once run test many times goal.

    Opinions might differ but the description of the hip phrase “Test Pain” actually describes all the issues around Unit Testing that TDD was to abolish. Including having to re-write tests, maintain tests, ensure tests kept up to design.

    In short TDD+Test_Pain=Unit_Testing.

  11. Lars Asplund says: -#1

    “TDD works in the functional domain, which is inherently untimed”

    ” Then, before you write the code for a unit, you write the test harness for it—again, working directly from the requirements and the software architecture”

    If you’re developing and unit testing RTL code you’re not likely implementing and testing many requirements per day but rather work with a number of smaller design decisions. Most often these are design decision below the level of detail documented in design documents. Some of the decisions are related to timing, for example throughput and latency, and there will be unit tests verifying this. Unit tests for software is a bit different since timing and performance is less deterministic. Such unit tests should avoid verifying timing.

    “TDD was an ideal that has not been met in practice in the software world. In principle it failed because the software needed to be designed to the point that test cases could be expressed in terms of invariants, pre-conditions, post conditions etc.”

    With the exception of safety-critical code I haven’t seen unit testing been driven by detailed up front design documents, it’s driven by the “live” design decisions you make when you’re about to implement a piece of code. I agree that such micromanagement would make testing painful but I haven’t seen it in practice. However, I see a lot of other rules and metrics that people feel that they must apply to do unit testing with or without TDD properly: You must always do unit testing the TDD way, a unit must not be more than x lines of code, units must be tested in isolation, test to production code line ratio must be at least x, you must have certain levels of code coverage and so on. It’s easy to get lost in following such rules and forgetting the basics.

    1. Code developers must test their own code because bugs becomes so much more expensive when handed over to others
    2. Code developers need frequent feedback. If you get feedback on your code much less frequent than you make mistakes you’ll spend a lot of time doing the wrong things and reverting mistakes once they are discovered

    Companies successful at this are those that adapt practices supporting these goals and remove the obstacles preventing them. What works for them may differ but in general they start testing earlier and more frequent by testing smaller things and they manage their test suites with automation.

    “… having to re-write tests, maintain tests, ensure tests kept up to design.”

    Unit testing is a first class activity when developing code. The test code is not a second class citizen, it’s a valuable piece of code that deserves attention just like the production code. We apply good design practices to our production code to ease the burden of maintenance, trying to make it modular for example. In the same way there are good practices for unit tests. For example, by testing behavior and not implementation we reduce the risk of test re-writes when refactoring production code.

  12. Karl Stevens says: -#1

    I give up: There is no way to run a functional test on hardware. There is only simulation which creates a waveform from a netlist.

    “Caution is reasonable, in a world where design automation has often over-promised. Bailey says, “If you just move software concepts to hardware, you can expect problems. Software is essentially done when you’ve debugged it. A hardware design still has to go through synthesis, place and route, test insertion, layout … you have many chances to break it still.””

    I previously wrote

    These two quotes taken together — with some thought and design/debug experience — tell me that the TDD approach is good but not do-able given the current tools and design approach.

    ” TDD works in the functional domain, which is inherently untimed.”

    “But within the confines of proving the functional correctness of the design, TDD is directly applicable. “Some software techniques do get mangled when you use them for hardware,” Johnson agrees. “But TDD works exactly the same whether you are writing a program in C or describing behavior in System Verilog, or even describing a structure in Verilog.”””

    Verilog has nothing to do with hardware function and SystemVerilog emphasizes verification rather than functional design.

    Sorry that I wasted my time reading gibberish. Goodbye All.

Write a Reply or Comment

Your email address will not be published.