GNAT GPL User's Guide: Creating Unit Tests with gnattest

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

# 27. Creating Unit Tests with `gnattest`

`gnattest` is an ASIS-based utility that creates unit-test skeletons as well as a test driver infrastructure (harness). `gnattest` creates a skeleton for each visible subprogram in the packages under consideration when they do not exist already.

In order to process source files from a project, `gnattest` has to semantically analyze the sources. Therefore, test skeletons can only be generated for legal Ada units. If a unit is dependent on other units, those units should be among the source files of the project or of other projects imported by this one.

Generated skeletons and harnesses are based on the AUnit testing framework. AUnit is an Ada adaptation of the xxxUnit testing frameworks, similar to JUnit for Java or CppUnit for C++. While it is advised that gnattest users read the AUnit manual, deep knowledge of AUnit is not necessary for using gnattest. For correct operation of `gnattest`, AUnit should be installed and aunit.gpr must be on the project path. This happens automatically when Aunit is installed at its default location.

> 27.1 Running `gnattest` |  |
> 27.2 Switches for `gnattest` |  |
> 27.3 Project Attributes for `gnattest` |  |
> 27.4 Simple Example |  |
> 27.5 Setting Up and Tearing Down the Testing Environment |  |
> 27.6 Regenerating Tests |  |
> 27.7 Default Test Behavior |  |
> 27.8 Testing Primitive Operations of Tagged Types |  |
> 27.9 Testing Inheritance |  |
> 27.10 Tagged Types Substitutability Testing |  |
> 27.11 Testing with Contracts |  |
> 27.12 Additional Tests |  |
> 27.13 Support for other platforms/run-times |  |
> 27.14 Current Limitations |  |

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.1 Running `gnattest`

`gnattest` has a command-line interface of the form

    $ gnattest -Pprojname [--harness-dir=dirname] [switches] [filename] [-cargs gcc_switches]

where

-Pprojname
  specifies the project defining the location of source files. When no file names are provided on the command line, all sources in the project are used as input. This switch is required.

filename
  is the name of the source file containing the library unit package declaration for which a test package will be created. The file name may be given with a path.

``gcc_switches'`
  is a list of switches for `gcc`. These switches will be passed on to all compiler invocations made by `gnattest` to generate a set of ASIS trees. Here you can provide ``-I'` switches to form the source search path, use the ``-gnatec'` switch to set the configuration file, use the ``-gnat05'` switch if sources should be compiled in Ada 2005 mode, etc.

switches
  is an optional sequence of switches as described in the next section.

`gnattest` results can be found in two different places.

- automatic harness: the harness code, which is located by default in "gnattest/harness" directory that is created in the object directory of corresponding project file. All of this code is generated completely automatically and can be destroyed and regenerated at will. It is not recommended to modify this code manually, since it could easily be overridden by mistake. The entry point in the harness code is the project file named `test_driver.gpr`. Tests can be compiled and run using a command such as:

      gnatmake -P<harness-dir>/test_driver
      test_runner

  Note that you might need to specify the necessary values of scenario variables when you are not using the AUnit defaults.

- actual unit test skeletons: a test skeleton for each visible subprogram is created in a separate file, if it doesn't exist already. By default, those separate test files are located in a "gnattest/tests" directory that is created in the object directory of corresponding project file. For example, if a source file my_unit.ads in directory src contains a visible subprogram Proc, then the corresponding unit test will be found in file src/tests/my_unit-test_data-tests-proc_<code>.adb. <code> is a signature encoding used to differentiate test names in case of overloading.

  Note that if the project already has both my_unit.ads and my_unit-test_data.ads, this will cause a name conflict with the generated test package.

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.2 Switches for `gnattest`

``--harness-only'`
  When this option is given, `gnattest` creates a harness for all sources, treating them as test packages.

``--additional-tests=projname'`
  Sources described in projname are considered potential additional manual tests to be added to the test suite.

``-r'`
  Recursively consider all sources from all projects.

``-Xname=value'`
  Indicate that external variable name has the value value.

``-q'`
  Suppresses noncritical output messages.

``-v'`
  Verbose mode: generates version information if specified by itself on the command line. If specified via GNATtest_Switches, produces output about the execution of the tool.

``--validate-type-extensions'`
  Enables substitution check: run all tests from all parents in order to check substitutability.

``--skeleton-default=val'`
  Specifies the default behavior of generated skeletons. val can be either "fail" or "pass", "fail" being the default.

``--passed-tests=val'`
  Specifies whether or not passed tests should be shown. val can be either "show" or "hide", "show" being the default.

``--tests-root=dirname'`
  The hierarchy of source directories, if any, is recreated in the dirname directory, with test packages placed in directories corresponding to those of the sources. If the dirname is a relative path, it is considered relative to the object directory of the project file. When all sources from all projects are taken recursively from all projects, directory hierarchies of tested sources are recreated for each project in their object directories and test packages are placed accordingly.

``--subdir=dirname'`
  Test packages are placed in a subdirectory of the corresponding source directory, with the name dirname. Thus, each set of unit tests is located in a subdirectory of the code under test. If the sources are in separate directories, each source directory has a test subdirectory named dirname.

``--tests-dir=dirname'`
  All test packages are placed in the dirname directory. If the dirname is a relative path, it is considered relative to the object directory of the project file. When all sources from all projects are taken recursively from all projects, dirname directories are created for each project in their object directories and test packages are placed accordingly.

``--harness-dir=dirname'`
  specifies the directory that will hold the harness packages and project file for the test driver. If the dirname is a relative path, it is considered relative to the object directory of the project file.

``--separates'`
  Bodies of all test routines are generated as separates. Note that this mode is kept for compatibility reasons only and it is not advised to use it due to possible problems with hash in names of test skeletons when using an inconsistent casing. Separate test skeletons can be incorporated to monolith test package with improved hash being used by using ``--transition'` switch.

``--transition'`
  This allows transition from separate test routines to monolith test packages. All matching test routines are overwritten with contents of corresponding separates. Note that if separate test routines had any manually added with clauses they will be moved to the test package body as is and have to be moved by hand.

``--tests_root'`, ``--subdir'` and ``--tests-dir'` switches are mutually exclusive.

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.3 Project Attributes for `gnattest`

Most of the command-line options can also be passed to the tool by adding special attributes to the project file. Those attributes should be put in package gnattest. Here is the list of attributes:

- Tests_Root is used to select the same output mode as with the --tests-root option. This attribute cannot be used together with Subdir or Tests_Dir.

- Subdir is used to select the same output mode as with the --subdir option. This attribute cannot be used together with Tests_Root or Tests_Dir.

- Tests_Dir is used to select the same output mode as with the --tests-dir option. This attribute cannot be used together with Subdir or Tests_Root.

- Harness_Dir is used to specify the directory in which to place harness packages and project file for the test driver, otherwise specified by --harness-dir.

- Additional_Tests is used to specify the project file, otherwise given by --additional-tests switch.

- Skeletons_Default is used to specify the default behaviour of test skeletons, otherwise specified by --skeleton-default option. The value of this attribute should be either "pass" or "fail".

Each of those attributes can be overridden from the command line if needed. Other `gnattest` switches can also be passed via the project file as an attribute list called GNATtest_Switches.

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.4 Simple Example

Let's take a very simple example using the first `gnattest` example located in:

    <install_prefix>/share/examples/gnattest/simple

This project contains a simple package containing one subprogram. By running gnattest:

    $ gnattest --harness-dir=driver -Psimple.gpr

a test driver is created in directory "driver". It can be compiled and run:

    $ cd obj/driver
    $ gnatmake -Ptest_driver
    $ test_runner

One failed test with diagnosis "test not implemented" is reported. Since no special output option was specified, the test package Simple.Tests is located in:

    <install_prefix>/share/examples/gnattest/simple/obj/gnattest/tests

For each package containing visible subprograms, a child test package is generated. It contains one test routine per tested subprogram. Each declaration of a test subprogram has a comment specifying which tested subprogram it corresponds to. Bodies of test routines are placed in test package bodies and are surrounded by special comment sections. Those comment sections should not be removed or modified in order for gnattest to be able to regenerate test packages and keep already written tests in place. The test routine Test_Inc_5eaee3 located at simple-test_data-tests.adb contains a single statement: a call to procedure Assert. It has two arguments: the Boolean expression we want to check and the diagnosis message to display if the condition is false.

That is where actual testing code should be written after a proper setup. An actual check can be performed by replacing the Assert call with:

    Assert (Inc (1) = 2, "wrong incrementation");

After recompiling and running the test driver, one successfully passed test is reported.

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.5 Setting Up and Tearing Down the Testing Environment

Besides test routines themselves, each test package has a parent package Test_Data that has two procedures: Set_Up and Tear_Down. This package is never overwritten by the tool. Set_Up is called before each test routine of the package and Tear_Down is called after each test routine. Those two procedures can be used to perform necessary initialization and finalization, memory allocation, etc. Test type declared in Test_Data package is parent type for the test type of test package and can have user-defined components whose values can be set by Set_Up routine and used in test routines afterwards.

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.6 Regenerating Tests

Bodies of test routines and test_data packages are never overridden after they have been created once. As long as the name of the subprogram, full expanded Ada names, and the order of its parameters is the same, and comment sections are intact the old test routine will fit in its place and no test skeleton will be generated for the subprogram.

This can be demonstrated with the previous example. By uncommenting declaration and body of function Dec in simple.ads and simple.adb, running `gnattest` on the project, and then running the test driver:

    gnattest --harness-dir=driver -Psimple.gpr
    cd obj/driver
    gnatmake -Ptest_driver
    test_runner

the old test is not replaced with a stub, nor is it lost, but a new test skeleton is created for function Dec.

The only way of regenerating tests skeletons is to remove the previously created tests together with corresponding comment sections.

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.7 Default Test Behavior

The generated test driver can treat unimplemented tests in two ways: either count them all as failed (this is useful to see which tests are still left to implement) or as passed (to sort out unimplemented ones from those actually failing).

The test driver accepts a switch to specify this behavior: --skeleton-default=val, where val is either "pass" or "fail" (exactly as for `gnattest`).

The default behavior of the test driver is set with the same switch as passed to gnattest when generating the test driver.

Passing it to the driver generated on the first example:

    test_runner --skeleton-default=pass

makes both tests pass, even the unimplemented one.

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.8 Testing Primitive Operations of Tagged Types

Creation of test skeletons for primitive operations of tagged types entails a number of features. Test routines for all primitives of a given tagged type are placed in a separate child package named according to the tagged type. For example, if you have tagged type T in package P, all tests for primitives of T will be in P.T_Test_Data.T_Tests.

Consider running gnattest on the second example (note: actual tests for this example already exist, so there's no need to worry if the tool reports that no new stubs were generated):

    cd <install_prefix>/share/examples/gnattest/tagged_rec
    gnattest --harness-dir=driver -Ptagged_rec.gpr

Taking a closer look at the test type declared in the test package Speed1.Controller_Test_Data is necessary. It is declared in:

    <install_prefix>/share/examples/gnattest/tagged_rec/obj/gnattest/tests

Test types are direct or indirect descendants of AUnit.Test_Fixtures.Test_Fixture type. In the case of nonprimitive tested subprograms, the user doesn't need to be concerned with them. However, when generating test packages for primitive operations, there are some things the user needs to know.

Type Test_Controller has components that allow assignment of various derivations of type Controller. And if you look at the specification of package Speed2.Auto_Controller, you will see that Test_Auto_Controller actually derives from Test_Controller rather than AUnit type Test_Fixture. Thus, test types mirror the hierarchy of tested types.

The Set_Up procedure of Test_Data package corresponding to a test package of primitive operations of type T assigns to Fixture a reference to an object of that exact type T. Notice, however, that if the tagged type has discriminants, the Set_Up only has a commented template for setting up the fixture, since filling the discriminant with actual value is up to the user.

The knowledge of the structure of test types allows additional testing without additional effort. Those possibilities are described below.

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.9 Testing Inheritance

Since the test type hierarchy mimics the hierarchy of tested types, the inheritance of tests takes place. An example of such inheritance can be seen by running the test driver generated for the second example. As previously mentioned, actual tests are already written for this example.

    cd obj/driver
    gnatmake -Ptest_driver
    test_runner

There are 6 passed tests while there are only 5 testable subprograms. The test routine for function Speed has been inherited and run against objects of the derived type.

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.10 Tagged Types Substitutability Testing

Tagged Types Substitutability Testing is a way of verifying the global type consistency by testing. Global type consistency is a principle stating that if S is a subtype of T (in Ada, S is a derived type of tagged type T), then objects of type T may be replaced with objects of type S (that is, objects of type S may be substituted for objects of type T), without altering any of the desirable properties of the program. When the properties of the program are expressed in the form of subprogram preconditions and postconditions (let's call them pre and post), the principle is formulated as relations between the pre and post of primitive operations and the pre and post of their derived operations. The pre of a derived operation should not be stronger than the original pre, and the post of the derived operation should not be weaker than the original post. Those relations ensure that verifying if a dispatching call is safe can be done just by using the pre and post of the root operation.

Verifying global type consistency by testing consists of running all the unit tests associated with the primitives of a given tagged type with objects of its derived types.

In the example used in the previous section, there was clearly a violation of type consistency. The overriding primitive Adjust_Speed in package Speed2 removes the functionality of the overridden primitive and thus doesn't respect the consistency principle. Gnattest has a special option to run overridden parent tests against objects of the type which have overriding primitives:

    gnattest --harness-dir=driver --validate-type-extensions -Ptagged_rec.gpr
    cd obj/driver
    gnatmake -Ptest_driver
    test_runner

While all the tests pass by themselves, the parent test for Adjust_Speed fails against objects of the derived type.

Non-overridden tests are already inherited for derived test types, so the --validate-type-extensions enables the application of overriden tests to objects of derived types.

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.11 Testing with Contracts

`gnattest` supports pragmas Precondition, Postcondition, and Test_Case, as well as corresponding aspects. Test routines are generated, one per each Test_Case associated with a tested subprogram. Those test routines have special wrappers for tested functions that have composition of pre- and postcondition of the subprogram with "requires" and "ensures" of the Test_Case (depending on the mode, pre and post either count for Nominal mode or do not count for Robustness mode).

The third example demonstrates how this works:

    cd <install_prefix>/share/examples/gnattest/contracts
    gnattest --harness-dir=driver -Pcontracts.gpr

Putting actual checks within the range of the contract does not cause any error reports. For example, for the test routine which corresponds to test case 1:

    Assert (Sqrt (9.0) = 3.0, "wrong sqrt");

and for the test routine corresponding to test case 2:

    Assert (Sqrt (-5.0) = -1.0, "wrong error indication");

are acceptable:

    cd obj/driver
    gnatmake -Ptest_driver
    test_runner

However, by changing 9.0 to 25.0 and 3.0 to 5.0, for example, you can get a precondition violation for test case one. Also, by using any otherwise correct but positive pair of numbers in the second test routine, you can also get a precondition violation. Postconditions are checked and reported the same way.

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.12 Additional Tests

`gnattest` can add user-written tests to the main suite of the test driver. `gnattest` traverses the given packages and searches for test routines. All procedures with a single in out parameter of a type which is derived from AUnit.Test_Fixtures.Test_Fixture and that are declared in package specifications are added to the suites and are then executed by the test driver. (Set_Up and Tear_Down are filtered out.)

An example illustrates two ways of creating test harnesses for user-written tests. Directory additional_tests contains an AUnit-based test driver written by hand.

    <install_prefix>/share/examples/gnattest/additional_tests/

To create a test driver for already-written tests, use the --harness-only option:

    gnattest -Padditional/harness/harness.gpr --harness-dir=harness_only \
      --harness-only
    gnatmake -Pharness_only/test_driver.gpr
    harness_only/test_runner

Additional tests can also be executed together with generated tests:

    gnattest -Psimple.gpr --additional-tests=additional/harness/harness.gpr \
      --harness-dir=mixing
    gnatmake -Pmixing/test_driver.gpr
    mixing/test_runner

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.13 Support for other platforms/run-times

`gnattest` can be used to generate the test harness for platforms and run-time libraries others than the default native target with the default full run-time. For example, when using a limited run-time library such as Zero FootPrint (ZFP), a simplified harness is generated.

Two variables are used to tell the underlying AUnit framework how to generate the test harness: `PLATFORM`, which identifies the target, and `RUNTIME`, used to determine the run-time library for which the harness is generated. Corresponding prefix should also be used when calling `gnattest` for non-native targets. For example, the following options are used to generate the AUnit test harness for a PowerPC ELF target using the ZFP run-time library:

    powerpc-elf-gnattest -Psimple.gpr -XPLATFORM=powerpc-elf -XRUNTIME=zfp

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

## 27.14 Current Limitations

The tool currently does not support following features:

- generic tests for generic packages and package instantiations
- tests for protected subprograms and entries

[ << ] | [ >> ] |  |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

This document was generated by GNAT Mailserver on April, 17 2014 using texi2html