JUnit4 Task

## JUnit4

### Description

This task runs tests from the JUnit testing framework in versions 4.10 and above. It to some extent mimics the style of the default ANT's `junit` task, but adds the following features:

- ability to run tests in multiple forked JVMs (isolated JVMs),
- ability to load-balance tests based on previous execution history or job-stealing,
- seamless integration with randomized runner to support a global seed for randomized tests and proper seed reporting,
- Synchronized and decluttered console output from child JVMs,
- Built-in reporting to HTML, JSON and various forms of plain text.

Note: The latest version of the JUnit framework can be found at http://www.junit.org. This task has been tested with JUnit 4.10 and later. Appropriate `junit*.jar` must be available in the task's classpath and in the tested code's classpath. See the examples below.

External references: More information about randomized testing.

### Parameters

`junit4` attempts to be a drop-in replacement for ANT's default `junit` task, but so attributes that make little sense have been deprecated and are ignored. Attributes specific to `junit4` are highlighted in light green.

Attribute | Description | Required
printsummary | Prints the summary of all executed, ignored and failed tests after all suites have been executed. To get per-suite summaries, configure a console report properly (see below). | No; default is `on`.
fork | Not used. ANT-junit drop-in compatibility only. Tests are always forked. | --
forkmode | Not used. ANT-junit drop-in compatibility only. Tests are always forked into one more more forked JVMs. Each suite is executed entirely on a single JVM. | --
haltonerror | Not used. ANT-junit drop-in compatibility only. Use `haltonfailure` which handles both errors and failures. | --
errorproperty | Not used. ANT-junit drop-in compatibility only. Use `failureproperty` which handles both errors and failures. | --
haltonfailure | Stop the build process if a test fails (errors are considered failures as well). | No; default is `on` (ANT's junit has the default set to 'off').
failureproperty | The name of a property to set in the event of a failure (errors are considered failures as well). | No.
filtertrace | Not used. ANT-junit drop-in compatibility only. Reports may provide their own filtering capabilities. | --
timeout | Not used. ANT-junit drop-in compatibility only. Test runners provide their own timeout facilities. | --
maxmemory | Maximum amount of memory to allocate to each forked VM (be careful if using lots of forks). Note: If you get `java.lang.OutOfMemoryError: Java heap space` in some of your tests then you need to raise the size like `maxmemory="128m"` | No
jvm | The command used to invoke the forked java virtual machines, the default is 'java'. The command is resolved by `java.lang.Runtime.exec()`. | No; default is `java`.
dir | The directory in which to invoke the forked JVMs in. This is by default the project's `basedir`. Because multiple JVMs may clash if they are generating files in the current working directory, one can use per-JVM directory by setting `isolateWorkingDirectories` attribute to `true`; in such case each JVM gets a sub-directory the provided `dir`. See `onNonEmptyWorkDirectory` attribute for action taken when the forked JVM's work directory is not initially empty. | No
isolateWorkingDirectories | If `true` current working directories are isolated for each forked JVM. See `dir` attribute too. | No; default is `true`.
onNonEmptyWorkDirectory | [Since version 2.5.0]. Specifies the action taken when forked JVM's work directory is not initially empty (contains files or directories, possibly from a previous run). The values permitted are: `fail` Fails the build. `wipe` Tries to wipe the work directory clean before tests begin. Fails the build if unsuccessful. `ignore` Ignores any existing files and proceeds (not recommended). | No; default is `fail`.
newenvironment | Do not propagate the old environment when new environment variables are specified. | No; default is `false`.
includeantruntime | Not used. ANT-junit drop-in compatibility only. | --
showoutput | Not used. ANT-junit drop-in compatibility only. Configure console reports appropriately. | --
outputtoformatters | Not used. ANT-junit drop-in compatibility only. Configure console reports appropriately. | --
tempdir | Specify where to store temporary files. These temporary files include a list of suites passed to each forked JVM so it is usually wise to just leave this attribute unset - the default is to take the value of the `dir` attribute or the project's basedir. | No
reloading | Not used. ANT-junit drop-in compatibility only. | --
clonevm | Not used. ANT-junit drop-in compatibility only. | --
logfailedtests | Not used. ANT-junit drop-in compatibility only. Configure console reports appropriately to get just the failing tests, their output etc. | --
enableTestListenerEvents | Not used. ANT-junit drop-in compatibility only. | --
parallelism | The number of parallel forked JVMs. Can be set to a constant `max` for the maximum number of cores returned from `Runtime.availableProcessors` or `auto` for sensible defaults depending on the number of cores. Note: this setting forks physical JVM processes so it multiplies the requirements for heap memory, IO, etc. | No; Can be set to any integer or: 'max', 'auto'. The default is '1' (sequential execution in a forked JVM).
dynamicAssignmentRatio | Specifies the ratio of suites moved to dynamic assignment list (job-stealing). A dynamic assignment list dispatches suites to the first idle forked JVM. Theoretically this is an optimal strategy, but it is usually better to have some static assignments to avoid communication costs. A ratio of 0 means only static assignments are used. A ratio of 1 means only dynamic assignments are used. The list of dynamic assignments is sorted by decreasing cost (always) and is inherently prone to race conditions in distributing suites. Should there be an error based on suite-dependency it will not be directly repeatable. In such case use the per-forked-jvm list of suites file dumped to disk for each forked JVM. See `leaveTemporary` attribute. | No; default is '0.25' (25% of all suites are assigned dynamically).
seed | Specify random seed for anything that is randomized in `junit4`. The order of suites execution and suite-JVM assignments are a result of this seed for example. The main seed is also passed to forked JVMs as a system property (to bootstrap randomized runner). | No; default is a randomly generated seed.
prefix | Initializes custom prefix for all randomized runner properties. This must be consistent across all junit4 invocations if done from the same classpath. Use only when REALLY needed. | No; default is randomized runner's prefix.
shuffleOnForkedJvm | Predictably shuffle tests order after balancing. This will help in spreading lighter and heavier tests over a single forked JVM execution timeline while still keeping the same tests order depending on the seed. See nested elements for configuring load balancers. | No; the default is 'true'.
leaveTemporary | Leave temporary `junit4` files after the task quits. This can be used to trace the exact order of suites executed on each forked JVM for example. | No; default is 'false'.
jvmOutputAction | What should be done on unexpected JVM output? JVM may write directly to the original descriptors, bypassing redirections of System.out and System.err. Typically, these messages will be important and should fail the build (permgen space exceeded, compiler errors, crash dumps). However, certain legitimate logs (gc activity, class loading logs) are also printed to these streams so sometimes the output can be ignored. Allowed values (any comma-delimited combination of): ignore, listeners, pipe, fail, warn. | No; default is 'warn, listeners'; append text report listener to get the actual JVM output or change to 'warn, pipe' to get the output piped to system out.
sysouts | If set to true, any sysout and syserr calls will be written to original output and error streams (and in effect will appear as "jvm output". By default sysout and syserrs are captured and proxied to the event stream to be synchronized with other test events (and properly handled by reports) but occasionally one may want to synchronize them with direct JVM output (to synchronize with compiler output or GC output for example). See examples below on examples of capturing full JVM output. | No; default is 'false'.
heartbeat | A duration (in seconds) before an event is dispatched about forked JVM inactivity. This can be useful to report hung or very long tests. Heartbeat information will include an approximate location inside the non-responding JVM (suite or test case). | No; default is '0'.
uniqueSuiteNames | Allow or disallow duplicate suite names in resource collections. By default this option is true because certain ANT-compatible report types (like XML reports) will have a problem with duplicate suite names (will overwrite files). | No; default is 'true'.
ifNoTests | Defines the behavior when no tests were executed successfully or failed (either there were no tests at all or all of them were filtered out). One of the following enum constants: 'ignore', 'warn' or 'fail'. | No; default is 'ignore'.

### Nested Elements

The `<junit4>` task supports a nested elements. Some of them are compatible with ANT's default `<junit>` task while others are unique to `<junit4>`.

#### Specifying classpath for tests

The `<classpath>` element represents a PATH like structure to be used for each forked JVM. Note that classpath should include suiteable `junit*.jar`, otherwise your tests wont run.

#### Specifying test suite classes

Unlike standard `<junit>` task, `<junit4>` defines suite classes as resources to `.class` files. These resources can originate from any resource collection and there can be any number of resource collections with such resources. For example this definition includes all suite classes from a ZIP file (JAR file), excluding nested classes:

#### jvmarg

Additional parameters may be passed to the new JVM via nested `<jvmarg>` elements. For example:

would run the JVM with a serial garbage collector on HotSpot JVM.

#### sysproperty

Use nested `<sysproperty>` elements to specify system properties required by the tests. These properties will be made available to the forked JVMs during their execution. The attributes for this element are the same as for environment variables.

An extended attribute `<ignoreEmpty>` is supported which, if defined, causes the property NOT to be defined in the forked JVM (as opposed to passing an empty property value).

would run the test in ANT's VM and make the project's `basedir` property available to the test (note the current working directory of a forked JVM will not be the basedir typically).

#### syspropertyset

You can specify a set of properties to be passed to the target JVM as system properties with syspropertysets. This target behaves much like the default in ANT but allows a dynamic set of (remapped) properties to be resolved at runtime. Similar to `sysproperty`, this tag also supports `ignoreEmpty` attribute to work around bugs with local properties in ANT.

#### env

It is possible to specify environment variables to pass to the forked VM via nested `<env>` elements. For a description of the `<env>` element's attributes, see the description in the exec task.

#### bootclasspath

The location of bootstrap class files can be specified using this PATH like structure.

#### assertions

You can control enablement of Java 1.4 assertions with an `<assertions>` subelement.

### listeners and reports

There is no notion of "reports" in `junit4`. Reports are just listeners that attach to the even stream and can produce something (to the console, file or otherwise). You can also attach your own listeners if you like (although you'd have to peek at the code of existing listeners to get the feeling how this is done -- communication is via events not interfaces).

#### report-text

There are a few predefined listeners to choose from. By far the most common will be `report-text` which produces console output (or plain text file). A plain text report configuration may look like this:

This listener supports stack trace filtering to reduce verbosity (console output, for example). For production or integration builds, it is recommended to keep the full stacks. For developer runs, the default set of filters (enabled by default) can declutter the output a bit. An example stack trace filter, with default options and two custom filters, is shown below.

The level of verbosity of the output can be fine-tuned to one's needs by disabling or enabling test statuses and suite summaries or each test's output. Experimenting highly recommended.

#### report-ant-xml

Another listener is `report-ant-xml` which can produce XML files suitable for junitreport task (although we highly recommend using the built-in JSON report.

#### report-json

Yet another built-in listener is producing a modern JSON output with test data and an accompanying HTML5 file for visualizing it:

An example of a HTML5 report produced by the above can be seen here, for example.

### runlisteners

You can attach custom JUnit 4 RunListeners to your tests like this:

Each RunListener defined is added to each class before execution and removed after

### Load balancing of test suites

Suites can be scattered across forked JVMs randomly or using a greedy load-balancing algorithm (and followed by job-stealing if needed). For load balancing, previous execution statistics will be needed. These statistics should ideally come from a single machine and multiple executions so that they average the influence of the environment etc.

A dedicated listener is used to collect and update statistics. The configuration below shows both the listener and a balancer configuration. The balancer uses multiple sources for collecting statistics (for example precomputed statistics and previous local runs):

### Capturing original JVM output

JUnit4 has been designed to run reports and aggregation of test events on the main JVM, not on the forked JVMs. This slightly complicates things when diagnostic JVM messages are used because these messages are printed directly to original stream descriptors by the JVM, bypassing System.out or System.err substitutions.

In those rare cases when the original JVM output streams are needed (possibly mixed with what the tests have printed), junit4 can be configured to leave the original streams on disk or pipe them to ANT's output. An example is shown below:

Should one need to preserve the original output files (in case they are large, for example), the configuration would then look as follows:

This would result in the following message being printed to ANT (note the output file name contains an execution-unique element to avoid clashing with other forked JVMs or builds).

    [junit4:junit4] JVM J0: stdout was not empty, see: /home/[...]/junit4-J0-1058309761ea5fafd.sysout
    [junit4:junit4] JVM J0: stderr was not empty, see: /home/[...]/junit4-J0-1058309761ea5fafd.syserr