..
    Copyright (C)      2013 Thomas Heller

    SPDX-License-Identifier: BSL-1.0
    Distributed under the Boost Software License, Version 1.0. (See accompanying
    file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)

=============
Testing |hpx|
=============

To ensure correctness of |hpx|, we ship a large variety of unit and regression
tests. The tests are driven by the |ctest|_ tool and are executed automatically 
on each commit to the |hpx| |hpx_github|_ repository. In addition, it is encouraged 
to run the test suite manually to ensure proper operation on your target system. 
If a test fails for your platform, we highly recommend submitting an issue on our 
|hpx_github_issues|_ tracker with detailed information about the target system.

Running tests manually
======================

Running the tests manually is as easy as typing ``make tests && make test``.
This will build all tests and run them once the tests are built successfully.
After the tests have been built, you can invoke separate tests with the help of
the ``ctest`` command. You can list all available test targets using ``make help
| grep tests``. Please see the |ctest_doc|_ for further details.

Running performance tests
=========================

We run performance tests on Piz Daint for each pull request using Jenkins. To
run those performance tests locally or on Piz Daint, a script is provided under
``tools/perftests_ci/local_run.sh`` (to be run in the build directory specifying
the |hpx| source directory as the argument to the script, default is
``$HOME/projects/hpx_perftests_ci``.

Adding new performance tests
============================

To add a new performance test, you need to wrap the portion of code to benchmark
with ``hpx::util::perftests_report``, passing the test name, the executor name
and the function to time (can be a lambda). This facility is used to output the
time results in a json format (format needed to compare the results and plot
them).  To effectively print them at the end of your test, call
``hpx::util::perftests_print_times``. To see an example of use, see
``future_overhead_report.cpp``.  Finally, you can add the test to the CI report
editing the ``hpx_targets`` variable for the executable name and the
``hpx_test_options`` variable for the corresponding options to use for the run
in the performance test script ``.jenkins/cscs-perftests/launch_perftests.sh``.
And then run the ``tools/perftests_ci/local_run.sh`` script to get a reference
json run (use the name of the test) to be added in the
``tools/perftests_ci/perftest/references/daint_default`` directory.

Issue tracker
=============

If you stumble over a bug or missing feature in |hpx|, please
submit an issue to our |hpx_github_issues|_ page. For more information on how to
submit support requests or other means of getting in contact with the developers,
please see the |support|_ page.

Continuous testing
==================

In addition to manual testing, we run automated tests on various platforms. We also 
run tests on all pull requests using both |circleci|_ and a combination of |cdash|_ 
and |pycicle|_. You can see the dashboards here: |hpx_circleci|_ and |hpx_cdash|_ .
