<!-- Copyright (c) 2014 John Biddiscombe                                          -->
<!--                                                                              -->
<!-- SPDX-License-Identifier: BSL-1.0                                             -->
<!-- Distributed under the Boost Software License, Version 1.0. (See accompanying -->
<!-- file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)        -->

Not Finished yet.


The network storage benchmark comes with several files

##################################################
# What the files are for
##################################################

------------------------------
network_storage.cpp
------------------------------
This is the code for the test.

------------------------------
network_storage.bat.in
------------------------------
This is a windows batch file template which will be filled in using cmake substitution
and then copied to your build/scripts directory.
The generated script can be invoked to run the benchmark in serial or parallel.
It spawns each task on the same node so although you may run 4 copies,
they are competing for resources.
The windows version is good for testing and debugging on a single node.
adding or removing /B for the launch command in the script
controls whether each task is launched in a new console window or not
example usage (2 tasks)
scripts/network_storage.bat 2

------------------------------
slurm-test-HPX-storage.sh.in
------------------------------
This is a bash script template which will be filled in using cmake substitution
and then copied to your build/scripts directory.
When executed, the script will loop over a number of parameter combinations
and for each create a directory with a name generated from the parameters, such as
  hpx-N0002-T02048-t04-ibverbs
and inside the dir there will be a "submit-job.bash" script which contains a single
slurm job submission. You can manually submit just one job using "sbatch submit-job.bash",
or in the root of the scripts folder there will be generated another script
which is called "run_jobs.bash". When this script is run, it will loop over
the jobs that were created by the first script and submit them all, you
can then sit back and wait until they complete.
Each job will write its results into "slurm.out" (and errors to slurm.err)
in the same subdirectory in which each individual job submission script was created.

The network_storage executable will produce several lines of output, but one line
contains the condensed information needed by the plotting script.
This line begins with the text "CSVData" to indicate comma separated data values.
Whilst jobs are running or when they have completed, you can execute a command
from the test root dir, such as
  find . -name slurm.out -exec grep CSV {} \;
and a list of results generated from the jobs will be produced.
For plotting of results, the output should be directed into a file using
  find . -name slurm.out -exec grep CSV {} \; >results-bgq-1a-2014-04-01.csv
Where you use a file name applicable to your current experiment.
The generated file will be suitable for use by the plotting script.

Important Note:
By default, the slurm jobs generated by the script have the 'exclusive' flag set.
This is because the jobs are intended to test BW of the network and it is
often desirable  to have only a single job running at a time.
The 'exclusive' flag only works (under slurm) if all jobs have the same name
so in your queue you will see many identical jobs and it will take a long time to drain.
If you can afford to run many job simultaneously, the job name may be set more
appropriately (details in script) and the exclusive flag removed.

------------------------------
plot-results.py
------------------------------
This is a python script which takes results generated by the test program
as described above and plots a number of graphs for different parameter studies.
The results.csv file generated contains BW measurements, timing ,thread, parcelport,
block size, etc information for the plots.
The python script parses the results and generates arrays (maps) of the data which
can be plotted in various ways. The scripts can be invoked as
  plot-results.py results.csv
optional arguments such as the figure size can be found by looking at the script.
The output from the script will be a series of svg and png files for each of the plots
created.

## Launch notes
greina14
bin/network_storage --localMB=1024 --transferKB=16384 --hpx:localities=2 -Ihpx.parcel.tcp.enable=1 --hpx:agas=148.187.83.44:7910 --hpx:hpx=148.187.83.44:7910 -Ihpx.parcel.tcp.enable=1 -Ihpx.parcel.mpi.enable=0 -Ihpx.parcel.verbs.enable=1 -Ihpx.parcel.bootstrap=tcp

greina15
bin/network_storage --localMB=1024 --transferKB=16384 --hpx:localities=2 -Ihpx.parcel.tcp.enable=1  --hpx:agas=148.187.83.44:7910 --hpx:worker --hpx:hpx=148.187.83.45:7910 -Ihpx.parcel.tcp.enable=1 -Ihpx.parcel.mpi.enable=0 -Ihpx.parcel.verbs.enable=1 -Ihpx.parcel.bootstrap=tcp

greina1
gdb --args /home/biddisco/build/hvtkm/bin/network_storage --localMB=64 --transferKB=1024 --hpx:localities=2 -Ihpx.parcel.tcp.enable=1 --hpx:agas=192.168.3.31:7910 --hpx:hpx=192.168.3.31:7910 -Ihpx.parcel.tcp.enable=1 -Ihpx.parcel.mpi.enable=0 -Ihpx.parcel.verbs.enable=1 -Ihpx.parcel.bootstrap=tcp --hpx:threads=1

greina2
gdb --args /home/biddisco/build/hvtkm/bin/network_storage --localMB=64 --transferKB=1024 --hpx:localities=2 -Ihpx.parcel.tcp.enable=1 --hpx:agas=192.168.3.31:7910 --hpx:hpx=192.168.3.32:7910 -Ihpx.parcel.tcp.enable=1 -Ihpx.parcel.mpi.enable=0 -Ihpx.parcel.verbs.enable=1 -Ihpx.parcel.bootstrap=tcp --hpx:worker --hpx:threads=1

mpi launch
mpiexec -n 2 -host greina14,greina15 bin/network_storage --localMB=1024 --transferKB=16384 --iterations=10 --hpx:localities=2 -Ihpx.parcel.tcp.enable=1 -Ihpx.parcel.tcp.enable=0 -Ihpx.parcel.mpi.enable=1 -Ihpx.parcel.verbs.enable=0 -Ihpx.parcel.bootstrap=mpi --hpx:threads=12

test TCP greina14
bin/network_storage --localMB=1024 --transferKB=16384 --hpx:localities=2 -Ihpx.parcel.tcp.enable=1 --hpx:agas=148.187.83.44:7910 --hpx:hpx=148.187.83.44:7910 -Ihpx.parcel.tcp.enable=1 -Ihpx.parcel.mpi.enable=0 -Ihpx.parcel.verbs.enable=0 -Ihpx.parcel.bootstrap=tcp --iterations=10  --hpx:threads=12

test TCP greina15
bin/network_storage --localMB=1024 --transferKB=16384 --hpx:localities=2 -Ihpx.parcel.tcp.enable=1  --hpx:agas=148.187.83.44:7910 --hpx:worker --hpx:hpx=148.187.83.45:7910 -Ihpx.parcel.tcp.enable=1 -Ihpx.parcel.mpi.enable=0 -Ihpx.parcel.verbs.enable=0 -Ihpx.parcel.bootstrap=tcp --iterations=10  --hpx:threads=12
