# RPP Test Suite

This repository contains four test suites for the RPP library: `image`/`voxel`/`audio`/`miscellaneous` augmentations processing. These test suites can be used to validate the functionality and performance of the RPP `image`/`voxel`/`audio`/`miscellaneous` augmentation functionalities.

## Prerequisites

* OpenCV Version `3.X`/`4.X`
  ```shell
  sudo apt install libopencv-dev
  ```

* [Turbo JPEG](https://libjpeg-turbo.org/) 
  * Source: `https://github.com/libjpeg-turbo/libjpeg-turbo.git`
  * Tag: [3.0.2](https://github.com/libjpeg-turbo/libjpeg-turbo/releases/tag/3.0.2) 
  ```shell
  sudo apt-get install nasm
  sudo apt-get install wget
  git clone -b 3.0.2 https://github.com/libjpeg-turbo/libjpeg-turbo.git 
  cd libjpeg-turbo
  mkdir build
  cd build
  cmake -DCMAKE_INSTALL_PREFIX=/usr \
        -DCMAKE_BUILD_TYPE=RELEASE  \
        -DENABLE_STATIC=FALSE       \
        -DCMAKE_INSTALL_DEFAULT_LIBDIR=lib  \
        -DWITH_JPEG8=TRUE           \
        ..
  make -j$nproc
  sudo make install
  ```

* Libsndfile
  ```shell
  sudo apt-get install libsndfile1-dev
  ```

* Imagemagick
  ```shell
  sudo apt-get install imagemagick
  ```

* Nifti-Imaging - [nifti_clib](https://github.com/NIFTI-Imaging/nifti_clib)
  ```shell
  git clone https://github.com/NIFTI-Imaging/nifti_clib.git
  cd nifti_clib
  mkdir build
  cd build
  cmake ..
  sudo make -j$nproc install
  ```

* Python3 and Python3 PIP
  ```shell
  sudo apt install python3-dev python3-pip
  ```

* Python: Openpyxl
  ```shell
  pip install openpyxl
  ```

* Python: Pandas
  ```shell
  pip install pandas
  ```

>[!NOTE]
> * All package installs are shown with the `apt` package manager. Use the appropriate package manager for your operating system.
> * Run `sudo ldconfig` to configure dynamic run-time bindings

## Rpp Image Test Suite
The image test suite can be executed under 2 backend scenarios - (HOST/HIP):
-   HOST backend - (On a CPU with HOST backend)
-   HIP backend - (On a GPU with HIP backend)

### Command Line Arguments (RPP Image Test Suite)
The image test suite accepts the following command line arguments:
-   input_path1: The path to the input folder 1. Default is $cwd/../TEST_IMAGES/three_images_mixed_src1
-   input_path2: The path to the input folder 2. Default is $cwd/../TEST_IMAGES/three_images_mixed_src2
-   case_start: The starting case number for the test range (0-91). Default is 0
-   case_end: The ending case number for the test range (0-91). Default is 91
-   test_type: The type of test to run (0 = Unit tests, 1 = Performance tests). Default is 0
-   case_list: A list of specific case numbers to run. Must be used in conjunction with --test_type
-   profiling: Run the tests with a profiler (YES/NO). Default is NO. This option is only available with HIP backend
-   qa_mode: Output images from tests will be compared with golden outputs - (0 / 1). Default is 0
-   decoder_type: Type of Decoder to decode the input data - (0 = TurboJPEG / 1 = OpenCV). Default is 0
-   num_runs: Specifies the number of runs for running the performance tests
-   preserve_output: preserves the output images or performance logs generated from the previous test suite run - (0 = remove output images or performance logs / 1 = preserve output images or performance logs). Default is 1
-   batch_size: Specifies the batch size to use for running tests. Default is 1

### Running the Tests for HOST Backend (RPP Image Test Suite)
The test suite can be run with the following command:
``` python
python runTests.py --input_path1 <input_path1> --input_path2 <input_path2> --case_start <case_start> --case_end <case_end> --test_type <test_type>
```

### Running the Tests for HIP Backend (RPP Image Test Suite)
The test suite can be run with the following command:
``` python
python runTests.py --input_path1 <input_path1> --input_path2 <input_path2> --case_start <case_start> --case_end <case_end> --test_type <test_type> --profiling <profiling>
```

### Modes of operation (Rpp Image Test Suite)
-   QA mode (Unit tests) - Tolerance based PASS/FAIL tests for RPP HIP/HOST functionalities checking pixelwise match between C/SSE/AVX/HIP versions after comparison to preset golden outputs. Please note that QA mode is only supported with a batch size of 3.
Note: QA mode is not supported for case 84 due to run-to-run variation of outputs.
``` python
python runTests.py --case_start 0 --case_end 91 --test_type 0 --qa_mode 1 --batch_size 3
```
-   QA mode (Performance tests) - Tolerance based PASS/FAIL tests for RPP HIP/HOST functionalities checking achieved improvement in performance percentage over BatchPD versions after comparison to a threshold percentage of improvement
``` python
python runTests.py --case_list 21 36 63 --test_type 1 --qa_mode 1 --batch_size 8 --num_runs 100
```
-   Unit test mode - Unit tests allowing users to pass a path to a folder containing images, to execute the desired functionality and variant once, report RPP execution wall time, save and view output images
Note: For testcase 82(RICAP) Please use images of same resolution and Batchsize > 1
      RICAP dataset path: rpp/utilities/test_suite/TEST_IMAGES/three_images_150x150_src1
``` python
python runTests.py --case_start 0 --case_end 91 --test_type 0 --qa_mode 0
```
-   Performance test mode - Performance tests that execute the desired functionality and variant 100 times by default, and report max/min/avg RPP execution wall time, or optionally, AMD rocprof kernel profiler max/min/avg time for HIP backend variants.
Note: For testcase 82(RICAP) Please use images of same resolution and Batchsize > 1
      RICAP dataset path: rpp/utilities/test_suite/TEST_IMAGES/three_images_150x150_src1
``` python
python runTests.py --case_start 0 --case_end 91 --test_type 1
```

To run the unit tests / performance tests for specific case numbers. please case use case_list parameter. Example as below

-   To run unittests for case numbers 0, 2, 4
``` python
python runTests.py --case_list 0 2 4 --test_type 0
```
-   To run performance tests for case numbers 0, 2, 4
``` python
python runTests.py --case_list 0 2 4 --test_type 1
```

To run performance tests with AMD rocprof kernel profiler for HIP backend variants. This will generate profiler times for HIP backend variants
``` python
python runTests.py --test_type 1 --profiling YES
```

### Summary of features (RPP Image Test Suite)
The image test suite includes:
-   Unit tests that execute the desired functionality and variant once, report RPP execution wall time and save output images
-   Performance tests that execute the desired functionality and variant 100 times by default, and report max/min/avg RPP execution wall time, or optionally, AMD rocprof kernel profiler max/min/avg time for HIP backend variants.
-   Unit and Performance tests are included for three layouts - PLN1 (1 channel planar NCHW), PLN3 (3 channel planar NCHW) and PKD3 (3 channel packed/interrleaved NHWC).
-   Unit and Performance tests are included for various input/output bitdepths including U8/F32/F16/I8.
-   Support for pixelwise output referencing against golden outputs, and functionality validation checking, by tolerance-based pass/fail criterions for each variant. (Current support only for U8 variants)
-   Support for TurboJPEG and OpenCV decoder for decoding input images

## RPP Voxel Test Suite
The 3D Voxel test suite can be executed under 2 backend scenarios - (HOST/HIP):
-   HOST backend - (On a CPU with HOST backend)
-   HIP backend - (On a GPU with HIP backend)

### Command Line Arguments (RPP Voxel Test Suite)
The voxel test suite accepts the following command line arguments:
-   header_path: Path to the nii header
-   data_path: Path to the nii data file
-   case_start: The starting case number for the test range (0-38). Default is 0
-   case_end: The ending case number for the test range (0-38). Default is 4
-   test_type: The type of test to run (0 = Unit tests, 1 = Performance tests). Default is 0
-   case_list: A list of specific case numbers to run. Must be used in conjunction with --test_type
-   profiling: Run the tests with a profiler (YES/NO). Default is NO. This option is only available with HIP backend
-   qa_mode: Output images from tests will be compared with golden outputs - (0 / 1). Default is 0
-   num_runs: Specifies the number of runs for running the performance tests
-   preserve_output: preserves the output images or performance logs generated from the previous test suite run - (0 = remove output images or performance logs / 1 = preserve output images or performance logs). Default is 1
-   batch_size: Specifies the batch size to use for running tests. Default is 1

### Running the Tests for HOST Backend (RPP Voxel Test Suite)
The test suite can be run with the following command:
``` python
python runVoxelTests.py --header_path <header_path> --data_path <data_path> --case_start <case_start> --case_end <case_end> --test_type <test_type>
```

### Running the Tests for HIP Backend (RPP Voxel Test Suite)
The test suite can be run with the following command:
``` python
python runVoxelTests.py --header_path <header_path> --data_path <data_path> --case_start <case_start> --case_end <case_end> --test_type <test_type> --profiling <profiling>
```

### Modes of operation (RPP Voxel Test Suite)
-   QA mode - Tolerance based PASS/FAIL tests for RPP HIP/HOST functionalities checking pixelwise match between C/SSE/AVX/HIP versions after comparison to preset golden outputs.
``` python
python runVoxelTests.py --case_start 0 --case_end 4 --test_type 0 --qa_mode 1 --batch_size 3
```
-   Unit test mode - Unit tests allowing users to pass a path to a folder containing nii fikes, to execute the desired functionality and variant once, report RPP execution wall time, save and view output images, gifs and nii files
``` python
python runVoxelTests.py --case_start 0 --case_end 4 --test_type 0 --qa_mode 0
```
-   Performance test mode - Performance tests that execute the desired functionality and variant 100 times by default, and report max/min/avg RPP execution wall time, or optionally, AMD rocprof kernel profiler max/min/avg time for HIP backend variants.
``` python
python runVoxelTests.py --case_start 0 --case_end 4 --test_type 1
```

To run the unit tests / performance tests for specific case numbers. please case use case_list parameter. Example as below

-   To run unittests for case numbers 0, 2, 4
``` python
python runVoxelTests.py --case_list 0 2 4 --test_type 0
```
-   To run performance tests for case numbers 0, 2, 4
``` python
python runVoxelTests.py --case_list 0 2 4 --test_type 1
```

To run performance tests with AMD rocprof kernel profiler for HIP backend variants. This will generate profiler times for HIP backend variants
``` python
python runVoxelTests.py --test_type 1 --profiling YES
```

### Summary of features (RPP Voxel Test Suite)
The image test suite includes:
-   Unit tests that execute the desired functionality and variant once, report RPP execution wall time and save output images, gifs and nii files
-   Performance tests that execute the desired functionality and variant 100 times by default, and report max/min/avg RPP execution wall time, or optionally, AMD rocprof kernel profiler max/min/avg time for HIP backend variants.
-   Unit and Performance tests are included for three layouts - PLN1 (1 channel planar NCHW), PLN3 (3 channel planar NCHW) and PKD3 (3 channel packed/interrleaved NHWC).
-   Unit and Performance tests are included for one input/output bitdepth F32.
-   Support for pixelwise output referencing against golden outputs, and functionality validation checking, by tolerance-based pass/fail criterions for each variant.

### References (RPP Voxel Test Suite)
RPP test suite uses a sample ".nii" image usage from the BraTS2020 Dataset at https://www.kaggle.com/code/rastislav/3d-mri-brain-tumor-segmentation-u-net/input as per the following Data Usage Agreement present at the dataset link above.

Data Usage Agreement / Citations:
You are free to use and/or refer to the BraTS datasets in your own research, provided that you always cite the following three manuscripts:

-   B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, et al. "The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)", IEEE Transactions on Medical Imaging 34(10), 1993-2024 (2015) DOI: 10.1109/TMI.2014.2377694
-   S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J.S. Kirby, et al., "Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features", Nature Scientific Data, 4:170117 (2017) DOI: 10.1038/sdata.2017.117
-   S. Bakas, M. Reyes, A. Jakab, S. Bauer, M. Rempfler, A. Crimi, et al., "Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge", arXiv preprint arXiv:1811.02629 (2018)

## RPP Audio Test Suite
The audio test suite can be executed to validate the functionality and performance of the AMD ROCm Performance Primitives (RPP) audio library.
-   HOST backend - (On a CPU with HOST backend)
-   F32 Bit Depth

### Command Line Arguments (RPP Audio Test Suite)
The audio test suite accepts the following command line arguments:
-   input_path: The path to the input folder. Default is $cwd/../TEST_AUDIO_FILES/eight_samples_single_channel_src1
-   case_start: The starting case number for the test range (0-0). Default is 0
-   case_end: The ending case number for the test range (0-0). Default is 0
-   test_type: The type of test to run (0 = QA tests, 1 = Performance tests). Default is 0
-   qa_mode: Output audio data from tests will be compared with golden outputs - (0 / 1). Default is 0
-   case_list: A list of specific case numbers to run. Must be used in conjunction with --test_type
-   num_runs: Specifies the number of runs for running the performance tests
-   preserve_output: preserves the output or performance logs generated from the previous test suite run - (0 = remove output or performance logs / 1 = preserve output or performance logs). Default is 1
-   batch_size: Specifies the batch size to use for running tests. Default is 1

### Running the Tests for HOST Backend (RPP Audio Test Suite)
The test suite can be run with the following command:
``` python
python runAudioTests.py --input_path <input_path> --case_start <case_start> --case_end <case_end> --test_type <test_type>
```

### Modes of operation (RPP Audio Test Suite)
-   QA mode - Tolerance based PASS/FAIL tests for RPP AUDIO HOST functionalities checking match between output and preset golden outputs. Please note that QA mode is only supported with a batch size of 3.
``` python
python runAudioTests.py --case_start 0 --case_end 0 --qa_mode 1 --batch_size 3
```

-   Performance test mode - Performance tests that execute the desired functionality and variant 100 times by default, and report max/min/avg RPP execution wall time.
``` python
python runAudioTests.py --case_start 0 --case_end 0 --test_type 1
```

To run the QA tests / performance tests for specific case numbers. please case use case_list parameter. Example as below

-   To run QA tests for case numbers 0, 1, 2
``` python
python runAudioTests.py --case_list 0 1 2 --qa_mode 1 --batch_size 3
```
-   To run performance tests for case numbers 0, 1, 2
``` python
python runAudioTests.py --case_list 0 1 2 --test_type 1
```

### Summary of features (RPP Audio Test Suite)
The audio test suite includes:
-   Performance tests that execute the desired functionality and variant 100 times by default, and report max/min/avg RPP execution wall time.
-   QA and Performance tests are included for one input/output bitdepth F32.
-   Support for output referencing against golden outputs, and functionality validation checking, by tolerance-based pass/fail criterions for each variant.

## RPP Miscellaneous Test Suite
The miscellaneous test suite can be executed to validate the functionality and performance of the AMD ROCm Performance Primitives (RPP) generic libraries that can process N-Dimensional inputs
-   HOST backend - (On a CPU with HOST backend)
-   HIP backend - (On a GPU with HIP backend)
-   F32 Bit Depth

### Command Line Arguments (RPP Miscellaneous Test Suite)
The miscellaneous test suite accepts the following command line arguments:
-   case_start: The starting case number for the test range (1-1). Default is 1
-   case_end: The ending case number for the test range (1-1). Default is 1
-   test_type: The type of test to run (0 = QA tests, 1 = Performance tests). Default is 0
-   qa_mode: Output audio data from tests will be compared with golden outputs - (0 / 1). Default is 0
-   case_list: A list of specific case numbers to run. Must be used in conjunction with --test_type
-   num_dims: The number of dimensions of input. Default is 2
-   num_runs: Specifies the number of runs for running the performance tests
-   preserve_output: preserves the output or performance logs generated from the previous test suite run - (0 = remove output or performance logs / 1 = preserve output or performance logs). Default is 1
-   batch_size: Specifies the batch size to use for running tests. Default is 1
-   profiling: Run the tests with a profiler (YES/NO). Default is NO. This option is only available with HIP backend

### Running the Tests for HOST Backend (RPP Miscellaneous Test Suite)
The test suite can be run with the following command:
``` python
python runMiscTests.py --case_start <case_start> --case_end <case_end> --test_type <test_type>
```

### Modes of operation (RPP Miscellaneous Test Suite)
-   QA mode - Tolerance based PASS/FAIL tests for RPP MISC HOST functionalities checking match between output and preset golden outputs. Please note that QA mode is only supported with a batch size of 3 and num dims 2 or 3
``` python
python runMiscTests.py --case_start 1 --case_end 1 --qa_mode 1 --batch_size 3
```

-   Performance test mode - Performance tests that execute the desired functionality and variant 100 times by default, and report max/min/avg RPP execution wall time.
``` python
python runMiscTests.py --case_start 1 --case_end 1 --test_type 1
```

### Summary of features (RPP Miscellaneous Test Suite)
The miscellaneous test suite includes:
-   Performance tests that execute the desired functionality and variant 100 times by default, and report max/min/avg RPP execution wall time.
-   QA and Performance tests are included for one input/output bitdepth F32.
-   Support for output referencing against golden outputs, and functionality validation checking, by tolerance-based pass/fail criterions for each variant.
