
## Table of Contents
1. [Batchnorm Forward](#Batchnorm Forward)
2. [Batchnorm Backward](#Batchnorm Backward)
3. [Batchnorm Finalize](#Batchnorm Finalize)
4. [BGenerate Stats](#Generate Stats)

### Batchnorm Forward
Batchnorm operation computes:
$$ output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias $$

Optionally the operation also computes:
```math
next\_running\_mean = (1 - momentum)*previous\_running\_mean + momentum*current\_running\_mean
```
```math
next\_running\_variance = (1 - momentum)*previous\_running\_variance + momentum*current\_running\_variance
```


The API to achieve above equations is:  
```
std::array<std::shared_ptr<Tensor_attributes>, 5> batchnorm(std::shared_ptr<Tensor_attributes>& input,
                                                            std::shared_ptr<Tensor_attributes>& scale,
                                                            std::shared_ptr<Tensor_attributes>& bias,
                                                            Batchnorm_attributes attributes); 
```
where the output array has tensors in order of: `[output, saved_mean, saved_invariance, next_running_mean, next_running_variance]`

Batchnorm attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:  
```
Batchnorm_attributes&
set_previous_running_stats(std::shared_ptr<Tensor_attributes>& previous_running_mean,
                            std::shared_ptr<Tensor_attributes>& previous_running_variance,
                            std::shared_ptr<Tensor_attributes>& momentum)

Batchnorm_attributes&
set_name(std::string const&)

Batchnorm_attributes&
set_compute_data_type(DataType_t value)
```

Python API: 
- batchnorm
    - input
    - scale
    - bias
    - in_running_mean
    - in_running_var
    - epsilon
    - momentum
    - compute_data_type
    - name

### Batchnorm Finalize

`bn_finalize` calculates the statistics required for the next iteration from the statistics generated by the genstat operation. 
```
    std::array<std::shared_ptr<Tensor_attributes>, 6> bn_finalize(std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  BN_finalize_attributes);
```

with outputs as `[EQ_SCALE, EQ_BIAS, MEAN, INV_VARIANCE, NEXT_RUNNING_MEAN, NEXT_RUNNING_VAR]`

### Batchnorm Backward(DBN)
DBN operation computes data gradient, scale gradient, bias gradient during backpropagation of batchnorm forward operation.

The API to achieve above is:  
```
std::array<std::shared_ptr<Tensor_attributes>, 3> batchnorm_backward(std::shared_ptr<Tensor_attributes> loss,
                                                                         std::shared_ptr<Tensor_attributes> input,
                                                                         std::shared_ptr<Tensor_attributes> scale,
                                                                         Batchnorm_backward_attributes);
```
where the output array has tensors in order of: `[input gradient, scale gradient, bias gradient]`.

DBN attributes is a lightweight structure with setters:  
```
Batchnorm_backward_attributes&
set_saved_mean_and_inv_variance(std::shared_ptr<Tensor_attributes> saved_mean,
                                std::shared_ptr<Tensor_attributes> saved_inverse_variance)
                                
Batchnorm_backward_attributes&
set_epsilon(std::shared_ptr<Tensor_attributes> epsilon)

Batchnorm_backward_attributes&
set_name(std::string const&)

Batchnorm_backward_attributes&
set_compute_data_type(DataType_t value)
```
Only setting either (saved mean and inverse_variance) or (epsilon) is necessary.

### Generate Stats
Genstats operation computes sum and sum of squares per-channel dimension.

The API to achieve above is:  
```
std::array<std::shared_ptr<Tensor_attributes>, 2>
cudnn_frontend::graph::genstats(std::shared_ptr<Tensor_attributes>, Genstats_attributes);
```
where the output array has tensors in order of: `[sum, square_sum]`

Genstats attributes is a lightweight structure with setters:  
```
Genstats_attributes&
set_name(std::string const&)

Genstats_attributes&
set_compute_data_type(DataType_t value)
```

Python API: 
- genstats
    - input
    - compute_data_type
    - name


### Layernorm Forward

Layer norm computes

$$ output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias $$

where normalization happens across features, independently for each sample.

The API to achieve above equations is:  
```
std::array<std::shared_ptr<Tensor_attributes>, 3> layernorm(std::shared_ptr<Tensor_attributes>& input,
                                                            std::shared_ptr<Tensor_attributes>& scale,
                                                            std::shared_ptr<Tensor_attributes>& bias,
                                                            Layernorm_attributes attributes); 
```
where the output array has tensors in order of: `[output, mean, variance]`

Layernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:  
```
Layernorm_attributes&
set_name(std::string const&)

Layernorm_attributes&
set_compute_data_type(DataType_t value)
```

Python API: 
- layernorm
    - norm_forward_phase
    - input
    - scale
    - bias
    - epsilon
    - compute_data_type
    - name


### Layernorm Backward

DLN operation computes data gradient, scale gradient, bias gradient during backpropagation of layernorm forward operation.

The API to achieve above is:  
```
std::array<std::shared_ptr<Tensor_attributes>, 3>
            layernorm_backward(std::shared_ptr<Tensor_attributes> dy,
                          std::shared_ptr<Tensor_attributes> x,
                          std::shared_ptr<Tensor_attributes> scale,
                          Layernorm_backward_attributes options);
```
where the output array has tensors in order of: `[input gradient, scale gradient, bias gradient]`.

Layernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:  
```
Layernorm_attributes&
set_name(std::string const&)

Layernorm_attributes&
set_compute_data_type(DataType_t value)
```

Python API: 
- layernorm
    - input
    - scale
    - loss
    - compute_data_type
    - name


### Instancenorm Forward

Instance norm computes

$$ output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias $$

where normalization happens across each sample.

The API to achieve above equations is:  
```
std::array<std::shared_ptr<Tensor_attributes>, 3> instancenorm(std::shared_ptr<Tensor_attributes>& input,
                                                            std::shared_ptr<Tensor_attributes>& scale,
                                                            std::shared_ptr<Tensor_attributes>& bias,
                                                            Instancenorm_attributes attributes); 
```
where the output array has tensors in order of: `[output, mean, variance]`

Instancenorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:  
```
Instancenorm_attributes&
set_name(std::string const&)

Instancenorm_attributes&
set_compute_data_type(DataType_t value)
```

Python API: 
- instancenorm
    - norm_forward_phase
    - input
    - scale
    - bias
    - epsilon
    - compute_data_type
    - name


### Instancenorm Backward

DIN operation computes data gradient, scale gradient, bias gradient during backpropagation of instancenorm forward operation.

The API to achieve above is:  
```
std::array<std::shared_ptr<Tensor_attributes>, 3>
            instancenorm_backward(std::shared_ptr<Tensor_attributes> dy,
                          std::shared_ptr<Tensor_attributes> x,
                          std::shared_ptr<Tensor_attributes> scale,
                          Instancenorm_backward_attributes options);
```
where the output array has tensors in order of: `[input gradient, scale gradient, bias gradient]`.

Instancenorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:  
```
Instancenorm_attributes&
set_name(std::string const&)

Instancenorm_attributes&
set_compute_data_type(DataType_t value)
```

Python API: 
- layernorm
    - input
    - scale
    - loss
    - compute_data_type
    - name