\section{Which MVA method should I use for my problem?}
\label{sec:whatMVAshouldIuse}

There is obviously no generally valid answer to this question. To
guide the user, we have attempted a coarse assessment of various MVA
properties in Table~\ref{tab:classQA}.  Simplicity is a virtue, but
only if it is not at the expense of significant loss of discrimination
power. A properly trained Neural Network with the problem dependent
optimal architecture or a Support Vector Machine with the appropriate
kernel size should theoretically give superior performance compared to
a more robust ``out of the box''  Boosted Decision
Trees\footnote{Note, in an earlier version we noted the problem of
  overtraining in BDTs when using large trees and trying to regulate
  them using pruning. This however is overcome completely when using
  tress with limited depth (say 2 to 4) and minimal leaf node size
  (some \% of training sample size) instead of pruning, which in
  addition gives superior performance.}, which in practice often
outperforms the former.


To assess whether a linear discriminant analysis (LDA) could be sufficient 
for a classification (regression) problem, the user is advised to analyse the 
correlations among the discriminating variables (among the variables and regression 
target) by inspecting scatter and profile plots (it is not enough to print the 
correlation coefficients, which by definition are linear only). Using an LDA 
greatly reduces the number of parameters to be adjusted and hence allow smaller
training samples. It usually is robust with respect to generalisation
to larger data samples. For moderately intricate problems, the function discriminant 
analysis (FDA) with some added nonlinearity may be found sufficient. It is always 
useful to cross-check its performance against several of the sophisticated nonlinear
methods to see how much can be gained over the use of the simple and very
transparent FDA.

\begin{table}[t]
{\small
\setlength{\tabcolsep}{0.0pc}
\begin{tabular*}{\textwidth}{@{\extracolsep{\fill}}p{1.3cm}p{2.7cm}p{1cm}p{1cm}p{1cm}p{1cm}p{1cm}p{1cm}p{1cm}p{1cm}p{1cm}p{1cm}} 
\hline
&&&&&&&&&\\[\BD]
  && \mc{9}{c}{MVA M{\footnotesize ETHOD}} \\[\AD]
  \mc{2}{c}{C{\footnotesize RITERIA}} 
                       & Cuts & Likeli-hood & PDE-RS~/ k-NN & PDE-Foam & H-Matrix & Fisher /~LD & MLP & BDT & Rule-Fit & SVM \\
&&&&&&&&&&\\[\BD]
\hline
&&&&&&&&&&\\[\BD]
$~~~~~$ Perfor- 
             & No\;or\;linear ~~~~~~ correlations   
                       & \OK & \Good   & \OK   & \OK   & \OK & \Good & \Good & \OK & \Good & \OK \\
mance        & Nonlinear ~~~~~~~~~~~ correlations
                       & \Bad & \Bad   & \Good & \Good & \Bad & \Bad & \Good & \Good & \Good & \Good \\
&&&&&&&&&&\\[\BD]                                       
\hline                                                 
&&&&&&&&&&\\[\BD]                                       
             & Training                                
                       & \Bad  & \Good & \Good & \Good & \Good   & \Good & \OK & \OK & \OK & \Bad\\
\rs{Speed}   & Response                                
                       & \Good & \Good & \Bad  & \OK   & \Good    & \Good & \Good & \OK & \Good & \OK \\
&&&&&&&&&&\\[\BD]                                       
\hline                                                 
&&&&&&&&&&\\[\BD]                                       
Robust-                                                
             & Overtraining                            
                       & \Good & \OK   & \OK   & \OK   & \Good     & \Good & \OK & \OK\footnote{Note, in earlier versions we had rated this as bad. However, only little care has to be taken to use ``small trees'' with restricted maximum depth and reasonably large leaf nodes (O(1\%) of the training sample) to literally eliminate overtraining for all cases} & \OK & \Good \\
ness         & Weak variables                          
                       & \Good & \OK   & \Bad  & \Bad  & \Good     & \Good & \OK & \Good & \OK & \OK \\
&&&&&&&&&&\\[\BD]                                       
\hline                                                 
&&&&&&&&&&\\[\BD]                                       
\mc{2}{l}{Curse of dimensionality}                     
                       & \Bad & \Good  & \Bad  & \Bad  & \Good    & \Good & \OK & \OK & \OK \\
&&&&&&&&&&\\[\BD]                                       
\hline                                                 
&&&&&&&&&&\\[\BD]                                       
\mc{2}{l}{Transparency}                                
                       & \Good & \Good & \OK   & \OK   & \Good    & \Good & \Bad & \Bad & \Bad & \Bad \\[\AD]
\hline      
\end{tabular*}
}
\caption[.]{\captionfont Assessment of MVA method properties. The symbols stand for 
        the attributes ``good'' ($\star\star$), ``fair'' ($\star$) and ``bad'' ($\circ$).
        ``Curse of dimensionality'' refers to the ``burden'' of required increase in 
        training statistics and processing time when adding more input variables. See also 
        comments in the text. The FDA method is not listed here since its properties
        depend on the chosen function. }
\label{tab:classQA}
\end{table}
For problems that require a high degree of optimisation
and allow to use a large number of input variables, complex nonlinear methods
like neural networks, the support vector machine, boosted decision trees and/or 
RuleFit are more appropriate. 

Very involved multi-dimensional variable correlations with strong nonlinearities 
are usually best mapped by the multidimensional probability density estimators 
such as PDE-RS, k-NN and PDE-Foam, requiring however a reasonably low number of 
input variables.

For RuleFit classification we emphasise that the TMVA implementation differs 
from Friedman-Popescu's original code~\cite{RuleFit}, with slightly better robustness 
and out-of-the-box performance for the latter version. In particular, the behaviour of the 
original code with respect to nonlinear correlations and the curse of dimensionality would 
have merited two stars.\footnote
{
   An interface to Friedman-Popescu's original code can be requested from the TMVA
   authors. See Sec.~\ref{sec:friedman}.
} 
We also point out that the excellent performance for by majority linearly correlated 
input variables is achieved somewhat artificially by adding a Fisher-like term to the 
RuleFit classifier (this is the case for both implementations, \cf\  Sec.~\ref{sec:rulefit}
on page~\pageref{sec:rulefit}). 

In Fig~\ref{fig:usingtmva:rejBvsS} (page~\pageref{fig:usingtmva:rejBvsS}) 
we have shown the example of ROC-curves obtained from various linear and non-linear classifiers 
on a simple toy Monte Carlo sample of linear correlated Gaussian distribution functions. 
Here all classifiers capable of dealing with such type of correlations
are equally performing. Only the projective likelihood method, which ignores all correlations,
performs significantly worse (it recovers optimum performance after prior decorrelation
of the variables). For such a problem, a Fisher (aka linear) discriminant is optimal by 
construction, and the non-linear methods effectively reduce in the training to linear discriminants
achieving competitive, but somewhat less robust performance. 
\begin{figure}[p]
\begin{center}
  \includegraphics[width=1.00\textwidth]{plots/Schachbrett_both.png}

  \vspace{0.5cm}
  \includegraphics[width=0.65\textwidth]{plots/Schachbrett_ROC.png}
\end{center}
\caption[.]{The top left plot shows the two-dimensional Gaussian-distributed 
  signal (blue) and background (red) distributions for the ``Schachbrett'' toy 
  Monte Carlo sample. In the top right plot is drawn as an example the Support Vector
  Machine's classification response in terms of the weights given to events in
  the two dimensional plane. Dark red signifies large MVA output values,
  \ie\   signal like event regions. The bottom plot gives the ROC curves 
  obtained with several TMVA classifiers for this example after training. 
  The theoretically best ROC curve is known and shown by the thick red outer
  line.  While all the non-linear classification methods (after
  carefully tuning their parameters) are able to approximate the
  theoretical limit, the linear methods strongly underperform. }
\label{fig:conclusions:schachbrett}
\end{figure}

For another academic toy Monte Carlo example however, which exhibits strongly non-linear 
correlations in form of a ``Schachbrett'' (chess board with two-dimensional 
Gaussian-distributed signal and background spots -- see Fig.~\ref{fig:conclusions:schachbrett}), 
the limitations of the linear classifiers, being unable to capture the features of the data,
become manifest. Non-linear classifiers, once appropriately optimised such as choosing 
a sufficiently complex network architecture for the MLP, or the proper kernel functions 
in the SVM, all non-linear methods reproduce or approximate the theoretical limit of 
this classification problem (indicated by the thick red line in the ROCc curve in 
the lower plot of Fig.~\ref{fig:conclusions:schachbrett}).

\section{TMVA implementation status summary for classification and regression}
\label{sec:classifierSummary}

All TMVA methods are fully operational for user analysis, requiring training, testing,
evaluating and reading for the application to unknown data samples. Additional 
features are optional and -- despite our attempts to provide a fully transparent analysis
-- not yet uniformly available. A status summary is given in Table~\ref{tab:methodStatus}
and annotated below.

Although since TMVA 4 the framework supports multi-dimensional MVA outputs it has not yet 
been implemented for classification. For regression, only a few methods are fully multi-target 
capable so far (see Table \ref{tab:methodStatus}). 

Individual event-weight support is now commonly realised, only missing (and not 
foreseen to be provided) for the less recommended neural network CFMlpANN. Support of
negative event weights occurring, \eg, in NLO MC requires more scrutiny as discussed in 
Sec.~\ref{sec:NegativeEventWeights} on page~\pageref{sec:NegativeEventWeights}.

Ranking of the input variables cannot be defined in a straightforward manner for all 
MVA methods. Transparent and objective variable ranking through performance comparison 
of the MVA method under successive elimination of one input variable at a time is under 
consideration (so far only realised for the naive-Bayes likelihood classifier).

Standalone C++ response classes (not required when using the Reader application) 
are generated by the majority of the classifiers, but not yet for regression analysis. 
The missing ones for PDE-RS, PDE-Foam, k-NN, Cuts and CFMlpANN will only be considered on explicit request.

The availability of help messages, which assist the user with the performance tuning 
and which are printed on standard output when using the booking option 'H', is 
complete. 

Finally, custom macros are provided for some MVA methods to analyse specific 
properties, such as the fidelity of likelihood reference distributions or the 
neural network architecture, etc. More macros can be added upon user request.
\begin{sidewaystable}
\begin{center}
{\small
\setlength{\tabcolsep}{0.3pc}
\begin{tabular}{lcccccccccccccc}\hline
&&&&&&&\\[\BD]
                     & Classi- & Regress- & \mc{2}{c}{Multi-class/target} & \mc{2}{c}{Treats event weights:}  & Variable & Standalone & Help & Custom \\
\rs{MVA method}      & fication
                            & ion 
                                   & classification
                                          & regression
                                                 &  positive 
                                                        & negative 
                                                               & ranking 
                                                                      & response class  
                                                                             & messages 
                                                                                    & macros\\[\AD]
\hline
&&&&&&&\\[\BD]
Cut optimisation     & \YES & \NO  & \NO  & \NO  & \YES & \NO  & \NO  & \NO  & \YES & \NO  \\[\AD]
Likelihood           & \YES & \NO  & \NO  & \NO  & \YES & \YES & \YES & \YES & \YES & \YES \\
PDE-RS               & \YES & \YES & \NO  & \NO  & \YES & \YES & \NO  & \NO  & \YES & \NO  \\
PDE-Foam             & \YES & \YES & \YES & \YES & \YES & \YES & \YES & \NO  & \YES & \YES \\
k-NN                 & \YES & \YES & \NO  & \YES & \YES & \YES & \NO  & \NO  & \YES & \NO  \\[\AD]
H-Matrix             & \YES & \NO  & \NO  & \NO  & \YES & \NO  & \YES & \YES & \YES & \NO  \\
Fisher               & \YES & \NO  & \NO  & \NO  & \YES & \NO  & \YES & \YES & \YES & \NO  \\
LD                   & \YES & \YES & \NO  & \NO  & \YES & \NO  & \YES & \YES & \YES & \NO  \\
FDA                  & \YES & \YES & \NO  & \NO  & \YES & \NO  & \NO  & \YES & \YES & \NO  \\[\AD]
MLP                  & \YES & \YES & \NO  & \YES & \YES & \NO  & \YES & \YES & \YES & \YES \\
TMlpANN$^{(\star)}$   & \YES & \NO  & \NO  & \NO  & \YES & \NO  & \NO  & \YES & \YES & \NO  \\
CFMlpANN             & \YES & \NO  & \NO  & \NO  & \NO  & \NO  & \NO  & \NO  & \NO  & \NO  \\[\AD]
SVM                  & \YES & \NO  & \NO  & \NO  & \YES & \YES & \NO  & \YES & \YES & \NO  \\[\AD]
BDT                  & \YES & \YES & \NO  & \NO  & \YES & \YES & \YES & \YES & \YES & \YES \\
RuleFit              & \YES & \NO  & \NO  & \NO  & \YES & \NO  & \YES & \YES & \YES & \YES \\[\AD]
\hline      
&&&&&&&\\[\BD]
\end{tabular}
}
\end{center}
\vspace{-0.4cm}
\footnotesize \hspace{0.8cm}$^{(\star)}$Not a generic TMVA method $-$ interface to ROOT class 
               \code{TMultiLayerPerceptron}.
\caption[.]{\captionfont Status of the methods with respect to various TMVA features. See 
            text for comments. Note that the column ``Standalone response class'' only refers
            to classification. It is yet unavailable for regression. }
\label{tab:methodStatus}
\end{sidewaystable}

\section{Conclusions and Plans}
\label{sec:conclusions}

TMVA is a toolkit that unifies highly customisable multivariate (MVA) classification and 
regression algorithms in a single framework thus ensuring convenient use and an objective 
performance assessment. It is designed for machine learning applications in high-energy 
physics, but not restricted to these. Source code and library of TMVA-v.3.5.0 and higher
versions are part of the standard ROOT distribution kit (v5.14 and higher). The 
newest TMVA development version can be downloaded from Sourceforge.net at 
\urlsm{http://tmva.sourceforge.net}.

This Users Guide introduced the main steps of a TMVA analysis allowing a user to optimise 
and perform her/his own multivariate classification or regression. Let us recall the main 
features of the TMVA design and purpose:
\begin{itemize}

\item TMVA works in transparent factory mode to allow an unbiased 
      performance assessment and comparison: all MVA methods
      see the same training and test data, and are evaluated following 
      the same prescription. 

\item A complete TMVA analysis consists of two steps:
      \begin{enumerate}

      \item {\bf Training:}
            the ensemble of available and optimally customised MVA methods are 
            trained and tested on independent signal and background data samples; the 
            methods are evaluated and the most appropriate (performing and concise) 
            ones are selected.

      \item {\bf Application:}
            selected trained MVA methods are used for the classification of data samples with
            unknown signal and background composition, or for the estimate of unknown 
            target values (regression).  

      \end{enumerate}
      
\item A Factory class object created by the user organises the 
      customisation and interaction with the MVA methods for the training, 
      testing and evaluation phases of the TMVA analysis. The training results 
      together with the configuration of the methods are written to
      result (``weight'') files in XML format.

\item Standardised outputs during the Factory running, and dedicated ROOT 
      macros allow a refined assessment of each method's behaviour and 
      performance for classification and regression.
     
\item Once appropriate methods have been chosen by the user, they can be 
      applied to data samples with unknown classification or target values. Here, 
      the interaction with the methods occurs through a Reader class 
      object created by the user. A method is booked by giving the path to its
      weight file resulting from the training stage. Then, inside
      the user's event loop, the MVA response is returned by the Reader for 
      each of the booked MVA method, as a function of the event values of the 
      discriminating variables used as input for the classifiers. Alternatively,
      for classification, the user may request from the Reader the probability 
      that a given event belongs to the signal hypothesis and/or the event's Rarity.

\item In parallel to the XML files, TMVA generates standalone C++ classes after the 
      training, which can be used for classification problems (feature not available 
      yet for regression). Such classes are available for all classifiers except for 
      cut optimisation, PDE-RS, PDE-Foam, k-NN and the old CFMlpANN.

\end{itemize}
We give below a summary of the TMVA methods, outlining the current state of their
implementation, their advantages and shortcomings.
\begin{itemize}

\item {\em Rectangular Cut Optimisation} \\
      The current implementation is mature. 
      It includes speed-optimised range searches using binary 
      trees, and three optimisation algorithms: Monte Carlo sampling,
      a Genetic Algorithm and Simulated Annealing. In spite of these 
      tools, optimising the cuts for a large number of discriminating variables 
      remains challenging. The user is advised to reduce the available 
      dimensions to the most significant variables (\eg, using a principal
      component analysis) prior to optimising the cuts.

\item {\em Likelihood} \\
      Automatic non-parametric probability density function (PDF) estimation through 
      histogram smoothing and interpolation with various spline functions and 
      quasi-unbinned kernel density estimators is implemented. The PDF description
      can be individually tuned for each input variable. 

\item {\em PDE-RS} \\
      The multidimensional probability density estimator (PDE) approach is in an advanced 
      development stage featuring adaptive range search, several kernel estimation 
      methods, and speed optimised range search using event sorting in binary trees.
      It has also been extended to regression.

\item {\em PDE-Foam} \\
      This new multidimensional PDE algorithm uses self-adapting phase-space 
      binning and is a fast realisation of PDE-RS in fixed volumes, which are
      determined and optimised during the training phase. Much work went 
      into the development of PDE-Foam. It has been thoroughly tested, and 
      can be considered a mature method. PDE-Foam performs classification 
      and regression analyses.

\item {\em k-NN} \\
      The k-Nearest Neighbour classifier is also in a mature state, featuring
      both classification and regression.
      The code has been well tested and shows satisfactory results. 
      With scarce training statistics it may slightly underperform in comparison
      with PDE-RS, whereas it is significantly faster in the application to large
      data samples.

\item {\em Fisher and H-Matrix}\\
      Both are mature algorithms, featuring linear discrimination for classification 
      only. Higher-order correlations are taken care of by FDA (see below).

\item {\em Linear Discriminant (LD)}\\
      LD is equivalent to Fisher but providing both classification and linear 
      regression.

\item {\em Function Discriminant Analysis (FDA)} \\
      FDA is a mature algorithm, which has not been extensively used yet. It 
      extends the linear discriminant to moderately non-linear correlations that
      are fit to the training data.

\item {\em Artificial Neural Networks} \\
      Significant work went into the 
      implementation of fast feed-forward multilayer perceptron algorithms
      into TMVA. Two external ANNs have been integrated as fully independent
      methods, and another one has been newly developed for TMVA, with emphasis 
      on flexibility and speed. The performance of the latter ANN (MLP) has been 
      cross checked against the Stuttgart ANN (using as an example $\tau$ 
      identification in ATLAS), and was found to achieve competitive performance.
      The MLP ANN also performs multi-target regression.

\item {\em Support Vector Machine}\\
      SVM is a relatively new multivariate analysis algorithm with a strong 
      statistical background. It performs well for nonlinear discrimination 
      and is insensitive to overtraining. Optimisation is 
      straightforward due to a low number of adjustable parameters (only two 
      in the case of Gaussian kernel). The response speed is slower than 
      for a not-too-exhaustive neural network, but comparable with other 
      nonlinear methods. SVM is being extended to multivariate regression.

\item {\em Boosted Decision Trees}\\
      The BDT implementation has received constant attention over the years of
      its development. The current version includes additional features like 
      bagging or gradient boosting, and manual or automatic pruning of 
      statistically insignificant nodes. It is a highly performing MVA method
      that also applies to regression problems. 

\item {\em RuleFit} \\
      The current version has the possibility to run either the original 
      program written by J.~Friedman~\cite{RuleFit} or an independent TMVA 
      implementation. The TMVA version has been improved both in speed and 
      performance and achieves almost equivalent results with respect to the 
      original one, requiring however somewhat more tuning.

\end{itemize}

The new framework introduced with TMVA~4 provides the flexibility to combine
MVA methods in a general fashion. Exploiting these capabilities for classification
and regression however requires to create so-called committee methods for each 
combination. So far, we provide a generalised Boost method, allowing to boost
any classifier by simply setting the variable \code{Boost_Num} in the configuration
options to a positive number (plus possible adjustment of other configuration
parameters). The result is a potentially powerful committee method unifying
the excellent properties of boosting with MVA methods that already represent 
highly optimised algorithms. 

Boosting is not the only combination the new framework allows us to establish. 
TMVA now also supports the categorisation of classification according to
the phase space, which allows a better modelling and hence simplifies the 
mapping of the feature space. This particularly improves simple methods, such 
las projective likelihood and linear discriminants. 

\subsubsection*{Acknowledgements}
\addcontentsline{toc}{section}{Acknowledgements}
\label{sec:Acknowledgments}

\begin{details}
The fast growth of TMVA would not have been possible without the 
contribution and feedback from many developers (also co-authors of this Users Guide)
and users to whom we are indebted. 
%
We thank in particular the CERN Summer students Matt Jachowski (Stanford U.) for the 
implementation of TMVA's MLP neural network, and Yair Mahalalel (Tel Aviv U.) for a 
significant improvement of PDE-RS. The Support Vector Machine has been contributed to 
TMVA by Andrzej Zemla and Marcin Wolter (IFJ PAN Krakow), and the k-NN method has 
been written by Rustem Ospanov (Texas U.). 
%
We are grateful to
%
Lucian Ancu,
Doug Applegate, 
Kregg Arms, 
Ren\'e Brun and the ROOT team, 
Andrea Bulgarelli,
Marc Escalier,
Zhiyi Liu, 
Colin Mclean,
Elzbieta Richter-Was, 
Alfio Rizzo,
Lydia Roos,
Vincent Tisserand,
Alexei Volk,
Jiahang Zhong
%
for helpful feedback and bug reports. Thanks also to Lucian Ancu for 
improving the plotting macros.
\end{details}
