Stacking

Stacking is an advanced feature of NeuroMiner that takes multiple learned decision values, combines them, and then learns again from this meta information (see this blog post for an in-depth overview). This technique has provided better solutions than using single models to learn the classification or regression problem. Stacking can be done in NeuroMiner if you have different data modalities (e.g., you can learn from clinical and neuroimaging separately and then stack the learners) or if you have run separate analyses on one (or more) modalities (e.g., if you run a linear SVM and then a Random Forest and then combine the learned models). If you have more than one modality or have run separate analyses on the same data, then an option will appear in the ”Define parameter template” menu entitled ”1| Define meta-learning/stacking options”.

1 | Activating stacking using prediction generated by other analyses [ ]

2 | Select 1st-level analyses to provide features to stacker [ selected ]

3 | Define feature names [ ]

4 | Extract predictions from …

1 | Activating stacking using predictions generated by other analyses

Turn stacking on or off.

2 | Select 1st-level analyses to provide features to the stacker

Of the analyses that have been conducted, select the models that you want to stack.

3 | Define feature names [ Var ’feats’]

Optionally define feature names for each of the models that you want to stack.

4 | Extract predictions from …

This gives the option to extract predictions from either the CV1 training data or the CV1 test data (the default). If CV1 test data is used, the higher-level algorithms should not be combined with wrappers because of overfitting. Cross-CV probabilistic feature extraction is helpful in this regard to prevent overfitting.

Once stacking is turned on and these options are defined, then all parameter settings now relate to the stacked models (e.g., all the preprocessing and training settings). These settings will need to be changed because the data inputs are now the decision scores for each of the models (i.e., you’ll have to change the preprocessing accordingly). Then you can return to the main menu, initialize the stacking models, and train as normal. The results will then relate to the stacked models rather than the original data.

It’s important to note that the results now pertain to the decision scores of the stacked models and therefore the interpretation of results is based on the models and not the original data (i.e., you can see how one model performed over another, but not information related to the individual variables).

Please note that stacking does not work with external validation data.