Visualization options

In NeuroMiner, visualization refers to the representation of the model weights for each feature. In the case of neuroimaging, this means that the brain maps are ’visualized’. This setting gives the user the option to derive Z scores and P values using permutation analysis.

The magnitude of the feature weights do not necessarily inform the user about the statistical significance (i.e., the degree to which the result may replicate)(see Bilwaj & Davatsikos, 2013]). To do this, permutation analyses can be conducted to create an empirical null distribution of weights for each feature and then the observed weight is compared to this distribution. This is described in the supplementary material of Koutsouleris et al. (2016).

The method is based on Golland & Fischl (2003) and involves the permutation of the outcome labels, features, or both. For each permutation, the models are retrained in the cross-validation framework using the respective feature/label subsets obtained from the observed-label analyses. For each permutation, the predictions are accumulated from the random models into a permuted ensemble prediction for each CV2 subject. Thus, a null distribution of out-of-training classification performance (BAC) for the prediction models is constructed. The significance of the observed out-of-training BAC is calculated as the number of events where the permuted out-of-training BAC is higher or equal to the observed BAC divided by the number of permutations performed. The significance of the model can then be determined by using a p-threshold (e.g., p>0.05).

When the option is selected, you will be prompted to enter the number of permutations conducted (this will be dependent on sample size, analysis complexity, and computational time) and then whether you would like to have the labels, the features, or both permuted (see Golland & Fischl, 2003 for details).

Important

It is important to note that using PCA with this method will result in overly conservative p-values and is not recommended.

Additionally, If you would like to test the significance of covariate correction methods (e.g., site effects removal), there is an additional option to apply the permutation analyses to the covariates. For each permutation, the preprocessing is rerun with the permuted covariate values and then the models are retrained using the preprocessed features from different permutations. The significance calculation is the same as the labels and features permutation analyses described above. To use this method, please specify in the visualization options.