The importance of the region of practical equivalence (ROPE)

The difference between two classifiers (algorithms) can be very small; however there are no two classifiers whose accuracies are perfectly equivalent. By using an null hypothesis significance test (NHST), the null hypothesis is that the classifiers are equal. However, the null hypothesis is practically always false! By rejecting the null …

Continue reading

Bayesian Signed-Rank Test for comparing algorithms in machine learning

This post is about Bayesian nonparametric tests for comparing algorithms in ML. This time we will discuss about Python module signrank in bayesiantests (see our GitHub repository). It computes the Bayesian equivalent of the Wilcoxon signed-rank test. It return probabilities that, based on the measured performance, one model is better …

Continue reading