The difference between two classifiers (algorithms) can be very small; however there are no two classifiers whose accuracies are perfectly equivalent. By using an null hypothesis significance test (NHST), the null hypothesis is that the classifiers are equal. However, the null hypothesis is practically always false! By rejecting the null hypothesis NHST indicates that the …
Continue reading The importance of the region of practical equivalence (ROPE)
This post is about Bayesian nonparametric tests for comparing algorithms in ML. This time we will discuss about Python module signrank in bayesiantests (see our GitHub repository). It computes the Bayesian equivalent of the Wilcoxon signed-rank test. It return probabilities that, based on the measured performance, one model is better than another or vice versa …
Continue reading Bayesian Signed-Rank Test for comparing algorithms in machine learning