2021
Benavoli, Alessio; Azzimonti, Dario; Piga, Dario
Choice functions based multi-objective Bayesian optimisation Technical Report
2021.
Abstract | Links | BibTeX | Tags: bayesian nonparametric, bayesian optimisation, Gaussian Process
@techreport{Benavoli2021bb,
title = {Choice functions based multi-objective Bayesian optimisation},
author = {Benavoli , Alessio and Azzimonti, Dario and Piga, Dario },
url = {https://arxiv.org/pdf/2110.08217.pdf},
year = {2021},
date = {2021-10-18},
abstract = {In this work we introduce a new framework for multi-objective Bayesian optimisation where the multi-objective functions can only be accessed via choice judgements, such as “I pick options x1, x2, x3 among this set of five options x1, x2, . . . , x5”. The fact that the option x4 is rejected means that there is at least one option
among the selected ones x1, x2, x3 that I strictly prefer over x4 (but I do not have to specify which one). We assume that there is a latent vector function f for some dimension ne which embeds the options into the real vector space of dimension ne, so that the choice set can be represented through a Pareto set of non-dominated
options. By placing a Gaussian process prior on f and deriving a novel likelihood model for choice data, we propose a Bayesian framework for choice functions learning. We then apply this surrogate model to solve a novel multi-objective Bayesian optimisation from choice data problem.},
keywords = {bayesian nonparametric, bayesian optimisation, Gaussian Process},
pubstate = {published},
tppubtype = {techreport}
}
In this work we introduce a new framework for multi-objective Bayesian optimisation where the multi-objective functions can only be accessed via choice judgements, such as “I pick options x1, x2, x3 among this set of five options x1, x2, . . . , x5”. The fact that the option x4 is rejected means that there is at least one option
among the selected ones x1, x2, x3 that I strictly prefer over x4 (but I do not have to specify which one). We assume that there is a latent vector function f for some dimension ne which embeds the options into the real vector space of dimension ne, so that the choice set can be represented through a Pareto set of non-dominated
options. By placing a Gaussian process prior on f and deriving a novel likelihood model for choice data, we propose a Bayesian framework for choice functions learning. We then apply this surrogate model to solve a novel multi-objective Bayesian optimisation from choice data problem.
among the selected ones x1, x2, x3 that I strictly prefer over x4 (but I do not have to specify which one). We assume that there is a latent vector function f for some dimension ne which embeds the options into the real vector space of dimension ne, so that the choice set can be represented through a Pareto set of non-dominated
options. By placing a Gaussian process prior on f and deriving a novel likelihood model for choice data, we propose a Bayesian framework for choice functions learning. We then apply this surrogate model to solve a novel multi-objective Bayesian optimisation from choice data problem.
Benavoli, Alessio; de Campos, Cassio
Bayesian Kernelised Test of (In)dependence with Mixed-type Variables Conference
The 8th IEEE International Conference on Data Science and Advanced Analytics (DSAA). Porto, Portugal, 2021.
Abstract | Links | BibTeX | Tags: bayesian nonparametric
@conference{benavoli2021bayesian,
title = {Bayesian Kernelised Test of (In)dependence with Mixed-type Variables},
author = {Benavoli, Alessio and de Campos, Cassio },
url = {https://arxiv.org/abs/2105.04001},
year = {2021},
date = {2021-10-04},
booktitle = {The 8th IEEE International Conference on Data Science and Advanced Analytics (DSAA). Porto, Portugal},
abstract = {A fundamental task in AI is to assess (in)dependence between mixed-type variables (text, image, sound). We propose a Bayesian kernelised correlation test of (in)dependence using a Dirichlet process model. The new measure of (in)dependence allows us to answer some fundamental questions: Based on data, are (mixed-type) variables independent? How likely is dependence/independence to hold? How high is the probability that two mixed-type variables are more than just weakly dependent? We theoretically show the properties of the approach, as well as algorithms for fast computation with it. We empirically demonstrate the effectiveness of the proposed method by analysing its performance and by comparing it with other frequentist and Bayesian approaches on a range of datasets and tasks with mixed-type variables. },
keywords = {bayesian nonparametric},
pubstate = {published},
tppubtype = {conference}
}
A fundamental task in AI is to assess (in)dependence between mixed-type variables (text, image, sound). We propose a Bayesian kernelised correlation test of (in)dependence using a Dirichlet process model. The new measure of (in)dependence allows us to answer some fundamental questions: Based on data, are (mixed-type) variables independent? How likely is dependence/independence to hold? How high is the probability that two mixed-type variables are more than just weakly dependent? We theoretically show the properties of the approach, as well as algorithms for fast computation with it. We empirically demonstrate the effectiveness of the proposed method by analysing its performance and by comparing it with other frequentist and Bayesian approaches on a range of datasets and tasks with mixed-type variables.
Benavoli, Alessio; Azzimonti, Dario; Piga, Dario
A unified framework for closed-form nonparametric regression, classification, preference and mixed problems with Skew Gaussian Processes Journal Article
In: Machine Learning, pp. 1-39, 2021.
Abstract | Links | BibTeX | Tags: bayesian nonparametric, Gaussian Process
@article{benavoli2021,
title = {A unified framework for closed-form nonparametric regression, classification, preference and mixed problems with Skew Gaussian Processes},
author = {Alessio Benavoli and Dario Azzimonti and Dario Piga},
url = {https://link.springer.com/article/10.1007/s10994-021-06039-x},
doi = {10.1007/s10994-021-06039-x},
year = {2021},
date = {2021-09-13},
journal = {Machine Learning},
pages = {1-39},
abstract = {Skew-Gaussian processes (SkewGPs) extend the multivariate Unified Skew-Normal distributions over finite dimensional vectors to distribution over functions. SkewGPs are more general and flexible than Gaussian processes, as SkewGPs may also represent asymmetric distributions. In a recent contribution we showed that SkewGP and probit likelihood are conjugate, which allows us to compute the exact posterior for non-parametric binary classification and preference learning. In this paper, we generalize previous results and we prove that SkewGP is conjugate with both the normal and affine probit likelihood, and more in general, with their product. This allows us to (i) handle classification, preference, numeric and ordinal regression, and mixed problems in a unified framework; (ii) derive closed-form expression for the corresponding posterior distributions. We show empirically that the proposed framework based on SkewGP provides better performance than Gaussian processes in active learning and Bayesian (constrained) optimization},
keywords = {bayesian nonparametric, Gaussian Process},
pubstate = {published},
tppubtype = {article}
}
Skew-Gaussian processes (SkewGPs) extend the multivariate Unified Skew-Normal distributions over finite dimensional vectors to distribution over functions. SkewGPs are more general and flexible than Gaussian processes, as SkewGPs may also represent asymmetric distributions. In a recent contribution we showed that SkewGP and probit likelihood are conjugate, which allows us to compute the exact posterior for non-parametric binary classification and preference learning. In this paper, we generalize previous results and we prove that SkewGP is conjugate with both the normal and affine probit likelihood, and more in general, with their product. This allows us to (i) handle classification, preference, numeric and ordinal regression, and mixed problems in a unified framework; (ii) derive closed-form expression for the corresponding posterior distributions. We show empirically that the proposed framework based on SkewGP provides better performance than Gaussian processes in active learning and Bayesian (constrained) optimization
Kania, Lucas; Schürch, Manuel; Azzimonti, Dario; Benavoli, Alessio
Sparse Information Filter for Fast Gaussian Process Regression Conference
European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases ECML PKDD, 2021.
Abstract | Links | BibTeX | Tags: bayesian nonparametric, Gaussian Process
@conference{Kania2021,
title = { Sparse Information Filter for Fast Gaussian Process Regression},
author = {Kania, Lucas and Schürch, Manuel and Azzimonti, Dario and Benavoli, Alessio},
url = {https://2021.ecmlpkdd.org/wp-content/uploads/2021/07/sub_854.pdf
http://alessiobenavoli.com/wp-content/uploads/2021/07/sub_854.pdf
},
year = {2021},
date = {2021-09-01},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases ECML PKDD},
abstract = {Gaussian processes (GPs) are an important tool in machine
learning and applied mathematics with applications ranging from Bayesian
optimization to calibration of computer experiments. They constitute
a powerful kernelized non-parametric method with well-calibrated un-
certainty estimates, however, off-the-shelf GP inference procedures are
limited to datasets with a few thousand data points because of their cubic
computational complexity. For this reason, many sparse GPs techniques
were developed over the past years. In this paper, we focus on GP regression tasks and propose a new algorithm to train variational sparse GP
models. An analytical posterior update expression based on the Information Filter is derived for the variational sparse GP model. We benchmark
our method on several real datasets with millions of data points against
the state-of-the-art Stochastic Variational GP (SVGP) and sparse orthogonal variational inference for Gaussian Processes (SOLVEGP). Our
method achieves comparable performances to SVGP and SOLVEGP while
providing considerable speed-ups. Specifically, it is consistently four times
faster than SVGP and on average 2.5 times faster than SOLVEGP.},
keywords = {bayesian nonparametric, Gaussian Process},
pubstate = {published},
tppubtype = {conference}
}
Gaussian processes (GPs) are an important tool in machine
learning and applied mathematics with applications ranging from Bayesian
optimization to calibration of computer experiments. They constitute
a powerful kernelized non-parametric method with well-calibrated un-
certainty estimates, however, off-the-shelf GP inference procedures are
limited to datasets with a few thousand data points because of their cubic
computational complexity. For this reason, many sparse GPs techniques
were developed over the past years. In this paper, we focus on GP regression tasks and propose a new algorithm to train variational sparse GP
models. An analytical posterior update expression based on the Information Filter is derived for the variational sparse GP model. We benchmark
our method on several real datasets with millions of data points against
the state-of-the-art Stochastic Variational GP (SVGP) and sparse orthogonal variational inference for Gaussian Processes (SOLVEGP). Our
method achieves comparable performances to SVGP and SOLVEGP while
providing considerable speed-ups. Specifically, it is consistently four times
faster than SVGP and on average 2.5 times faster than SOLVEGP.
learning and applied mathematics with applications ranging from Bayesian
optimization to calibration of computer experiments. They constitute
a powerful kernelized non-parametric method with well-calibrated un-
certainty estimates, however, off-the-shelf GP inference procedures are
limited to datasets with a few thousand data points because of their cubic
computational complexity. For this reason, many sparse GPs techniques
were developed over the past years. In this paper, we focus on GP regression tasks and propose a new algorithm to train variational sparse GP
models. An analytical posterior update expression based on the Information Filter is derived for the variational sparse GP model. We benchmark
our method on several real datasets with millions of data points against
the state-of-the-art Stochastic Variational GP (SVGP) and sparse orthogonal variational inference for Gaussian Processes (SOLVEGP). Our
method achieves comparable performances to SVGP and SOLVEGP while
providing considerable speed-ups. Specifically, it is consistently four times
faster than SVGP and on average 2.5 times faster than SOLVEGP.
Corani, Giorgio; Benavoli, Alessio; Augusto, Joao; Zaffalon, Marco
Time series forecasting with Gaussian Processes needs priors Inproceedings
In: European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases ECML PKDD , 2021.
Links | BibTeX | Tags: bayesian nonparametric, Gaussian Process, Gaussian processes
@inproceedings{corani2020automatic,
title = {Time series forecasting with Gaussian Processes needs priors},
author = {Giorgio Corani and Alessio Benavoli and Joao Augusto and Marco Zaffalon},
url = {https://arxiv.org/abs/2009.08102},
year = {2021},
date = {2021-06-01},
booktitle = { European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases ECML PKDD },
keywords = {bayesian nonparametric, Gaussian Process, Gaussian processes},
pubstate = {published},
tppubtype = {inproceedings}
}
2020
Benavoli, Alessio; Azzimonti, Dario; Piga, Dario
Skew Gaussian Processes for Classification Journal Article
In: Machine Learning, vol. 109, pp. 1877–1902, 2020.
Abstract | Links | BibTeX | Tags: bayesian nonparametric, Gaussian Process
@article{benavoli2020skew,
title = {Skew Gaussian Processes for Classification},
author = {Alessio Benavoli and Dario Azzimonti and Dario Piga},
url = {https://arxiv.org/abs/2005.12987},
doi = {10.1007/s10994-020-05906-3},
year = {2020},
date = {2020-09-04},
journal = {Machine Learning},
volume = {109},
pages = {1877–1902},
abstract = {Gaussian processes (GPs) are distributions over functions, which provide a Bayesian nonparametric approach to regression and classification. In spite of their success, GPs have limited use in some applications, for example, in some cases a symmetric distribution with respect to its mean is an unreasonable model. This implies, for instance, that the mean and the median coincide, while the mean and median in an asymmetric (skewed) distribution can be different numbers. In this paper, we propose Skew-Gaussian processes (SkewGPs) as a non-parametric prior over functions. A SkewGP extends the multivariate Unified Skew-Normal distribution over finite dimensional vectors to a stochastic processes. The SkewGP class of distributions includes GPs and, therefore, SkewGPs inherit all good properties of GPs and increase their flexibility by allowing asymmetry in the probabilistic model. By exploiting the fact that SkewGP and probit likelihood are conjugate model, we derive closed form expressions for the marginal likelihood and predictive distribution of this new nonparametric classifier. We verify empirically that the proposed SkewGP classifier provides a better performance than a GP classifier based on either Laplace's method or Expectation Propagation. },
keywords = {bayesian nonparametric, Gaussian Process},
pubstate = {published},
tppubtype = {article}
}
Gaussian processes (GPs) are distributions over functions, which provide a Bayesian nonparametric approach to regression and classification. In spite of their success, GPs have limited use in some applications, for example, in some cases a symmetric distribution with respect to its mean is an unreasonable model. This implies, for instance, that the mean and the median coincide, while the mean and median in an asymmetric (skewed) distribution can be different numbers. In this paper, we propose Skew-Gaussian processes (SkewGPs) as a non-parametric prior over functions. A SkewGP extends the multivariate Unified Skew-Normal distribution over finite dimensional vectors to a stochastic processes. The SkewGP class of distributions includes GPs and, therefore, SkewGPs inherit all good properties of GPs and increase their flexibility by allowing asymmetry in the probabilistic model. By exploiting the fact that SkewGP and probit likelihood are conjugate model, we derive closed form expressions for the marginal likelihood and predictive distribution of this new nonparametric classifier. We verify empirically that the proposed SkewGP classifier provides a better performance than a GP classifier based on either Laplace's method or Expectation Propagation.
2015
Benavoli, Alessio; de Campos, Cassio P.
Statistical Tests for Joint Analysis of Performance Measures Inproceedings
In: Suzuki, Joe; Ueno, Maomi (Ed.): Advanced Methodologies for Bayesian Networks: Second International Workshop, AMBN 2015, Yokohama, Japan, November 16-18, 2015. Proceedings, pp. 76–92, Springer International Publishing, Cham, 2015, ISBN: 978-3-319-28379-1.
Links | BibTeX | Tags: bayesian nonparametric, hypothesis testing
@inproceedings{Benavoli2015i,
title = {Statistical Tests for Joint Analysis of Performance Measures},
author = {Benavoli, Alessio and de Campos, Cassio P.},
editor = {Suzuki, Joe and Ueno, Maomi},
url = {http://dx.doi.org/10.1007/978-3-319-28379-1_6},
doi = {10.1007/978-3-319-28379-1_6},
isbn = {978-3-319-28379-1},
year = {2015},
date = {2015-01-01},
booktitle = {Advanced Methodologies for Bayesian Networks: Second International Workshop, AMBN 2015, Yokohama, Japan, November 16-18, 2015. Proceedings},
pages = {76--92},
publisher = {Springer International Publishing},
address = {Cham},
keywords = {bayesian nonparametric, hypothesis testing},
pubstate = {published},
tppubtype = {inproceedings}
}