bayesian methods for machine learning

Bayesian Methods for Machine Learning
Zoubin Ghahramani
* Tutorial Notes Now Available Here *
Topic
Many topics in Machine Learning (e.g. kernel methods, clustering, semi-
supervised learning, feature selection, active learning, reinforcement learning)
can be addressed within the framework of Bayesian statistics. While the
proportion of work in machine learning based on statistical modelling has
grown over the past few years, there remains a good degree of skepticism
with respect to taking a fully Bayesian approach. This tutorial aims at
introducing fundamantal topics in Bayesian statistics as they apply to machine
learning problems, and addressing some misconceptions about Bayesian
approaches. The tutorial will also attempt to present a balanced view of the
limitations of Bayesian approaches. Finally the tutorial will delve into some of
the practical issues in current Bayesian machine learning including the role of
approximation algorithms, sampling methods, and nonparameterics.
Intended audience
The tutorial is intended for the broad ICML community. Little prior knowledge
is assumed (other than basic probability theory). The participants will hopefully
get both a big picture of Bayesian approaches to machine learning and some
insight into specific state-of-the-art methods.
Detailed Outline
[total 180 minutes, outline subject to revision]
1. Three canonical problems [10 minutes]
o Linear Classification
o Coin Toss
o Clustering with Gaussian Mixtures
2. Foundations [30 minutes]
o Representing beliefs and the Cox Axioms
o Dutch Book Theorem
o Asymptotic Convergence and Consensus

o Occam's Razor
o Priors: Objective, Subjective, Hierarchical and Empirical Bayes
o Exponential Family and Conjugate Priors
o How to choose priors?
3. Intractability [10 minutes]
o Bayesian inference in Gaussian mixtures and linear classifiers
o Hidden variables, parameters and partition functions
4. Approximation Tools [40 minutes]
o BIC
o Laplace Approximation
o Variational Approximations
o MCMC
o Exact Sampling
break
5. Feature Selection, Model Selection and Bayesian Methods [20
minutes]
o Do we need to select features?
o Automatic Relevance Determination
o Model selection criteria and model averaging
6. Bayesian Discriminative Modelling [20 minutes]
o Myth: Bayesian methods = Generative models
o Bayes Point Machines vs Support Vector Machines
o Bayesian Neural Networks
7. From Parametric to Nonparametric Bayes [20 minutes]
o Gaussian Processes
o Dirichlet Processes and Infinite Mixtures
o Other non-parametric Bayesian models
8. Further Topics [15 minutes]
o Bayesian Active Learning and Bayesian Decision Theory
o Bayesian Semi-supervised Learning

o Reconciling Bayesian and Frequentist Views
9. Open Discussion of Limitations and Criticisms [10 minutes]
o Philosophical
o Practical
o Computational
10. Other Questions from the Audience
Format
The format will be data-projected slides; I will also occasionally use the
whiteboard and have some simple Matlab demos to illustrate some ideas.
However, the focus won't be on algorithms but rather on concepts.
Presenter
Zoubin Ghahramani is a Reader in Machine Learning at the Gatsby Unit in
London, and an Associate Research Professor at CALD at CMU. He has
given tutorials at NIPS, ICANN, the Machine Learning Summer School in
Canberra, and various other summer schools. He is interested in Bayesian
machine learning, computational approaches to sensorimotor control, and
applications of machine learning to bioinformatics.
Zoubin Ghahramani
Gatsby Computational Neuroscience Unit
University College London
17 Queen Square, Room 403
London WC1N 3AR
United Kingdom
Tel +44 (0)20 7679 1199
Fax +44 (0)20 7679 1173
Email zoubin "AT" gatsby.ucl.ac.uk
/>