Prsentation PowerPoint

1
CHAP 8 MODEL INFERENCE AND AVERAGING 8.1
Introduction 8.2 The bootstrap and maximum
likelihood methods 8.3 Bayesian methods 8.4
Relationship between the bootstrap and bayesian
inference 8.5 The EM algorithm 8.6 MCMC for
sampling from the posterior 8.7 Bagging 8.8
Model averaging and stacking 8.9 Stochastic
search bumping
2
8.1 Introduction General exposition of
the maximum likelihood approach and the Bayesian
method for inference ? Link with the
bootstrap Some related techniques for model
averaging and improvement Committee
methods, Bagging, Stacking, Bumping
3
8.2 The bootstrap and maximum likelihood
method Example one-dimensional smoothing
(N50 training data) Fit a cubic spline with 3
knots
4
8.2 The bootstrap and maximum likelihood
method Least squares
where ?
?
where
5
8.2 The bootstrap and maximum likelihood
method Nonparametric bootstrap Repeat B200
times - draw a dataset of N50 with replacement
from the training data zi(xi,yi) - fit a cubic
spline
Construct a 95 pointwise confidence
interval At each xi compute the mean and find
the 2,5 and 97,5 percentiles
Conclusion least squares ? nonparametric
bootstrap
6
8.2 The bootstrap and maximum likelihood
method Parametric bootstrap We assume that
the model errors are Gaussian Repeat B200
times - draw a dataset of N50 with replacement
from the training data zi(xi,yi) - fit a cubic
spline on zi - simulate new responses
zi(xi,yi) - fit a cubic spline on zi
Construct a 95 pointwise confidence interval At
each xi compute the mean and find the 2,5 and
97,5 percentiles
7
8.2 The bootstrap and maximum likelihood
method Parametric bootstrap
Conclusion least squares parametric
bootstrap as B ? ? (only because of Gaussian
errors)
8

• 8.2 The bootstrap and maximum likelihood
  method
• Maximum likelihood
• Probability density
• Likelihood function
• Log-likelihood function
• PRINCIPLES
• estimate ? by the value that maximises the
  log-likelihood function
• use likelihood function to assess the
  precision of .
• where
• approximated by or

9

• 8.2 The bootstrap and maximum likelihood
  method
• Maximum likelihood
• (1-2?) CONFIDENCE INTERVAL
• use the approximated distribution for
  or
• or
• use the approximated distribution
  where p is the number of
• components in ?

10

• 8.2 The bootstrap and maximum likelihood
  method
• Maximum likelihood
• EXAMPLE
• ? (?,?2) and yi N(h(x)T ?,?2)
  
• - Likelohood
• Log-likelohood
• et ? et
• I(?) is block-diagonal and the block
  corresponding to ? is
• ?

Conclusion least squares maximum likelihood
11
8.2 The bootstrap and maximum likelihood
method Bootstrap and maximum likelihood
Advantage of the bootstrap over the maximum
likelihood allows to compute maximum likelihood
estimates of standard errors in settings where no
formulas are available. Example If the number
and positions of knots, ?, are not fixed but
adaptively chosen by cross- validation, bootstrap
allows to account for the adaptive choice of ?
when computing standard errors and confidence
bands.

• Conclusions
• least squares ? nonparametric bootstrap
• least squares parametric bootstrap as B ? ?
  (only because
  of Gaussian errors)
• least squares maximum likelihood

12
8.3 Bayesian Methods
prior distribution
posterior distribution
sampling model
reflète notre connaissance des paramètres avant
de voir les données
représente notre connaissance des paramètres
après avoir examiné les données
expression de lincertitude avant de voir les
données incertitude restante après avoir examiné
les données exprimée par la  posterior
distribution 
13
8.3 Bayesian Methods
Prédictions
predictive distribution
au lieu de
maximum likelihood
on ne tient pas compte de lincertitude dans
lestimation des paramètres
14
8.3 Bayesian Methods
Example smoothing
Modèle
Gaussian prior
posterior est aussi gaussienne avec
15
8.3 Bayesian Methods
Example smoothing
Gaussian prior pour les paramètres ?
prior de ?(x) gaussienne avec une covariance
16
8.3 Bayesian Methods
Example smoothing
choix de la matrice de corrélation ??
pour obtenir fonction lisse (parfois besoin
dune pénalité supplémentaire comme pour
smoothing splines)
17
8.3 Bayesian Methods
Example smoothing
semblable au résultat obtenu par bootstrap
noninformative prior
posterior distribution bootstrap distribution
18
8.3 Bayesian Methods
Noninformative prior (cas gaussien)
maximum likelihood parametric bootstrap
analyse bayesienne
car modèle gaussien
modèles gaussiens noninformative prior
voir section 8.4
19
8.3 Bayesian Methods
Noninformative prior (cas général)
noninforamtive prior aussi appelé constant prior
posterior ? likelihood prior
constant prior
posterior ? likelihood
maximum likelihood analyse bayesienne
20
8.4 Relationship between the Bootstrap and the
Bayesian Inference
example
prior
posterior
posterior
idem parametric bootstrap où on génère les
échantillons z à partir dune densité N(z,1)
21
8.4 Relationship between the Bootstrap and the
Bayesian Inference
Conditions pour que parametric bootstrap
bayesien

• noninformative prior
• la log-vraissemblance ne dépend des données quà
  travers lestimateur des paramètres
• symétrie de la log-vraissemblance

example gaussien
22
8.4 Relationship between the Bootstrap and the
Bayesian Inference
Example distribution de Dirichlet
probabilité dappartenir à la catégorie j
proportion observée dans la catégorie j
prior
c-à-d
car
posterior
bootstrap