A Tutorial on Learning with Bayesian Networks Part 1

1
A Tutorial on Learning with Bayesian Networks
Part 1

• Paper by David Heckerman
• Presented by Michael Verdicchio
• 16 April 2007

2
Overview

• Well cover this tutorial over 3-4 talks
• No prior experience/knowledge assumed
• Talk 1 Background, probability, Bayesian
  approach to probability
• Talk 2 Probabilistic inference algorithms,
  learning probabilities, incomplete data
• Talks 3/4 Learning parameters and structure,
  model selection, causal relationships, pertinent
  topics

3
Todays Talking Points

• Introduction Why and what
• Review of classical probability theory
• Bayesian approach to classical probability and
  statistics

4
Why This Tutorial?

• Computational Systems Biology lab will benefit
  from knowledge and application of Bayesian
  approaches to biomedical data
• We will expand our repertory of data analysis
  techniques for our own research
• We will be better able to critically evaluate the
  works of others

5
What is a Bayesian Network (BN)?

• Graphical model for probabilistic relationships
  among a set of variables
• Many data analysis techniques exist, so what
  advantages do BNs have?

6
Advantages of BNs

• BNs can readily handle incomplete data
• BNs allow learning about causal relationships
• BNs facilitate combination of domain knowledge
  with data
• Bayesian methods with BNs and other models avoid
  the over fitting of data

7
Todays Talking Points

• Introduction Why and what
• Review of classical probability theory
• Bayesian approach to classical probability and
  statistics

8
Some Basic Definitions

• The sample space is the collection of all
  outcomes for a well-defined experiment
• Every subset of a sample space is called an
  event, with singleton subsets called elementary
  events
• A function assigning a real number p(E) to each
  event E in a sample space O is called a
  probability function
• The pair (O,p) is called a probability space

9
Theorem

• Let (O,p) be a probability space. Then
• p(O) 1
• 0 p(E) 1 for every E in O
• For E and F in O s.t.
• Example Choose a queen or a king from a deck of
  cards

10
Definitions

• Let E and F be events such that p(F) ? 0. Then
  the conditional probability of E given F, denoted
  p(EF), is given by
• Two events E and F are conditionally independent
  if one of the following holds
• 1. p(E) ? 0, p(F) ? 0 and p(EF) p(E)
• 2. p(E) 0 or p(F) 0

11
Total Probability

• Suppose we have n events such that each pair-wise
  intersection is empty and the union of all events
  is the sample space
• Such events are mutually exclusive and
  exhaustive, and for any other event F,
• and if p(Ei) ? 0 for all i,

12
Bayes Theorem

• Given two events E and F s.t. p(E) ? 0 and p(F) ?
  0, we have the following
• Given n elementary events E1, E2, , En s.t.
  p(Ei) ? 0 for all i, we have for 1 i n,

13
Todays Talking Points

• Introduction Why and what
• Review of classical probability theory
• Bayesian approach to classical probability and
  statistics

14
Notation
15
Classical------------------Bayesian

• Probability is a physical property
• Requires repeated trials for probability

• Probability is ones degree of belief
• Only cares about the probability of next trial

16
The Thumbtack Problem

• In the classical analysis, we estimate the
  physical probability of heads from the N
  observations, using this probability for the
  N1th toss
• In the Bayesian analysis, we also assess some
  physical probability of heads, but we encode our
  uncertainty using Bayesian probabilities, and use
  the rules of probability to compute our
  probability of heads on the N1th toss

17
The Thumbtack Problem

• We define T to be a variable whose values ?
  correspond to the possible true values of the
  physical probability of tossing heads
• We express uncertainty about T using the
  probability density function p(??), where ?
  represents a state of information
• In Bayesian terms, the thumbtack problem reduces
  to computing p(xN1D,?) from p(??)

18
The Thumbtack Problem

• First, use Bayes rule to get the probability
  distribution for T given D and ?
• To make predictions, average over possible values
  of T to determine the probability for the N1th
  toss

19
The Thumbtack Problem

• In the Bayesian approach, D is fixed and we
  imagine all values of ? that could have been
  generated
• Given ?, the estimate of the physical probability
  is just ? itself
• Nonetheless, we are uncertain about ?, and so our
  final estimate is the expectation of ?

20
The Thumbtack Problem

• In contrast, in the classical approach, ? is
  fixed and we imagine all data that may be
  generated by sampling the ? distribution
• Each data set will occur with some probability
  and produce an estimate
• An estimator is chosen to balance bias and
  variance
• This estimator is applied to observed data

21
Coming up in future talks

• Bayesian networks (construction)
• Inference in a Bayesian network
• Learning probabilities in a Bayesian network
• Much, much more!