1 / 37

Basic Concepts of Discrete Probability

- (Theory of Sets Continuation)

Functions

- If X is a set and Y is a set, and there is a

sequence of well-specified operations for

assigning a well-defined object to

every element , and by applying these

sequence of operations to every member of a set X

we obtain a set Y, then this sequence of

operations forms a rule, which is commonly known

as function.

Functions

- The set X is called the domain of the function,

and the set Y is called the range. - Functions that have numerical values ere called

numerical functions. - A numerical function establishing the

correspondence between the elements of a set

having a finite number of elements and a set of

natural numbers will be of our particular

interest

The number of elements in a set

- The number of elements in a set may or may not be

finite. - If the elements of the set can be placed in

one-to-one correspondence with the set of natural

numbers 1, 2, 3,, we say that the set has a

denumerable or countable number of elements

otherwise the set is nondenumerable (e.g., a set

of points on any line or the set of points in

0,1).

The number of elements in a set

- If a set is denumerable and

is a finite number, we say that the power of

the set X (the cardinality of the set X) is X

(the number of elements in the set X). - If a set is equivalent to the set of points in

0,1 that is, it is denumerable, we say that the

set has the power of continuum.

The number of elements in a set

- The following properties hold for finite sets
- If A and B are two disjoint sets, then
- For any finite sets A, B, and C

Basic Concepts of Discrete Probability

- Elements of the Probability Theory

Sample Space

- When probability is applied to something, we

usually mean an experiment with certain outcomes. - An outcome is any one of the possibilities that

may be expected from the experiment. - The totality of all these outcomes forms a

universal set which is called the sample space.

Sample Space

- For example, if we checked occasionally the

number of people in this classroom on Tuesday

from 7pm to 9-45pm, we should consider this an

experiment having 15 possible outcomes

0,1,2,,13,14 that form a universal set. - 0 nobody is in the classroom, 14 all

students taking the Information Theory Class and

the instructor are in the classroom

Sample Space

- A sample space containing at most a denumerable

number of elements is called discrete. - A sample space containing a nondenumerable number

of elements is called continuous.

Sample Space

- A subset of a sample space containing any number

of elements (outcomes) is called an event. - Null event is an empty subset. It represents an

event that is impossible. - An event containing all sample points is an event

that is certain to occur.

Probability Measure

- The probability measure is a specific type of

function which can be associated with sets. - Since an experiment is defined so that to each

possible outcome of this experiment there

corresponds a point in the sample space, the

number of outcomes of the experiment is assumed

to be at most denumerable.

Probability Measure

- Let us consider an event of interest A as the set

of outcomes ak. - Let a real function m(ak ) be the probability

measure of the outcome ak. - The probability measure of an event is defined as

the sum of the probability measures associated

with all the outcomes ak of that event. It is

also referred to as the additive probability

measure

Probability Measure

- The additive probability measure has the

following important property the measure

associated with the union of two disjoint sets is

equal to the sum of their individual measures.

Probability Measure

- Two events A and B are called disjoint if they

contain no outcome in common two disjoint events

cannot happen simultaneously. The probability

measure has the following assumed properties

Probability Measure

- If with an

additive probability measure, then the following

relations are valid

Probability Measure

- For three disjoint sets

- For three sets in general

Frequency of events

- Let us consider a specific event Xk among all

the possible events of the experiment under

consideration. If the basic experiment is

repeated N times among which the event Xk has

appeared times, the ratio - is defined as the relative frequency of the

occurrence of the event Xk.

Frequency of events. The Probability

- If N is increased indefinitely, that is if

then, intuitively speaking, - is the probability of the event Xk.

The Probability

- The classical definition of Laplace says that the

probability is the ratio of the number of

favorable events to the total number of possible

events. - All events in this definition are considered to

be equally likely e.g., throwing of a true die

by an honest person under prescribed

circumstances - but not checking of the number of people in the

classroom.

The Probability

- The following properties are important

The Probability

- To verify whether our empiric definition of

probability satisfies the properties of the

probability measure, we have to consider the

probability of two events. - Let A and B be the events among the ones

resulting from the experiment. Let the experiment

be repeated n times.

The Probability

- Each observation can belong to only one of the

four following categories - 1) A has occurred, but not B ? AB
- 2) B has occurred, but not A ? BA
- 3) Both A and B have occurred ? AB
- 4) Neither A nor B has occurred ? AB

The Probability

- If the number of events of each category from the

mentioned four ones is denoted by - then
- relative frequency of A

independent of B - relative frequency of

B independent of A

The Probability

- relative

frequency of either A, B or both - relative frequency of A and B

occurring together - relative frequency of A

under condition that B has occurred - relative frequency of B

under condition that A has occurred

The Probability

- When the number of experiments tends to infinity,

these relations lead to - If A and B are mutually exclusive events, then

PAB0 PABPAPB

The Probability

- Thus, we have proved that
- Let us compare the latter with the properties of

the probability measure

The Probability

- We have just shown that our empirical definitions

of the frequency and probability, respectively,

satisfy all the properties of the probability

measure.

Theorem of addition

- For two events A and B of the sample space
- The additive property of the probability measure

suggests that - If A and B are mutually exclusive events, then

Theorem of addition

- For two opposite events A and A AAU AA0,

then

Theorem of addition

- For the three events A, B, and C
- In general, for a number of events

Theorem of addition

- If the events are mutually exclusive, then
- It is also clear now that

Conditional Probability

- Let A and B be two events. The conditional

probability of event A based on the hypothesis

that event B has occurred is defined by the

following relation

Conditional Probability

- Returning to the example of two events
- 1) A has occurred, but not B ? AB ?n1
- 2) B has occurred, but not A ? BA ?n2
- 3) Both A and B have occurred ? AB ?n3
- 4) Neither A nor B has occurred ? AB ?n4

Conditional Probability

- The two events A and B are called mutually

independent if PABPA and PBAPB. - For mutually independent events PABPAPB.

Theorem of multiplication

- The multiplication rule for the case of two

events A and B can be obtained through the

definition of the conditional probability

PABPAPBAPBPAB. - For three events A,B, and C PABCPABPCABP

APBAPCAB

Theorem of multiplication

- For n events
- For a countable infinite number of the mutually

independent events