Advertisement
English transcript
PATRICK WINSTON: Here we are, down to the final sprint.
Three to go.
And we're going to take some of the last three, maybe two
of the last three, to talk a little bit about stuff having
to do with probabilistic approaches--
use of probability in artificial intelligence.
Now, for many of you, this will be kind of a review,
because I know many of you learned about probability over
the [? sand ?] table and every year since then.
But maybe we'll put another little twist into it,
especially toward the end of the hour when we get into a
discussion of that which has come to be called belief nets.
But first, I was driving in this morning, and I was quite
astonished to see, as I drove in, this thing here.
And my first reaction was, oh my god, it's the world's
greatest hack.
And then I decided, well, maybe it's a piece of art.
So I'd like to address the question of how I could come
to grips with that issue.
There's a distinct possibility that this thing is a
consequence of a hat, possibly the result of
some kind of art show.
And in any event, some sort of statue appeared, and statues
don't usually appear like that.
So I got the possibility of thinking about how all these
things might occur together or not occur together.
So the natural thing is to build myself some sort of
table to keep track of my observations.
So I have three columns in my table.
I've got the possibility of a statue appearing, a hack
having occurred, and some sort of art show.
And so I can make a table of all the combinations of those
things that might appear.
And I happen to have already guessed that there are going
to be eight rows in my table.
So it's going to look like this.
And this is the set of combinations in this row where
none of that occurs at all.
And down here is the situation where all of
those things occur.
After all, it's possible that we can have an art show and
have a hack be a legitimate participant in the art show.
That's why we have that final row.
So we have all manner of combinations in between.
So those are those combinations.
Then we have F, F, T, T, F, F, T, T, F, T, F, T, F, T, F, T.
So it's plain that the number of rows in the table, or these
binary possibilities, is 2 to the number of variables.
And that could be a big number.
In fact, I'd love to do a bigger example, but I don't
have the patience to do it.
But anyhow, what we might do is in order to figure out how
likely any of these combinations are, is we might
have observed the area outside the student center and rest of
campus over a long period of time and keep track of what
happens on 1,000 days.
Or maybe 1,000 months or 1,000 years.
I don't know.
The trouble is, these events don't happen very often.
So the period of time that I use for measurement needs to
be fairly long.
Probably a day is not short enough.
But in any case, I can keep a tally of how often I see these
various combinations.
So this one might be, for example, 405, this one might
be 45, this one might be 225, this one might
be 40, and so on.
And so having done all those measurements, kept track of
all that data, then I could say, well, the probability
that at any given time period one of these things occurs
will just be the frequency--
the number of tallies divided by the
total number of tallies.
So that would be a number between 0 and 1.
So that's the probability for each of these events.
And it's readily calculated from my data.
And once I do that, then I can say that I got myself a joint
probability table, and I could perform all manner of miracles
using that joint probability table.
So let me perform a few of those miracles,
while we're at it.
There's the table.
And now, what I want to do is I want to count up the
probability in all the rows where the statue appears.
So that's going to be the probability
of the statue appearing.
So I'll just check off those four boxes there.
And it looks like the probability of the statue
appearing is about 0.355 in my model.
I don't think it's quite that frequent, but this is a
classroom exercise, right?
So I can make up whatever numbers I want.
Now, I could say, well, what's the probability of a statue
occurring given that there's an art show?
Well, I can limit my tallies to those in which art show is
true, like so.
And in that case, the probability
has just zoomed up.
So if I know there's an art show, there's a much higher
probability that a statue will appear.
And if I know there's a hack as well as an art show going
on, it goes up higher still to 0.9.
We can also do other kinds of things.
For example, we can go back to the original table.
And instead of counting up the probability we've got a
statue, as we just did, we're going to calculate the
probability that there is an art show.
I guess that would be that one and that one, not that one,
but that one.
So the probability there's an art show is one chance in 10.
Or we can do the same thing with a hack.
In that case, we get that one off, that one on, that one
off, that one on, that one off, that one
on, that one off.
So the probability of a hack on any given time period is
about 50-50.
So I've cooked up this little demo so it does the "ands" of
all these things.
It could do "ors," too, with a little more work.
But these are just the "ands" of these various combinations.
Then you can ask more complicated questions, like
for example, you could say, what is the probability of a
hack given that there's a statue?
And that would be limiting the calculations to those rows in
which the statue thing is true.
And then what I get is 0.781.
Now, what would happen to the probability that it's a hack
if I know that there's an art show?
Will that number go up or down?
Well, let's try it.
Ah, it went down.
So that's sort of because the existence of the art show sort
of explains why the statue might be there.
Now, just for fun, I'm going to switch to another
situation, very similar.
And the situation here is that a neighbor's dog often barks.
It might be because of a burglar.
It might be because of a raccoon.
Sometimes, there's a burglar and a raccoon.
Sometimes, the damn dog just barks.
So let's do some calculations there and calculate the
probability that a raccoon is true, similar to
what we did last time.
Looks like on any given night--
it's kind of a wooded are-- there's a high probability of
a raccoon showing up.
And then we can ask, well, what is the probability of the
dog barking given that a raccoon shows up?
Well, in that case, we want to just limit the number of rows
to those where a raccoon--
or where the dog is barking.
Looks like the probability of the dog barking, knowing
nothing else, is about [? 3/7. ?]
But now we want to know the probability of the raccoon--
that's these guys here need to get checked.
These are off.
So that's the probability of a raccoon.
Did I get that right?
Oh, that's probability of a burglar.
Sorry, that was too hard.
So let me go back and calculate--
I want to get the probability of a raccoon.
That's true, false, true, false, true, false, true.
So the probability of a raccoon, as I
said before is 0.5.
Now, what happens to that probability if I know the dog
is barking?
Well, all I need to do is limit my rows to those where
the dog is barking, those bottom four.
And I'll click that there, and you'll notice all these
tallies up above the midpoint have gone to zero, because
we're only considering those cases
where the dog is barking.
In that case, the probability that there's a raccoon--
just the number of tallies over the
total number of tallies--
gee, I guess it's 225 plus 50 divided by 370.
That turns out to be 0.743.
So about 75% of the time, the dog barking is accounted for--
well, the probability of a raccoon under those conditions
is pretty high.
And now, once again, I'm going to ask, well, what is the
probability of a raccoon, given that the dog is barking
and there's a burglar?
Any guess what will happen there?
We did this once before with the statue.
Probability first went up when we saw the statue and then
went down when we saw another explanation.
Here's this one here.
Wow, look at that.
It went back to its original condition, its a priori
probability.
So somehow, the existence of the burglar and the dog
barking means that the probability of a raccoon is
just what it was before we started this game.
So those are kind of interesting questions, and
there's a lot we can do when we have this table by way of
those kinds of calculations.
And in fact, the whole miracle of probabilistic inference is
right in front of us.
It's the table.
So why don't we go home?
Well, because there's a little problem with this table--
with these two tables that I've shown you by way of
illustration.
And the problem is that there are a lot of rows.
And I had a hard time making up those numbers.
I didn't have the patience to wait and make observations.
That would take too long.
So I had to kind of make some guesses.
And I could kind of manage it with eight rows--
those up there.
I could put in some tallies.
It wasn't that big of a deal.
So I got myself all those eight numbers
up there like that.
And similarly, for the art show calculations, produced
eight numbers.
But what if I added something else to the mix?
What if I added the day of the week or
what I had for breakfast?
Each of those things would double the number of rows of
their binary variables.
So if I have to consider 10 influences all working
together, then I'd have 2 to the 10th.
I'd have 1,000 numbers to deal with.
And that would be hard.
But if I had a joint probability table, then I can
do these kinds of miracles.
But Dave, if I could have this little projector now, please.
I just want to emphasize that although we're talking about
probabilistic inference, and it's a very powerful tool,
it's not the only tool we need in our bag.
Trouble with most ideas in artificial intelligence is
that their hardcore proponents think that they're the only
thing to do.
And probabilistic inference has a role to play in
developing a theory of human intelligence.
And it certainly has a practical value, but it's not
the only thing.
And to illustrate that point, I'd like to imagine for a few
moments that MIT were founded in 1861 BC instead of 1861 AD.
And if that were so, then it might be the case that there
would be a research program on what floats.
And this, of course, would be a problem in experimental
physics, and we could imagine that those people back there
in that early MIT would, being experimentally minded, try
some things.
Oh, I didn't know that's what happened.
It looks like chalk floats.
Here's a rock.
No, it didn't float.
Here's some money.
Doesn't float.
Here's a pencil.
No, it doesn't float.
Here's a pen.
Here's a piece of tin foil I got from Kendra.
That floats.
That's a metal.
The other stuff's metal, too.
Now I'm really getting confused.
Here's a little wad of paper.
Here's a cell ph-- no, actually,
I've tried that before.
They don't float.
And they also don't work afterward, either.
I don't need to do any of that in the MIT of 1861 AD and
beyond, because I know that Archimedes
worked this all out.
And all I have to do is measure the volume of the
stuff, divide that by the weight, and if that ratio is
big enough, then the thing will float.
But back in the old days, I would have to try a lot of
stuff and make a big table, taking into account such
factors as how hard it is, how big it is, how heavy it is,
whether it's alive or not.
Most things that are alive float.
Some don't.
Fish don't, for instance.
So it would be foolhardy to do that.
That's sort of a probabilistic inference.
On the other hand, there are lots of things where I don't
know all the stuff I need to know in order to make the
calculation.
I know all the stuff I need to know in order to decide if
something floats, but not all the stuff I need to know in
order, for example, to decide if the child of a Republican
is likely to be a Republican.
There are a lot of subtle influences there, and it is
the case that the children of Republicans and the children
of Democrats are more likely to share the political party
of their parents.
But I don't have any direct way of calculating whether
that will be true or not.
All I can do in that case is what I've done over here, is
do some measurements, get some frequencies, take some
snapshots of the way the world is and incorporate that into a
set of probabilities that can help me determine if any given
parent is a Republican, given that I've observed the voting
behavior their children.
So probability has a place, but it's not the
only tool we need.
And that is an important preamble to all the stuff
we're going to do today.
Now, we're really through, because this joint probability
table is all that there is to it, except for the fact we
can't either record all those numbers, and it becomes
quickly a pain to guess at them.
There are two ways to think about all this.
We can think about these probabilities as probabilities
that come out of looking at some data.
That's a frequentist view of the probabilities.
Or we could say, well, we can't do those measurements.
So I can just make them up.
That's sort of the subjective view of where these
probabilities come from.
And in some cases, some people like to talk about natural
propensities, like in quantum mechanics.
But for our purposes, we either make them up, or we do
some tallying.
Trouble is, we can't deal with this kind of table.
So as a consequence of not being able to deal with this
kind of table, a gigantic industry has emerged for
dealing with probabilities without the need to work up
this full table.
And that's where we're going to go for
the rest of the hour.
And here's the path we're going to take.
We're going to talk about some basic overview of basic
probability.
Then we're going to move ourselves step by step toward
the so-called belief networks, which make it possible to make
this a practical tool.
So let us begin.
The first thing is basic probability.
Let us say basic.
And basic probability--
all probability flows from a small number of axioms.
We have the probability of some event a has got to be
greater than 0 and less than 1.
That's axiom number one.
In a binary world, things have a probability of being true.
Some have a probability of being false.
But the true event doesn't have any possibility of being
anything other than true, so the probability of true is
equal to 1, and the probability of false--
the false event, the false condition--
has no possibility of being true, so that's 0.
Then the third of the axioms of probability is that the
probability of a plus the probability of b minus the
probability of a and b is equal to the
probability of a or b.
Yeah, that makes sense, right?
I guess it would make more sense if I didn't switch my
notation in midstream--
a and b.
So those are the axioms that mathematicians love to start
up that way, and they can derive everything there is to
derive from that.
But I never can deal with stuff that way.
I have to draw a picture and think of this stuff in a more
intuitionist type of way.
So that's the formal approach to dealing with probability,
and it's mirrored by intuitions that have to do
with discussions of spaces, like so, in which we have
circles, or areas, representing a and b.
And to keep my notation consistent,
I'll make those lowercase.
So you can think of those as spaces of all possible worlds
in which these things might occur.
Or you can think of them as sample spaces.
But in any event, you associate with the probability
of a the size of this area here relative to the total
area in the rectangle--
the universe.
So the probability of a is the size of this circle divided by
the size of this rectangle in this picture.
So now all these axioms make sense.
The probability that a is certain is just when that
fills up the whole thing, and there's no other place for a
sample to be, that means it has to be a.
So that probability goes all the way up to 1.
On the other hand, if the size of a is just an infinitesimal
dot, then the chances of landing in that world is 0.
That's the bound on the other end.
So this--
axiom number one-- makes sense in terms of that
picture over there.
Likewise, axiom number two.
What about axiom number three?
Does that make sense in terms of all this stuff?
And the answer is, sure, because we can just look at
those areas with a little bit of colored chalk.
And so the probability of a is just this area here.
The probability of b is this area here.
And if we want to know the probability that we're in
either a or b, then we just have to add up those areas.
But when we add up those areas, this intersection part
is added in twice.
So we've got to subtract that off in order to make this
thing make a rational equation, so that makes sense.
And axiom three makes sense, just as
axioms one and two did.
So that's all there is to basic probability.
And now you could do all sorts of algebra on that, and it's
elegant, because it's like circuit theory or
electromagnetism, because from a very
small number of axioms--
in this case three--
you can build an elegant mathematical system.
And that's what probability subjects do.
But we're not going to go there, because we're sort of
focused on getting down to a point where we can deal with
that joint probability table that we
currently can't deal with.
So we're not going to go into a whole lot of algebra with
these things.
Just what we need in order to go through that network.
So the next thing we need to deal with is conditional
probability.
And whereas those are axioms, this is a definition.
We say that the probability of a given b is equal to, by
definition, the probability of a and b.
I'm using that common notation to mean [INAUDIBLE]
as is conventional in the field.
And then we're going to divide that by the probability of B.
You can take that as a definition, and then it's just
a little bit of mysterious algebra.
Or you could do like we did up there and take an intuitionist
approach and ask what that stuff means in terms of a
circle diagram and some sort of space.
And let's see, what does that mean?
It means that we're trying to restrict the probability of a
to those circumstances where b is known to be so.
And we're going to say that--
we've got this part here, and then we've got the
intersection of a with b.
And so it does make sense as a definition, because it says
that if you've got b, then the probability that you're going
to get a is the size of that intersection--
the pink and orange stuff--
divided by the whole of b.
So it's as if we restricted the universe of consideration
to just that part of the original universe
as covered by b.
So that makes sense as a definition.
And we can rewrite that, of course, as P of a and b is
equal to the probability of a given b times the
probability of b.
That's all basic stuff.
Now, we do want to do a little bit of algebra here, because I
want to consider not just two cases, but what if we divide
this space up into three parts?
Then we'll say that the probability of a, b, and c is
equal to what?
Well, there are lots of ways to think about that.
But one way to think about it is that we are restricting the
universe to that part of the world where b
and c are both true.
So let's say that y is equal to b and c--
the intersection of b and c, where a and b are both true.
Then we can use this formula over here to say that
probability of a, b, and c is equal to the probability of a
and y, which is equal to the probability of a given y times
the probability of y.
And then we can expand that back out and say that P of a
given b and c is equal to the probability--
sorry, times the probability of y, but y is equal to the
probability of b and c, like so.
Ah, but wait--
we can run this idea over that one, too, and we can say that
this whole works is equal to the probability of a given b
and c times the probability of b given c times the
probability of c.
And now, when we stand back and let that sing to us, we
can see that some magic is beginning to happen here,
becaus
…
Three to go.
And we're going to take some of the last three, maybe two
of the last three, to talk a little bit about stuff having
to do with probabilistic approaches--
use of probability in artificial intelligence.
Now, for many of you, this will be kind of a review,
because I know many of you learned about probability over
the [? sand ?] table and every year since then.
But maybe we'll put another little twist into it,
especially toward the end of the hour when we get into a
discussion of that which has come to be called belief nets.
But first, I was driving in this morning, and I was quite
astonished to see, as I drove in, this thing here.
And my first reaction was, oh my god, it's the world's
greatest hack.
And then I decided, well, maybe it's a piece of art.
So I'd like to address the question of how I could come
to grips with that issue.
There's a distinct possibility that this thing is a
consequence of a hat, possibly the result of
some kind of art show.
And in any event, some sort of statue appeared, and statues
don't usually appear like that.
So I got the possibility of thinking about how all these
things might occur together or not occur together.
So the natural thing is to build myself some sort of
table to keep track of my observations.
So I have three columns in my table.
I've got the possibility of a statue appearing, a hack
having occurred, and some sort of art show.
And so I can make a table of all the combinations of those
things that might appear.
And I happen to have already guessed that there are going
to be eight rows in my table.
So it's going to look like this.
And this is the set of combinations in this row where
none of that occurs at all.
And down here is the situation where all of
those things occur.
After all, it's possible that we can have an art show and
have a hack be a legitimate participant in the art show.
That's why we have that final row.
So we have all manner of combinations in between.
So those are those combinations.
Then we have F, F, T, T, F, F, T, T, F, T, F, T, F, T, F, T.
So it's plain that the number of rows in the table, or these
binary possibilities, is 2 to the number of variables.
And that could be a big number.
In fact, I'd love to do a bigger example, but I don't
have the patience to do it.
But anyhow, what we might do is in order to figure out how
likely any of these combinations are, is we might
have observed the area outside the student center and rest of
campus over a long period of time and keep track of what
happens on 1,000 days.
Or maybe 1,000 months or 1,000 years.
I don't know.
The trouble is, these events don't happen very often.
So the period of time that I use for measurement needs to
be fairly long.
Probably a day is not short enough.
But in any case, I can keep a tally of how often I see these
various combinations.
So this one might be, for example, 405, this one might
be 45, this one might be 225, this one might
be 40, and so on.
And so having done all those measurements, kept track of
all that data, then I could say, well, the probability
that at any given time period one of these things occurs
will just be the frequency--
the number of tallies divided by the
total number of tallies.
So that would be a number between 0 and 1.
So that's the probability for each of these events.
And it's readily calculated from my data.
And once I do that, then I can say that I got myself a joint
probability table, and I could perform all manner of miracles
using that joint probability table.
So let me perform a few of those miracles,
while we're at it.
There's the table.
And now, what I want to do is I want to count up the
probability in all the rows where the statue appears.
So that's going to be the probability
of the statue appearing.
So I'll just check off those four boxes there.
And it looks like the probability of the statue
appearing is about 0.355 in my model.
I don't think it's quite that frequent, but this is a
classroom exercise, right?
So I can make up whatever numbers I want.
Now, I could say, well, what's the probability of a statue
occurring given that there's an art show?
Well, I can limit my tallies to those in which art show is
true, like so.
And in that case, the probability
has just zoomed up.
So if I know there's an art show, there's a much higher
probability that a statue will appear.
And if I know there's a hack as well as an art show going
on, it goes up higher still to 0.9.
We can also do other kinds of things.
For example, we can go back to the original table.
And instead of counting up the probability we've got a
statue, as we just did, we're going to calculate the
probability that there is an art show.
I guess that would be that one and that one, not that one,
but that one.
So the probability there's an art show is one chance in 10.
Or we can do the same thing with a hack.
In that case, we get that one off, that one on, that one
off, that one on, that one off, that one
on, that one off.
So the probability of a hack on any given time period is
about 50-50.
So I've cooked up this little demo so it does the "ands" of
all these things.
It could do "ors," too, with a little more work.
But these are just the "ands" of these various combinations.
Then you can ask more complicated questions, like
for example, you could say, what is the probability of a
hack given that there's a statue?
And that would be limiting the calculations to those rows in
which the statue thing is true.
And then what I get is 0.781.
Now, what would happen to the probability that it's a hack
if I know that there's an art show?
Will that number go up or down?
Well, let's try it.
Ah, it went down.
So that's sort of because the existence of the art show sort
of explains why the statue might be there.
Now, just for fun, I'm going to switch to another
situation, very similar.
And the situation here is that a neighbor's dog often barks.
It might be because of a burglar.
It might be because of a raccoon.
Sometimes, there's a burglar and a raccoon.
Sometimes, the damn dog just barks.
So let's do some calculations there and calculate the
probability that a raccoon is true, similar to
what we did last time.
Looks like on any given night--
it's kind of a wooded are-- there's a high probability of
a raccoon showing up.
And then we can ask, well, what is the probability of the
dog barking given that a raccoon shows up?
Well, in that case, we want to just limit the number of rows
to those where a raccoon--
or where the dog is barking.
Looks like the probability of the dog barking, knowing
nothing else, is about [? 3/7. ?]
But now we want to know the probability of the raccoon--
that's these guys here need to get checked.
These are off.
So that's the probability of a raccoon.
Did I get that right?
Oh, that's probability of a burglar.
Sorry, that was too hard.
So let me go back and calculate--
I want to get the probability of a raccoon.
That's true, false, true, false, true, false, true.
So the probability of a raccoon, as I
said before is 0.5.
Now, what happens to that probability if I know the dog
is barking?
Well, all I need to do is limit my rows to those where
the dog is barking, those bottom four.
And I'll click that there, and you'll notice all these
tallies up above the midpoint have gone to zero, because
we're only considering those cases
where the dog is barking.
In that case, the probability that there's a raccoon--
just the number of tallies over the
total number of tallies--
gee, I guess it's 225 plus 50 divided by 370.
That turns out to be 0.743.
So about 75% of the time, the dog barking is accounted for--
well, the probability of a raccoon under those conditions
is pretty high.
And now, once again, I'm going to ask, well, what is the
probability of a raccoon, given that the dog is barking
and there's a burglar?
Any guess what will happen there?
We did this once before with the statue.
Probability first went up when we saw the statue and then
went down when we saw another explanation.
Here's this one here.
Wow, look at that.
It went back to its original condition, its a priori
probability.
So somehow, the existence of the burglar and the dog
barking means that the probability of a raccoon is
just what it was before we started this game.
So those are kind of interesting questions, and
there's a lot we can do when we have this table by way of
those kinds of calculations.
And in fact, the whole miracle of probabilistic inference is
right in front of us.
It's the table.
So why don't we go home?
Well, because there's a little problem with this table--
with these two tables that I've shown you by way of
illustration.
And the problem is that there are a lot of rows.
And I had a hard time making up those numbers.
I didn't have the patience to wait and make observations.
That would take too long.
So I had to kind of make some guesses.
And I could kind of manage it with eight rows--
those up there.
I could put in some tallies.
It wasn't that big of a deal.
So I got myself all those eight numbers
up there like that.
And similarly, for the art show calculations, produced
eight numbers.
But what if I added something else to the mix?
What if I added the day of the week or
what I had for breakfast?
Each of those things would double the number of rows of
their binary variables.
So if I have to consider 10 influences all working
together, then I'd have 2 to the 10th.
I'd have 1,000 numbers to deal with.
And that would be hard.
But if I had a joint probability table, then I can
do these kinds of miracles.
But Dave, if I could have this little projector now, please.
I just want to emphasize that although we're talking about
probabilistic inference, and it's a very powerful tool,
it's not the only tool we need in our bag.
Trouble with most ideas in artificial intelligence is
that their hardcore proponents think that they're the only
thing to do.
And probabilistic inference has a role to play in
developing a theory of human intelligence.
And it certainly has a practical value, but it's not
the only thing.
And to illustrate that point, I'd like to imagine for a few
moments that MIT were founded in 1861 BC instead of 1861 AD.
And if that were so, then it might be the case that there
would be a research program on what floats.
And this, of course, would be a problem in experimental
physics, and we could imagine that those people back there
in that early MIT would, being experimentally minded, try
some things.
Oh, I didn't know that's what happened.
It looks like chalk floats.
Here's a rock.
No, it didn't float.
Here's some money.
Doesn't float.
Here's a pencil.
No, it doesn't float.
Here's a pen.
Here's a piece of tin foil I got from Kendra.
That floats.
That's a metal.
The other stuff's metal, too.
Now I'm really getting confused.
Here's a little wad of paper.
Here's a cell ph-- no, actually,
I've tried that before.
They don't float.
And they also don't work afterward, either.
I don't need to do any of that in the MIT of 1861 AD and
beyond, because I know that Archimedes
worked this all out.
And all I have to do is measure the volume of the
stuff, divide that by the weight, and if that ratio is
big enough, then the thing will float.
But back in the old days, I would have to try a lot of
stuff and make a big table, taking into account such
factors as how hard it is, how big it is, how heavy it is,
whether it's alive or not.
Most things that are alive float.
Some don't.
Fish don't, for instance.
So it would be foolhardy to do that.
That's sort of a probabilistic inference.
On the other hand, there are lots of things where I don't
know all the stuff I need to know in order to make the
calculation.
I know all the stuff I need to know in order to decide if
something floats, but not all the stuff I need to know in
order, for example, to decide if the child of a Republican
is likely to be a Republican.
There are a lot of subtle influences there, and it is
the case that the children of Republicans and the children
of Democrats are more likely to share the political party
of their parents.
But I don't have any direct way of calculating whether
that will be true or not.
All I can do in that case is what I've done over here, is
do some measurements, get some frequencies, take some
snapshots of the way the world is and incorporate that into a
set of probabilities that can help me determine if any given
parent is a Republican, given that I've observed the voting
behavior their children.
So probability has a place, but it's not the
only tool we need.
And that is an important preamble to all the stuff
we're going to do today.
Now, we're really through, because this joint probability
table is all that there is to it, except for the fact we
can't either record all those numbers, and it becomes
quickly a pain to guess at them.
There are two ways to think about all this.
We can think about these probabilities as probabilities
that come out of looking at some data.
That's a frequentist view of the probabilities.
Or we could say, well, we can't do those measurements.
So I can just make them up.
That's sort of the subjective view of where these
probabilities come from.
And in some cases, some people like to talk about natural
propensities, like in quantum mechanics.
But for our purposes, we either make them up, or we do
some tallying.
Trouble is, we can't deal with this kind of table.
So as a consequence of not being able to deal with this
kind of table, a gigantic industry has emerged for
dealing with probabilities without the need to work up
this full table.
And that's where we're going to go for
the rest of the hour.
And here's the path we're going to take.
We're going to talk about some basic overview of basic
probability.
Then we're going to move ourselves step by step toward
the so-called belief networks, which make it possible to make
this a practical tool.
So let us begin.
The first thing is basic probability.
Let us say basic.
And basic probability--
all probability flows from a small number of axioms.
We have the probability of some event a has got to be
greater than 0 and less than 1.
That's axiom number one.
In a binary world, things have a probability of being true.
Some have a probability of being false.
But the true event doesn't have any possibility of being
anything other than true, so the probability of true is
equal to 1, and the probability of false--
the false event, the false condition--
has no possibility of being true, so that's 0.
Then the third of the axioms of probability is that the
probability of a plus the probability of b minus the
probability of a and b is equal to the
probability of a or b.
Yeah, that makes sense, right?
I guess it would make more sense if I didn't switch my
notation in midstream--
a and b.
So those are the axioms that mathematicians love to start
up that way, and they can derive everything there is to
derive from that.
But I never can deal with stuff that way.
I have to draw a picture and think of this stuff in a more
intuitionist type of way.
So that's the formal approach to dealing with probability,
and it's mirrored by intuitions that have to do
with discussions of spaces, like so, in which we have
circles, or areas, representing a and b.
And to keep my notation consistent,
I'll make those lowercase.
So you can think of those as spaces of all possible worlds
in which these things might occur.
Or you can think of them as sample spaces.
But in any event, you associate with the probability
of a the size of this area here relative to the total
area in the rectangle--
the universe.
So the probability of a is the size of this circle divided by
the size of this rectangle in this picture.
So now all these axioms make sense.
The probability that a is certain is just when that
fills up the whole thing, and there's no other place for a
sample to be, that means it has to be a.
So that probability goes all the way up to 1.
On the other hand, if the size of a is just an infinitesimal
dot, then the chances of landing in that world is 0.
That's the bound on the other end.
So this--
axiom number one-- makes sense in terms of that
picture over there.
Likewise, axiom number two.
What about axiom number three?
Does that make sense in terms of all this stuff?
And the answer is, sure, because we can just look at
those areas with a little bit of colored chalk.
And so the probability of a is just this area here.
The probability of b is this area here.
And if we want to know the probability that we're in
either a or b, then we just have to add up those areas.
But when we add up those areas, this intersection part
is added in twice.
So we've got to subtract that off in order to make this
thing make a rational equation, so that makes sense.
And axiom three makes sense, just as
axioms one and two did.
So that's all there is to basic probability.
And now you could do all sorts of algebra on that, and it's
elegant, because it's like circuit theory or
electromagnetism, because from a very
small number of axioms--
in this case three--
you can build an elegant mathematical system.
And that's what probability subjects do.
But we're not going to go there, because we're sort of
focused on getting down to a point where we can deal with
that joint probability table that we
currently can't deal with.
So we're not going to go into a whole lot of algebra with
these things.
Just what we need in order to go through that network.
So the next thing we need to deal with is conditional
probability.
And whereas those are axioms, this is a definition.
We say that the probability of a given b is equal to, by
definition, the probability of a and b.
I'm using that common notation to mean [INAUDIBLE]
as is conventional in the field.
And then we're going to divide that by the probability of B.
You can take that as a definition, and then it's just
a little bit of mysterious algebra.
Or you could do like we did up there and take an intuitionist
approach and ask what that stuff means in terms of a
circle diagram and some sort of space.
And let's see, what does that mean?
It means that we're trying to restrict the probability of a
to those circumstances where b is known to be so.
And we're going to say that--
we've got this part here, and then we've got the
intersection of a with b.
And so it does make sense as a definition, because it says
that if you've got b, then the probability that you're going
to get a is the size of that intersection--
the pink and orange stuff--
divided by the whole of b.
So it's as if we restricted the universe of consideration
to just that part of the original universe
as covered by b.
So that makes sense as a definition.
And we can rewrite that, of course, as P of a and b is
equal to the probability of a given b times the
probability of b.
That's all basic stuff.
Now, we do want to do a little bit of algebra here, because I
want to consider not just two cases, but what if we divide
this space up into three parts?
Then we'll say that the probability of a, b, and c is
equal to what?
Well, there are lots of ways to think about that.
But one way to think about it is that we are restricting the
universe to that part of the world where b
and c are both true.
So let's say that y is equal to b and c--
the intersection of b and c, where a and b are both true.
Then we can use this formula over here to say that
probability of a, b, and c is equal to the probability of a
and y, which is equal to the probability of a given y times
the probability of y.
And then we can expand that back out and say that P of a
given b and c is equal to the probability--
sorry, times the probability of y, but y is equal to the
probability of b and c, like so.
Ah, but wait--
we can run this idea over that one, too, and we can say that
this whole works is equal to the probability of a given b
and c times the probability of b given c times the
probability of c.
And now, when we stand back and let that sing to us, we
can see that some magic is beginning to happen here,
becaus
…