YouSub
Untitled

en English

Untitled English subtitles

0 downloads · 1,017 lines · added Nov 18, 2015

Report a problem
What's wrong?
Advertisement

English transcript

The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
PROFESSOR: Today we are introducing
an exciting new pledge in 6034.
Anyone who has already looked at any of the neural net problems
will have easily been able to see that even though Patrick
only has them back up to 2006 now,
there's still-- well out of four tests, perhaps two or three
different ways that the neural nets were drawn.
Our exciting new pledge is we're going to draw them
in a particular way this year.
And I will show you which way, assuming that this works.
Yes.
We are going to draw them like the one on the right.
The one on the left is the same as the one on the right.
At first, not having had to explain
the difference between the two of them,
you might think you want the one on the left.
But you really want the one on the right,
and I'll explain why.
The 2007 quiz was drawn, roughly similarly, to this.
Although if you somehow wind up in tutorial or somewhere else
doing one of the older quizzes, a lot of them
were drawn exactly like this.
In this representation, one thing I really don't like,
is that the inputs are called x's, and the outputs
are called y's, but there's two x's, so the inputs are not
x and y, and then they often correspond to x's of a graph,
and then people get confused.
Additional issues that many people have
are the fact that the summation and the multiplication
with the weight is implied.
The weights are written on the edges, where outputs and inputs
go, and the summation of the two inputs into the node
are also implied.
But take a look here.
This is the same net.
These w's here would be the w's that are written
onto these lines are here.
Actually the better way to draw it
would be like so, since each of these
can have their own w, which is different.
So each of the w's that are down here,
are being explicitly set to a multiplier.
Where as here, you just had to remember
to multiply the weight by the input that was coming by.
Here you see an input, comes to a multiplier,
you multiply by the weight, then once you multiplied
all the inputs by the weight, then you send them to a sum,
so the sigma is just a sum, you sum them, add them
all together, send the result of that into the sigmoid function,
our old buddy, 1 over 1 plus e to the negative whatever
our input was, with a weight for an offset,
and then we send the result of that
into more multipliers with more weights, more sums, more
sigmoids.
So this is how it's going to look like on the quiz.
And this is a conversion guide from version 0.9 data
into version 1.0.
So if you see something that looks
like this, on one of the old quizzes that you're doing,
see if you can convert it, and then solve the problem.
Chances are if you can convert it,
you're probably going to do fine.
We'll start off not only with this conversion guide,
but also-- I'll leave that up here-- also
I'm going to work out the formulas for you guys one more
time.
These are all the formulae that you're
going to need on the quiz.
And then we're going to decide what
will change in the formulae, if, and this is a very
likely if, there seems to be good amount of times
that this happens, is that the sigmoid function in those
neurons out there was ever changed
into some other kind of function.
Hint.
It's changed into a plus already in the problem
we're about to do.
People change it all the time into some bizarro function.
I've seen arc tangent, I think.
So here we go.
Let's look at the front of it.
First of all, sigmoid.
Well our old buddy, sigmoid, I just said it a moment ago,
sigmoid is 1 over 1 plus e to the minus x.
Also, fun fact about sigmoid, the derivative of sigmoid,
is itself-- the derivative of sigmoid
is-- let's say that the sigmoid-- we'll just
turn sigmoid into like the letter say y.
Y is the result, right?
So if you say y equals 1 over 1 plus e to the negative x,
then the derivative of sigmoid is y times 1 minus y.
You can also write out the whole nasty thing,
it's 1 over 1 plus e to the negative x times 1 minus 1
over 1 plus e to negative x.
So the nice property of sigmoid it's
going to be important for us in the very near future,
and that future begins now.
So now the performance function.
This is a function we used to tell neural nets when
they inevitably act up and give us really crappy results.
At first we tell them just how long they are,
with our performance function.
The first function can be any sane function
that gives you a better score, where better can be decided
as lower or higher, if you feel like, that gives you a better
score, if your answers are closer to the answer you're
looking for.
However, in this case, we have, for a very sneaky reason,
chosen the performance function to be
1/2 d, which is the desired output, minus o,
the actual output squared.
So we want a small, well it's negative,
So we want a small negative or 0.
That would mean we performed well.
So why this?
Well the main reason is ddx of performance
is, the 2 comes down, the o is the variable that we're
actually, so maybe I should say ddo, that negative comes out,
we get a simple d minus o.
And yeah, we're using derivatives here.
So those are fine.
These are two assumptions.
They could be changed on your test.
We're going to figure out what happens, if we change them,
if we change the performance, if we change the sigmoid, that
is if we change the sigmoid to some other function, what's
going to happen to the next three functions, which
are basically the only things that you need
to know to do backpropagation.
So let's look at that.
First, w prime.
This is the formula for a new weight.
After one step of backpropagation.
A new weight in any of these positions
that you can see up here on this beautiful neural net.
That w-- each of the w's will have to change step by step.
That's, in fact, how you do the hill climbing neural nets.
You change the weights incrementally.
You step a little bit in the direction
towards giving you your desired results until eventually, you
hope, you have an intelligent neural net.
And maybe you have many different training examples
that you run it on, in a cycle, hoping
that you don't over fit to your one sample, on a computer.
But on the test, we will probably will not do that.
So let's take a look at how you calculate
the weights for the next level.
And then you have the weights for the current level.
So first things first.
New weight, weight prime equals--
starts with the old weight.
That has to go there because otherwise we're
just going to jump off somewhere at random.
We want to make a little step in some direction,
so we want to start where we are, with the weight.
And then we're going to add three things.
So if we're talking about the weight
between some i and some j-- there's
some examples of the names of weights.
So this is w 1 i, that's the weight between 1
and-- so this is w 1 a, it's the weight between 1 and a.
This is w 2 b, which is the weight between 2 and b .
Makes sense?
Well makes sense so far, but what
if it's just called w b, then it's
the weight between-- these w's that only have one letter,
we'll get to later.
They're the bias.
They're the offset.
They are always attached to a negative 1.
So you can pretty much treat them as being a negative 1
here, that is then fed into a multiplier with this w b,
if you like.
This is implied to be that.
All of the offsets are implied to be that.
So w plus sum of alpha-- why is this Greek letter?
Where does it come from?
How do we calculate it?
Well alpha is just some value told to you on the quiz.
You'll find it somewhere.
There's no way you're going to have to calculate alpha.
You might be asked to try to give us an alpha,
but probably not.
Alpha is supposed to give the size of our little steps
that we take when we're doing hill climbing.
Very large alpha, take a huge step.
Very small alpha, take tentative steps.
So alpha is there, basically, to change this answer
and to make the new value either very close to w, or far from w,
depending on our taste.
So plus alpha times i, so i is the value
coming in into the node.
We're changing the weight here.
So i is the value, for instance, i sub 1 here,
i would be the value of WAC, i would be the value coming
output of node a.
WBC, i would be the output of node b.
i is sometimes as little as i is the input coming in
to meet that weight at the multiplier.
And then it's multiplied by delta j.
Your delta is the delta that belongs
to these neural net nodes.
What is a delta, you said?
Funny you may ask.
It is a strange Greek letter.
It sort of comes from the fact that we're
doing some partial derivatives and stuff,
but the main way you're going to figure out what the deltas are
are these two formulae that I've not written in yet.
So hold off on trying to figure out
what the delta is until-- well right now, I'm
about to tell you the delta is.
So the delta is basically, think of the delta
as using partial derivatives to figure out
which way you're going to step, when
you're doing hill climbing.
Because you know when you're doing hill climbing,
you look around, you figure out, OK, this
is the direction of the highest increase,
and then you step off in that direction.
So the deltas are telling you which way
to step, with the weights.
And the way they do that is by taking the partial derivative
of-- basically you try to figure out
how the weight that you're currently looking at
is contributing to the performance of the net.
Contributing to, either the good performance
of the net, or the bad performance of the net.
So when you're dealing with the weights, like WBC, WAC,
that pretty much directly feed into the end of the net.
They feed into the last node, and it then comes out.
It's the output.
That's pretty easy.
You can tell exactly how much those weights,
and the values coming from them, are contributing to the end.
And we do that by essentially, remember
what the partial derivative, so partial derivative here
is, in fact, the way that the final weights are contributing
to the performance, is just the performance function.
Partial derivative-- I've already
figured out the derivative here, it's just d minus o.
This is for sort of final weights,
the weights in the last level.
D minus o, except we're not done yet,
because when we do derivatives, remember the chain rule.
To get from the end to these weights, we pass through,
well it should be a sigmoid, here it's not,
we're going to pretend it is for the moment,
we pass through a sigmoid, and since we
passed through the sigmoid, we had better
take the derivative of the sigmoid function.
That is, y times 1 minus y.
Well what is y?
What is the output of the sigmoid?
It's up.
So that's also multiplied by o times 1 minus o.
However, there is a-- let me see,
let me see, yes-- sorry, I'm carefully studying this sheet
to make sure my nomenclature is exactly
right for our new nomenclature, which so new and brave,
that we're doing it, that we only knew for sure we're
going to do it on Wednesday.
So we have d minus o times o times 1 minus o.
So you say, that's fine, that can get us these weights
here, even this w c, how are we going
to get the deltas for the new weights here?
Oh, I realize-- yeah, I got it.
So the delta-- by the way, this is a delta c,
how is neuron c contributing to the output?
Well it's directly contributing to the output ,
and it's got a sigmoid in it.
It doesn't really, but we're pretending it does for now.
d minus o times 1 minus o.
What about inner node?
Node d, node a, what are we going to have to do?
Well the way they contribute to the output is
that they contribute to node c.
So we can do this problem recursively.
So let's do this recursively.
First of all, as you have probably figured out,
all of them are going to have an o times 1 minus o factoring
from the chain rule, because they're all sigmoid, pretending
that they're all sigmoids.
We also have a dearth of good problems
that are actually sigmoid on the web right now.
There's only 2007.
But here's o times 1 minus o, what
are we going to do for the rest of it?
How does it contribute to our final result?
Well it contributes to our final result recursively.
So we're talking about delta i.
I is an inner node.
It's not a final node.
It's somewhere along the way.
So sum over j of w, going from i to j, times delta j.
Now sum over all j, j such that i leads to j.
I needs to have a direct path into j.
So if i, in this instance, was j,
everyone, the only possible j in this would be c.
That's right.
We would not sum over b as one of the j.
i does not lead to b, or a does not lead to b, a only leads
to c.
Also note that c does not need to be here.
That's going backwards.
So you just-- to figure out which j you're looking at,
look directly forwards at the next one.
So if there was another d here, or that a does not go to d,
a goes to c.
You only look at the next level children,
and you sum over all those children,
the weight between them, multiplied
by the child's delta.
That makes sense, right?
Because the way we affect, if the child's delta is the way
the child affects the output, calling these children
for a moment, and then if this one directly
affects the output, then the way this one
affects it is-- it affects it because it affects this,
but it's also multiplied by it's weight.
So in fact, for instance, if the weight between a and c were 0,
then a doesn't affect the output at all, right?
Because its weight is 0, and when we do this problem,
we go this times 0, and then we try to add it in there,
doesn't affect anything.
It's weight is very high, it's going to really
dominate c, and that is taken into account here,
and then multiply by the delta for the right node.
So the following question, and since I
spent a lot of time with formulae and not that much time
starting on the problem, I will not call on someone at random,
but rather take a volunteer.
If no one volunteers, I'll eventually
tell you, which is, we've got some nice formulae
on the bottom three.
If we change the sigmoid function, what has to change?
That's the only thing that changes
in this crazy assed problem right here, which by the way,
changes the sigmoid functions into adders,
is that we take all of the o times 1
minus o in delta f and the delta i,
and we change it to a new derivative.
We then do the exact same thing that we would've done.
Correct.
And on a similar note, if you change the performance
function, how many of these equations
at all have to change out of the bottom three.
Yeah.
That's right, just one, just delta f.
Take the d minus o, make it the new derivative
of the new performance function.
And in fact, delta i doesn't change at all.
Does everyone see that?
Because it is very common for something to be replaced,
I think three of the four the quizzes that we have,
replaced in some-- changed something in some way.
All right.
Let's go.
We're going to do 2008 quiz, because it
has a part of the end that screwed up everyone,
and so let's make sure we get to that part.
That's going to be the part that you probably care
about the most at this point.
So these are all adders instead of sigmoids.
That means that they simply add up everything as normal,
for a normal neural net, and then there's
no sigmoid threshold.
They just give some kind of value.
Question?
STUDENT: So we talked about those multiplier things,
we don't have those in nodes?
PROFESSOR: They're not neural net nodes.
That is one of the reasons why that other form that you
can see over there is elegant.
It only has the actual nodes on it.
It is very compact.
It's one of the front we've used in the previous tests.
The question is, do those multipliers count as nodes?
However by not putting in the multipliers,
we feel it sometimes confuses people of explicitness.
The ones that are nodes will always
have a label, like a or here, you see
there's a sigmoid and an L1.
The multipliers are there for your convenience,
to remind you to multiply, and also
those, if you look those sigmoids that are over there,
are there for your convenience to remind you to add.
In fact, the only thing that counts
as a node in the neural net-- and that's
a very good question-- is usually the sigmoids,
here it's the adders.
We've essentially taken out the sigmoids.
These adders are the-- oh, here's the way to tell.
If it's got a threshold weight associated with it,
then it's one of the actual nodes.
A threshold weight.
I guess the multipliers look like they have a weight,
but this is just the weight that is being multiplied in.
This is our witness be multiplied in with the input,
but if it has a threshold weight, like wa,
wb-- oh, I promised I would tell you guys
the difference between the two weights.
So let's do that very quickly.
The kinds of weights that, say w2b or w1a, our weight the
comes between input 1 and a or between a
and c, then mentally multiplying the input by this weight,
and then eventually that's added together.
The threshold weights, they just have like wb, wa, wc.
They are essentially to decide the threshold for a success
or failure, for a 1 or a 0, or anything in between,
at any of the given nodes.
So the idea is maybe you at some node
want to have a really high cut off,
you have to very high value coming in, or else it's a 0.
So you put a high threshold.
The weight is multiplied by negative 1.
And in fact, the threshold weight won't-- one could
consider if you wanted to that the threshold weight times
negative 1. was also added in it that sum,
instead of putting at the same location as the node.
If that works better for you, when you're converting it,
you can also think of it that way.
Because the threshold weight is essentially
multiplied by negative 1 and added in
at that same sum over there.
So that's another way to do it.
There's a lot of ways to visualize these neural nets.
Just make sure you have a way that makes sense to you,
and that you can tell pretty much whatever
we write, as long as it looks vaguely like that,
how to get it in your mind, into the representation
that works for you.
Because once you have the representation right for you,
you're more than halfway to solving these guys.
They aren't that bad.
They just look nasty.
They don't bite.
OK.
These are just adders.
So if it's just an adder, then that
means that, if we take all the x inputs coming in--
let's do x and y for the moment, so we can figure out
the derivative-- then what comes out
after we just add up the x, what comes out, y equals x, right?
We're just adding it up.
Adding up all the input, we're not doing anything to it.
Y equals x is what this node does.
You people see that?
So the derivative is just one.
So that's pretty easy, because the first problem says,
what is the new formula, delta f.
So I'll just tell you.
You guys probably figured it out.
o times 1 minus o.
Because we replaced d minus o with 1.
OK?
Makes sense so far?
Please ask questions along the way,
because I'm not going to be asking you guys.
I'll do it myself.
Question?
STUDENT: Why to we use d minus o of 1?
PROFESSOR: That's a good question.
The reason is because I did the wrong thing.
So see, it's good that you guys are asking questions.
It actually should be replaced with o times 1 minus o with 1.
The answer is delta f equals d minus o.
So yes, perhaps I did it to trick you.
No, I actually messed up.
But yes, please ask questions along the way.
Again, I don't have time to call on you guys
at random to figure out if you guys are following along.
So I'll do it myself.
We're placing the o times 1 minus o with 1 because
of the fact that the sigmoid is gone,
and we get just delta f equals d minus o.
So great.
We now want to know what the equation is for delta i,
at the node a.
So delta a.
Well let's take a look.
The o times 1 minus o is gone.
Now we just have the sum over j, which you guys already
to
…

Need subtitles from another video?