YouSub
Untitled

en English

Untitled English subtitles

0 downloads · 1,041 lines · added Sep 29, 2015

Report a problem
What's wrong?
Advertisement

English transcript

Hello.
I'm here today with Charity Majors of the Parse division
of Facebook.
And this is a very different use case of MongoDB
than we've seen in the class.
So Charity, please tell us about Parse and what it does.
Parse is basically a mobile backend as a service.
So if you're a mobile developer and you want to build an app,
you can use Parse.
We have SDKs for all the major platforms.
We have a REST API, and we do core data storage,
we do developer-defined analytics,
we do push notifications.
Basically everything that you need
to build a really rich, fully-featured mobile app
without having to run your own backend servers
or your own databases, your own API servers.
OK.
So let's say I wanted to create an iOS app.
I would open up Xcode, I'd create a new application.
Maybe it's a multi-view application.
And I wanted to store data persistently,
and I was using Parse.
How would that work?
Well to you, you'd just-- you would look at our Data Browser.
It makes it look like a relational database, basically.
You can create what look like tables to you.
They're MongoDB-backed collections
on the storage side.
We also create certain sort of default collections for apps.
Like if you're going to send push notifications,
you need to have an installation collection.
So there are some collections that we
create for you on the Parse backend side
if you want to do certain things.
If you want to have installations, or users,
or ACLs, or-- there's some things
that we create implicitly.
But besides that, you can create basically any type
of collection that you want.
You can define your own schema.
You can define the data types.
And you can perform-- it almost looks like relational queries
against them.
And we, at this point, have over half a million apps on Parse.
Right.
So it's a multi-tenant system--
Yes.
Where each of the apps is a tenant with all the other apps
that are running on the servers.
Yeah.
So you said it was similar to relational.
But from the end user's perspective,
they can store data that's hierarchical as well, right?
There are ways to put arrays and dictionaries--
Yep, absolutely.
As values of keys.
Mhm.
Yeah.
And I mean, they can do online schema changes
at any time, which is really nice.
They could have a collection that
has hundreds of millions of rows,
and they can just decide to add another field in the Data
Browser, or in the brand-new schema API.
And we just do all that magically
on the backend for them.
So if a user-- if a developer saves something
to parse to their developer account for that app,
and using their API key, does it create a new collection
behind the scenes, or does it commingle the data
with other people's collections?
The collections are distinct.
The UUID for the API key for each app
is baked into the collection name.
So there will never be any collections that contain data
from more than one app.
OK.
So you create a collection that's
like the Andrew collection for your app.
Right.
It'll be like your UUID colon Andrew.
Which lets us-- when you log in or when you app authenticate,
you use your keys.
So you only have access to the collections that
are prefixed by your API key.
I see.
And you also offer some-- I looked
at the docs a little bit-- some higher-level services
for the users.
It isn't just storing and retrieving data from MongoDB.
There's stuff for creating a new end user account--
Mhm.
Absolutely.
And logging people in--
Yeah.
And resetting their password.
Yep.
Absolutely.
This was really the vision of Parse for the founders.
They started creating mobile apps themselves,
and they're just like, why do we have
to re-implement these very basic functions over and over
and over again.
This doesn't make any sense.
This is tedious.
We can make this a better experience
for mobile developers.
So yeah, we absolutely try to think
really hard about what operations
as a mobile developer do you find yourself just implementing
over and over and over again, and how could we just
bake them into the SDKs so that you have to write, like,
one line of code.
Right.
That makes perfect sense.
So when you do this behind the scenes,
is it-- what does the architecture look like?
Is it replicated, is it sharded?
What does it look like?
Sure.
Well the short answer is, it's very complicated.
Our architecture diagram looks like spaghetti.
OK.
But the very basic API request path is-- so we write on AWS.
Mhm.
So like we have an ELB endpoint that everyone accesses,
api.parse.com.
ELB talks to the web server, which
talks to HAProxy, which talks to the API server, which either
talks directly to MongoDB or it talks
to Cloud Cloud, which is a third-party-- third party--
which is the JavaScript ServerSide hosting service,
basically.
Cloud Code can also make-- return calls back into the API.
And then the object is returned to the user, or the code can--
they can recurse many times.
So the basic API request path is just like your standard website
path.
And you talked about Cloud Code.
So-- and this is an evolution, right?
When Parse first started, you could store data
and retrieve data, but you couldn't execute code
on Parse servers.
Yeah.
I mean it's scary, right?
Whenever you let people run arbitrary code on your servers,
it's a little bit terrifying.
Right.
But it's been out there for almost two years now.
So it's pretty solid.
There are some restrictions, like duration of time
that the request is allowed to run.
We do have longer-running background jobs that will
let you run like 15 minutes.
But we restrict it to like 12 seconds for the basic Cloud
Code API request because fundamentally,
if it's running for longer than 12 seconds,
it's a bad mobile experience.
Right.
That makes sense.
So with Cloud Code, what's the implementation language
that the developer can use to--
JavaScript.
All JavaScript.
It's a V8 engine.
And what are-- and Parse itself, what language is it written in?
Ah, fun story.
When we started out, we were written in Ruby on Rails.
Mhm.
There are many great things about Ruby.
I personally love Ruby as a language.
But it has this fundamental problem
where you have a fixed pool of HTTP workers, right?
And when you have one or two backends, maybe
that isn't such a big deal because, you know,
backends get slow.
The problem with a fixed pool of workers,
no matter what the language is, is
that if any backend that they're all talking to gets slow,
all of your workers fill up with requests to that backend.
And the site goes down.
I see.
Not good.
So we've just basically completed
a year-and-a-half-long project to rewrite every element
in our stack from Ruby to Go.
Interesting.
It's amazing.
It's--
Wow.
Oh my god, it's been revolutionary for us.
We love Go.
We love the Go Mongo driver.
I love Gustavo.
I think it's the best driver that's out there.
Our reliability has improved by over an order of magnitude
since completing the rewrite.
But it was hard, right, because you're rewriting endpoint
by endpoint--
Yeah.
And it's really challenging to-- you know.
And your mobile developers have baked
in-- Ruby is a very, let's call it, assumption-rich language.
It makes a lot-- it just like has default behaviors that you
never explicitly specified, it just decided to do this.
And we have been slowly discovering every single one
of those over the last 18 months because, as a platform,
we cannot break this behavior for our developers.
Maybe their mobile app hasn't been updated in two years.
We can't break it.
We can't break any of these implicit assumptions.
So we had to do this by using a splitter,
by shadowing production traffic, by moving endpoint
after endpoint from the Ruby-- we'd
roll it out some percentage of time-- of requests
from the Ruby stack to the Go stack and compare the results,
return the Ruby one to the developer, log any differences,
and then go back and manually resolve those.
Oh, wow.
That is a painful transition process to keep it--
Writes are one thing.
Rewrites are always harder than you think they will be.
It has to be bug-compatible, if you will.
Exactly.
Bug-compatible is a great way of putting that.
Yeah.
So just to make it clear to the people watching-- and this
is a complicated stack, right?
So on your side, it was written in Ruby.
Of course, the end developers who
are writing Android apps and iOS apps
were writing mostly in Java, Objective-C or Swift.
Yeah.
And they weren't exposed to any of that Ruby.
Yeah, no.
But the issue is that when they made requests
to store data and retrieve data from Parse--
Yeah.
They needed to get back exactly the same results regardless
of whether your backend was written in Ruby
or written in Go.
Yes.
Yeah.
And so that's the challenge.
Yeah.
And they'll bake in assumptions like oh, well this is totally
going to return a null.
Well if you return an empty string, your app crashes.
Right.
Details matter.
Yeah, details definitely matter.
So you said that you don't use sharding behind the scenes.
Correct.
You use-- but you do use replication.
Yes.
We definitely use replication.
We use replica sets.
We have a topology that is essentially--
I mean we started out using three-node replica sets.
We then-- so we started out in MongoDB 2.0.
Back in the bad old days of the global lock.
OK.
Moving from 2.0 to 2.2 with a database level lock
was really revolutionary for us.
And 2.4 and 2.6 have continued to, generally speaking,
get better.
But when we upgraded from 2.0 to 2.2,
we kept encountering categories of bugs
that would cause all secondaries to crash, which would leave us
without a primary, so all requests would start failing.
So we modified our topology to include two arbiters.
This also had the ancillary benefit
of making us more robust to availability zones going down.
Because with the primary and two arbiters, we're alive.
No matter what happens.
If all the secondaries crash, if any AZ goes down--
So a primary, some secondaries, and two arbiters.
It wasn't just the primary and two arbiters.
Primary, two secondaries, two arbiters.
OK.
Primary, two secondaries, and two arbiters.
And then we realized-- because one of these node
would be a snapshot node, right, where
we'd do all of our backups.
All right, we'll get to that.
Backups, OK.
Yeah.
And then we realized OK, well, we don't actually
want to be reading off the node that is serving backups
because it will stall.
It's a bad experience for our users.
So we ended up adding-- and also,
if one secondary goes down, you're
left with only-- if you have two secondaries and one goes down,
you're left with only one, and that's
actually not a great place to be in either because you have
to have a node taking backups.
And in our drivers, we'll set things like Prefer Secondaries.
We don't want to create roaming herds where
all of the old secondary traffic is converging
on the primary at the same time as maybe
some other reliability event is going on.
So we've actually moved to a model where
we have the two arbiters, one primary, two
visible secondaries, and one hidden secondary.
All right--
Which has no votes and takes backups.
And we haven't talked a lot about that, hidden secondaries,
in this class.
But the goal for you is really-- do
you use the secondaries for some user traffic?
Yes, we do.
You do.
Right.
Yeah.
So--
There are some read queries that are not efficient.
In part this is because there are some things that you just
can't do with databases efficiently,
and our customers do them anyway.
Like we can't-- we don't have code review for their queries.
Sometimes they just implement things
that do full table scans.
They'll say, give me all the objects in this 10-million item
collection.
We can't prohibit them from doing that.
Or doing like nested series are like not equals to this,
not equals to this.
Like we try to tell people not to do that,
but we can't really enforce it.
They can still do bad things.
So we try to automatically detect
these inefficient queries and systematically limit them
or offload them to a secondary so that they don't impact
everyone else on that tenant.
OK.
So that's actually a good segue--
and we'll get back to how that works.
Let's talk about indexing.
Yes.
Right?
So the users of this service, do they create the indexes?
They have no idea the indexes exist.
They have no idea the indexes-- so they write
queries using a subset of the MongoDB query language
that's been translated into the Parse API.
API, mhm.
And they-- so if they don't know if the indexes exist, how do
you deal with indexing at all?
Well there are basically two answers to this question,
because we wrote one version of our smart indexing
like almost three years ago.
And it worked really well for us for a while.
And I'll tell you what that was.
And then I'll tell you the ways that we've
found that it sort of fails, and the system that we're building.
So the first version of smart indexing
was basically computed on every single API request.
We would look at the expected cardinality of the data type,
you know, like booleans, not very indexable.
Right.
Strings, very indexable.
And the operator type.
Like not equals to, not a great index.
Equals to is a great index.
So we look at the operator type and the data type,
and we would compute how valuable
we thought this index would be.
And we would ensure that the three best indexes
for every API request existed.
We would just create them inline.
And we have--
So the system would learn, basically.
Yeah.
We have a caching layer where it caches which
indexes are expected to exist.
So this was the first implementation.
There were some minor-- there were
some big problems with this and some small problems with this.
One small problem was rate-limiting index creation.
Because you can do background indexing on primaries,
and if the collections are small,
it has super low impact on your performance.
But obviously you can't-- you couldn't do background indexing
on secondaries for a long, long time.
And you didn't want to build more than one index
at a time per replica set.
So we shifted from doing it inline in the API request
to dropping them in a queue.
And then using, like, ZooKeeper to pick up a lock on a per--
we call them shards.
They're not shards in the MongoDB sense,
but it's like every app is coherent to a single replica
set.
So we wouldn't build more than one index
at a time on our replica set.
So this worked, honestly, pretty freaking well for a really long
time.
We did-- just over the last year,
we started to run into some pretty critical issues, though.
where we turned on Full Query Logging, like log level one,
for all queries.
And we noticed that-- well there were a few things.
So you get long-running queries--
you'll see the long-running queries in the logs.
Or you see all the queries in the logs.
We see all the queries in the logs,
and we process them and ship them off
to various magical Facebook logging systems
where we can do real-time queries on what's going.
We can see things like lock percentage, and scanned, et
cetera, which helps us detect congestion
with the multi-tenant setup.
But for indexing, we were noticing
that, number one, performance, write
performance, degraded linearly with the number of indexes
that were created for a collection.
OK.
And since we were just creating all the indexes
that we thought might be useful, performance is taking a hit.
Right.
And what you mean by that is that any update or write
needs to update every index.
Yep.
Right?
Yep.
So if you have one index and you're only using that index,
great.
If you have 10, well, maybe you need all 10.
But if you don't, you're taking the write hit anyway.
Yeah, you're taking the-- yeah, you're
taking the performance hit of increasing
the latency of writes.
And we have some users who would literally
auto-generate their column names systematically.
And they would write into-- the max
is 64 indexes per collection for a very good reason,
because that's way too many indexes
to have in a collection.
And their write performance would degrade,
and then they couldn't create any new indexes.
And performance would fall off a cliff.
So we realized that what we needed
was a way to detect which indexes were actually valuable
and delete the rest.
And not automatically create them
on every single API request.
What we needed basically was to create and maintain
an indexing schema for every application, as well
as a schema schema.
So when you said automatically create them
on every API request, you're saying
that if the user made an API request
and it didn't hit an index, then you would create the index--
We would just ensure--
Right there?
That all of-- the three best indexes for every API request
existed.
Period.
So if you have predictable, consistent traffic,
this is mostly OK.
But if you go into the Data Browser
and happen to click Sort on a column,
it's going to go and create that index.
And maybe you never actually need that index,
and you're paying the write performance forever.
Right.
So you need to manage that.
Or you like are doing a bunch of testing with your app
and you just send a bunch of wacky REST API requests.
Well, we're going to index everything.
Even though we may never hit those indexes again,
we pay the write performance forever.
So we've been developing a new system
where we use the full query logs to detect which indexes are
actually valuable, which ones actually get hit,
and we store those to an indexing schema.
So we then reap the old ones that are not getting hit.
This is also going to be really helpful for us.
So-- I didn't mention this earlier,
but Parse is really an outlier in terms
of installations, MongoDB installations,
on the axes of the number of collections we have
and the number of indexes we have.
We don't have the largest Mongo deployment in the world,
but I'm pretty sure we have the most collections and the most
indexes.
Right.
We're talking like tens of millions.
Right.
Which is very unusual.
There are lots of people out there
who will have-- there are people out there who
have hundreds of shards.
They have billions of records.
We have billions of records.
We have only 30 replica sets.
We have an order of magnitude fewer replica sets
than the largest installation that I
know of, but like tens of orders of magnitude more collections
and indexes.
Yeah.
So this is really fun and interesting,
because we tend to fight all of your query planner and index
intersection bugs.
Because query planners are generally written for people
who think about their indexes and manage them sanely,
and we don't.
We don't.
We create them and manage them in a very automated,
very insane way.
Yeah.
So there's lots of collisions.
Yeah, I'm pretty sure we've found
most of your greatest-hits bugs for the 2.6 release,
where the entire query planner was rewritten.
So you have this system with, if I understand the architecture
right, a replica set may hold maybe multiple applications--
Anywhere from one to tens of thousands.
Now how much of a problem is the noisy neighbor
problem, which is that--
Yes.
You're on a replica set and, you know, unbeknownst to you,
there's another application that's really, really popular
and it's just using all this capacity--
Yep.
And now you're not getting very good performance--
It's been a really fun set of problems, for sure.
And part of this problem is just the fact
that mobile apps are unpredictable.
Someone will wake up and their app
will be featured on the iTunes store
overnight without them even knowing about it.
Right.
And they go from having a couple hundred users and their mom
and their friends to having millions of users.
This is one of the reasons that Parse is powerful,
because we're used to solving these problems very
naively at scale.
We've seen it happen a million times-- not a million times.
We've seen it happen many times.
And that we've gotten really good at automatically
remediating these types of problems.
So the greatest indicator of performance degradation
in up to 3.0, up to the document level locking storage,
is lock percentage.
So we've put a lot of instrumentation
in
…

Need subtitles from another video?