Advertisement
English transcript
>>Ankit Mehta: Good morning. >>Audience: Good morning.
>>Ankit Mehta: Seems like everyone's had a late night.
Welcome. I'm getting old. I'm going to hold on to this, mostly to stick to the script
so you don't make this keynote a three-hour one.
And, yeah, what I was going to be talking about today is a topic which is very near
and dear to my heart --- how do you move fast and don't break things. Because I think each
one of these things are interesting to do in isolation, but to do them together poses
some interesting challenges and opportunities. And we have had a fun ride for about the last
decade where we have done this. And we think we have it down to the point where we would
like to share this with all of you, so you guys can get a perspective of how we try to
do this at Google, but, more importantly, you can help us make this even better going
ahead. Quick thing. GTAC, last time we did this,
this was in New York, it was wildly successful. The New York one was based on a theme. I don't
know how many of you were there. And we somewhat felt constrained with the theme. So this year,
we decided to keep it open-ended. We roughly had about 1500 speaker applications
this time, which was roughly about 3X of last year, which to us, gave us a signal that the
community is still very much engaged with GTAC, which is a great thing. That really
enables us to keep doing this. Out of the 1500, about 200 were Googlers'
applications, with about 1300 were external. And of those, you -- 250 of you are the unlucky
ones or lucky ones. You'll find out in a couple of days, who get to spend two great days in
this room. And the speakers. We have about 15 speakers.
What do I like about GTAC? The booze is great.
[ Laughter ] We can't put it on the slide. Oops. This is
recorded. [ Laughter ]
But on a more serious note, I think what has amazed me about is GTAC to really get a chance
to talk with most of you guys from the industry, and it's an interesting variance in how testing
is approached differently in different industries and different segments of the tech industry.
And I think we value at Google at sharing what we have done within Google, but I think
we have gained more from learning from the experiences some of you guys have shared at
GTAC. So again, thank you in advance to all of you
who are going to be talking about -- the other speakers who are going to be talking about
stuff. So why am I here?
Yeah, I've become one of those guys who spends a decade at a company, but Google is a pretty
damn good company to spend that decade at. Over the years, due to the kind of work I've
done at Google, I'm super passionate about, again, the balance between velocity and quality,
because it's almost like a two-edged sword, and you need to balance that. And I think
I just want to be able to share with you guys what we have done.
A quick history of my career at Google. I started about ten years back leading testing
efforts for Google desktop, there used to be this thing. There was a time people used
to build software for desktops. So did Google. And then I moved on to leading efforts for
everything Google. Anything Google that you install on any operating system, toolbar,
desktop, Picasa, so on, and so forth. After that, I moved on to Gmail which was
great. For the first month I literally tried to figure out how to find a dong on Gmail,
but then I realized I couldn't. And then after about three years on Gmail,
I led testing efforts for -- we bucket these projects under an umbrella called social at
Google, but projects like Google+, news, blogger, hangouts, voice, et cetera. Things like that.
So very quickly, My first automation project at Google, I call it MAD (Millions of Automated
Documents), because it sounded like it was mad, but we had to create content, a lot of
content which we could stress as the Google search appliance and Google desktop, and so
that's UI automating, Word, Excel, AIM. I don't know how many of you remember what AIM
is, but yes AIM. Back then, Google used to use a variant of
Bugzilla as their internal bug database. It was running under someone's desk. That was
Google ten years back. It started getting really slow because it
turns out one-fifth of the bugs on the bug database were from my team, so she's like,
yeah, you guys need to take care of this. Then it kept getting slower, and I essentially
(indiscernible) just made it 10x faster. What did I do? I just moved it to a faster machine
in a data center. That helps. It's amazing what technology can do, I guess. I wrote down
a 20% project called surveytool. The reason I talk about this is this was my first real-life
example of how not to do things in terms of how this grew organically much to initial
joy, later to dismay. Gint action. In that day and age, there was no AppEngine, no Google
forms, no Google+ Polls , and so it grew to the point that at one point, Google H.R. wanted
to use it as the tool for our annual survey. Within literally I believe it was less than
three minutes, the tool melted after we sent out this email to Googlers. I spent the next
two weeks pretty much recovering data and then I happily retired the tool after a month.
But that was a good learning experience. And one of the reasons I am doing what I am
doing and I like test engineering is when I was doing my bachelor's, I worked with a
buddy and we worked on a microprocessor control pick-and-place manupulator, which is a mouthful
way to say Robot but that really gave me a perspective that while software is great,
hardware is a lot more difficult to mold. really -- for those of you who work in hardware,
I salute you. I almost remember spending 10x more time on hardware
So what is test engineering? It's -- We think we have a dual purpose at Google. We think
we want to build infrastructure and tooling that is going to enable us to test things
at a scale that no one else tests things at. And the other focus we have is we want to
test and automate projects right from things like the autonomous cars to things like Glass
to things like, you know, one of my teams test these, I like to use like how blue is
your blue in terms of photo editing software. Which is a very tricky problem to automate,
if you come to think of it. But that's kind of how we think test engineering
at Google is all about. This is how Google looks from the outside,
you know? Everyone is like, yeah, this is the world of Gmail, Chrome, YouTube, free
food, shuttles, massages, paddleboats. Wow, it's awesome. I'm sure it's probably a fun
place to sit and code, too. And it is, depend ongoing what your definition of fun is.
[ Laughter ] And what part of the code base you are in.
So but this is more like how it is. [ Laughter ]
It surely does feel at least three of the five days of the week.
I guess great things come with great pains. I don't know if that's a saying, or I just
made it up. But with rich products -- I mean, we have great products which have rich integrations
but that leads to pretty and complexing tasks. We have legacy codebases, and the moment that
resonated with me was probably a few years back, when I was on Gmail, I wanted to add
a feature in Gmail to add this thing called Google feedback into Gmail. If you go to Gmail
right now you'll see send feedback. And I thought that should be a two-day thing. I'm
just adding a module, every feature in Gmail was a module, and I'll have it done easily.
Turns out I added a module, but turns out, of course, there was a lot of refactoring
which had to be done in another module. And then I did that there was race conditions
between the module. And then turns out you have to handle the JavaScript. You have to
lazily load it, so you also have to ensure the JavaScript is not too much because you're
causing every user to download this every day.
And then you have to write a unit test. And then you have to write a UI automated test
and then you have to write an integration test. By the time I was done with all of this
it was like two months. I was like, wow, that is shockingly sad.
I did get that done but I very rightly went back to all the guidelines and principles
we had in terms of how to write a feature and what are the things you need to do, and
revised that. It was good to get that perspective. So that's kind of really, I think, a lot of
what Google is. Why this slide? I wanted to take an opportunity
to highlight -- the code throughput at Google continues to grow linearly.
Ever since, I think, Ari spoke about this in the last keynote, and I've been tracking
this metric. I like graphs. And I've been hoping or assuming that this number would
actually slow down as we add more people. Bigger companies, things get harder to do.
That's kind of what you hear. Except at Google, that doesn't seem to be the case.
And another interesting tidbit I found, and this is probably true for every tech company
-- I guess, but it's interesting to see the peaks. You know, there's a peak right before
the holidays, after the holidays, and right before Google I/O. So kind of shows -- yeah,
it's working as intended. That's what engineers do.
[ Laughter ] I got my slides together 5:00 a.m. yesterday,
so that's what happens. And there's the trend down correlates, too.
I mean, it's the holidays. But I have no clue who is still checking in those 10,000 CLs?
Who are those people and why -- Yeah. So that's something we've got to fix. That's way too
much code going in when you shouldn't be getting code in.
Larry strongly believes in velocity and we've had a very sustained push over the years on
velocity and we've made a bunch of things around that, that in turn has made us really
efficient in how we do our releases. And we've had a bunch of other new product offerings.
Google Fiber, things like balloons and whatnot. It's still shock to go see we're doing almost
twice as many releases. I think this is another testament to -- I
mean, I want to give a shout out to the release engineers we have at Google. I think they
have done an incredible job in automating a large part of our release tooling and work
flow to make it really easy for the teams, individual teams, to really sustain themselves
and a lot of their tasks needed for release. Of course, you know, our SREs rock. I mean,
while we continue to push out all these releases, they ensure Google and all our infrastructures
stay up always and have a very high availability. Google actually publishes these number on
our apps -- dashboard for our apps, and it's kind of insane, the reliability we have.
As I've been talking to teams within Google and to folks in the industry, I think theme
emerged to me. A theme about when I started hearing or I heard Googlers who leave Google
and join startups, and they're like, wait, we kind of feel stuck. We're a startup. We
should be able to release quickly; right? A lot of people end up having long release
cycles, and the definition of long can vary from a few weeks to a few months before but
as a result of that, everyone wants to get stuff in because they're like wait, the next
release is a ways out so let's get this out with this release. I call it a lack of discipline.
It leads to further problems. It leads to further iterations because you're trying to
put in stuff which isn't totally baked. It leads to delays. It's kind of a hole you keep
digging yourself into. Once you're in there and if you encounter problems, you're so deep
in the hole that you -- there's no way to isolate it. The only way to get out of that
is to fix that problem and -- fix it and work around it.
The context, this slide is important because this slide is maybe the reason why you guys
are going to be sitting and listening to me for an hour, about a bunch of years back on
Gmail, we suddenly realized we need to iterate faster. While we thought we had for a release
cycle which was weekly, we realized even though we have weekly cycles, by the time our code
makes it to prod, from the time it's cut, it's checked in, it's three weeks.
So while we were cutting weekly, that code took three weeks to get to prod. And that
had a bunch of problems. That model had a bunch of problems because it hindered our
ability to react quickly when we needed to. It also meant that there were three parallel
tracks, which means if you find issues somewhere, you have to ensure you down integrate them
on all the branches, there are conflicts. One specific example I can remember is we
pushed a certain Gmail release to prod, and there was -- the initial page load, aka the
inbox when you load the page, it slowed down by 20% on IE. We had a laser focus on latency
and we still do at Google, and so that wasn't acceptable to us.
We did have a state of the art latency regression system, which could help us detect regressions
of even 25 milliseconds, but that thing didn't show a blip, but on production, we saw a big
spike. And then that led to an interesting wild goose chase for two months, because then
we were iterating on production, it further accentuated the three-week delta problem we
had. And not to mention we were literally pushing code to our prod users and continuously
updating it. After almost a two-month goose chase, we narrowed
it down to a changelist which introduced a pseudo CSS rule which in IE causes all the
rules to be reevaluated every time you move the cursor from one element to another. And
the reason our system didn't catch it is we didn't have hover based automated test back
then because they can be tricky to automate. We did get it added after this.
But that's kind of what really was the aha moment for us. And we said this isn't great.
We need to iterate faster. We need to ship daily.
And that's how we set about this mission. .
Okay. We call this a bio break for the deck. Okay. Moving fast is good; right? Everyone
likes to move fast. But I wanted to add a bit more color to that.
Innovation is no longer just the thing to do. I think you need to do it to survive.
We all know what happens to companies who don't innovate, especially in the tech industry.
It also really is a great way to retain and hire talent because people like to work for
companies that innovate and move fast. And I think moving fast really enables you to
do that. The ability to react quickly to critical bugs
is huge, and I think moving fast gets you there.
This is a chicken and egg problem in my opinion, where you can't react quickly until you move
fast, but you can't move fast till you know if you can react quickly. And at some point
you really just have to take a leap of faith and hope for these two to come together, and
they do. We have tried this umpteen times and it's come through in some or the other
shape or form, in some places painfully, in some places less painfully.
Another side effect of moving faster is productivity is, automation is a key in getting to make
this happen, if you want to move fast, you have the automation in place which means if
you are making changes in your work space, if you actually write code that doesn't compile,
you're immediately going to find out about it. If you're writing a code where you break
a test, you will find out because the tests are continuously running in the background.
And I talk a bit more about how we do that. Or if you try to check in a changelist without
tests in it, we have tools at Google which actually will yell at you, though our human
reviewers do a pretty good job of doing that, too.
And now, I think -- I think this come up in the last couple of years, we have tools at
Google which show incremental code coverage for every change list, which I think is extremely
useful. Lastly, I think why moving fast really helps
is it helps get the focus on code health. In my opinion; Every software product has
a 15-year cycle. It's zero to two years to build it depending on how complex it is, about
a seven-year shelf live, and then three years it's in maintenance mode. And then there are
the last three years where it's in deprecation and migration mode.
Having the focus on code health right from the start really makes this journey on the
later part a lot easier. We know by experience in Google, like, we've had to deal with this
on projects which didn't have code health and had code health and it's a day and night
difference. So again, that's why moving fast is good.
But breaking things isn't. Obviously. The most important part is --, we owe it to
our users to ship high-quality products. And all companies and businesses do. You know,
unbaked features really lead to a crappy user experience, or low quality products just build
up bloat, feel clunky all the time, and again that doesn't result in successful products
in the long term. But also, really a lot of big tech companies
are getting into this phase where they are having a hard time shipping -- shipping, their
launches slip into years, and I think if you have the focus, really, the one way to not
get in that state is really ensuring that you have the focus on code health and we move
fast but you don't ignore quality. So Alister is here somewhere. Yes.
So this -- I'm borrowing this slide, this image from him, because over the years, I
think I found this is a great way to communicate to people what I think about my testing philosophy
or automation philosophy. I believe a lot of teams in companies -- when
they talk about automation, there's a lot more emphasis on UI-based automation and that
is unfortunate because in my opinion, UI automation is actually the hardest thing to do with all
the moving pieces. Very often the people don't just stop at the UI automation. They want
to test the whole stack. My litmus test is I know a team is doing too
much UI automation when they are catching the null pointer exceptions via the UI test,
essentially. And so I believe this cone needs to be inverted
in more teams than not. Manual session-based testing is, in my opinion,
a fancy way of saying exploratory testing, but I believe exploratory testing is great
because -- and I wish my teams can do a lot more of it because UI automation is hitting
the same code paths over and over again and exploratory testing is the way you are going
to be hitting these other code paths which don't get exercise otherwise.
And I strongly believe that if the testers are running the same set of test cases over
and over again, those need to be automated unless in rare circumstances the return on
investment is not there. API tests are, I think, great bang for the
buck. These are relatively simple tests to write if the APIs are defined cleanly. They
are almost always stable and almost always have a binary signal.
And everyone likes integration tests. Everyone is like oh, more integration tests. But again,
I've seen beyond the point integration tests stop being an asset and become a liability
because the signal-to-noise ratio flips. You just have too much noise. So you want to have
integration tests but you want to have a healthy dose of it and not too much.
The component tests are essentially if you have a stack, you start isolating, and I talk
a bit about it later. You test each component in isolation and fake all the other things,
which helps you really not -- run the real world in terms of tests. And, yes, -- unit
tests of course are essential. Google has always had a high bar on developer testing,
and I think we continue to do that. So the rest of this slide is going to be talking
about this flowchart. I tried to take a step back and summarize what are the things we
did to take our journey of moving faster and not breaking things.
I could think of ten things. So let's talk about them.
So the first one is what I call push on amber. But before I talk about that I want to take
a detour and talk about push on green. This should be a concept most of you would have
heard. Kent, I think in 2003, came up with this concept
and I think it's a great one. If you or your company and team is not doing this, by all
means, please start with push on green. At Google what we started r
…
>>Ankit Mehta: Seems like everyone's had a late night.
Welcome. I'm getting old. I'm going to hold on to this, mostly to stick to the script
so you don't make this keynote a three-hour one.
And, yeah, what I was going to be talking about today is a topic which is very near
and dear to my heart --- how do you move fast and don't break things. Because I think each
one of these things are interesting to do in isolation, but to do them together poses
some interesting challenges and opportunities. And we have had a fun ride for about the last
decade where we have done this. And we think we have it down to the point where we would
like to share this with all of you, so you guys can get a perspective of how we try to
do this at Google, but, more importantly, you can help us make this even better going
ahead. Quick thing. GTAC, last time we did this,
this was in New York, it was wildly successful. The New York one was based on a theme. I don't
know how many of you were there. And we somewhat felt constrained with the theme. So this year,
we decided to keep it open-ended. We roughly had about 1500 speaker applications
this time, which was roughly about 3X of last year, which to us, gave us a signal that the
community is still very much engaged with GTAC, which is a great thing. That really
enables us to keep doing this. Out of the 1500, about 200 were Googlers'
applications, with about 1300 were external. And of those, you -- 250 of you are the unlucky
ones or lucky ones. You'll find out in a couple of days, who get to spend two great days in
this room. And the speakers. We have about 15 speakers.
What do I like about GTAC? The booze is great.
[ Laughter ] We can't put it on the slide. Oops. This is
recorded. [ Laughter ]
But on a more serious note, I think what has amazed me about is GTAC to really get a chance
to talk with most of you guys from the industry, and it's an interesting variance in how testing
is approached differently in different industries and different segments of the tech industry.
And I think we value at Google at sharing what we have done within Google, but I think
we have gained more from learning from the experiences some of you guys have shared at
GTAC. So again, thank you in advance to all of you
who are going to be talking about -- the other speakers who are going to be talking about
stuff. So why am I here?
Yeah, I've become one of those guys who spends a decade at a company, but Google is a pretty
damn good company to spend that decade at. Over the years, due to the kind of work I've
done at Google, I'm super passionate about, again, the balance between velocity and quality,
because it's almost like a two-edged sword, and you need to balance that. And I think
I just want to be able to share with you guys what we have done.
A quick history of my career at Google. I started about ten years back leading testing
efforts for Google desktop, there used to be this thing. There was a time people used
to build software for desktops. So did Google. And then I moved on to leading efforts for
everything Google. Anything Google that you install on any operating system, toolbar,
desktop, Picasa, so on, and so forth. After that, I moved on to Gmail which was
great. For the first month I literally tried to figure out how to find a dong on Gmail,
but then I realized I couldn't. And then after about three years on Gmail,
I led testing efforts for -- we bucket these projects under an umbrella called social at
Google, but projects like Google+, news, blogger, hangouts, voice, et cetera. Things like that.
So very quickly, My first automation project at Google, I call it MAD (Millions of Automated
Documents), because it sounded like it was mad, but we had to create content, a lot of
content which we could stress as the Google search appliance and Google desktop, and so
that's UI automating, Word, Excel, AIM. I don't know how many of you remember what AIM
is, but yes AIM. Back then, Google used to use a variant of
Bugzilla as their internal bug database. It was running under someone's desk. That was
Google ten years back. It started getting really slow because it
turns out one-fifth of the bugs on the bug database were from my team, so she's like,
yeah, you guys need to take care of this. Then it kept getting slower, and I essentially
(indiscernible) just made it 10x faster. What did I do? I just moved it to a faster machine
in a data center. That helps. It's amazing what technology can do, I guess. I wrote down
a 20% project called surveytool. The reason I talk about this is this was my first real-life
example of how not to do things in terms of how this grew organically much to initial
joy, later to dismay. Gint action. In that day and age, there was no AppEngine, no Google
forms, no Google+ Polls , and so it grew to the point that at one point, Google H.R. wanted
to use it as the tool for our annual survey. Within literally I believe it was less than
three minutes, the tool melted after we sent out this email to Googlers. I spent the next
two weeks pretty much recovering data and then I happily retired the tool after a month.
But that was a good learning experience. And one of the reasons I am doing what I am
doing and I like test engineering is when I was doing my bachelor's, I worked with a
buddy and we worked on a microprocessor control pick-and-place manupulator, which is a mouthful
way to say Robot but that really gave me a perspective that while software is great,
hardware is a lot more difficult to mold. really -- for those of you who work in hardware,
I salute you. I almost remember spending 10x more time on hardware
So what is test engineering? It's -- We think we have a dual purpose at Google. We think
we want to build infrastructure and tooling that is going to enable us to test things
at a scale that no one else tests things at. And the other focus we have is we want to
test and automate projects right from things like the autonomous cars to things like Glass
to things like, you know, one of my teams test these, I like to use like how blue is
your blue in terms of photo editing software. Which is a very tricky problem to automate,
if you come to think of it. But that's kind of how we think test engineering
at Google is all about. This is how Google looks from the outside,
you know? Everyone is like, yeah, this is the world of Gmail, Chrome, YouTube, free
food, shuttles, massages, paddleboats. Wow, it's awesome. I'm sure it's probably a fun
place to sit and code, too. And it is, depend ongoing what your definition of fun is.
[ Laughter ] And what part of the code base you are in.
So but this is more like how it is. [ Laughter ]
It surely does feel at least three of the five days of the week.
I guess great things come with great pains. I don't know if that's a saying, or I just
made it up. But with rich products -- I mean, we have great products which have rich integrations
but that leads to pretty and complexing tasks. We have legacy codebases, and the moment that
resonated with me was probably a few years back, when I was on Gmail, I wanted to add
a feature in Gmail to add this thing called Google feedback into Gmail. If you go to Gmail
right now you'll see send feedback. And I thought that should be a two-day thing. I'm
just adding a module, every feature in Gmail was a module, and I'll have it done easily.
Turns out I added a module, but turns out, of course, there was a lot of refactoring
which had to be done in another module. And then I did that there was race conditions
between the module. And then turns out you have to handle the JavaScript. You have to
lazily load it, so you also have to ensure the JavaScript is not too much because you're
causing every user to download this every day.
And then you have to write a unit test. And then you have to write a UI automated test
and then you have to write an integration test. By the time I was done with all of this
it was like two months. I was like, wow, that is shockingly sad.
I did get that done but I very rightly went back to all the guidelines and principles
we had in terms of how to write a feature and what are the things you need to do, and
revised that. It was good to get that perspective. So that's kind of really, I think, a lot of
what Google is. Why this slide? I wanted to take an opportunity
to highlight -- the code throughput at Google continues to grow linearly.
Ever since, I think, Ari spoke about this in the last keynote, and I've been tracking
this metric. I like graphs. And I've been hoping or assuming that this number would
actually slow down as we add more people. Bigger companies, things get harder to do.
That's kind of what you hear. Except at Google, that doesn't seem to be the case.
And another interesting tidbit I found, and this is probably true for every tech company
-- I guess, but it's interesting to see the peaks. You know, there's a peak right before
the holidays, after the holidays, and right before Google I/O. So kind of shows -- yeah,
it's working as intended. That's what engineers do.
[ Laughter ] I got my slides together 5:00 a.m. yesterday,
so that's what happens. And there's the trend down correlates, too.
I mean, it's the holidays. But I have no clue who is still checking in those 10,000 CLs?
Who are those people and why -- Yeah. So that's something we've got to fix. That's way too
much code going in when you shouldn't be getting code in.
Larry strongly believes in velocity and we've had a very sustained push over the years on
velocity and we've made a bunch of things around that, that in turn has made us really
efficient in how we do our releases. And we've had a bunch of other new product offerings.
Google Fiber, things like balloons and whatnot. It's still shock to go see we're doing almost
twice as many releases. I think this is another testament to -- I
mean, I want to give a shout out to the release engineers we have at Google. I think they
have done an incredible job in automating a large part of our release tooling and work
flow to make it really easy for the teams, individual teams, to really sustain themselves
and a lot of their tasks needed for release. Of course, you know, our SREs rock. I mean,
while we continue to push out all these releases, they ensure Google and all our infrastructures
stay up always and have a very high availability. Google actually publishes these number on
our apps -- dashboard for our apps, and it's kind of insane, the reliability we have.
As I've been talking to teams within Google and to folks in the industry, I think theme
emerged to me. A theme about when I started hearing or I heard Googlers who leave Google
and join startups, and they're like, wait, we kind of feel stuck. We're a startup. We
should be able to release quickly; right? A lot of people end up having long release
cycles, and the definition of long can vary from a few weeks to a few months before but
as a result of that, everyone wants to get stuff in because they're like wait, the next
release is a ways out so let's get this out with this release. I call it a lack of discipline.
It leads to further problems. It leads to further iterations because you're trying to
put in stuff which isn't totally baked. It leads to delays. It's kind of a hole you keep
digging yourself into. Once you're in there and if you encounter problems, you're so deep
in the hole that you -- there's no way to isolate it. The only way to get out of that
is to fix that problem and -- fix it and work around it.
The context, this slide is important because this slide is maybe the reason why you guys
are going to be sitting and listening to me for an hour, about a bunch of years back on
Gmail, we suddenly realized we need to iterate faster. While we thought we had for a release
cycle which was weekly, we realized even though we have weekly cycles, by the time our code
makes it to prod, from the time it's cut, it's checked in, it's three weeks.
So while we were cutting weekly, that code took three weeks to get to prod. And that
had a bunch of problems. That model had a bunch of problems because it hindered our
ability to react quickly when we needed to. It also meant that there were three parallel
tracks, which means if you find issues somewhere, you have to ensure you down integrate them
on all the branches, there are conflicts. One specific example I can remember is we
pushed a certain Gmail release to prod, and there was -- the initial page load, aka the
inbox when you load the page, it slowed down by 20% on IE. We had a laser focus on latency
and we still do at Google, and so that wasn't acceptable to us.
We did have a state of the art latency regression system, which could help us detect regressions
of even 25 milliseconds, but that thing didn't show a blip, but on production, we saw a big
spike. And then that led to an interesting wild goose chase for two months, because then
we were iterating on production, it further accentuated the three-week delta problem we
had. And not to mention we were literally pushing code to our prod users and continuously
updating it. After almost a two-month goose chase, we narrowed
it down to a changelist which introduced a pseudo CSS rule which in IE causes all the
rules to be reevaluated every time you move the cursor from one element to another. And
the reason our system didn't catch it is we didn't have hover based automated test back
then because they can be tricky to automate. We did get it added after this.
But that's kind of what really was the aha moment for us. And we said this isn't great.
We need to iterate faster. We need to ship daily.
And that's how we set about this mission. .
Okay. We call this a bio break for the deck. Okay. Moving fast is good; right? Everyone
likes to move fast. But I wanted to add a bit more color to that.
Innovation is no longer just the thing to do. I think you need to do it to survive.
We all know what happens to companies who don't innovate, especially in the tech industry.
It also really is a great way to retain and hire talent because people like to work for
companies that innovate and move fast. And I think moving fast really enables you to
do that. The ability to react quickly to critical bugs
is huge, and I think moving fast gets you there.
This is a chicken and egg problem in my opinion, where you can't react quickly until you move
fast, but you can't move fast till you know if you can react quickly. And at some point
you really just have to take a leap of faith and hope for these two to come together, and
they do. We have tried this umpteen times and it's come through in some or the other
shape or form, in some places painfully, in some places less painfully.
Another side effect of moving faster is productivity is, automation is a key in getting to make
this happen, if you want to move fast, you have the automation in place which means if
you are making changes in your work space, if you actually write code that doesn't compile,
you're immediately going to find out about it. If you're writing a code where you break
a test, you will find out because the tests are continuously running in the background.
And I talk a bit more about how we do that. Or if you try to check in a changelist without
tests in it, we have tools at Google which actually will yell at you, though our human
reviewers do a pretty good job of doing that, too.
And now, I think -- I think this come up in the last couple of years, we have tools at
Google which show incremental code coverage for every change list, which I think is extremely
useful. Lastly, I think why moving fast really helps
is it helps get the focus on code health. In my opinion; Every software product has
a 15-year cycle. It's zero to two years to build it depending on how complex it is, about
a seven-year shelf live, and then three years it's in maintenance mode. And then there are
the last three years where it's in deprecation and migration mode.
Having the focus on code health right from the start really makes this journey on the
later part a lot easier. We know by experience in Google, like, we've had to deal with this
on projects which didn't have code health and had code health and it's a day and night
difference. So again, that's why moving fast is good.
But breaking things isn't. Obviously. The most important part is --, we owe it to
our users to ship high-quality products. And all companies and businesses do. You know,
unbaked features really lead to a crappy user experience, or low quality products just build
up bloat, feel clunky all the time, and again that doesn't result in successful products
in the long term. But also, really a lot of big tech companies
are getting into this phase where they are having a hard time shipping -- shipping, their
launches slip into years, and I think if you have the focus, really, the one way to not
get in that state is really ensuring that you have the focus on code health and we move
fast but you don't ignore quality. So Alister is here somewhere. Yes.
So this -- I'm borrowing this slide, this image from him, because over the years, I
think I found this is a great way to communicate to people what I think about my testing philosophy
or automation philosophy. I believe a lot of teams in companies -- when
they talk about automation, there's a lot more emphasis on UI-based automation and that
is unfortunate because in my opinion, UI automation is actually the hardest thing to do with all
the moving pieces. Very often the people don't just stop at the UI automation. They want
to test the whole stack. My litmus test is I know a team is doing too
much UI automation when they are catching the null pointer exceptions via the UI test,
essentially. And so I believe this cone needs to be inverted
in more teams than not. Manual session-based testing is, in my opinion,
a fancy way of saying exploratory testing, but I believe exploratory testing is great
because -- and I wish my teams can do a lot more of it because UI automation is hitting
the same code paths over and over again and exploratory testing is the way you are going
to be hitting these other code paths which don't get exercise otherwise.
And I strongly believe that if the testers are running the same set of test cases over
and over again, those need to be automated unless in rare circumstances the return on
investment is not there. API tests are, I think, great bang for the
buck. These are relatively simple tests to write if the APIs are defined cleanly. They
are almost always stable and almost always have a binary signal.
And everyone likes integration tests. Everyone is like oh, more integration tests. But again,
I've seen beyond the point integration tests stop being an asset and become a liability
because the signal-to-noise ratio flips. You just have too much noise. So you want to have
integration tests but you want to have a healthy dose of it and not too much.
The component tests are essentially if you have a stack, you start isolating, and I talk
a bit about it later. You test each component in isolation and fake all the other things,
which helps you really not -- run the real world in terms of tests. And, yes, -- unit
tests of course are essential. Google has always had a high bar on developer testing,
and I think we continue to do that. So the rest of this slide is going to be talking
about this flowchart. I tried to take a step back and summarize what are the things we
did to take our journey of moving faster and not breaking things.
I could think of ten things. So let's talk about them.
So the first one is what I call push on amber. But before I talk about that I want to take
a detour and talk about push on green. This should be a concept most of you would have
heard. Kent, I think in 2003, came up with this concept
and I think it's a great one. If you or your company and team is not doing this, by all
means, please start with push on green. At Google what we started r
…