The discussion centers on the pragmatic approach to software development, particularly debating the necessity and effectiveness of unit tests versus alternative strategies like robust tooling, code reviews, and focusing on user needs, with a strong emphasis on developer productivity and shipping value quickly.
Mind Map
クリックして展開
クリックしてインタラクティブなマインドマップを確認
oh yeah my last point there is i i trust
the history of what my developers did
more than the history of what they think
they did and merge is what happened
rebase is what they think happened and
what they want to pretend happened if
you're exclusively working with
developers whose imagination is better
than their commits rebase away but i
want to make sure that our history is true
true
is the true history so i think this is
where this is this bleeds right into the
next topic which is i don't believe my
imagination plays a single role because
i am an aggressive unit tester
and i have all my code unit tested and
perfectly working so when i make a
refactor i'm confident in this moment
that i have not changed
anything or the behavior without having
to go through integration now we may be
using the term integration or end-to-end
tests uh slightly different right people
tend to use these phrases in all sorts
of ways i don't think i've ever heard
someone use the term functional test in
the correct way
ever it's i i don't think anyone has any
idea what that actually means uh but
all right so this this might be part of
the problem which is you
don't think
unit tests
should be done in fact i believe you
said something along the lines of
someone being a delusional if they do
unit tests probably not that intense but
it was pretty intense
i thought i was vehemently anti-unit
test oh you kept going i believe you
even tossed some shade my way at one point
point
you probably even said like my hair
wasn't as good as yours or something
like that
the tick tock came out great though
thank you for that it's one of the best
ones i've done so far don't don't
compliment this take talk this is not
the time for tick tock it's it's your
tick tock to be fair i just stole your content
content
oh that's great i didn't put my message
in there oh really just you but yeah it
was wonderful oh that's awesome
oh yeah i got like that that was
fantastic i'm always fine i'm always
fine being a vic a collateral victim of
tick-tock damage i'll i'll accept it in
fact you know i was the only victim
there you call me an ignorant [ __ ] i'm
not saying it's inaccurate i'm just
saying that i'm the victim i know you
were you were the victim in the moment
okay but uh you know what i deserved one
that was the nicest thing someone has
posted about me in a while right usually
when people take something it is in a
negative fashion i felt like
at least there was some love there maybe
some some anger yeah i mostly love you
there's my community has a lot of love
for you i was surprised because i've
definitely seen you catch some heat but
i think we're on like the same
wavelength and a lot of ways there where
it's like we're not we're not here to
make people happy we're here to talk
about how [ __ ] gets done yeah and
it gets spicy if it has to yeah all
right all right
just for the unit tests though i
i i want again i'm gonna preface this
with something oh wait can i do can i do
one quick thing sorry i'm sorry that i
just interrupted you right there i'm
gonna do a quick poll in chat that says
who likes unit tests or uh do you
think unit tests
are good
yes and no so answer truthfully because
afterwards i would like to see the outcome
outcome
to this if anybody was swayed obviously
this is a bias test in the sense that
new people are going to join towards the
end they'll vote on it like oh yeah of
course they're great right and so we
won't get accurate results but it'll be
interesting to see
if uh you know if you've changed
people's minds if things have if things
have changed
so go on
start start your unit test
test these so the
the [Music]
[Music]
issue i i have a handful that you need
to test but i want to give like one
important addendum at the beginning here
which is a lot of my stance here
is trying to be
the opposite extreme from what i
consider the problem which is the
everything should be unit test people
i think that no unit tests is
significantly better than 100 code
coverage in the sense that
where i am the
wrong extreme you might believe i am at
prevents my engineers from being
hampered by [ __ ] and i am always
looking for [ __ ] to trim i want my
engineers working on things and i want
the teams i'm in working on things that
make them more productive effective and
confident as they solve problems for our
users and where
most of the code i see shipping goes
unit tests are slowing them down more
than helping and they're often reached
for due to other problems my concern is
there are tests being written to
create contracts and expectations where
the codes architecture itself should be
enforcing them okay so i think we do
fall on the same page that i think we
both would universally agree that 100
coverage is just terrible right it's
easy to have a bug and have 100 coverage
just do a list and access a list and
have the second thing be undefined right
you can just bam you know hit it
undefined right very very easy and
number two i have up
okay so i want to go off on this one for
a little bit because i totally agree
like the undefined in a list problem is
so real and we hit it in production and
it turns out and this is really fun
learning there's actually a typescript
rule for accessing an index and not
checking if it's defined or not because
when you're grabbing from an array you
can't know yeah and we turn that on we
found like three other places that that
bug could have happened in the code base
as a result and once again my stance of
good tools are better than okay tests
was proven out because we could have
written 100 unit tests and not caught
that but we turned on one typescript
rule and caught three okay so i do i do
think that good tools
help prevent a class of bugs or
something along those lines so i i'm
gonna agree with you on that one but
okay tests i mean it sounds like you
have poor test writers so we'll just
we'll ignore that for a second uh the
second thing is that i think we both
agree i try to find the common ground
before we get on the argument here is
that we both agree that tdd defining the
test first and then building your
architecture around the test is the
devil's tool and should be thrown away
immediately i personally will never
define my architecture based on how i'm
going to write my tests i usually do it
a little bit you know the other way around
around
uh are you you must not be on the td d
d
yeah no i can't
can't
i i'm a big udd fan user driven
development we do things because our
users need them and tests are not the
best way to define those things like the
problems we're solving exist much more
in user land than they do in like
like
if i was a back-end developer building
long-running systems and i needed to
make sure things were staying super
stable and never leaking probably a
little cooler with test driven
development and unit tests but
building product for users tdd
tdd
pulls you far enough away from the user
that you lose track of what you're
building pretty regularly and that's
been the experience with every tdd team
building user first software that i've
ever worked in okay
uh so i mean i i think you might be
drawing like a some sort of like false
dichotomy here that somehow udd is
opposed well tdd let's just put that in
the trash okay nobody likes tdd we all
agree that tdd sucks but tests
incorporated along with writing code
somehow takes you away from the user
so good example i'll throw out an
example here uh one of my jobs was we
had an ipad in which we were one of the
first people to kind of do like a google
docs like presentation where you could
view you know a document on several
different platforms including one that's
written in the old uh action script way
back in the day and also in html5 when
html5 was hot and or html4 maybe it was
at that point i can't even remember what
it was at but either way we did this and
if you touched the ipad and drug your
finger dragged drugged drug edited your
finger across it you would highlight
some text and that text would be in the
exact same pixel position as it would be
in action script on say a computer or
using the app versus on the ipad you
could make comments and all that now
touching the screen obviously involves a
couple points coming in which then takes
this huge block of text and now we have
to find exactly the point translation to
text position now i could have just
whipped out that feature in a day but
instead i took two three days and made a
very comprehensive set of tests and that
feature still actually is running
untouched and used by a lot of people
and no one's had to touch or change that
code and it's been a decade simply
because it is well thought through and
well tested so everyone knows the exact
ins and outs of the whole nine yards so
is udd so it's actually never been
changed or when it or was it never
changed or easy to fix when things broke
because those are different both easy to
fix because there was a couple bugs
because i had a very comprehensive set
of tests so they knew where the bug was
and how to fix that right and second uh
in a sense having a set of kind of tests
especially when it comes to complicated
data structures and how you use them uh
you have to uh
uh
need a way to kind of like describe that
to the developer i'm not convinced
writing a description of how to search
through a document is the best versus an
example that actually is a test right to
me that is a much better developer
experience in my personal book to
actually see the thing get used exactly
how to pinpoint it and understand it and
so i both lean on tests as a way to
prove correctness and as a way for
someone to get effectively tech
documentation for free
yeah i use typescript for the latter and
i use
documents plus explain the latter loom
to do quick recordings uh typescript
definitions give you 90
plus of what you need when you're
consuming something like you can hover
over you see what it takes and then it
tells you what it returns as long as
they're named okay and i would say like
the big thing we lean on hardest is code
reviews and like pushing for us to be
like raising the bar constantly i
obviously cannot say that that scales
forever but if you build a culture
around it it keeps you from running into
a lot of the problems that like unit
tests solve in those ways i i don't like
unit tests as a demonstration of how
code works and should be used because
more often than not unit tests are
breaking those assumptions with the
weird ways they mock [ __ ] and that's the
problem with unit tests is to test an
individual unit you're definitely
mocking something and you're probably
mocking it poorly because every mock
i've ever seen was bad every single
phone okay so that might be how your you
might be running into people writing it
poorly so i do have a rule when writing
tests which is anything that interacts
with the environment is usually behind
an interface so there is no mocking for
me i actually provide an object in which
abstracts it out right so it's just
simple calls in and out whereas i don't
actually have to worry about a lot of
the mocking type at least any sort of
like uh dependency injection style ones
which i find to be where all the
confusion happens when you have these
weird just import mocks and you're
taking all the imports and you're
rewriting the imports and then you're
hoping it goes all the way through i
find that to be where a lot of the
confusion is versus like hey when i'm
gonna do some sort of like say i'm
setting the audio render rate on a
television i'm gonna have this as part
of the config and that way i can provide
the render rate to my player and not
actually have it query some sort of
outside item and so kind of inverting
the control if you will i think they
have a name for that specifically yeah
yeah
the amount of thought that has to go
into these things and developing these
patterns and onboarding people to use
them properly and maintaining them and
then realizing they're bad and gutting
them and figuring out which tests
actually bring value and not and it
becomes a whole
additional thing to maintain that is
relatively big i actually keep a list i
don't think i have it here uh so i guess
the invert of this would be integration
right i mean integration tests
themselves are extremely complicated to
set up they break all the time data is
unreliable you have to do this whole
data mocking layer which is always out
of date it's like never end date i've
never seen someone data mock successful
ever we've done it a lot of times at
netflix it is just one of the worst
things services are going down you have
to have an entire staging environment
then the staging environment has to be
scaled because if your tests all run at
once or a bunch of developers come in at
once you have horizontal scaling issues
you're doing you're solving production
in not production at all points so i'm
not convinced that the other way is even
remotely easier
in any sense and creating tests that
often especially with webrtc you're in
webrtc world like how do you do uh
sequence out of order in rtp you know
how do you do uh forward air correction
uh read solomon how do you take care of
all these really complicated things if
you were to own it without something
that more defines or controls the
network and puts it in in a very
systematic way
i wouldn't i would outsource with
somebody who does and i'm almost
positive everybody will ever outsource
our webrtc info to or if we start hiring
enough people doing that ourselves they
will probably be much more pro unit
tests than me and they will probably
ignore my stance here and go and do
those things i think that unit tests are
reached to way too often for software
that the majority of is not things that
have sequence out of or order errors
they have the items in my cart aren't
going into the order errors the vast
majority of the web is e-commerce and
mostly static sites the vast majority of
servers are just hosting wordpress and
postgres the vast majority of the things
that we do
are slowed down by unit tests and that's
why i take the extreme stance i do i
think you need a good reason to have
unit tests rather than a good reason to
not and i think you're describing things
that are good reasons to have unit tests
i think that your job and your role that
makes sense a lot of the time i think
the majority of people even in your chat
here are probably writing too many unit
tests at their job
so how do you know so so we'll go we'll
go with your theory here how do you know
when you've written too many unit tests
if you say none we're fist fighting i
i really don't think this is it
because because i almost feel like it's
harder yours than mine mine is just
simply i'm gonna think of a few edge
conditions that i know will exist say i
have a bunch of items that could be
added in various orders and put into a
cart i expect if i handed this list it
just produces me the list in the same
order right that's a pretty happy case
very simple if someone decides i want to
add a new filtering mechanism it's going
to break and they go oh crap we broke
the default user experience either i
have to change the test
or make mine right as some sort of
option right it's like a catch because
they don't know
the history
so how would you how would you know when
that's too much
that already sounds like too much to me
like you have a list you use the list
the list is the same i don't i wouldn't
test that i i trust react to do i this
is the thing i see the most of that i
saw a ton of a twitch basically testing
if you pass a component props it gets
the props well yeah the act works
what yeah no no i agree that that is not
a very good problem but i'm just saying
like specific you know because there's
always filtering and all that like say
you want to do something smart some sort
of logic on top of it to make this list
into something else at some point how do
you know somebody's not breaking that
because it may not show up in user
metrics right it doesn't always just
because you make something doesn't mean
people quit using the product they'll
find a way around it you'll often see
this where people just do the just
dumbest things you're like why are you
doing that they're like well actually
it's quit working and then this is how
we solve it
you're like oh yeah you just worked
around it i think this is where like for
us user obsession is super important we
are constantly hopping into users rooms
seeing how they use the product see
what's going right and wrong and we've
already had times where feedback didn't
get to us here that we had to like find
ourselves by digging in really hard and i
i i have we've had one bug so far total
since we started shipping in august that
a unit test reasonably could have caught
and the amount of inertia we would have
had to just
eat to
start integrating
tests into our workflows like our time
to like our timed response for a
production issue that a user reports to
us is under seven minutes for a
production fix from when it's reported
and i'm aiming to get that lower right
now i want us on top of our [ __ ] and
unit tests don't help there my general
philosophy is build safety nets not
guard rails and test unit tests are the
most guard raily of like it feels like
bumpers being put on like bowling alleys
except half the time you hit them
they're just blocking you from doing the
thing you intend to do and you have to
remove the bumper yourself and now you
don't trust bumpers at all anymore and
the experience especially for newer devs sucks
sucks
i want to build the best safety nets
possible so when things do go wrong we
can revert the thing that caused it to
go wrong quick and we can address the
problem as fast as possible and if we
cause another problem in doing that we
can address that as well obviously as
systems get bigger this gets harder but
i have seen time and time again
especially at companies like twitch that
you can do this if you put your time and
energy into the right places you can
build in a way that is focused on fixing
problems when they come up rather than
guiding people to solutions and
solving problems with guardrails that
you don't know about yet i don't trust
my developers to guess the edge cases
better than i trust my users to run into them
them
okay so then i i do
so how do you test right so how do you
have confidence that when someone makes
a change it doesn't blow up so you know
obviously there's the integration
there's literal manual qa
what is your what and if you do
integration what's your integration
story look like
we don't we open up the pr build and hop
in a call if it's something that touches
any of the scary parts the vast majority
of the code we're shipping nowadays is
css changes and feature flags being
turned like on and off and moved from
place to place we
we
like here's a fun example of a
production issue we had we added this
one just happened uh yesterday actually
i'm sure you might have noticed in ping
right now i'm flashing and yesterday
when we talked i wasn't we had added a
network test in the preview before you
joined the call that instantiated the
webrtc client and then killed it and
instantiated a new one when you joined
the call
that broke the signaling process that
passes us the volume indication levels
there is literally no way in hell we
could have tested that because it was an
external sdk with a memory error inside
of it that
when killed and reinstantiated breaks
this one pipe inside of it and those are
the things that we tend to run into in production
production
those are the bugs we have
and being able to get bisect find the
last time it worked
quickly comment out that hook and have
everything working in production
it's pretty good
so then i guess the follow-up question was
was
you don't you don't do any integration
testing would have an integration test
simply caught this
like a puppeteer a puppeteer would not
have been able to start two streams and
no a puppeteer would not at least not
trivially be able to spin up an audio
and video device and then send down
arbitrary audio and detect that the
levels are being picked up on the other
side that would be
i would have lost two of my engineers
for three months trying to do that but
we fixed every five minutes instead and
that's just not a worthwhile trade and i
would be hard to convince that there's a
scale where that makes sense other than
you are selling these services to other
companies i
don't believe the amount of effort it
would take to
properly mock and rebuild those things
would be worthwhile almost ever when we
could build our own sdk for our webrtc
stuff instead when we run into problems
like that where it feels like a test
would bring value
fixing the thing that makes us so
uncomfortable we want to test is better
every single time but how do you know
you even got it right besides for just
manual testing it sounds like you're
annual testing so you instead of so if
you had to fix one place a hundred times
you would have to manually test that one
place 100 times
yeah and i probably tested our call
logic over 100 times at this point would
that not slow you down in the greater end
end
i mean i guess that some parts are just
you can't you can't lock out right like
i do understand that some things are
just they're just too hard and webrtc i
can kind of buy that obviously there's g stream
stream
right well how do you know you get it
right when you rewrite it like how do
you know any part of the things you do
that you haven't introduced small bugs
that aren't causing customer friction
which ultimately like personally
uh every time i've entered used a
product that has a series of small bugs
over and over again i find that more
frustrating than a place that has one
big bug every now and then
right like small things drive me way
more nuts when i just use it on my crap
this thing's broken oh okay today this
thing's broken like how do you prevent
that as your team grows to say 20 50 100 engineers
