So today's standup, we are going to talk
about TDD. Okay, we're not going to have
some topical topic. We're going to go
after the big stuff. Okay, TDD. Is it
good? Is it bad? Do you use it? TJ's
just smiling, looking like an absolute
champ in that freeze frame, dude.
Somehow TJ's the only human I know that
gets freeze framed and it's good
picture. Yep. Yeah. Look at DJ. Look at
that. I can handle this right now.
I cannot handle this right now.
All right, Casey, you probably have the
strongest opinion potentially out of all
of us.
Okay. Okay.
I doubt that, but okay. All right.
Because I also have a strong opinion,
but I am I'm willing probably you then.
Yeah, I think Prime Prime, you got to
lead off the strong opinions here. I
I'll lead off with the strong one, which
is So, I recently just tried another
round of TDD. Okay, Prime Prime, you
have to say, "Hey everyone, welcome to
the standup. We got a great show for you
tonight. I'm your host of Prime Agen.
With me as always are TJ. Hello trash.
Hey everyone, sir. Great to be here
Prime. Thanks. Today's topic is tester.
Have you ever heard a podcast
before? Thanks everybody for joining.
You're watching the hottest podcast on
the greatest software engineering
topics, the standup. Today we have with
us Tee who's in the basement fixing something.
Is that a Home Depot ad? Look, he's
showing us his router. We have with
those trash dev broke a toe winning a
judo competition still living off that
high. And Semor Moratory the actual good
programmer among us. Oh, all right. I
got a lot to live up to. Yeah, you got
Don't worry, it's not hard to live up
to, at least in this crowd. I'll kind of
start us off on this one. So, we've been
developing a game called The Towers of
Mordoria. We came up with the name from
an AI and AI generated generated that
name for us. And effectively, I decided
at one point I have this kind of really
blackbox experience. I have a deck of
cards that I need to be able to draw
cards out. When they're played, they
either need to go to the discard pile or
the exhaust pile. When there's no more
cards in my hand or not enough to be
able to draw cards, I need to be able to
take all the discard cards. Dude, teach
is just flexing on us touching grass
right now. Mhm. put uh put all the
discard cards the shuffle everything up
and then draw new cards back out.
Replenish the draw pile, right? This
sounds like a
TDD pinnacle problem. There's black
boxes. There's a few operations. I'm
going to be able to do this. And so I
just did this like a week and a half ago
on stream. It was amazing. I did the
whole thing, designed the entire API in
the test, then set up all the tests, set
up it all to fail, then it did work or
then it failed. Then I made it work. and
it worked and it was great and
everything was perfect and I was like
there we go, we just did it. I just TDD
and it was a good experience. So then I
took what I made and I went to integrate
it into the game and realized I made an
interface that's super good for testing
and not actually good for the actual
thing I was making and I had to redo it.
And so it always, you know, that's kind
of been a lot of my experience with TDD
is that either the thing is simple
enough and it's super blackbox enough
that I can make it and both the testing
interface and the usage interface is the
same or I create an interface that's
really great for testing and it's
actually not that great for the
practical use case. And I know people
talk about how it's a great way to
develop your, you know, your
architecture and all this kind of stuff.
I've just yet to really buy it. But I
will say the thing that I do like doing
is on problems that are hard to develop
and have many moving constraints. I do
like to cocreate tests with it. Meaning
that I come up with an idea. How do I
want to use it? I program it out. Then I
go create a test which is backwards from
TDD. Go, okay, this is what I want to
see. Am I even correct on this? Can I
get a fast iteration loop being like,
no, no, no, yes. Okay, got it. Next
thing I want to make sure it does this.
No, no, no. Yes. Okay, got it. And so I
do like the fast um iteration cycle of
getting tests in early but few problems
feel like I can actually have that and I
feel like this all things need to be
tested immediately and need to be
written first. You just write dumb
interfaces everything becomes inversion
of control. Everything's an interface
and you just have this abstract mess at
the end of it because testing was the
requirement not hey use it when it's
good. So, that's my TDD experience and I
feel fairly strong about that. And if
you say TDD is good, I feel like I'm
I can respond to that. Or Casey, if you
have strong feelings. No, I think Trash
Let's Let's uh let's let's go to trash
You first. You couldn't like deter me
from doing any task than to tell me I
need to like start off by writing tests
first off. Second, I kind of had a very
similar experience recently building.
Hold on. Can you Can you rewind for a
second? What What did you just say for
your first point? I said you couldn't
deter me from doing work if you let off
with me having to write tests first.
I don't even know what that means. You
mean you couldn't deter me anymore? Like
that's the most deterring thing or Oh,
man. Am I stupid? You are. Can you try
using a differentuse?
Okay, let's let's reward this. Flip,
please take this out. Flip, leave it in.
I don't like test driven development.
However, okay, however, however, I just
built something recently at work which
was very kind of similar to what Prime
was saying. Um, I ended up doing a
but you wouldn't notice this, but I'm
actually really kind of passionate about
testing. So I'm going to kind of like
when I test I would
say 2% of the time I actually lead off
with test driven development and the
rest is usually post implementation because
because
one requirements and all these things
change afterwards, right? So it's like
why write tests for things that are
potentially going to change and you
don't really know what's going to
happen. But I do write tests for very
granular things that I think are going
to be used all the time. So for
instance, like if I wrote a parser for
like my CLI, the parser is going to be
reused by multiple commands, probably
never going to change. So I typically
will write that and I'll also also write
tests for things that take multiple
steps to test. So like, you know, people
argue for end to end tests because you
test just like a user. So that's kind of
how I like to develop. I like to use it
like I would use it for anybody else.
But if I got to do like three steps to
get to the point I want to test, I'm
like, "All right, screw this. I'm just
going to write a test specifically for
this like instance." and then I save
what save time that way. So that's
usually the time I arrive at test-driven
development is specifically in those
scenarios. Otherwise it's purely just
like you know uh what do you call it
integration testing and stuff like that
because you end up with stuff like like
prime said like you when you write tests
you write everything so granular and
everything works so modular that you
just don't even know what it's like when
you actually put them together. So
that's why that's kind of like the way I
approach things. That being said, like
people that kind of
like live and die by test-driven
development, I don't I don't get it. Um,
proTdd. That's all I got for right
now. I'm I'm uh pretty much right there
with trash on this one. My My feeling on
TDD is pretty much the same as my
feeling on oop, right? The problem is
not if you end up with an object that or
something that looks like an object in
your code in oop that's fine. The
problem is the object-oriented part the
like thinking in terms of that wastes a
lot of your time and leads you to bad
architecture. I think TDD is the same
way. I think testing is very good but
test driven development meaning forcing
the programmer to think in terms of
tests during development like that is
what your primary thing is as I'm making
tests to drive the development the TD
part that's actually the problem. I
think the reason for that pretty
straightforward. Uh, and it's kind of
weird because usually the people who
advocate for TDD are the same people who
would poo poo any similar suggestion.
Like any suggestion that like, oh, we
should measure performance during
development. They'll be like, that's
ridiculous. You're wasting so much
program time. Like, what about the 8,000
tests that no one ever needed because we
ended up deleting that system anyway.
And they're like, well, you know, test
driven development is very good. So, I
think like the problem with all of those
sorts of things, any of them, is just
putting emphasis on something instead of
talking about it like what it really is,
which is a trade-off. If you're spending
time making tests or you're orienting
things towards making tests, that is
necessarily programming time that isn't
being spent doing something else. Like
programming is zero sum. If you spend
time doing one thing, you're not
spending time doing another thing. And
so, for example, if you end up spending
a lot of your time making the interface
revolve around tests, you weren't
spending a lot of your time making the
interface revolve around actual use
cases because actual use cases don't
usually look like tests. Therefore,
sometimes you get the exact situation
that Prime was talking about where you
do test driven development and you end
up with a completely unusable mess,
which is actually very common in my
experience. Right. I wouldn't call it a
mess. You know, I I write pretty good. I
mean, it was it was like pretty dizzy.
Yeah, I just want No, the API the API is
a mess. Not the not the code, right?
Like the code has been tested and it
probably works, but the API is a mess,
right? I mean, that's what happens. Uh
because it wasn't designed specifically
for the kinds of use cases it was meant
to tackle. And so you end up with that
problem. And again, it was shifting the
developer focus was all it took to make
that happen. Good APIs are hard to make
in the first place, right? So if you're
adding more complexity to the
programmer's workflow when they're
trying to make an API that's good for
the, you know, the actual use case,
you're just going to get a worse result.
Again, zero sum. So what I would say is
like testing is better something that
you do as the thing takes shape when
you're like, okay, we now have figured
out like how this system works properly.
We're really happy with it. The API is
working nicely. We check the performance
and we think we've got like reasonable
we've got a reasonable path towards
having this perform really well. Now,
let's start seeing what are the what is
the overlay like what are the parts we
can now start to add some little
substrate to to test that we think are
going to have bugs that will be hard to
find because that's usually the way I
look at like what are the parts of this
system whose bugs will be either very
hard to find or really catastrophic like
they'll cost us a lot of money or
they'll crash the user's computer or
whatever right and then you put the
tests for that right and that's how I've
always approached it and I think that
works pretty well and uh you do want to
do that step, but I don't think you want
testdriven development. It just seems
like a really bad idea. And to be
honest, when I use software that comes
from people who are like TDD advocate,
this organizations TD, whatever, it's
full of bugs all the time, right? Like
I'm not getting these pristine software
things. I mean, I'm I would imagine that
the people at YouTube do a lot of
testing and code reviews and yet for the
past eight years, the play button cannot
properly represent whether something is
playing or not. for eight years, right?
Sometimes it's paused and it has the
play thing and sometimes it's playing
and it has the play thing and and
sometimes both of those times it shows
the pause thing. It's just it doesn't
seem like the testing is working, guys.
So, again, I'd say, you know, I think
you have to kind of rethink how you're
allocating that time. That's what I
would say. Can we get TJ in here? Can
Can Playboy TJ on this one? Can we Am I
Am I coming in loud and clear? Yeah,
your voice sounds good. Your voice
sounds like trash. Looks good. Thank
you. Thank you. So, I would I'm I'm uh I
think I agree mostly with what everybody
is saying. Although, I would say for a
few kinds of
problems, the problem is I I I think
people when you say TDD, they just mean
10,000 different things. So, this is
like one of the problems, right? There's
like people who are like TDD is only if
you do red green cycle before you write
the code. you have to write the tests
and like you can't write any code till
you've written a failing test and blah
blah blah which like I get
uh that maybe they're the people who
invented the word so they can define it
that way. Uh but like I find that not
not very useful for some stuff but uh
there are like testing or like projects
where I think I test like basically
everything and there's just ways that
you can do it that are like more
effective versus less effective. Like my
favorite kind of testing is if you can
find a way to do like snapshot or like
golden testing whatever you know people
have different names for it but where
you can basically just write a test and
then say assert that the output is the
same as it used to be right uh like so
and you usually I've seen those called
snapshot tests or expect tests I find
that those are likeing often yeah yeah
or and like people will use them in
regression but like the kinds that I've
done it before like uh maybe I'm
building up a complex data structure to
like represent the state of like a bunch
of users or maybe it's the parse tree of
something or maybe it's the type system
what like whatever it is if you can find
a way to represent that in like a human
readable way usually just by like
printing it in tests that I've done you
can just like compare the diff between
this one and before you made the changes
and they're they should be fast at least
if they're not fast then that's a
separate problem like your tests have to
be fast for people to run them otherwise
because they don't run them. Um then
then those kinds of things tend to be I
find like very effective and give you a
lot of confidence that you haven't like
scuffed anything throughout the system
with your changes, right? Because you're
sort of like asserting the entire state
of something and if something changes
it's really easy to update because you
just say oh snapshot update this cool
done one second all your tests are
updated. You walk through them and
change them. You get some other side
benefits too like in code review you
literally see the diff of the snapshot
and you can be like um that looks wrong
that used to say false and now it says
true those things that shouldn't have
been flipped between those. So there are
like certain problems where I've employed
employed
like nearly TDD in this snapshot style
in the sense that the snapshot always
fails on the first time you run it
because there's nothing to
expect and then you accept that snapshot
if it looks good or keep changing the
code until it does look good and then
you like move on to the next thing. So,
but that but it's kind of like a unique
case. Like I wouldn't want to snapshot
test a
like HTML page, right? It would just
like there's no way that they're
reproducible it feels like. Uh anyways,
uh but like so there's there's it's only
for like certain subsets of problems
which have been more effective like for
me because I've made a lot of dev tools
in my career and stuff like that. So you
want to do like verify that all of the
references of something look like XYZ.
Okay. very easy to do with snapshots.
Um, that's so that's kind of the main
the the one thing that I wanted to throw
out there is like that style of testing
I find really powerful and I've used it
a lot like across projects you wouldn't
really expect as well. I've always had
the opposite experience with snapshot
tests to I I feel like it's okay maybe
if like it's you and maybe some other
people that actually know the expected
output of a snapshot. My experience is
it changes and they're like, "Oh, let's
just update it." And then the test pass
again and then you're kind of just left
with this snapshot and you're just like,
"All right." So, it's to the point like
where it's like, "All right, let's just
get rid of them because they're just
changing every time requirements change
and no one actually knows what the
expected output is." Um, one more point
I want to make. Someone said something
about like test helper refactoring. I
think that's true after a product is
actually mature because you're just
gonna be it's it's kind of like a
snapshot test where you're just gonna
keep updating your tests every time
something changes to the point where
it's kind of pointless. So I always find
when tests are like that I usually just
delete them immediately. Oh yeah, go
ahead try or I called you trash all my
trash. Your your comment filled Prime
with so much joy it translated into
physical up and down motion. Yeah, I I
was literally watching him as I was
talking and I was like, "Let's just see
how hype we can get Pride." Yeah, I wish
I could see you guys. I see nothing.
Yeah. Yeah, we see you. We see you guys.
Yeah, we could. We can see you, DJ.
Enjoy. TJ, if you see a strange mushroom
in the grass, do not eat it. Oh, yeah.
Don't eat a strange mushroom, DJ. When
you're mowing, there's a strange
mushroom. Guys, do I I don't have my AI
glasses. Do I eat this or not? Do I eat
it or not, guys? I No, don't eat the
mushroom. Eat the mushroom. That's all I
heard. Okay, I'm going for it. Prime,
say your thing. All right.
All right. So, the reason why I actually
agree with Trash a lot on this one is
that I've done a lot of snapshot testing
and I think there's probably does exist
a world where this makes a lot of sense.
It's just all the times I've ever done
it, whenever a requirement change or I
have to make a change that warrants
changing the snapshot, I've also had to
make some decent some non-minor change
also to some logic throughout the
program. So when my snapshot breaks, I
also don't know if I got it right. So
now I'm like trying to hand eyeball
through this this thing like one at a
line like okay, is this is this really
what I meant it for it to be in this
moment? How do I know I should change
this from true to false? Like I have to
re I have to re-reason about everything.
And I find that true snapshot testing on
big things is is feels very difficult
though. I love it. Like I want like
ultimately that's what I always want. I
I fall into the hey let's just make a
golden trap constantly. Like that is
like my default go-to is just like it's
golden time and then I always get sad.
As far as like the whole um unit test
breaking and mature product thing. I
just find that testing generally I do
not think testing helps for refactoring
unless if it's a true endto-end test.
Like you use the product from the very
tippity top and you only have the very
bottom of the output. You have nothing
else you look at. But often I don't like
that's really hard to test because I'll
write a function that I know is very
difficult to get right and I just want
to test those five things. So, I'm gonna
make a I'm gonna make a test that's
highly coupled to that piece. And if you
decide to refactor it, yeah, the tests
all break. Like, that's just a part of
it. I wrote it so that this one thing
works this one way and I wrote a test
that runs that one thing to make sure
you didn't screw it up if you had to
make a minor bug fix to it. I don't try
to do that whole um I just really think
this whole idea of unit testing and oh,
it will prevent you from having
headaches while refactoring has just
been a lie from the beginning. And then
what you'll hear is people do the exact
same thing every time. Oh, that's
because you're testing at the wrong
level. It's just like, I'm just testing
the things that I want to test. I don't
test all the things. And what happens if
you're testing on the wrong level just
simply means that you broke a whole
bunch of tests, but then some of your
tests didn't and that gave you the
confidence that you did the right thing.
And I just don't I just am not going to
test the universe. I'm going to have a
few end to end. I'm going to have the
hard parts and that's that. No more. I
also think like I mean when I'm thinking
back to times when I've when I've done testing
testing
uh like a lot of times when I when I
will break out testing it's for things
that I know aren't really all that
refactorable or changeable anyway. It's
like okay I you know I'm made this
memory allocator and we pretty much know
what it does right you have these like
two ways you can allocate memory from it
and you have one way to free or
something like that and then I just have
these tests to make sure that you know
it doesn't end up in bad states or
whatever it is and and that's a really
useful thing right I've definitely done
that I do that for math libraries like
we kind of all know what the sign of
something should return so if I'm making
a math library that implements the sign function
function
That's not like I don't have to think
too hard about refactoring sign because
math has been around for like you know
hundreds of years and isn't going to
change tomorrow. Like oh bad news guys
like React updated and now the
definition the mathematical definition
of sign is different today. So I'm so
happy that's not even a possibility that
React does stuff with math because we'd
be guys there's still time. There's
still time.
Yeah. So uh so I do I do like them for
that and and actually like there was a I
can think of a specific time uh when I
was shipping game libraries when uh we
had a really bad bug uh that caused a
problem for one particular customer that
took forever to find and it was because
it was in a thing that I had added to
the library just for that customer as
like a back door, right? They they like,
"We need this one thing really quick."
And I was like, "Okay." And I put it in
there. And it had a bug in it. And of
course, it wasn't in any of my standard
testing or anything. It wasn't even
really in anyone's use case, right? And
uh and so like, you know, that was to me
I was like, "Okay, that's a pretty
that's pretty good evidence that like
the testing that I was doing before was
good and like wasn't a waste of time
because like the one time it didn't
happen, it caused a significant problem,
right? And so, you know, presumably I
would have had more of those in actual
other customers had the main had all the
stuff that normally ships to the library
not been tested in the way that I
thought it should have been. And, you
know, from then I was always like, you
know, don't never do a one-off ad. Like,
if I'm going to add something for a
customer, I'll add it like the right way
and actually put in the testing for it
or I'll just tell them I'm sorry, I
can't do that right now because, you
know, I'm too busy or whatever. And so I
do think like yeah when you can actually
talk about this is isolated this is what
this thing does I'm pretty sure it's not
going to change I think the tests are
great but before then yeah it's just and
also this was just kind of going off on
a random anecdote but uh just to circle
back the thing that prime was saying
absolutely true it's like refactor
testing not really like that's only you
can only really get like a very high
level confidence from that you can't
really know there could be all sorts of
things that aren't really working
because whole system testing like that
is going to miss it's going to miss like
weird edge cases that can overlap and
the only way you catch those with test
development is knowing about those edge
cases and and making tests to kind of
target them and you can't do that if you
just refactored everything because now
the edge cases are different right uh so
yeah I think I think TDD people
overstate the degree to which that
really works that's why I'm um I'm a
huge fan of PDD prodriven development I
think that Every time you release and
you have a bunch of people that use your
product, you should do a slow release.
You should measure the outcome because
it's just going to test all the weird
things you're never going to be able to
think of or set up. And if it starts if
it starts breaking, you know that you
screwed up. Previous version worked
better than the new version, you can't
release this new version. You got to
back it up. Not PD PDD. Okay? Stop
saying there's no baby oil in this
situation. Okay? But I mean to me shots
fired at Sonos or Sonus or whatever that
company is called. I don't know what
that I don't know what they did. I don't know
know
what you don't you guys are supposed to
be the webdev people. What's going the
hell are you talking about Casey? Sonos
the company that makes the speakers. Nobody.
Nobody.
Are you kidding me? You
guys beat by Dre. No, this is a this is
like legendary. Like everyone knew about
this. So there's this company called
Sonos. They make like the most popular
internet powered speakers in the world.
Like they everyone has these things. I
mean, not me because I'll There's no way
in hell I'm connecting speakers to the
internet. That's just a way to make them
not work, which Sonos went ahead and
demonstrated for me. They had a product
that everyone liked. They were extremely
popular speaker. They then did a big
software update where like they were
like, "We're rolling out our new system
like, you know, kind of like how Google
does every couple years." are like,
"Hey, guess what? Gmail is going to suck
way harder than before today. Like
you're you have five, you know, you'll
get five months of being able to switch
between them and then you're on to the
new bad one." They did that only there
was no five months. They just basically
said like, "Here's the new rollout." And
it was awful. Like tons of feature
regressions. It didn't really work.
Customers couldn't do most of the things
they used to. And it like tanked the
company. like it was just like
immediately everyone hated it and nobody
wanted to buy their stuff and they had
to do this walk back and they they
couldn't roll back because they had
redeployed all their servers and I guess
they didn't know how to like undeploy
them so they couldn't you couldn't go
back to the old app. It was Go read
about this. I can't believe no one knows
about this but me. It was a massive
failure. Never heard of them. I heard
it. Okay, Casey. I don't know if you
guys can hear me but I heard that. Okay,
we can hear you. We can hear you TJ. We
can't really see you. You have very few
expressions. That's fine. But I do.
Okay. Your point is that hardware is
definitely different than software. You
know, if Twitter has a small breaking
thing because they made something
Twitter, all these major sites, they
break all the time, right? I have much
more forgiveness for that. This is, you
know, it's that's obviously different
than, you know, you got to play the
field a little bit different when I say
break production. If I'm building a game
that only, you know, so many people
play, you don't want it just crashing
non-stop while you're testing features,
there's probably some levels. But if
you're Twitter and all of a sudden you
can't post for 5 minutes and only 0001%
of the people couldn't post for 5
minutes, it's just not going to make a
huge splash and it's probably not going
to deter them from actually using the
product again. Well, I was actually No,
I was agreeing with you with the Sonos
thing. What I was saying was they should
have done exactly what you're talking
about. They should have taken and
deployed a small server bank for what
they were doing. Oh, I see. Given it to
1% of their users and their users would
have been like, "This is utter garbage
and I do not want it." And they would
have been like, "Okay, pull that on
back." Nobody saw it. We're fine. Right.
So, exactly what you're saying. I was
totally agreeing with you. I'm like,
Sonos, if they had done that, they would
have totally saved their company
literally like six months of hell. Had
they just given a slice of users this
thing for a while, they would have been
to, you know, some significant
percentage above beta test, right? Like
so like you said, you know, 5 10% of the
people got this. They would all have
been like, "Nope, nope, nope, nope,
nope." And then they would have like,
you know, been like, "Okay, okay."
So that's all I do want to Can I Can I
say one thing just about the testing
part though? Is that okay from before?
Maybe. Sure. Sure. That's what the
podcast is about. Exactly. I like this.
I like this. Go for it, TJ. Um, my
favorite project that I've ever worked
on, Neoim, by the way. Um, can you go
back to the grass? You were much more
like better in the grass. Really? My
phone if I got I got a notification that
said your phone is overheating cuz I was
in the sunshine. I was actually going to
bring that up in the beginning.
Like, it literally said your phone is
over. Make sure your body's the one
covering. Okay. All right. Sit over.
Create a shadow, TJ. So they
can't get your phone. You got to plank
plank over your phone. Hey, what's up,
boys? Hey yo, what's up, boys? This is
This is like getting into like TJ
profile pick territory where it's like
on the dating app. Can you tell us some
of your pet peeves, TJ? Do you like long
walks on the beach? I'm a I'm a long
walks enjoyer. I'm really not a fan of
short conversations. I want to really
find out about the real you. Um I don't
know what
else. I don't even know if any of my
words are coming through. TJ's actually
just describing himself right now.
That's all TJ's doing. I love laughing.
I love having fun. I like to make you
laugh. This is just me to prime. This is
my pitch for getting on the on the podcast.
It's true. I'll make fun of trash.
Yo, if if there's a guy named Trash on
the podcast, I will make sure to make
fun of him every episode. He uses one
password. Are you joking?
All right. All right. All right. All
right. TJ, what's your take? My take is
my favorite project that I've worked on
and like definitely the one with the
most weird edge cases and probably
users. Neoim like the test suite for
Neovim is amazing. We have tons of we
have different like styles of tests. We
have different ways to run them. We have
all this different stuff. Tons of work
went into making it. And maybe that's
just partially because Neo Vim is like
trying to maintain compatibility with
Vim on some stuff like literally
forever, right? So there's some other
requirements that are different, but
man, every time I did a PR and a random
test broke, it was my fault. It was not
the other test. It was not the snapshot
tests. It was nothing else. Like, and
like random stuff. I mean, maybe it's
just cuz Neoim is also like a really old
C project. So you like change something
and then a random global is destroyed. I
don't know, you know, it's like
whatever. It's maybe not always the
best, but like so I don't in in my
experience where people like actually
cared about it and like cared to
maintain snapshots and make those happen
like no one's even getting paid to work
on Neoim. You know what I'm saying? And
like the snapshots were valuable, the
unit tests are valuable, the functional
tests were valuable, and I used them in
a ton of different PRs and across like
really large refactors, adding Lua to
auto commands, like doing lots of
different stuff where I had to change
tons and tons of the code. I was just
wrong every time until the test went green.
green.
I think that probably goes to show that
tests are as valuable as the people who
wrote them. I was literally going to say
that. Yeah, that's what I've been
waiting to say too is I just wanted to
say but since I've been lagging I didn't
want to yell. I just wanted to yell
skill issue to everything Trash has been saying.
saying.
I would I would disagree actually. I
would I will defend Trash actually.
Casey's smarter than you. Because here's
the thing. That's true. Dang it. I
suspect I suspect that uh some of what
you're seeing though is not so much the
skill of the people who did it, although
they may have been very skilled. It's
more just about when the tests were
added. So, at least for me, a lot of
times what I'll do is I'll go, okay, if
the if I like let's say I have a bug
that I actually work on for a day, like
like this is because most bugs are like,
oh yeah, I know what it is and you just
fix it, right? But you get one and
you're just like, what the hell is
happening? And you actually, it's like
that once every few months where you're
just like, "Okay, this is really weird
and I'm not sure what it is." When I
find those, I typically try to add a
test or assertions or whatever I think
would have caught it, right? Because I'm
like, "Okay, this one's nasty and I
wasn't expecting it. What can I add that
will trip this kind of thing up and I'll
add it." And I suspect that much of what
gets seen in like mature products like
Neov are just like, hey, if you've got
some post test discipline, like after we
made the product, when we're finding
bugs, we add the test that would have
caught the bugs. Those are often very
good because they come out of what are
the things that are likely to break when
we do stuff and we put in a net there to
catch them. And so those are like very
good and they're the opposite of TDD.
They're like, "Oh, they're TDD. They're
testdriven debugging." Uh, maybe you
could say, or or something like that.
But, but, uh, so, so I do think there's
something to that uh, that that's a lot
more. I had a couple other hot takes
that I'll get to a second, but I just
want to leave it. I want to say that one
based on what you say. I don't know if
that's true about Neoim, but I I might
suspect it, right? It definitely is. The
thing though that like I was more
pushing back on is like I find the
snapshot tests in Neoim invaluable. Like
I they're amazing. I find a bunch of
these good end toend tests truly end to
end. Like literally it runs a test and
does user I don't know if I just dropped
uh it does like user input and then
asserts the outputs right like I found
those super valuable in Neovim. Probably
just dropped. Yeah. I also No, you
didn't you didn't drop. We s we heard
your point. I think also to put on top
of it is think about the the horizon of
Neoim versus you starting up a new dev
tool at work. And maybe that maybe
there's something to be said about when
are the goldens introduced during
exploration phase or like Neoim's just
not going to be changing. My assumption
is that a lot of Neovim doesn't change
super hard. It's not like hey guys let's
do a whole new data model today. Wrong
changes. All the ones that's all the
ones that ever everything you guys have
poo pooed on that I found them the most
valuable when I changed literally the
entire data model and how all auto
commands work all inside of Neoim. It
was the most useful to have the snapshot
tests then. I mean your testing is going
to depend greatly on what you're
building 100%. Sure. Yeah. Yeah. But I'm
just saying like we were fundamentally
changing the data models. Yeah. That's
Yeah. I'm just saying that's what
happened. Yeah. I thought it was a
pretty smart statement. power.
You're really smart, trash. I wouldn't
poo that. I wouldn't poo poo that at
all. I You've noticed I have not poo
pooed snapshot testing. True. Uh, good
job, Casey. I love I love you. You're
I love that I'm not getting ded. Can we
start rearranging the windows based on
who T Te thinks is smartest at any given
time? And I'm guessing that it has a
very high correlation to who is agreeing
with him. It seemed like I'm getting
that sense. I'll just always put trash
at the end. It's fine, Casey. You don't
have to worry. Today, I'm gonna tweet
that Casey protected me from Tee. Yes.
This is gonna This is a pinnacle moment
of my career. I can't even see your
guys' videos, so I have no idea what's
happening. I'm talking to a black screen
right now. Yeah. Yeah. Dude, I've been
just cracking up at your feed this whole
time. I just can't handle it. It's
pretty good. It's very good. Like, the
result is is topnotch of the feed.
Unfortunately, I think this will be a
stream only feature. I don't know if TJ
will actually have this effect on the
video because Riverside will be
uploading it up. That's true, right?
Yeah, that's true. He'll still be in in
weird like uh it in his uh profile his
dating site profile pick phase, but but
it will be much smoother. Yeah. So, so
anyway, uh I I wanted to add a I I would
agree that snapshot testing can be good.
It depend. It just depends on whether
those snapshots are going to be stable.
I mean, that's the bottom line. If you
have uh if you're I I would imagine what
Prime and Trash were talking about is if
you're trying to do snapshot testing at
a level
below something that's actually
consistent across the refactor,
obviously that's not going to work,
right? And so if your snapshot testing
is like someone mentioned, a parse tree.
Well, if it's a parse of something that
could be parsed different ways depending
on how you chose to implement that
subsystem, those snapshots are useless.
If the parse always has to be the same
because it's like parsing for the C
language specification and it can't
change because like you know it's
hardcoded. So if the thing's in C99 mode
then it had better produce the exact
parse tree. That's a great snapshot
test, right? And so I think the
difference boils down to those things.
It it has to be about something that
actually is um you know not going to
change item potent to the to the release
of the project otherwise it's a limited
time test. But you know that's just how
it is. So probably also the scope of the
breaking, right? If only 10% of your
golden is invalid because you made this
change, that's a lot different than a
100% of your goldens are invalid and you
have to like rework all of them because
of big kind of refactors. That was my
last my last experience is that I had to
do a bunch of testing against this uh
games reporting thing at Netflix and to
be able to do this autod do and to be
able to do it I had to like create the
data model of the incoming events and
then they are changing the incoming
events like once a week because it was
still an exploration phase. So it's just
like well not only is my input incorrect
now my output's incorrect. So I have to
like change both sides. So I'm just like
I guess I'm just redoing all the things
again. Here we go. Y and so then it just
became super hard to know if I was
actually right or if I was breaking
things. So I I think at the outset I
made it pretty clear I don't like test
different development as an idea. I
don't like focusing the developers
attention on tests. I think they should
understand tests and know how to deploy
them smartly. But I don't think right I
do want to say there is a place where I
do recommend literal test driven
development and it is as follows. That
is if the programmer is finding that
they are low productivity on a
particular task and that task does not provide
provide
feedback. So for example, if you have to
implement some low-level feature
component that has no visual output, no
auditory output, no anything. It's just
like an abstract thing that happens
inside the computer, right? Then
sometimes programmers like they get
bored working on these sorts of things.
But a lot of
programmers will like focus on things
like numbers that come out of a thing.
Like if a thing if you run a thing and
the number like you know 37% pops out
suddenly the programmer is like super
motivated to get it up to like 45% and
50% and right. So sometimes it's like,
oh, okay, if you can turn this problem,
if you can make a test that turns
whatever you're working on into
something with a concrete output that
you feel some reward about when it goes
up or
improves. That can be a valuable use of
test-driven development. And we do this
in performance a lot. We'll put make a
little test around something that
outputs things like how many cache
misses did it have or how many, you
know, uh, cycles did it take to complete
or whatever. and it's like oh now I'm
like trying to get that number down and
that's a lot more fun than I am just
trying to make this system faster or
whatever. So like a lot of times you can
get a you can make it so that those
things go together and then you get the
test or the good statistics gathering or
whatever it is that you actually wanted
anyway and you get this motivation for
the programmer and it they all kind of
work together in this nice way. So I I
just wanted to put that out. There's one
place where I think it does kind of work
to drive development by a test of some
kind. First thing that comes to my mind
is like code coverage reports and you
see that number really low and I have
the opposite reaction where I just don't
quit. I don't think I've ever saw like a
low code coverage report and I was like,
you know what, today we're making that
number higher. I'm kind of just like,
"Guys, we suck and I'm not going to help
us get better. There's no way." Oh, no.
All right. I I mean, to be fair, if I go
to a place like 100% code coverage, I
also go, "Uhoh." Right. Like, I'm also
kind of worried about the 100% code
coverage uh code cases as well. I'm not
going to say the company I was at, I've
been at many. Netflix and there was a
team that required 100% test coverage to
the point where people were just like
writing the most worthless test just to
get like this like PR bot to just shut
up and
was and to make it even like funnier
like bugs every day like in production
just like I was just like what's
happening here? I don't know if anyone's
ever experienced like a 100% like test
coverage quota by their team.
Yes, I did with Falkor when I was at
Netflix. 100% code coverage bugs every
day of my life. Oh, her Falkor the one
Wait, F. Did you say Falor? Uhhuh. Like
the luck dragon from the Never- Ending Story.
Story.
What? Why was it called that? You don't
know what Falkor is? Because it was
named after I do know what Falkor is.
It's the luck dragon from the Never-
Ending Story. Yes. named after the luck
dragon from the never- ending story.
Tell the story. Tell it all so Casey can
hear it and then we'll be done. I'll do
a quick one. Casey Casey Snow, how how
it works is that you would provide
effectively an array with keys in it as
the path into your data model in the
back end. So say you want videos, some
ID, title. It would return to you the
title in that kind of as a tree. Videos
or as a graph or maybe as a forest is a
better way to say it. Videos ID, title.
So if you wanted say a list, I want list
0 through 10. title, it would go list
0ero through 10, which would be ids,
which would then link to videos by ID,
title. So, it's kind of like a
graph-like data structure that makes it
so that if you make duplicate requests,
it's able to go, oh, okay, I already
have videos, whatever. I already have
list item zero. I can kind of not
request so much from the back end. The
problem was is that what happened if
list uh seven doesn't exist? You don't
know. So, we have to return something
back to you that says it does not exist.
So we return something called like a
it's a it's a boxed value. It's an atom
of undefined. It it it lets you know hey
there is a value there and the value
there is nothing. You don't need to
request it again. It's not going to be
something. So we created something
called materialize on the back end. That
means you need to materialize missing
values. Well what happens if you request
list 0 through 10,000 through 10,000
through 10,000 title? Well you just
created a gigantic number that we're
going to materialize everything for you.
And so with one simple line of code, a
while loop and bash, you can do DOSs or
at one point 2016, you could have dossed
down and took down Netflix permanently.
There was no rolling back. It was in
production for like two years. You could
do it for all endpoints, for all
devices, for everything and just it
would never run again. And that's it. It
was a simple while loop. And I I wrote a
lot of beautiful code about it and I did
a lot of beautiful stuff with it and it
was very nice code. And then I also
discovered it and reported it. And then
later on it got named the repulsive
grizzly attack. Repulsive grizzly. Yeah.
So does has no one at Netflix watched
the never- ending story? Shouldn't it be
called the Gamorc or something like it?
It's a movie. It's probably on Netflix.
Okay. That No. No. This wasn't some this
wasn't some 9,000. It was a completely
separate team that when they found out
about it, they had to do a bunch of work
and and it ended up being, you know, a
fix on the Zool Gateway and all this
kind of stuff. All right. because there
are no grizzly bears in the never-
ending story. I didn't read the original
book because I think it's in German or
something. I don't know. But yeah. Yeah.
But the nice news is I I I originally
didn't come up with the materialized
idea. Uh but I did do a lot of
refactoring that while refactoring and
recreating everything, I stopped and
went, "Wait a second. Something seems
funny here." So then I trashed uh
staging for a day where no one could use
staging for a day. And then that's when
I realized we made a mistake.
When I say we, I mean me. Yeah, I didn't.
didn't.
And that's it. And he's off.
I told my story and I'm not
participating in this podcast anymore.
The transitions at the end of this
podcast are just getting worse and
worse. We just end stream. He walks off.
What's happening? All right. Do you have
anything you want to add to this all?
He's stream. Are we still live? We're
still live, dude. Find a friend that
smiles at you like TJ smiles at us. I
think today's big takeaway is don't
bring trash dev into a project that
needs help. He'll just quit.
Yeah. No, actually I will help your your
product get better. Bye. TJ drive your
code coverage metrics down to zero.
Yeah. Code coverage is a lie anyways.
Are we live? And then quit. Oh, TJ's
back again.
All right. Well, what is happening?
I think we had a really productive
meeting today, everyone. Yeah. Everybody
enjoy your weekend. Next quarter is
going to be big. It's going to be the
biggest quarter of our lifetime.
Absolutely. We got to hit our OKRs. Yep.
Well, okay
then. Bye, ladies and gentlemen. Another
professional podcast from the standup.
All right. And ending it. All right.
Hold on one second. Let me I'm going to
I'm going to end the stream. Hey, thanks
everybody for being around here. And TJ
Chrome Extension