This content explains experimental design, differentiating between a completely randomized design and a more sophisticated randomized block design, emphasizing how blocking helps account for variability and increases the likelihood of detecting significant results.
Mind Map
Click to expand
Click to explore the full interactive mind map • Zoom, pan, and navigate
okay so before we get into the actual
content of our lesson for today all I
want to talk just really briefly about
what I did with these section here this
topic on blocking and experiments is
technically part of section 4.2 still
but I took this piece of the section and
I kind of pushed it back so that when I
give you guys a quiz over 4.2 blocking
will not be on it so at the first time
you guys are watching this video for me
on in class we will talk about just one
quick thing in these slides that'll help
you guys review for in general
completely randomized design experiment
stuff that'll get you ready for your 4.2
quiz so I would expect that the first
time you're watching this it'll be for
like the first five minutes of the video
and then you can revisit this and watch
the whole new lesson um later on for the
next class period so anyway the one I
want thing I want to talk about in this
first part here um for the first time
you're watching this is introducing this
problem right here and talking about how
we would use a completely randomized
design as an experiment so remember a
completely randomized design is just a
general experiment like you would
probably think to do um where there's
nothing fancy going on blocking which is
the subject of this lesson is like a
more sophisticated or fancier way of
running running in experiments so the
context for this problem is that we have
20 students available and those 20
students have taken the ACE the SAT
before and they're gonna participate in
an experiments with an online in an
in-class prep course for the SAT so we
want to see if these kids are gonna do
better with their SATs if they're either
in an online class or an in-person class
and we need to describe a completely
randomized design um to get this done
now I've done this a couple times with
you guys where we've made a little exam
experiment diagram for the randomized
design but I want to show you because I
haven't done this in a lesson view yet
how you would actually describe this
with like a little list because if the
AP test asks you to do a question like
this you're generally going to want to
do it via like just
Crypton rather than having to make a
picture um so pay attention as I go over
this because your four point to quiz
will have a problem where I asked you to
design an experiment like this and use a
completely randomized design so I've got
20 students at my disposal and they need
to go to online and in class those are
going to be buying two treatment groups
right here so the first thing I'm going
to do um you can do it where you put
everybody's name on it next card and
shuffle those up or you can give
everybody a unique number I'm gonna go
ahead and do it the number way although
either way is cool so assign each
volunteer a unique number from 1 to 20
so unique number from 1 to 20 when you
specify this make sure you're saying
unique which gets rid of like the
technicality of oh I gave somebody the
same number twice so everybody gets a
unique number here after I give
everybody a number what I'm gonna do is
use randant on my calculator use some
sort of random number generator so Rand intz
intz
the command would be 1 comma 20 I'm
gonna pick a number from 1 to 20 and I'm
gonna do this to generates 10 unique
numbers again and I'm specifying that
the numbers are unique which means that
there are no repeats allowed it's always
a good thing when you do problems like
this to explicitly say if repeats are
allowed or not here I definitely don't
want repeats I wouldn't want to pick the
same person twice after you talk about
how you're gonna pick your 10 people um
didn't get myself in early enough space
here the next step is going to be to us
say what you're gonna do with those 10
people so those 10 people the 10 people
corresponding to those numbers so the 10
people corresponding to the selected numbers
- in person it doesn't matter which ten
go to which group as long as the
selection of the ten was random so if
you would reverse the order of this that
would be totally fine as well
so they go to the class they take their
class I'm running out of space here so
I'll delete this when I continue with my
video but the next step in my process I
would have to give them the SAT so they
take their little class after class have
subjects take the SAT and measure their
improvements okay so they take the class
they retake the SAT again we see how
they improve because everybody's taking
it before the very last thing you have
to do when you talk about an experiment
is talk about what happens with your
results the comparison step that's
really really important that kids tend
to forget so my last step in this
process is going to be to compare the
improvement scores between the two
groups so when I asked you to design me
a completely randomized experiments you
need to talk about labels how do they
get their labels so I said assign each
volunteer a number from 1 to 20 that's
unique talk about how the randomness
takes place it could be if you're
shuffling names out of a hat doing that
it could be talking about randant with
repeats or no repeats then you need to
talk about what happens to the people
once you've randomly selected them so
where do they go what do they do talk
about the experiment itself
oh yeah collect data blah blah blah and
then talk about a comparison at the end
you're gonna compare you two
so this is what I would expect to see
when you are asked to design a
completely randomized experiments I'm
gonna go ahead and delete this and I'm
gonna continue with the video right now
but if you haven't had your 4.2 quiz yet
this is where you should stop for now
I'm gonna carry on though and you'll
pick up from this point in the next
lesson that you is on you Arendt's so
I'm talking to you in the future now
compared to when you first watch this
video this is where you pick up to talk
about our new concept of blocking so
what we just finished diagramming right
there um it was just written on the
screen is an example of a completely
randomized design but the issue with
this design um in our little fictitious
problem about the SAT it turns out that
out of those 20 people who volunteered
to be in our study they were in
different math classes 10 of the kids
were in precalc six were in algebra two
and four are only in geometry what kind
of a problem with this cause in terms of
SAT scores and the value of this class
so think about that for a second and
hopefully come up with something in your
head the biggest thing I'm telling you
the answer now the biggest thing that
this would cause is depending on how
much math you already know this class
probably isn't gonna be as valuable if
you don't know as much okay if I put a
third grader through an SAT class it's
not gonna help them at all because they
just don't know enough math that it's
not gonna make sense anyway if I put
somebody in precalc through this class
who already knows the stuff and may have
forgotten things the review is gonna be
really beneficial and I would expect the
creeks out kids to benefit more than the
geometry kids who haven't learned those
tougher things like logarithms etc etc
okay so going in the amount of math you
already have learned in the past is
probably gonna have a pretty significant
impact on how much this class actually
helps you so the amounts of math you
know so the amount of math you know
we'll have or could have a major impact
on how much a review class helps again
if you've never learned the content in
the first place seeing it in a quick
review SAT class isn't gonna make you
understand it well enough you have to
have been in the content and seen the
class for real so that is not really an
ideal thing right here because those
precalc kids may end up just getting way
more out of this class than the other
kids and that couldn't make things
trickier when we're actually analyzing
our results one way to solve this
problem the best way to solve this
problem is to use a concept called
blocking so blocking is a different type
of experimental design as opposed to a
completely randomized design and that is
what I need to teach you about in this lesson
lesson
so let's go on to the next page and I'll
explain what blocking actually is so a
block as a general vocab term right here
a block is a group established by a researcher
group established by a researcher with a
common characteristic and the big thing
about a block is that it's established
before you do random assignments okay so
a block is established before random
assignment this will make more sense
when I actually break down this problem
um and I'll show you how it works
but it completely a randomized block
design is an experiments involving this
is a bad definition right here but it's
an experiment involving blocks so how
would this work rather than have all 20
of my people and be like a random
assignment 10 here 10 here if I do that
it could be that I just end up putting
all that most of the precalc kids like
what if 7 or 8 precalc kids just
magically end up in one group by random
chance and the other group only is 2 or
3 freaked out kids well this group with
a lot of precalc kids is just gonna look
really good maybe if it's online or in
person it could just be the fact that
they're in precalc that's making this
group look better than this one right
here so what we do if we're pretty
confident that math class is going to
make a difference we take each group
separately we take our precalc kittens
all 10 of them and then we would divvy
them up so like randomly assign 5
randomly assign 5 to each group there
were six kids and algebra two randomly
assigned three here randomly assigned
three here so rather than relying on
just my big pool and hoping that they
end up about even when you deal them out
you break them into little piles called
blocks and then from those blocks what
you do is you randomly assign parts to
each this is actually a lot like
stratified sampling so this is a concept
that kids sometimes confused with
stratified sampling the big difference
with stratified sampling is that you are
sampling it says it in the UM name when
you sample you pull out of the
population so again my promise
sample I have freshmen sophomores
juniors seniors I'm sampling people to
be in my study so I would pull from the
list of freshmen I would pull from the
list of sophomores with blocking let's
say I was doing a problem in experiment
based on school somehow and I wanted to
make sure I got some freshmen to each
group I am assigning putting it into
groups rather than sampling and pulling
it but it's a very similar concept here
so creating a diagram outlining this
problem with our randomized block design
for SAT we would start out with our 10
precalc kids that would be one block we
would have six kids who are in algebra
two and then we have my little toolbars
on the way here
I got four kids who are in that does not
look like a four for kids in Geo and
then what you're basically gonna do is
do a very similar process I'm not
necessarily gonna write this whole thing
here um but what you would do is you
would randomly assign it so that's five
of the kids end up in each group so you
would have five of the kids go to
treatment one which is I don't know the
online class and then you have the
remaining five go to the in-person class
and you would have the same sort of deal
with the algebra two except this time
there were only six of them so you have
three and three so you that way you're
basically making sure you have five
freaked out kids in each group
three algebra two kids in each group and
we'll talk about why this is beneficial
as the lesson progresses here but that
is what blocking essentially needs now
one more thing because I'm not writing
this whole thing out here for my
experimental design but after you get
your results so after they take their
class blah blah blah when you make your
comparison what you're gonna end up
doing is comparing within the block
firsts so compare
within the block what does that mean
basically when you get your results
compare apples to apples precalc kittens
versus precalc kids algebra 2 kids
versus algebra 2 kids
you compare those different parts of the
same block to each other because that is
more likely to tell you if something is
really going on so let's keep things
going in our lesson right here and what
I have on this next slide I'm pretty
sure yeah it's like hypothetical results
for what actually happened in this study
so um the kids took their classes life
was good and then what we did is we went
ahead I think they did use blocking in
this yeah so that this was with
blockings have you counted out there are
five precalc kids in each group three
algebra two kids in each group two Geoje
group and this shows when they took
their SAT again hypothetically what
their results ended up be so we need to
make ourselves dot plots right here side
by side so we can look at our results
our whole next chapter we will talk
about is all about graphs and what
they're good for
but when you look at this it's kind of
hard to tell if one is necessarily
better than the other so I'm gonna go
ahead and graph these on top of each
other right here and see what's going on
so I'll start at 0 10 20 30 40 50 and
you guys should be graphing these as
well so while I'm just talking here you
can start making your own graph I don't
know why I started numbering all these
so yeah make your graph and try to make
them stack on top of each other because
when you do it that way it'll be easier
to see if there is a difference between
the groups we talk about overlap in the
pictures and we did that in the previous
lesson here so that should be a little
familiar for you guys I'm gonna make my
first group this will be the online
group and this will be the classroom
group and what I'm gonna do is put a dot
for each kid where they end up so precalc
precalc
kids start out with online I have two
kids who got hundreds points
improvements that's good
etc so go ahead and graph them and make
sure you put them on the appropriate
axis here and another 100 those are my
freakout kids then it switches to
all rights that's pre couch
algebra to online is 50 60 40 50 60 for
he's right there okay um algebra 2 in
class is 30 40 20 and then GE Oh online
30 30 and then geo classroom 0 20 okay
so you guys should have these graphed on
your own here um and then what we're
gonna do is look at these pictures and
try to decide is there compelling
evidence that the online class is better
than the in-person class or vice versa
and we will learn much more
sophisticated ways of doing this using
like actual statistics later on
primarily second semester for right now
we're just kind of using our gut and
we're looking at the picture and the key
like I told you is over left if there's
a ton of overlap between the two groups
the results aren't compelling enough for
me to be like wow they're probably
actually is a difference there couldn't
be just a small difference if there's
only like a very small difference
between the two groups
I could assume that difference or I
could suspect that that difference only
occurred due the chance of the random
assignments so if you look at these two
pictures here that's what I care about
when you compare dot plots is the
overlap and if you look at the bulk of
these points these points go from here
to here these points go from here to
here these points right here have an
awful lot of overlap to them there's a
very strong amount of overlap right here
where if I was looking at this picture
yes the kids
and ooh nobody got a 9-year 100 at all
in the class group but I have five kids
who got it in the online you might be
like ah online can be a little better
but there's so much overlap between the
two groups that this probably isn't
enough to be convincing okay so what I'm
gonna write here just informally I'm
gonna say there's too much overlap to be
confidence of a difference if I was
writing this as like a full free
response question I would say there's
too much overlap between the scores
context is really important too much of
overlap between the scores we can't be
confident that one class is necessarily
better than the other
okay so when your data is spread out
that's a lot of variability we talked
about that before
variability is not our friend in
experiments because when you're more
spread out it's more likely you're gonna
have that overlap and then your results
aren't gonna need to mincing then I need
to erase this picture I think let me see
if I can keep it here actually my tabs
my next okay it's a little bit of a mess
right here but this is important and I
want you guys to see what's going on
when we made our dot plots by hands we
kind of looked throughout the fact that
kids were in different math classes like
you might remember oh those kids I think
we're in precalc
but it's not very evident by looking at
these dots right here that they're in
different math classes we kind of
disregarded that so what this picture
did is it took that same data you can
see the shape of this graph is the same
although their classroom and their
online are reversed for mine I should
have looked at that ahead of time so be
aware that this matches up with here and
that this matches up with here but you
can see that the shape of the graph is
very much the same all they did
differently instead of dots for
everybody though is they put different
symbols for each Club math class that
the kid was in so our precalc id's are
the triangles right here our algebra two
kids are the grey dots and then the
squares are the geometry kids here is
why this is actually more beneficial if
you look at the groups
individually earlier in this video I
said the whole point of blocking is that
you're comparing apples to apples
oranges to oranges etc but when I looked
at my graph right over here I didn't
really do that I just like looked at
everything as a whole look at the
group's individually so if I look here
are my precalc is for in class
here are my pre calc id's for online no
overlap between those groups this group
is all higher than all of these kids
right here if this was just random
assignment here it's very unlikely that
every single kid would beat every single
kid right here so what that tells me is
man that online class might have
actually helps same thing when you look
at out to be two granted there are less
dots so not grates making conclusions
based on six data points but there's
quite a bit of separation right here
only a little bit of overlap with you
through kids tied and then with the Geo
I got this versus this so when you look
at the groups individually apples to
apples it becomes a lot more clear that
the online class in this scenario was better
better
it appears that online led to higher
improvements within students of the same
math class so out of all the precalc
kids the kids in the online class did do
better and same with algebra two and
same with geometry so when you actually
look at the blocks and compare within
the blocks apples to apples it becomes
easier to see if your results could be
statistically significant where things
get cloudy when you look at the data
overall so this is kind of a visual and
demonstration of like why blocking is
helpful it helps us determine if things
are statistically significant or not by
accounting for a source of variability
in our data that was a big mouthful
right there
source of variability in our data is
what math class you're in depending on
what math classroom and that's gonna
swing how much you improve on this class
quite a bit well if you kind of take
that off the table and you balance it
out so you've got this versus this right
here I'll freak out
then you don't have to worry about that
clouding your results anymore so it
accounts for a source of variability in
the data I'm gonna have you write that
at some point in the slides but I want
to see if I did that elsewhere so what I
want to do right here is show you guys
just a quick way that you can this is an
alternative kind of extension sort of a
deal so just a straight up law there is
another way we could kind of put
everybody on the same playing fields so
what they did if you look at this right
here because this is gonna take a little
bit of explaining for you guys to
understand it says right here that the
average improvements for students in
pre-k was 86 I'm gonna start right there
so if you take all those stuff those
numbers right here 100 hundred 99 270 is
70 all those precalc kids in you average
their scores together turns out that
average is an 86 so no matter online or
classroom everybody together who was in
precalc improved by an average of 86
points and in algebra 2 so if you add up
all these numbers right here for the
algebra 2 kids those numbers average out
to 40 and geo averaged out to 20 so
another option this is like kind of a
just above and beyond sort of thing that
you can do if you are in precalc your
score is expected to improve by 86
points just by virtue of being in pre-k
that's just the average of everybody in
pre-k okay so this is kind of like the
head start or the bonus or whatever that
you get the benefit you get just from
being a kid in pre-k what we do a lot a
lot a lot in statistics is we analyze
the difference after you account for
things like we do a lot with differences
in statistics
so in this context what I'm gonna do is
I'm going to take each of my freakout
kids and I'm gonna subtract 86 points
right here so if I subtract 86 from this
I wish I had written these out ahead of
time but that's a 14 that's a 14 for for
14 and then we have negative 16 negative
16 negative 6 negative 6 negative 6 ok
so this is the number these are the
numbers after we take out that average
improvement of freaked out kids I'm
gonna do the same thing with algebra too
but by being an algebra 2 your score
improved by an average of 40 points
across the board so by being an algebra
2 this is like your algebra 2 bonus
right here so if I subtract out that's I
get 10 20 so I'm subtracting 40 each
time right here um so I have negative 10
I have 0 I have negative 20 and then
with the Gio kids I have a minus 20 for
each of them so I should change colors
and then I'm gonna have 20 points
subtracted off from each of them this is
gonna end up being a 10 a 10 a 0 that's
negative 20 and a 0 so what this did is
it basically leveled the playing field
in a way by removing um the advantage of
being in that class so to speak so these
ones got 86 points knocked off these
ones got 40 points knocked up and he's
got 20 points knocked off now that all
the kids are more or less on the same
playing field and we eliminated the
advantage to being in a certain math
less what we could do now is we could
look at all of our online kids so these
guys are online um these guys are online
and these guys are online and we could
graph all of these against all of the
not circled problems I'm not gonna graph
it but if you look most of the kids with
positive numbers are in the online
and there's a whole lot of negative
numbers in the in-person groups would be
another way of looking at your data
holistically and seeing that maybe there
is an advantage to being in the online
group okay so this slide right here a
little bit more above and beyond just a
different way of approaching it but
looking at things within blocks like we
started off on these slides very very
important but I've set a few of these
things already but this is good used to
be able to talk about it again and have
it here in your notes blocking in
experiments is similar to stratified
sampling I've already talked about that
in the difference you can backpedal in
this video and hear me say that again if
you need it but picture it's like
stratified is for sampling blocking is
for experiments so blocking is a form of
random or blocking goes well with random
assignments stratified goes well with
random sampling and then um blocking is
a good way to increase your chances of
finding convincing evidence we've talked
about that it was easier to see there
was a difference when we compare apples
to apples block should be choosing
chosen like strata units within a block
should be similar and different than the
other blocks this line right here is
important you should only block when you
expect that the blocking variable is
associated with the response variable
okay so in my same problem with my
online class my in-person class blocking
by somebody's para color would be a
stupid thing to do because there's no
real like um incentive like there's no
real thought that people with brown hair
are gonna do differently an online or in
person if you block by a poor choice of
a variable one that doesn't actually
influence things like you were thinking
what you've done is you've fragmented
your data instead of looking at twenty
in having a 10 and a 10 if I went by
hair color now I've got three people
with brown here three people with brown
hair three versus three is not as good
as 10 versus 10 so if you block poorly
by a variable it doesn't have to be as
ridiculous as my example right there but
if you block by something that doesn't
actually matter
you run the risk of fragmenting your
data too much to find anything meaningful
meaningful
I mean blocks are not formed at random
they are chosen in advance by the
experimenters there's one other phrase
that's super importance I want you to
put a star by it's in your notes I said
it out loud earlier but now it's
actually here for you guys
blocking accounts for a source of
variability in your data the math class
you're in is gonna be a source of
variability and how much you improve
blocking balances it out and make sure
that you're comparing apples to apples
all right so something to think about
here at the bottom what are some
variables that we can block for in the
caffeine experiment that we did so let's
go with our hypothetical one we didn't
actually do with the caffeine so I'm
gonna give you guys cups of soda so
think about what would be good a good
choice of a blocking variable okay so a
couple things we can do here now that
you've thought about it one probably
pretty good one would be caffeine
tolerance so if I have 10 of you guys
who have caffeine allots and 8 of you
guys who don't have caffeine as much I
would take the 10 that'd be my block and
make sure 5 of you hit the caffeine 5
you get the caffeine free then I take my
8 kids who are not big with caffeine and
you 4 and 4 and then you compare those
little groups instead of comparing the
whole big pile um you can set other
things like weighed athletic ability
etcetera etc as an experiment or what
you would do is you would put people
into piles based on some variable you
can block my more than one variable but
you fragment your data to like into two
smaller groups so generally you'll see
it just with one variable and then what
you do is you deal out the people in the
piles randomly in general how can we
determine which variables might be best
for blocking just kind of common sense
logic maybe if you have prior research
in the area you may know that something
is going to influence things you don't
want to just guess something a new block
and just because it's a good idea
because you think it's a good thought
you have to have some motivations and
reason to believe that what you're doing
will be helpful all right so I believe
that this guy right here's my last slide
let me
yet is so we have a fresh context right
here and I'm not gonna write this I'm
just going to talk you through it so we
have um a popcorn aficionado here and
she has four types of popcorn and she's
gonna see if the popcorn button on the
microwave does better with popping the
kernels than using the amount of time
printed on the back and when she goes to
the store she's gonna buy four kinds of popcorn
popcorn
there's movie butter like butter natural
cattle for 40 bags altogether
why would randomized design be
preferable to a completely randomized
design this I do want to write at least
something on because this is important
and something I could very well ask you
to do one quiz the type of popcorn is
how many kernels will blocking accounts
okay some types of popcorn probably pop
better than other types of popcorn if
you've got all sorts of butter wrapped
around it maybe that stops it from
popping as well or whatever so rather
than comparing all 40 bags 20 verses 20
and oh maybe I ended up with more of
butter in this kind of grip right here
you break it up so you get half and half
when you do like if I would have done
just a randomized design with all 40
bags my results are not going to be
biased and they are not you're not
running the risk of confounding so those
are two things students will incorrectly
assume sometimes your process is still
random so you're not favoring one side
over the other consistently just in one
trial you may have more in one group
than the other so your results if you
repeat this you replicate it multiple
times will have more variability because
you could get more butter over here this
time and more butter over here the next
so your results are more spread out and
I'm not harder to tell there's overlap
for statistical significance okay
outline a randomized block designed for
this experiments we've talked about that
before I don't want to write it all out
right here but basically what you would
do is you could give me a play-by-play
with this you would start by putting
them in groups of 10 so take the 10
movie butter randomly assign 5 give a
number to each one and randomly assign 5
to this group 5 to this group blah blah
blah proceed and do the same thing
repeat the process with light butter
natural etc you just talk in words about
what you're gonna do making sure you
highlight those key things we talked
about earlier in the lesson very last
thing I didn't want to waste an extra
piece of paper this is the eighth slide
but this is actually important and it
looks not important because I like
tacked it on the end right here but put
a star by this last definition matched
pairs design is important and it's the
last thing we have to talk about
Internet's right here matched pairs
design is blocks of size 2
with very similar individuals in each
grade so I'm gonna give you a few
examples of this match pairs is a
special type of blocking where you have
two people in each group and what you do
is you take those two people and you
give one one treatment and the other the
other treatment and then you compare
those two people okay um very easy
example to understand so you had
identical twins and you wanted to test
um I don't know some sort of medication
since they're genetically so similar you
could give one twin one kind of medicine
and once when the other and compare
between the two of them how their
responses are gonna be now naturally you
don't always have identical twins and
that's not even necessary to do a mashed
pairs um let's take our SAT example
let's say I was gonna do an SAT study on
you guys in class what I could do
instead of doing a blocking since you
guys are all in the same class what I
could do is take the two people with the
highest SAT scores in the class or a CT
scores and put one in each group
randomly then the next to highest score
is one here one here next two highest
scores here here etc so you basically
are pairing people up who are very
similar and then you're making sure one
goes to each treatment there's also an
example in our book it doesn't even have
to be two separate people let's say I
wanted to test like two kinds of
deodorant to see which one works better
I can use myself as a matched pairs and
randomly assign one arm one kind one arm
the other and compare on myself so you
can even get creative with things like
that but match pairs even further
accounts for variability because if I'm
testing both things on myself all those
other factors about sweat levels
athletic ability etc etc etc are the
same so match pairs is a nice way to
account for a lot of a variability but
blocking in general whether it's regular
blocking or mashed pairs basically in a
nutshell makes it easier to tell if what
we are testing is actually
Click on any text or timestamp to jump to that moment in the video
Share:
Most transcripts ready in under 5 seconds
One-Click Copy125+ LanguagesSearch ContentJump to Timestamps
Paste YouTube URL
Enter any YouTube video link to get the full transcript
Transcript Extraction Form
Most transcripts ready in under 5 seconds
Get Our Chrome Extension
Get transcripts instantly without leaving YouTube. Install our Chrome extension for one-click access to any video's transcript directly on the watch page.
