This content explains the fundamental principles of good experimental design, emphasizing the importance of comparison, random assignment, replication, and control to ensure valid and reliable results.
Mind Map
Click to expand
Click to explore the full interactive mind map • Zoom, pan, and navigate
all right so we are continuing our
discussion in this section on
experimental design and what goes into
making a really good experiments on our
slides in this section you guys are
gonna see a lot more typing and I
usually like to do on these slides right
here this is more just for later for you
guys to have something to look back at
and read in your notes but I'm gonna try
to summarize each of these things as
we're going through them so we are still
in the context of our caffeine
experiments from last time so if you
guys can think again the idea was I was
gonna see if having caffeine makes a
difference in your pulse rates um and
then we were talking about different
things I was doing or needed to do to
make sure my results were worthy of
actually being analyzed to make sure my
design was good so last time we only
talked about one aspect only one aspect
of experimental design and that was the
idea of having a comparison you have to
have at least two groups if you don't
have two separate groups to compare you
can't say whether the results that
you're seeing are because of the
variable of interest or because of some
other factor my example last time was if
I give every caffeine I weight and I
measure on maybe my awesome stats
lecture in the meantime is what
increases that pulse rates so I need to
make sure that I have at least two
groups that both experienced the same
thing two groups that were experiencing
all the same things except for that one
thing that matters which is Kathy so I'd
have a group that got caffeine and I'd
have a group that did not that all heard
the same lecture so if my lecture was
really what was going on I would see
both groups having a high rate and if
there isn't still a difference it would
be because of the caffeine not because
of that factor that was the same between
them but even if I have two groups if
I'm not careful about how those two
groups are made up I could still run
into issues so I need to make sure that
not only do I have two groups but that
those two groups are evenly balanced we
have to do what is called R and
assignments so we randomly assign
subjects to treatments to create groups
that are roughly equivalent at the
beginning of the experiments underline
and put a star by that line roughly
equivalent groups so let me talk about
what this means let's say I was gonna do
the caffeine study and I realized okay
yeah god I have two groups got I have a
comparison but then I just let people
self assigned to their own groups it
seems plausible to me that if I just let
you guys assign to your own groups
you probably wander over to the group
whichever group you end up being and
maybe you wouldn't even know what group
it is a or b but like you would probably
migrate with your friend group -
whichever station you're gonna go to so
people who are friends in the same group
would probably go over to the same
treatments well let's say you and your
friend group are just going to get
coffee every single day before school
you're getting a triple espresso mocha
whatever and you're just really really
tolerant to caffeine and all of your
friends go there together so you guys
are all pretty tolerant to caffeine um
so you're all in the same group right
now and you probably would experience
almost nothing due to the fact that the
you're used to the caffeine so what
could that do to my results if you guys
work in the caffeine group I would see
that me like wow caffeine actually
didn't make a difference at all cuz
you're so used to it or if you were in
the non caffeine group um then I would
be like really confused with what's
going on or on the reverse let's say you
never F caffeine your friends are all
anti caffeine and you guys get the
caffeine I'm gonna be like what is going
on here this caffeine makes such a big
difference we could talk about
athleticism on people who are physically
fit going into the same group etc etc if
I let you choose your own groups that
concept of confounding comes back I
won't know if I put you let you choose
your own groups if it's your caffeine
status or what blah blah or athleticism
or blah blah blah or whatever so you
need to make sure that when you make
your groups you're splitting it up so
you have people who have caffeine
tolerance in both groups so you have
people who are athletes in both groups
and the only way to do this just like we
talked about last section you can't
trust yourself as an experimenter to
break that up you have to trust a random
process the only way to be sure that it
wasn't because of this or this or this
or this that are confounded with your
variable is to break your people up
randomly why does that matter if they
were randomly assigned you should get
about the same number of athletes the
same number of caffeine people in each
group okay is it possible that you'd get
a few more in one than the other well
yeah it is it could work out that way
but if you have enough data you're gonna
be okay and basically what random
assignment does is it stopped people
from being like oh did you consider this
or oh what about this oh did you think
about what people are eating for
breakfasts and you can be like hey I
randomly assigned treatments so random
assignment is like a kind of just quick
boom Hey I did I did random assignments
I'm good so let's move on to our next
the principle of experiments in design
Oh actually wait I forgot I did this
first so let's talk about designing
experiments here and the concept of
completely randomized design when you
have a completely randomized design in
your experiment it's basically what you
probably think of or what I just
described on my last slide where I break
you guys up into groups randomly if I
asked you to describe a good experiment
there's a pretty strong chance it would
be this kind um kind of a difference
type of experiment that we will talk
about later in this chapter is blocked
experimental design a randomized block
design and we'll get to that but this
generalized randomized design is when
your experimental units are assigned
treatment entirely by chance and you
will be asked sometimes on the AP test
to describe a randomized design either
by like a lists or by making a diagram
and I prefer writing down like a
bulleted list like boom-boom-boom this
is what I would do but I'm actually
gonna do it via diagram right here so at
the time I'm making this video I don't
know who all is in I don't know the
number of kids in my class yet so I'm
gonna say there
are 18 of you guys who are in my
hypothetical caffeine study I start out
with my 18 students and the first thing
I need to do with my 18 kids is I need
to break you guys up into the treatment
groups so I need to figure out somehow
how to do random assignments that could
happen a lot of ways I could give you
guys each a number from a 1 to 18 and I
can randomly picking 9 numbers no
repeats those 9 numbers those people go
in the first treatment group then the
remaining 9 go in the net I could
instead take index cards or papers put
your names on the papers put them in a
hat shake and let pull out in the name
and there's nothing that says you have
to do the same a lot of people in each
group but unless there's a compelling
reason to do otherwise that's what I'm
gonna do right here so I'm gonna use
some sort of random assignment and I'm
gonna break you guys up into two groups
of 9 so we're gonna Group one um and the
variable n is one we use a lot and his
sample size so I'd have nine people in
this group right here so it's the size
of my group and idea of group two which
would also have nine people in it each
of those groups is then gonna get their
treatments so the treatment in this case
it doesn't really matter which group is
which say treatment one is gonna be the
caffeine group so I give you guys
caffeine and then treatment to you guys
are not gonna get caffeine we're gonna
talk later in the slides about how
giving like a caffeine free soda is
better than giving no soda at all and
we'll talk about that later but for
right now I'm just gonna write down no
caffeine and leave it at that
and after we actually get our data so we
do the caffeine we do the no caffeine
the part people always forget when
describing experiments is that you need
to make sure you actually compare the
results between the two groups so
measure pulses measure your pulse change
and then compare your two groups the
whole point of doing the experiment is
to pit with this group against this
group to see if there's actually a difference
difference
in pulse rate so once you collect your
data you need to make sure you actually
compare the two groups to see if there
is a difference alright so that is an
example of an experimental diagram right
there um you could have done that same
thing by making like a little list of
things to do and I like that better but
now you've at least seen one of those
diagrams let's move on to our next
principle of experimental design which
is a concept called replication so
replicate up replication is what I want
you guys to write on this blank up here
on the top and replication in a nutshell
means making sure you have enough data
you got to make sure you have enough
data when you do a study let's say I
have 4 people so I only have 4 people
that I'm gonna do my experiment on if I
only have 4 people and I randomly break
them up into two and two there are so
few people that it's totally plausible
that these two are athletes and these
two are not these two like coffee have
it all the time these two never have
caffeine when your groups are too small
random assignments doesn't get to kick
in and do its job you gotta have lots of
people and if you have lots then when
you break it up the chances of one group
being drastically different than the
other very very minimal but when your
groups are too small you could have a
flukey thing where this group is
actually different in some way than this
group over here
so having too little data is a huge
issue in experiments and honestly 18
people and class really probably isn't
enough for good study no scientist is
going to publish my study because we
just don't have the data at onions to
make it work in addition to not having
enough data points the other thing going
on with replication is it needs to be
possible to repeat the process not only
in like your one trial but to take the
whole thing and to do it again those of
you who are involved in science research
you've probably heard the concept of
replicating results in an experiments
basically if I'm a scientist and I do
some crazy study I find just the most
amazing results if other researchers
can't repeat my process and produce the same
same
it's my results are worthless okay you
hear about like like I feel like in the
news all the time there are studies
about the next new one your drug and
this awesome thing that's gonna solve
all these problems and you hear about it
in the news but then it kind of goes
away um it's something where the results
worked at the one time but they couldn't
be replicated reliably so that's a
problem that you need to you need to be
able to do it again and again for the
results to actually be significant so
when we do have good replication it can
help us feel more confident more
confident applying the results of our
experiment to a population all right so
let's keep things moving yeah and talk
about our next principle of experimental
design and this is a huge wall of text
right here so you can pause me if you
want to to read through this or read
through it afterwards but you read it at
some point right here I'm gonna give you
guys kind of a summary though of this
principle which is called control so if
you can go ahead and write control on
that top link I will now kind of break
down what that means control is a really
big concept and it's not the same thing
as having a control group control group
is like a buzz word oh you got to have a
group that doesn't get anything at all
and then people just think that's what
this is about
control is a much bigger issue than just
that what it basically means is making
sure everything is the same between your
treatment groups except for that one
thing that you're actually testing okay
so let me give you several examples of
that and you guys can see on the slides
right here there are two primary
benefits to control one is that it
prevents compoundings and one is that it
reduces variability so I'll talk about
those two things separately let's start
out with confounding this first one
right here so um still haven't gotten to
why but it's not a good idea for me to
give one group caffeine and one group
absolutely nothing
it's the placebo effect which we'll talk
about later on in these slides right
here so let's say I have these two
groups and I have the caffeine soda and
I've done this before actually I've gone
to Chinooks or whatever and bought some
caffeine Pepsi and then all I could find was
was
Diet Pepsi decaffeinated so that it's
harder to find and all they had in stock
to his diets well then I've got caffeine
regular Pepsi and I've got diet
caffeine-free caffeine Pepsi has sugar
in its diet does not have like
artificial sweeteners so everybody who
got caffeine also got sugar everybody
who didn't get caffeine also got like
fake sugar so if I see a difference in
my groups its confounded with sugars
confounded with caffeine I can't break
those two apart and if I see a
difference I don't know if it was the
sugar or if it was the caffeine that
made that happen so it's dangerous when
you have studies to let more than one
variable run wild because if you do that
you can't say for sure which of the
variables is actually responsible for
that difference okay so that is one
factor you definitely need to consider
the other kind of thing going on with
control in addition to like compounding
and having that going on if you don't
control four variables you will have
higher variability in your responses so
let's say I decide that I'm gonna do my
caffeine study but I'm not gonna specify
how much caffeine you have you guys just
go wild take however much so so do you
want you want to drink like this much
that's cool you want to drink a whole
two-liter have at it it's just up to you
you decide well then logically the
people who had more caffeine if you
drank a whole two-liter during that time
period you're gonna experience more of a
change presumably if you drink your soda
really fast so instead of just on the
health of soda being the same if you
just chug your soda I don't know maybe
that makes a difference and makes you
feel the caffeine more than if you sip
it gradually other things at play if
like certain cans of soda are at
different temperatures maybe the colder
soda lowers your pulse or whatever I'm
recording this video summer of 2020 so I
don't even know yet if I'm gonna be
seeing you guys in person for this
lesson or if it's gonna be like a
virtual deal if like some of you are at
home doing this study and you've got
music playing in the background
that's pumping you up and some of you
don't or somebody's yelling in your
house there are lots of different
variables that I wouldn't be able to
control for and those variables changing
will affect responses and just make my
results more spread out than I want them
to be okay so this slide talks about
that in a bunch of detail so let me
expand like um kind of explain a little
more that second thing about reducing
variability and why we want lower
variability in our responses so this dot
plot on the next page has two
hypothetical scenarios this is focusing
specifically on the amount of caffeine
so making sure that I keep the caffeine
consistent for everybody so in my first
little dot plot right here these people
were given the same amount of caffeine
and these people were given the same
amount of like let's go with caffeine
soda and then diet or not diet sorry
caffeine free same soda want to make
sure they drink the same amount of soda
not more or less depending if you have
caffeine or not same temperature same
everything same rate of drinking only
difference is these guys got X amount of
caffeine you can see in the results of
this study that these people over here
have positive changes in their pulse now
when you talk about a change in the
pulse I'm assuming they took the after
minus the before although didn't super
specify right here but that wouldn't
seem logical to me so if my pulse beats
for you times and however long that time
forward afterwards and it was 35 before
that means it changed by five so this is
the difference of their pulse rates
right here these numbers for the
caffeine europe are mostly higher than
these numbers right here for the new
caffeine group so I would look at this
and be like oh man caffeine made a
difference that's what this would lead
me to believe so it seems like caffeine
seems to make a difference okay that's
what I would think if I saw this these
results right here
meanwhile in the second scenario this
one kids were allowed to choose their
my caffeine so if you look at this right
here what you're gonna see is more
variability or spread in your results
results are a lot more all over the
place this kid over here probably drank
a two liter of soda where this kid was
probably sipping on just a little bit
right there so if caffeine actually
makes a difference you see a lot more
spread in your results okay spreading
your results is not desirable you don't
want that if you can avoid it and the
reason for that is the first one it was
pretty clear that this group is bigger
than this one but in the second one look
at how many of these people experience
very similar results these people on
these people got about the same results
right here there are a few people on the
high end and a few people on the low end
right here but if you have like
basically that makes this data less
relevant look at how many fewer data
points we're looking at now is opposed
to everything as a whole so if we do
that picture it's like one of these two
these people who are not in this study
anymore all of a sudden you really can't
tell that this group is better than this
one so when we look at that plots like
these we don't want to see a lot of
overlap not a lot of overlap means
there's probably a difference lots of
overlap between that means it's harder
to see what's going on so if I saw these
results I would be unsure of what was
actually happening okay
so you have to have control to lower
control allows you to lower variability
in responses if you don't control you
get increased variability which makes
things unclear I have one more question
on this scenario right here would
students weight be a confounding
variable in this experiments so what do
I mean by that weight possibly seems
like it would make a difference in no
change from your pulse rate if you weigh
more the caffeine that you have if
everybody's giving the same amount of
caffeine that caffeine probably isn't
gonna have as big of an impact on you
because you've got more mass for the
like caffeine to spread out across when
you weigh more in general things have
less of an effects on the other side of
things if you don't weigh very much that
same amount of caffeine may hit harder
for you and have more of an effect so
the question was is caffeine a
confounding Miriam
now remember what it means to be a
confounding variable it means that you
have these two factors that are tied
together that you can't separate to say
which variable is actually causing the
change that you're seeing so in this
context it would mean that weights and
caffeine are tied together in a way that
I can't tell which one made the
difference that's not what's going on in
this study I randomly assigned people to
treatment at the beginning so if I
randomly assign people to treatments
people who more should be balanced
between this group in this room who
don't weigh as much balance between this
group and this group random assignment
is basically our way of ensuring that
variables don't get links like that and
we can actually tell what's wrong not so
would weights be a confounding factor no
treatment was randomly assigned so sorry
treatment was randomly assigned so
weight should be balanced the balance
I just can't write between groups so if
you randomly assign treatments you avoid
that possibly you can sidestep the worry
of oh gosh which of those variables
wasn't the group should be even except
for the caffeine the thing you actually
care about so let's keep things moving
on our slide right here and the next
thing we have to talk about is more text
right here
I've already referenced one of these
things so I'll remind you guys again um
when you do a survey it's important that
subjects in both groups are blind to
what treatment they receive now I'm
gonna put blind in light quotes right
here this doesn't mean like physically
lines um the idea is that if I'm doing
this caffeine study I shouldn't know if
I'm getting caffeine you're nuts if I
know I'm getting caffeine my pulse may
actually start
to raise just because I expect it should
your mind can actually play tricks on
you would cause real responses by in
your body just because you think
something is supposed to happen that's a
very famous concepts called the placebo
effects so if you are not blind to
treatments you run the risk of being
fooled by the placebo effect if somebody
expects something to happen it may
actually happen even if there wasn't
caused by the caffeine you may have a
higher pulse just because you think
you're supposed to that can happen with
medications too even with like crazy
things like it can be very concrete
things like anxiety but also like I
don't know some sort of illness that you
have getting better faster your mind
actually is very powerful and just
because you think you're getting
treatments it may actually cause like
you to get better just because your mind
convinces it to happen so you want to
make sure you don't know what treatment
you're getting as a person involved in
the experiments sometimes it also makes
sense for the researcher to not know who
gets the treatments and in that case it
is called a double-blind study so when
the researcher doesn't know somebody
knows if nobody knows what's going on
and how are you ever gonna find out what
treatment you got it's like the results
are lost to you so there's somebody
behind the scenes or it's recorded
somewhere but let's say I'm the
researcher sometimes it may be
beneficial for me not to know either in
the caffeine problem it doesn't super
matter if I know where if I don't what's
going on because I can't have any sort
of influence on the results we're
counting your pulse rates um but let's
say instead I was measuring how relaxed
or help calm you looked or how anxious
you look there's something like that if
I knew what treatments you got I may
subconsciously be like oh they got the
good treatment so yeah they do look more
relaxed or they look more anxious and my
own biases can kind of cloud my
judgement right there so you want your
study to be double-blind not all the
time because it's not always helpful but
it's useful when the researcher must be
subjective so when the researcher must
the subjective you want to make sure
they're blind so they can't have their
own bias he's coming to play all right
couple more disclaimers here not every
single experiment needs to have a
control group that gets no treatment
needs to have a placebo as long as you
have a comparison so people mistakenly
say oh not a good experiment there was
no control group
there was no placebo um basically let me
give you a few examples of why that may
be the most important thing is that you
have a comparison you have to have
multiple groups but you don't always
have to have a group that gets no
treatment at all okay so let's talk
about medication if I'm testing a new
cholesterol medication and I want to
know if this new medication is better
than what's already out there it's kind
of unethical for me to make people who
need medication having no medication at
all hey guys you're not taking any
cholesterol medicine I'm giving you fake
pills so you won't know what's going on
that's not right so instead of testing
new pill versus no treatment at all
you can test new pill against treatment
that's already used and is already
commonplace so you can test the status
quo against new treatments it doesn't
have to mean new treatment against
nothing at all and even easier example
let's say I'm testing two kinds of
paints and I want to know which one does
better and feigns less in the rain I
don't have to have a section of pavement
with no paint at all before I do my
first paints in my second paint I can
just do first verse ii have to have a
comparison don't always have to have a
group with no treatment at all and this
last part right here as long as you have
these things that I'm talking about the
big ones that we talked about our lesson
here in your study um your group should
be about identical and then you can get
actually to testing what you want to
test so if you see changes in your
variable in your response variable then
you can begin to talk about what whether
the explanatory variable actually cause
those to happen now it talks at the end
here about there being two possibilities
and stuff we'll get to those later on so
there's a little disclaimer for that but
all this bottom part right here is
saying is if you do a good job
designing your experiments then you can
actually begin to talk about cause and
effect talking about whether the
explanatory variable cause the
difference you are seen so to close
things out in this section No close
things out in this section I have just
really quick here my little thing moves
but it's just a summary of the four big
principles of experimental design so
these four things that are written right
here make sure you read through these
make sure you understand them and
anytime you're designing your own
experiments these are all things you
Click on any text or timestamp to jump to that moment in the video
Share:
Most transcripts ready in under 5 seconds
One-Click Copy125+ LanguagesSearch ContentJump to Timestamps
Paste YouTube URL
Enter any YouTube video link to get the full transcript
Transcript Extraction Form
Most transcripts ready in under 5 seconds
Get Our Chrome Extension
Get transcripts instantly without leaving YouTube. Install our Chrome extension for one-click access to any video's transcript directly on the watch page.
