Large Language Models (LLMs) are powerful tools with significant capabilities, but understanding their specific limitations is crucial for effective and responsible use to avoid potential pitfalls and misapplications.
Mind Map
Clic para expandir
Haz clic para explorar el mapa mental interactivo completo
J of AI is an amazing technology but it
can't do everything in this video we'll
take a careful look at what LMS can and
cannot do we'll start off with what I
found to be a useful mental model for
what it can do and after that let's look
together at some specific limitations of
LMS I found that understanding the
limitations can lower the chance that
you might get tripped up trying to use
them for something that they're really
not good at so that let's Dive In
if you're trying to figure out what
prompting an L can do here's one
question that I found to provide a
useful mental framework which is I ask
myself can a fresh college grad
following only the instructions in the
prompts complete the task you want for
example can a fresh College grat follow
instructions to read an email to
determine if an email is a complaint
well I think a fresh College Dr could
probably do that and Al can do that
pretty well too or can a fresh College
drad read the restaurant review to
determine if it's a positive or negative
sentiment I think they could do that
quite well too and so too can prompting
an LM here's another example can a fresh
colge Strat write a press release
without any information about the CEO or
your company well this fresh college
grad just graduated from college they
only just met you and don't know
anything about you or your business and
so the best it could do is maybe write
it really generic and not quite satisi
press release like this but on the flip
side if you were to give them more
context about your business and about
the CEO then we can ask can this fresh
College Strat write a press release
given the basic relevant context and I
think they may able to do that decently
well and so too can a launch language
model when you're picturing an OM as
doing many of the things that a fresh
College Strat might be do think of this
fresh College gr as having lots of
background knowledge that they know lots
of general knowledge off the internet
but they have to complete this task
without access to a web search engine
and they don't know anything about you
or your business for clarity this mental
model thought experiment fresh College
grat has to complete the task with no
training specific to your company or
your business and every time you prompt
your LM the LM does not actually
remember earlier
conversations and so is as if you're
getting a different fresh College Drive
for every single T so you don't get to
train them up over time on the specifics
of your business or the style you want
them to write this Ru of thumb of asking
what the fresh College strad can do is
an imperfect rule of thumb there are
things College strads can do that Elms
cannot and vice versa but I found this
to be a useful starting point for
thinking through what Els can and cannot
do and while we're focus on this slide
on what prompting and LM can do next
week when we talk about gen of a
projects we'll talk about some slightly
more powerful techniques that might be
able to expand what you can do with
generative AI Beyond this fresh college
grad concept now let's take a look at
some further specific limitations of Ls
first is knowledge cut offs and's
knowledge of the world is Frozen at the
time of his training more precisely a
model trained on internet data scraped
by January 2022 will have no information
about more recent events so given such a
model if you would ask it what was the
highest grossing film of the Year 2022
it would say it doesn't know even though
now they were well past 2022 we know
that it was the movie Avatar the way of
water that was the highest grossing film
around July 2023 there were claims of uh
research lab having discovered a room
temperature superconductor called lk99
you may have seen this picture um in
some of the views this claim turned out
not quite to be right but if if you were
to ask an LM about
lk99 even though it's widely covered in
the news if the LM learned only from
text on the internet as of January 2022
it won't know anything about this so
This is called a knowledge cut off where
the LM knows things about the world only
up to a certain moment in time when it
was trained or when text from the
internet was L downloaded for the lm's
training a second limitation of lm's is
that they will sometimes just make
things up and we call these
hallucinations I found that if I asked
an OM to give me some quotes from
well-known people in histories it often
make up the quotes for example if you
ask it give me three quotes that
Shakespeare wrote about Beyonce since
Shakespeare lived and died well before
Beyonce I don't think Shakespeare said
anything about Beyonce but n will
confidently give you back some quotes
like a voo Shine Like the Sun all hell
the queen she SMS we the of love so
these are hallucinated Shakespearean
quotes or if you ask it to list court
cases tried in California about AI it
might give authoritative sounding
answers like this and in this case it
turns out the first case is real there
was a wayo versus umber case but I was
not able to find an ingason versus
chevron case and so the second case is a
hallucination sometimes om's can
hallucinate things or make things up in
a very confident authoritative sounding
tone and this can mislead people into
thinking that this madeup thing may
actually be real hallucinations can have
serious consequences there was a lawyer
that unfortunately used chat GPT to
generate text for a legal case and
actually submit it to the court not
knowing that he was submitting to the
court illegal filing with lots of madeup
court cases and in this New York Times
headline we see in this cringe inducing
court hearing the lawy who rely on AI
said he did not comprehend that the
chatbot could lead him aray and this
particular lawyer was sanctioned for
submitting a co-founding for madeup
things so understanding his limitations
is important if you are using this for
documents of real consequence Els also
have a technical limitation in that the
input length that is the length of the
prompt is limited and so is the length
of the text that can generate many LS
can set set a prompts of up to only a
few thousand words and so the total
amount of context you can give it is
limited so if you were asking it to
summarize a paper and the paper's length
is much longer than this input length
limitation the OM May refuse to process
that input in this case you may have to
give it one part of the paper at a time
and ask it to summarize parts of the
paper at a time
or sometimes you can also find an L with
a longer input limit length some will go
up to many tens of thousands of words
and technically LS have a limitation on
What's called the context length and the
context length is actually a limits on
the total input plus output size when I
use OMS I rarely have it generate so
much output that I run into limitation
really on the output length but I do hit
input length limits sometimes if I have
many many thousands of words of context
I want to give it lastly one major
limitation of J of AI is that they do
not currently work well with structured
data and by structured data I mean
tabular data like s the data that you
might store in an Excel or Google Sheets
spreadsheet for example here is a table
of home prices with data on both the
size of the house in square feet as well
as the price of the house if you were to
feed all of these numbers into an LM and
then ask it I have a house that's a
th000 square feet what do you think is a
good price LM are not really good at
that instead if you call the size the
input a and the price the output B then
supervised learning would be a better
technique with which to estimate the
price as a function of the size here's
another example of structure data of
tabular data showing when different
visitors may be visiting your website
how much offered a product to them and
whether or not they purchased it then
again supervised learning would be a
better technique than trying to copy
paste all of this time and price and
purchase information into the prompt of
a large language model in contrast to
structured data J of AI tends to work
best with unstructured data structured
data refers to tablet data of the S you
would store in a spreadsheet whereas
unstructured data refers to text images
audio video and J does apply to all of
these types of data although the impact
is the largest and that's why we focus
mostly on Text data in this course
finally large language models can bias
output and can sometimes output toxic or
other harmful speech for example large
language models were trained on text off
the internet and unfortunately text on
the internet can reflect biases that
exist in society so if you were to ask
an LM complete the sent sergeon walks to
parking lot and took out the L might
output his car keys but you will say the
nurse walks to the parking lot and took
out it may say her phone so in this case
the LM has assumed that the surgeon is
male and the nurse is female whereas we
know that clearly surgeons and nurses
can be any gender and so if you're using
an L in an application where such biases
could cause harm I would use care in how
we prompt and apply the LM to make sure
we don't contribute to such undesirable
biases finally some Elms can also
occasionally output toxic or other
harmful speech for example some Elms
will sometimes teach people how to do
undesirable sometimes even illegal acts
fortunately all the major large language
providers have been working hard on the
safety of these models and so most
models have gotten much safer over time
and if you use the web interfaces so the
major LM providers has actually been
getting much harder over time to get
them to Output these types of harmful
speech so that summarizes what prompting
an LM can and cannot do and as I
mentioned next week we'll take a look at
some techniques for overcoming some of
these limitations to make what OMS can
do even broader and more powerful but
first let's take a look at some tips on
prompting OMS and I hope that the tips I
share in the next video will be useful
right away to how you use STS I'll see
Haz clic en cualquier texto o marca de tiempo para ir directamente a ese momento del video
Compartir:
La mayoría de las transcripciones están listas en menos de 5 segundos
Copia con un clicMás de 125 idiomasBuscar en el contenidoIr a marcas de tiempo
Pega la URL de YouTube
Ingresa el enlace de cualquier video de YouTube para obtener la transcripción completa
Formulario de extracción de transcripción
La mayoría de las transcripciones están listas en menos de 5 segundos
Instala nuestra extensión para Chrome
Obtén transcripciones al instante sin salir de YouTube. Instala nuestra extensión de Chrome y accede con un clic a la transcripción de cualquier video directamente desde la página de reproducción.
