0:04 J of AI is an amazing technology but it
0:07 can't do everything in this video we'll
0:09 take a careful look at what LMS can and
0:12 cannot do we'll start off with what I
0:14 found to be a useful mental model for
0:17 what it can do and after that let's look
0:19 together at some specific limitations of
0:22 LMS I found that understanding the
0:24 limitations can lower the chance that
0:26 you might get tripped up trying to use
0:27 them for something that they're really
0:30 not good at so that let's Dive In
0:32 if you're trying to figure out what
0:35 prompting an L can do here's one
0:37 question that I found to provide a
0:39 useful mental framework which is I ask
0:42 myself can a fresh college grad
0:43 following only the instructions in the
0:47 prompts complete the task you want for
0:49 example can a fresh College grat follow
0:51 instructions to read an email to
0:53 determine if an email is a complaint
0:54 well I think a fresh College Dr could
0:56 probably do that and Al can do that
0:58 pretty well too or can a fresh College
1:01 drad read the restaurant review to
1:03 determine if it's a positive or negative
1:05 sentiment I think they could do that
1:08 quite well too and so too can prompting
1:11 an LM here's another example can a fresh
1:14 colge Strat write a press release
1:17 without any information about the CEO or
1:19 your company well this fresh college
1:22 grad just graduated from college they
1:24 only just met you and don't know
1:25 anything about you or your business and
1:27 so the best it could do is maybe write
1:30 it really generic and not quite satisi
1:33 press release like this but on the flip
1:35 side if you were to give them more
1:37 context about your business and about
1:39 the CEO then we can ask can this fresh
1:41 College Strat write a press release
1:43 given the basic relevant context and I
1:45 think they may able to do that decently
1:47 well and so too can a launch language
1:50 model when you're picturing an OM as
1:52 doing many of the things that a fresh
1:54 College Strat might be do think of this
1:56 fresh College gr as having lots of
1:58 background knowledge that they know lots
2:00 of general knowledge off the internet
2:03 but they have to complete this task
2:05 without access to a web search engine
2:06 and they don't know anything about you
2:10 or your business for clarity this mental
2:13 model thought experiment fresh College
2:15 grat has to complete the task with no
2:17 training specific to your company or
2:20 your business and every time you prompt
2:22 your LM the LM does not actually
2:23 remember earlier
2:26 conversations and so is as if you're
2:27 getting a different fresh College Drive
2:29 for every single T so you don't get to
2:32 train them up over time on the specifics
2:34 of your business or the style you want
2:37 them to write this Ru of thumb of asking
2:39 what the fresh College strad can do is
2:41 an imperfect rule of thumb there are
2:43 things College strads can do that Elms
2:46 cannot and vice versa but I found this
2:47 to be a useful starting point for
2:50 thinking through what Els can and cannot
2:52 do and while we're focus on this slide
2:55 on what prompting and LM can do next
2:57 week when we talk about gen of a
2:59 projects we'll talk about some slightly
3:01 more powerful techniques that might be
3:02 able to expand what you can do with
3:06 generative AI Beyond this fresh college
3:08 grad concept now let's take a look at
3:12 some further specific limitations of Ls
3:15 first is knowledge cut offs and's
3:17 knowledge of the world is Frozen at the
3:20 time of his training more precisely a
3:22 model trained on internet data scraped
3:25 by January 2022 will have no information
3:28 about more recent events so given such a
3:30 model if you would ask it what was the
3:33 highest grossing film of the Year 2022
3:36 it would say it doesn't know even though
3:39 now they were well past 2022 we know
3:41 that it was the movie Avatar the way of
3:44 water that was the highest grossing film
3:48 around July 2023 there were claims of uh
3:50 research lab having discovered a room
3:53 temperature superconductor called lk99
3:55 you may have seen this picture um in
3:57 some of the views this claim turned out
4:00 not quite to be right but if if you were
4:02 to ask an LM about
4:04 lk99 even though it's widely covered in
4:07 the news if the LM learned only from
4:11 text on the internet as of January 2022
4:14 it won't know anything about this so
4:16 This is called a knowledge cut off where
4:19 the LM knows things about the world only
4:21 up to a certain moment in time when it
4:23 was trained or when text from the
4:26 internet was L downloaded for the lm's
4:29 training a second limitation of lm's is
4:30 that they will sometimes just make
4:33 things up and we call these
4:35 hallucinations I found that if I asked
4:37 an OM to give me some quotes from
4:39 well-known people in histories it often
4:41 make up the quotes for example if you
4:44 ask it give me three quotes that
4:46 Shakespeare wrote about Beyonce since
4:49 Shakespeare lived and died well before
4:51 Beyonce I don't think Shakespeare said
4:54 anything about Beyonce but n will
4:56 confidently give you back some quotes
4:58 like a voo Shine Like the Sun all hell
5:00 the queen she SMS we the of love so
5:02 these are hallucinated Shakespearean
5:06 quotes or if you ask it to list court
5:09 cases tried in California about AI it
5:10 might give authoritative sounding
5:13 answers like this and in this case it
5:15 turns out the first case is real there
5:19 was a wayo versus umber case but I was
5:22 not able to find an ingason versus
5:25 chevron case and so the second case is a
5:27 hallucination sometimes om's can
5:29 hallucinate things or make things up in
5:32 a very confident authoritative sounding
5:34 tone and this can mislead people into
5:37 thinking that this madeup thing may
5:39 actually be real hallucinations can have
5:43 serious consequences there was a lawyer
5:47 that unfortunately used chat GPT to
5:49 generate text for a legal case and
5:51 actually submit it to the court not
5:53 knowing that he was submitting to the
5:56 court illegal filing with lots of madeup
5:59 court cases and in this New York Times
6:01 headline we see in this cringe inducing
6:04 court hearing the lawy who rely on AI
6:06 said he did not comprehend that the
6:08 chatbot could lead him aray and this
6:11 particular lawyer was sanctioned for
6:14 submitting a co-founding for madeup
6:17 things so understanding his limitations
6:20 is important if you are using this for
6:23 documents of real consequence Els also
6:25 have a technical limitation in that the
6:27 input length that is the length of the
6:29 prompt is limited and so is the length
6:32 of the text that can generate many LS
6:34 can set set a prompts of up to only a
6:37 few thousand words and so the total
6:39 amount of context you can give it is
6:42 limited so if you were asking it to
6:45 summarize a paper and the paper's length
6:48 is much longer than this input length
6:51 limitation the OM May refuse to process
6:53 that input in this case you may have to
6:56 give it one part of the paper at a time
6:58 and ask it to summarize parts of the
6:59 paper at a time
7:03 or sometimes you can also find an L with
7:05 a longer input limit length some will go
7:08 up to many tens of thousands of words
7:11 and technically LS have a limitation on
7:13 What's called the context length and the
7:15 context length is actually a limits on
7:18 the total input plus output size when I
7:21 use OMS I rarely have it generate so
7:24 much output that I run into limitation
7:27 really on the output length but I do hit
7:30 input length limits sometimes if I have
7:32 many many thousands of words of context
7:35 I want to give it lastly one major
7:39 limitation of J of AI is that they do
7:41 not currently work well with structured
7:44 data and by structured data I mean
7:47 tabular data like s the data that you
7:50 might store in an Excel or Google Sheets
7:53 spreadsheet for example here is a table
7:56 of home prices with data on both the
7:58 size of the house in square feet as well
8:00 as the price of the house if you were to
8:02 feed all of these numbers into an LM and
8:04 then ask it I have a house that's a
8:06 th000 square feet what do you think is a
8:09 good price LM are not really good at
8:12 that instead if you call the size the
8:15 input a and the price the output B then
8:17 supervised learning would be a better
8:19 technique with which to estimate the
8:22 price as a function of the size here's
8:24 another example of structure data of
8:26 tabular data showing when different
8:28 visitors may be visiting your website
8:30 how much offered a product to them and
8:32 whether or not they purchased it then
8:34 again supervised learning would be a
8:36 better technique than trying to copy
8:38 paste all of this time and price and
8:41 purchase information into the prompt of
8:44 a large language model in contrast to
8:48 structured data J of AI tends to work
8:50 best with unstructured data structured
8:52 data refers to tablet data of the S you
8:54 would store in a spreadsheet whereas
8:57 unstructured data refers to text images
9:00 audio video and J does apply to all of
9:02 these types of data although the impact
9:05 is the largest and that's why we focus
9:08 mostly on Text data in this course
9:11 finally large language models can bias
9:14 output and can sometimes output toxic or
9:17 other harmful speech for example large
9:19 language models were trained on text off
9:21 the internet and unfortunately text on
9:24 the internet can reflect biases that
9:27 exist in society so if you were to ask
9:30 an LM complete the sent sergeon walks to
9:32 parking lot and took out the L might
9:35 output his car keys but you will say the
9:37 nurse walks to the parking lot and took
9:40 out it may say her phone so in this case
9:43 the LM has assumed that the surgeon is
9:46 male and the nurse is female whereas we
9:48 know that clearly surgeons and nurses
9:52 can be any gender and so if you're using
9:55 an L in an application where such biases
9:59 could cause harm I would use care in how
10:02 we prompt and apply the LM to make sure
10:05 we don't contribute to such undesirable
10:08 biases finally some Elms can also
10:10 occasionally output toxic or other
10:13 harmful speech for example some Elms
10:15 will sometimes teach people how to do
10:18 undesirable sometimes even illegal acts
10:21 fortunately all the major large language
10:23 providers have been working hard on the
10:25 safety of these models and so most
10:28 models have gotten much safer over time
10:30 and if you use the web interfaces so the
10:32 major LM providers has actually been
10:34 getting much harder over time to get
10:36 them to Output these types of harmful
10:39 speech so that summarizes what prompting
10:42 an LM can and cannot do and as I
10:43 mentioned next week we'll take a look at
10:45 some techniques for overcoming some of
10:48 these limitations to make what OMS can
10:51 do even broader and more powerful but
10:53 first let's take a look at some tips on
10:56 prompting OMS and I hope that the tips I
10:57 share in the next video will be useful
11:00 right away to how you use STS I'll see
