Hang tight while we fetch the video data and transcripts. This only takes a moment.
Connecting to YouTube player…
Fetching transcript data…
We’ll display the transcript, summary, and all view options as soon as everything loads.
Next steps
Loading transcript tools…
Best AI for Literature Reviews? Only ONE Passed the Test | Andy Stapleton | YouTubeToText
YouTube Transcript: Best AI for Literature Reviews? Only ONE Passed the Test
Skip watching entire videos - get the full transcript, search for keywords, and copy with one click.
Share:
Video Transcript
Video Summary
Summary
Core Theme
This analysis evaluates three AI tools (Manis, GenSpark, and Gemini AI) for their effectiveness in generating literature reviews, focusing specifically on their tendency to "hallucinate" or fabricate references.
Mind Map
Click to expand
Click to explore the full interactive mind map • Zoom, pan, and navigate
There are three AI tools that I've been
really impressed with with literature
reviews recently, but here's the thing.
You need to know which ones are
hallucinating and how often they just
make up references. You see, when you're
doing a literature review, you want
exactly that, literature. So, here are
the best three AI tools that I've tested
and how much they actually lie to you.
So I took three tools, Manis, Gen Spark,
and Gemini AI from Google. And I wanted
to know, are they just lying to you? So
the prompt that I put in for every
single research tool was this. If we
head over to Manis to look at it, it
says here, generate a structured
literature review on the topic and then
the topic. that I had. Please include,
and this is the important bit, the key
research themes, how they've evolved,
important studies, conflicting
viewpoints, gaps in the current
literature, suggestions for future
research. And I says, please use
peer-reviewed sources. Present the
output in a format suitable for
inclusion into the introduction of a
thesis. Add intex citations in I e
e format even if they are placeholders
and make the review as long as it needs
to be to cover the literature
effectively. So I did this for the three
things and then if you can blur this
editor these are the results. Oh let's
get into it. So the first thing was uh
Manis was uh the fastest to kick it out.
It did it in about 3 minutes. So that's
really good. But did it actually produce
enough references? And were those
references completely made up? So let's
have a look here. These were the things
that it kicked out. I love Manis because
it gives you all of the kind of files
that it used to generate the literature
review. But this is what we're really
interested in. Literature review PDF. We
can open it up here. Let's go full
screen on that. And you can see that it
is a 14page document. And we'll go all
the way down to the bottom. and it
produced 38 references and I wanted to
know are those references first of all
actually sort of like saying what they
want to say but also are they just made
up. So overall this literature review is
good. It contains all of the important
things that I want to know about
critical analysis. Um and then it's got
you know different kind of themes that
pop up in this kind of research field.
So it's a really good first start. But
let's head over to my magical Excel
document so we can see what has actually
happened. So I went through every single
reference and I actually looked on
Google Scholar to see if it exists. And
here are the results for Manis IM. So
here it was 38 references down here. And
you can see that it did an good enough
job and it was actually um making up
references and hallucinating for 16% of
the time. So, I went through and you can
see here where I put across this
reference. It was just sort of like the
wrong journal and the wrong year, but
the title was right. And then I went
down here and it actually repeated two
references that were repeated up here in
four and five. And then these ones, it
just made up completely. Here it doesn't
exist. Doesn't exist. And here the
journal year and title was wrong, but it
was kind of like a plausible one. So,
let's have a look at 18. And what I did
for each one is if I couldn't find it
like just by using Google Scholar, I
went to the actual um journal where it
was published. So, uh where are we? This
one here. Look, morphology control blah
blah. I was like, hm, I wonder if that
exists. So, I went to journal of fizz
chem and I tried to find it. And uh
yeah, journal material chemistry. I used
to put in the page number and the year
and you can see that records not found.
So, ultimately this is what I did. I
went in after using Google Scholar if I
couldn't find it there. I went to the
actual journal and uh yeah, it just
didn't exist. So overall, Manis did a
pretty good job. And then you can see
after that, I was expecting it to get
worse as it kind of like went along as
it was using more and more of its memory
to hold on to different references. But
in fact, you know, early on we got this
one, which was wrong in a certain way.
These ones repeated, then these three
were wrong, and I was like, uh-oh, this
doesn't bode well. But then all the rest
of them were absolutely fine. So overall
this gives Manis a 16% hallucination
rate or just failure rate in some sense.
Now let's have a look to see how Gen
Spark did. So Gen Spark is another tool
that I've been absolutely amazed by for
all sorts of different academic tasks.
Let's see if it's lying to us. So I put
in the exact same prompt as before and
then I just waited. This one was the
second to finish and it did it in about
sort of like 5 to 7 minutes. So overall
it was still relatively quick and you
can see it understood the fact I wanted
I e e e e e e e e e e e e e e e e e e e
e e e e e e e e e e e e e e e e e e e e
e e e e e e e e e e e e e e e e e e e e
e e e e e e e e e e e e e e e e e e e e
e e e ref referencing and you can see
that, you know, it did a relatively good
job at getting all of this. One thing I
don't like about GenSpark is that it's
hard to extract it into like a a file.
At least Manis gave us a PDF. Genspark.
Um, yeah, it's not really a useful
format. You can ask it to put it out in
maybe like a latte or latte, whatever,
however you want to say it. It's
completely up to you. Um, but um, yeah,
it it is a little bit annoying. But if I
scroll down, you can see it gave us 19
references, which isn't quite as good as
uh, Manis, which did it quicker, but Gen
Spark uh, did uh, lie to us a little bit
more. Let's check it out. So, here, this
is what we ended up. GenSpark actually
ended up lying to us
26% of the time. And if I go through
each of the references, you can see yes,
yes, it was fine. It was fine. It
existed. Then it just didn't exist. And
then it just didn't exist. And then it
just didn't exist. So this one lies more
because over on Manis, we can see that
we only had sort of two that didn't
exist. Then we had ones that was
repeated. And then here and here, um,
you know, it uh, it just kind of like
got it a little bit wrong. there was a
very similar article but uh not exactly
the same. So uh yeah, we just need to I
think be aware of how different tools
are actually hallucinating. And
unfortunately, Gen Spark was the worst
because it literally just didn't exist.
There wasn't sort of like anything
similar apart from the last title where
it was like similar. It was close to
another peer-reviewed article, but
really it just yeah, didn't really
exist. So overall, this gave us a 26%
um sort of like hallucination or failure
rate of Gen Spark. So at the moment for
me, Manis is winning. But there's one
more tool that I've tested in the past
that uh is just really great. Check out
this one. So up till now, we need to
make sure that every single reference
that these tools give us, we need to
double check. Now, I have produced a
literature review, a really deep,
detailed literature review with Gemini.
Here's Gemini. I've paid for the
advanced one and I used deep research. I
used exactly the same prompt in this one
and this is what it gave me. Now, this
took the longest. This probably took
about 20 minutes and it is detailed. It
is deep. It has um lots of tables um you
know presenting the information in a
really sort of like useful way. This is
something that the other AI tools I've
tested in this video didn't do. They
just gave us text. Here we've got sort
of like tables and not just one table,
two tables. Then we got it split up into
really useful kind of like um uh
sections. It really is detailed. And if
we scroll all the way down here, we can
see that the references are actually
there we go are actually links. So, do
these links go somewhere and do they
actually sort of like lead to real
research? That's what I wanted to know.
So, I actually exported this to docs
because I did want to see how long it
was. This is a massive document. It's 61
pages and 105 references. H, that was a
long time going through each one, but I
did it for you because I love you. Um,
and so you can see that uh, yeah, it's a
really detailed document. You've got all
of the different um, things referenced
here. It's not E though. it is just kind
of like numbers. So, it didn't really
understand that brief. And also, you'll
see in a minute that it didn't stick to
peer reviewed in a sort of like very
strict sense. It did include other
references, but let's talk about that.
So, over here, um, yeah, you can see
that it's just a really detailed thing.
And if, look, let's just cut to the
chase. If you were wanting to do a
literature review at the moment and you
are only interested in sort of like the
uh finding the literature, getting
themes, um coming up with a generated
document, there's no doubt at the moment
that Gemini AI is just doing so so well
in that space. But that's not what we're
here for today. Is it lying to us? I
went through all of them and let's uh
column. Boom. Here we are. Gemini was a
1% hallucination, but it wasn't really
hallucination. And you'll see in a
minute. Stay around because uh yeah, it
went I went through each one and you can
see that it didn't sort of like strictly
stick to peer-reviewed papers that were
published in journals. Um it did include
thesis. It did include a website. I did
like this that it did include it was the
only one where I was like, "Oh, wow.
They've used 2025 research this year."
the other ones um they got close but I
don't think I saw any like really up
to-date references. I really like that
for Gemini. And then the one it failed
down here was just like a book that
wasn't accessible and that was the sort
of like issue is that it did take me to
a like a live web page but I couldn't
access what it was actually citing. But
I would take that over a completely
madeup reference any day. So here you
can see we got a thesis thesis and then
down here we have another thesis but
ultimately it gave us 105 references and
all of them existed in some like way but
this one did not exist. Now uh there are
some drawbacks like I said you know you
can't extract this into something you
work with in terms of like uh mendlay or
endnote. You'd have to sort of like go
and find each source and put it in to
your reference manager um manually. But
look ultimately it is a really great
introduction and a great literature
review. Look I love this. I love the
tables. I love it that you know you can
see here it's actually referenced. It
says I E. You're just lying to me a
little bit there, matey. That's a bit
cheeky, isn't it? It says I there in
reference, but it's not even in IIE. Oh
well, it's okay. It's okay. I'll let you
get away with that because you're doing
well in other ways. But ultimately,
yeah, this is just such a really great
literature review. It's got all of the
things. It's detailed. It's deep. it is
the definition and I think the benchmark
at the moment for academic um kind of uh
searching the literature and producing a
literature review. So really really
impressive. Give them a go and let me
know what you think. So overall this is
where we're at. Gemini there is no
hallucination. It sort of like got a
little bit wrong. Um Manis I really
like. If you're going to use Manis just
go through and double check. It's got
about a 16% hallucination or like
failure rate. I guess I can't strictly
call it hallucination because it was
only two titles that really didn't
exist. The other one kind of got a
little bit wrong, but there was kind of
a reference like it. Um, and then it did
repeat. So overall, this is probably
less of a, you know, hallucination rate.
Um, and then we've got Jen Spark, which
yeah, it really did just make up stuff
and it didn't give us many references.
So in order, Gemini wins the crown.
put the crown on. Manis in the second
place. Oh, thanks very much. And then
Jen Spark in third place. Oh, I'm sorry
I did so badly, but I I tried my best.
All right, then. That's where we're at
with those. Uh yeah, give them a go for
yourself and let me know what you find.
If you like this video, go check out
this one where I talk about writing a
Click on any text or timestamp to jump to that moment in the video
Share:
Most transcripts ready in under 5 seconds
One-Click Copy125+ LanguagesSearch ContentJump to Timestamps
Paste YouTube URL
Enter any YouTube video link to get the full transcript
Transcript Extraction Form
Most transcripts ready in under 5 seconds
Get Our Chrome Extension
Get transcripts instantly without leaving YouTube. Install our Chrome extension for one-click access to any video's transcript directly on the watch page.
