The core theme revolves around the strategic implementation of Artificial Intelligence (AI) and Machine Learning (ML) within institutional investing, emphasizing the dual goals of enhancing operational efficiency and improving decision-making, while underscoring the critical importance of robust governance and continuous education to navigate the complexities and potential pitfalls.
Mind Map
คลิกเพื่อขยาย
คลิกเพื่อสำรวจ Mind Map แบบอินเตอร์แอคทีฟฉบับเต็ม
in every organization where you have um
uh internally you've got people who
don't really have any knowledge of AI
and machine learning who are in
positions of authority and your board
has various levels of sophistication and
comfort with you know AI and machine
learning and so that you you're stuck
with we need to innovate here and move
with the times but also we've got to
bring people along but people who have
other responsibilities also and I think
you do have to move forward with
governance in mind Because one thing
that will kill innovation really quickly
is just poor governance and poor oversight.
oversight.
>> Hey everyone, I'm Angelo Calllo, host of
the institutional edge, a podcast in
partnership with pensions and
investments. Thanks for joining us for
another episode in our series on
artificial intelligence and
institutional investing. And I'm pleased
to say that my guest today is Mark
Steed, CIO of the Arizona Public Safety
Personal Retirement Systems. I'm excited
to have Mark on the episode. Mark is one
of the few asset owners with an academic
background in predictive analytics and
firsthand experience integrating AI into
his plans, investment decisionmaking. In
this episode, Mark discusses PSRS's
AI use cases, implementation strategy,
and future wish list. Hey Mark, it's
great to have you on today's show.
>> Yeah, I'm happy to be here. This is fun.
Anytime we can talk about AI, it's a
good time. You know, I got to say I'm
grateful that our uh our mutual friend
Mark Bombgardner, who's also a friend to
the show, recommended you as a guest.
And I got to say, you know, after
talking to you in our precall, I'm
surprised our paths really hadn't
crossed previously.
>> Yeah, me too. Yep.
>> And I say, I mean, for me, the surprise
is you're one of the few senior asset
owners who has firsthand knowledge of
and firsthand experience with AI
investment use cases. And I would have
thought our approach at Rosetta
Analytics would have caused our paths to
cross before, but hey, here we are. And
let's jump right into the topic. And the
topic today, asset allocators, AI use
cases and wish list. So let's start with
the use cases. Man,
>> give me your top use cases and their
benefits. And if you could, you know,
kind of take your time and break it down
for me.
>> So I think there are really sort of two
main use cases. One of them being just
the ability to make things more
efficient. So that just operational
efficiency. The other main benefit is
really decision- making. So when I think
about operational efficiency, that's
just using AI to some extent machine
learning to
streamline and automate a lot of the
routine task that the investment office
executes. So that's collecting documents
from proprietary data sites, getting
around the two-factor authentication. So
going in and retrieving documents,
downloading those documents, then
extracting data from those documents
because all of us have PDFs and that's
really what we're mostly worried about
are just the PDFs. It's unstructured
data. So then extracting relevant
information from that data. So just to
give you an example, uh I'm looking at a
private equity fund. They give me access
to their data room. I go in there and
they've got a mountain of documents,
ppms, DDQs, whatever spreadsheets
and uh I can go in there automatically
download all those documents and then
within those documents start to extract
useful information. How many partners,
how many partners to each portfolio
company, portfolio company operating
metrics, things about the track record,
right? Fund sizes, you know, vintages,
things like that. who uh who the
compliance officer is, right? Anything
you might want to pull out and stick
into a a database that you can then use
to build um predictive models to then
help inform the front-end decision-m.
So, that's sort of operational
efficiency and there's a lot of other
applications writing investment memos
automat automatically writing those
investment memos once you have uh a due
diligence packet. So, lots of ways to to
use the automation features that uh AI
offers. And then in terms of decision-m
there's kind of two components of this.
is just uh kind of the machine learning
component which I uh isn't necessarily
different from traditional kind of
statistical techniques in some sense in
that you know you've got most of the
time you've got structured data and I
mean you know kind of what your data set
looks like and you want to load it in
and you want to figure out hey what
variables matter you know I'm trying to
ascertain there's an output variable
whether it's quartile performance or
just absolute performance I'm trying to
predict you know the predictor variable
and you have all these input variables
And those are the you know the machine
learning techniques can help with that.
Machine learning technically is part of
AI and the reason why we have machine
learning versus say traditional linear
regression which is what most of us are
accustomed to is because a lot of the
relationships are nonlinear and in fact
most of the data we have violates the
assumptions of normality which is sort
of what you need to run a lot of the
traditional regression models. And so
when you look at small sample sizes and
a lot of us have small sample sizes just
in the traditional sense of statistical
measures right we don't have enough data
to really make high confidence claims at
sort of you know the 5% level or the 1%
level uh and the and a lot of the data
is just not normally distributed right
or it's not independent right so so we
have all those problems and I think
machine learning helps with that part
but then there's this other avenue of AI
which is deep learning where you really
are kind of in the traditional kind of
black box and that's where you're just
pointing it in the direction of
information whether that's in a document
or that's it it's unstructured data or
it's a data set of some type but you
know it could be spreadsheets whatever
and you're tell you're asking it to tell
you what what patterns matter right and
I think that's a tremendously powerful
technique that might highlight some
patterns that the traditional investment
office wouldn't be aware of so that's
kind of where our brain is at I know a
long answer to that first question but
um I think those are the applications I see
see
>> let's go back to the uh uh the efficiency
efficiency
>> that's gained what type of AI are you
using there
>> and it's proprietary Mark you know I mean
mean >> no
>> no
>> no so I I can say like all of this is
just in R&D for us
>> uh and so we just now um got
authorization to put the uh LLMs on our
local machines so part of the you know
part of the problem uh historically has
just been data security and and and
still is right with data security and um
you know uploading stuff to chat GPT and
things like that and what do they do
with it? But um we uh just got the
authorization to put the large language
models on our local machines. So the so
for us um uh what we're going to start
to do once we have access to the once we
point to the in the direction of the
documents. So you can use these uh
robotic processes to kind of go in and
just access the documents and you can
you know write those scripts internally
to just go in and access documents and
pull them down and and and uh set them
in a specific location on your S drive.
But then uh what we're going to do next
uh is just use the uh LLMs uh point
those to the direction of our documents.
And we're probably going to start with
uh there's Gemma, right, which is the um
the Google or Llama, which is metas uh
that are are both fairly robust. And uh
so I'm just kind of sharing a little bit
of like our research and development
here. And we're going to point one of
those in the direction of our documents
and see how it does and start to train
it on our own documents locally because
you don't need an internet connection
and your data is staying local uh to
start to build um and extract that
documentation just for just for that
proof of concept because right now it's
just manual if there's something you
know or you can compartmentalize it in
your DDQs and just make everything hey
just tell us who this person is or what
this is and what this is which is what
we're currently doing so that we can
then move the information over to a
database that people can serve. search,
but now we're going to start to automate
that with the LLMs and uh and that'll
just be the start of it and then we'll
start to train the LLMs on writing the
investment memos and things like that.
>> So, Gen AI is going to really be a kind
of a foundational approach.
>> Yeah, I hope so. Yeah. And these are and
these are and these are like exercises
that you where you can confirm and
verify what the machine is doing. So,
we're not fussing with the blackbox
element yet. We're just saying, hey, you
know, did you label the compliance
officer the right person? uh you know
did you identify carry as 20% because
some groups call it carry some will call
it the performance bonus or whatever
right so there's these different
nomenclatures for the same thing and uh
the nice thing about having the machine
do a lot of that is just that uh we can
confirm whether it was accurate or not.
>> Yeah. So using this uh this generative
AI, you know, you talked about document
analysis. Are you using it also for
manager selection process? I mean
there's a lot of data that you have to
be pulling in and I'm not sure if you
use a consultant or not, but either way
there's going to be a lot of data.
>> We do. Yeah, we do use a consultant and
we've always just run parallel
processes. So we do our own work and
then you know hopefully they kind come
to the same conclusion and I think for
for the most part uh you know we haven't
had any uh issues with that. So on the
screening on the screening side,
I'd say yes and uh no at the same time.
So when uh I when we get inquiries, what
we don't have and I I think this is a
change that we're making is the ability
for managers to enter information
uh without contacting a staff member,
but just enter information say through
like a web portal or some other location
about their fund. And I imagine what uh
what will happen for us is uh after we
we have enough observations
from the machine learning AI uh
algorithms that can sort of start to say
these factors matter, right? So they're
sort of like kind of what you your hunch
terms of uh what you think matters in
terms of of performance but you look for
the models to help inform that and you
need a number of observations before
you're going to be confidence that hey
this you know this actually might be
accurate here. Then what you do is you
put that on your web portal and you say
hey this these are the five or six
things that we think matter most. Uh
some of them might be you know you might
you might you might imagine what they
are. Uh you know u top performance and
prior funds is maybe a good indication
that it might continue or maybe not but
things like that and uh you put them on
a you know a portal so that managers can
kind of enter that information and then
you have a really good way uh to just
you know scan for new ideas. We don't
have that portal set up yet. And so
right now what we're doing is when we
have a a GP that's contacted us um you
know we're taking their information. our
due diligence is really sort of is
frontloaded. So we send a spreadsheet
with all sorts of quantitative requests
and then they send that that back to us
and then we're crunching it through um
our models and uh so we're not quite at
the point yet where we've been able to
sort of filter based on three or four
criteria that matter. I think we'll
probably get there soon, but you just
need a number of observations before you
have any confidence that the um that the
model outputs are um accurate. I think
when we get that, we'll start to say,
okay, here's the filter. This is going
to, you know, it's an 8020 kind of a
thing. So, we might, you know, there
might be some false negatives. Um, but
we're not too worried about that.
That's, you know, the deals we don't do
that turn out to be okay. Um, we're only
worried about the deals that we do, and
we need to make sure that those ones are okay.
okay.
>> It's kind of interesting, you know, a
theme that's run throughout your
comments is this idea of verification.
And the the word I'll use is
explainability. You want to be able to
understand the decisions, the models,
you know, the LLMs or if you're using
some kind of NLP, you want to
understand, you know, where the
decisions coming from. Is that just a a
kind of a bedrock for you like a like a
ground truth? You need to have that
verification and explainability in the process.
process.
>> I mean, I think you want it if you can
get it. You want to be able to explain
how things are going and just like we do
with with kind of human reasoning,
right? where ideally what you want is
someone to kind of explain their view
cogently organized thinking right
adhering to the best practices of kind
of logic but sometimes that doesn't
happen sometimes you can't get there and
sometimes you're just sometimes you're
left with these arguments where you have
to say I don't know it's just kind of my
gut I just I can't articulate it uh you
know and so I think that's kind of an
advantage that we have at our
organization where every recommendation
that's relevant we record and you have
to be very specific about what what it
is you're recommending ing or what you
think will happen. So, if you're a PM
recommending uh you know, investment in
a company or a fund or whatever, you've
just got to say, I think this uh here
here's real a real clear definition of
success. For example, this fund will
outperform the S&P 500 by at least 2%
over the next 12 months and I'm 75%
confident. And then we benchmark people
and we ask them to explain why they
think that way. So, there's always this
ability to kind of go back and piece
together sort of their logic. And the
reason you want to do that is because
it's more likely to be repeatable and
you're less likely to have surprises.
But sometimes
you just can't articulate it. And
there's sometimes as a CIO where you're
just you just feel like I just feel this
like this is compelling. I can't quite
articulate it or maybe it violates kind
of our traditional rules, but I still
feel uh exercised about it and and uh
that it's going to perform. So we write
these things down so that over time we
can sort of look at these gut decisions
as well as the ones that weren't gut
decisions and start to say like well
actually they are pretty accurate. So
what's going on here? Maybe we can pull
this apart and understand it. And I
think that's the same uh the same the
same discipline we apply to the models
which is we want to be able to explain
what's going on here. Sometimes we
can't. So we're going to benchmark these
things. We're going to track them uh
just to see hey if it's if it's saying
we should be 80% confident here. It's
going to be 80% accurate. Is it actually
right? 80% of the times it says that and
that does help us. But it's just a way
of of managing surprises and if you can
explain it potentially you can you have
better control over it and that's why
we're interested in it. Although again
just like with humans there's an element
where sometimes you just can't explain
it. I I expect that's going to be the
case with models too.
>> I know that firsthand given the
reinforcement learning model we were
using was a dark black box. Man, it was
you know it was a challenge.
>> Yeah. And the more complicated and and
ironically I think some of the more
accurate models are are easy like like
the hardest to explain. And so uh you
know if you look at like you know with
um some of the neural networks you know
yeah there are these these uh devices
you can use uh you know whether it's um
what's called feature attribution right
or these activation layers where with
feature attribution whether it's the
lime or shape you you can kind of pull
apart you know how much each of these
features is sort of contributing to the
output or with the activation layers you
can actually see where the what what
patterns the neural network is you know
is is keying in on for example shapes it
might be keen on like outlines or things
like that. So there are things you can
do to some of these models. I don't want
to say that the entire thing is a black
box, right? But decision trees and
neural networks I feel like are the are
easy are on the more explainable side of
kind of the AI machine learning, but
like some of the reinforcement learning,
deep learning, gen that stuff uh is
really kind of hard to explain at a deep
level just because of all I mean it's
effectively like trying to explain the
synapses in the human brain uh that you
know that are that are going on at any
point in time. They're just so fast and
there's so many layers and parameters,
it's really really kind of hard to
explain. So I think what's important
there is you come up with a governance
framework as to how you're going to
approach these models and these outputs
and and how you benchmark them and and
what decisions these models are actually
making and whether you're they're doing
them without uh you know without human
oversight or not which at in our shop
isn't happening.
>> Really you're building a a a governance
framework around the use of AI is what I
hear you saying. I mean, yeah, you have to
to
>> and and documentation. You've got, you
know, documentation to support the structure.
structure. >> Yeah.
>> Yeah.
>> But go ahead. I mean, you have to. I I
don't know if a lot of people are
thinking this way. Some people just use
it. And of course, they use it within,
you know, a compliance framework. Uh you
mentioned, you know, you're putting your
own data, but it's not a machine
connected to the internet, etc. But talk
about that as a governance uh framework.
>> Yeah. So I think this is a tough one
because in every organization you know
you've experienced this uh where you
have um uh internally you've got people
who don't really have any knowledge of
AI and machine learning who are in
positions of authority and so internally
you have that dynamic going on at your
organization and then for people like me
where we report to a board your board
has various levels of of sophistication
and comfort with you know AI and machine
learning and so that you you're stuck
with you know this awkward heard, you
know, hey, uh, we we need to innovate
here and move with the times, but also
we've got to bring people along, but
people who have other responsibilities
also. And so, like, how do we how do we
kind of bring these groups together and
and I think you do have to move forward
with governance in mind because one
thing that will kill innovation really
quickly is just poor governance and poor
oversight. So, oversight is as much I
think it's kind of paradoxical, but it's
as much part of the problem as it is
part of the solution. And so you can it
can be ownorous and prevent any growth
or innovation from occurring but uh and
it and and it can also be like you know
too lax and you can and it could be
non-existent. So for us, we've we've
started to say, look, with every
decision, it's written down and it's
tracked. And that's that's point number
one. And that goes a long way. And in
debiasing people in the discussion
because now it you have kind of an
objective score and it's not so
subjective and you can start to say,
hey, this machine is saying it's 70, you
know, here's its predictions for at the
75% threshold. Here's it predictions at
the 70 65 and we start to track it and
we can say hey how many uh how many
forecasts does it have at 70%. Well,
it's we've done 10 of them. Okay, small
sample size but maybe it's right, you
know, we'd expect seven out of the 10 to
be right if it were appropriately
calibrated. And that goes a long way.
You know, if after you've done say 50 of
these and you can say look um it's done
uh you know we've got we've got 50 of
these. it's about 70% accurate for 70%
confidence. That gives you some level of
comfort. But the other thing you can do
too is is start to just give it simple
tasks that like I said earlier that you
can just verify right is it is it
identifying the right things in
documents um running certain analyses
doing value attribution bridges and
things like that say in private equity
that you've done yourself and just
double-checking its work. And that's
that's one way to build it. But I think
along sort of like just like those hard
rules internally about hey we're
benchmarking decisions and things like
that. I think you also have to educate
your constituents whether that's your
board uh your executive directors other
people on your investment team and just
create that fluency with the vernacular
to start to get them comfortable with it
because it's like going to another
country and you're listening to people
talk and you don't understand what
they're saying. there's going to be a
natural level of distrust and so if you
don't understand the language of AI and
machine learning there's just going to
be a natural distrust. So I I think
that's that's sort of another prong to
the approach.
>> Do you use uh like workshops to kind of
build that educational level or does
that occur you know for example in a
board meeting? I mean how do you get
them fluent? right now it's it's uh it's
mentioning it uh on the occasion during
the board meetings and just you know our
board meetings are are fairly um uh uh
the they're spelt you know there's not a
lot of Mickey Mouseing going around
we're talking about performance and you
know governance you know and so on the
occasion I'll do my best to you know
sort of drop in a nugget about how we're
doing things and why we're doing it um
and I suspect we've been sort of waiting
to to kind of get the LLM initiative
launched and once we get there I think
we'll have a lot more to And probably
what we'll do is is some sort of
semiannual education uh with the board
to just sort of formally kind of go
through again set the stage for the
ecosystem and the various aspects of AI
and how it's impacting their day-to-day
with ways that they can you know
understand and then talk specifically
about what we're doing with it uh and
what decisions it's making because um on
the I've got some trustes who are really
comfortable and and like the idea that
we're kind of moving in this direction
and comfortable with my background and
experience in in us doing that and then
others who are are are are sort of um
cautiously optimistic and I think would
be a generous interpretation a little a
little more cynical and uh you know you
have our articles uh like the one
yesterday in the Wall Street Journal
about the uh you know AI overriding its
code so you can't shut it down and then
it kind of sets you back uh you know so
>> yeah exactly yeah so we're you know
that's so I suspect you know as as we
broadly roll it out it'll be under very
controlled circumstances and then when
we get to sort of the predictive side of
it and letting it sort of make
decisions. We'll run those in parallel
with the human uh you know and this is
just the deep learning side of it. You
know, we're already using the machine
learning algorithms and things like
that, but I those are easily I think
easy to explain and interpret. It's the
deep learning where you're basically
just giving it to a black box and
saying, "Hey, here's a bunch of due
diligence material. Tell me which fund
is going to outperform say the other
funds and uh you know or what variables
matter most." um where I think you have
to run those in parallel with the human
decisions and and over a number of uh of
observations to get any comfort there.
>> Let me go back to use cases for a
minute. You know, I hear from managers
and I also hear from allocators that
they're looking at sentiment analysis,
you know, trying to scrape the web and
then, you know, kind of detect sentiment
because sentiment is pre-ric, you know,
it it it it's there before prices
manifest. What do you think about this?
I mean, and and whether you're using it
or not, I mean, just intellectually, is
that a tool that you want to focus on? I
well I'm a little more cynical because I
don't know that I come from the
perspective that it the sentiment is
sort of pre-trade. I I think there's a I
think you can argue that causation goes
the other way that people people make
the trades and then and then they got to
talk the book and and create sentiment
if it's not going the right direction.
But I think anyway, let's just assume
that sentiment is sort of pre-trade. I
looked into this a number of years ago.
Uh it was probably 10 years ago when
sentiment analysis seemed like I mean
you know someone's probably been doing
this for 20 years but I know somebody
has been doing this for 20 years but I
mean it felt like it really entered the
mainstream in in the predictive analytic
circles about 10 years ago and back then
everyone was having trouble you know um
with hey if I'm looking at a review for
a vacuum cleaner and it says this vacuum
cleaner sucks is you know is the model
interpreting that the right way or not a
good review or a bad review and um you
know and so there's a number there
there's that but I I think I think you
know my my bias would be to say I'm not
so sure that like you know the sentiment
is actually being reflected in in sort
of the aggregate levels of flow. I I
think there are a lot of regulatory
requirements that are also driving
allocation decisions in terms of what
you have to rebalance you know what you
have to buy or sell to rebalance to stay
in compliance and I think you know who's
got to buy treasuries uh because they
have to have a certain number of you
know credits you know a certain amount
of credits in at this level and so I
don't I I'd be a little I've always been
a little dubious of sentiment analysis
because I just feel like there's too
much erosion between like the sentiment
and then like what's actually happening
in the trade. um you know pro I mean I
think relevant but I don't know how relevant
relevant
>> you know I agree with you historically
it's been around for a little bit what's
kind of gotten my attention just as an
aside is the amount of disinformation
that is out there >> right
>> right
>> and it's very difficult to detect you
know uh for a machine to detect the the truthfulness
