0:02 This is In the Black, a leadership,
0:05 strategy and business podcast brought to
0:20 Australia. Welcome to In the Black. I'm
0:22 Gareth Hanley and in today's show we're
0:24 talking with Miranda about the world of
0:26 artificial intelligence and its
0:29 implications for cyber security.
0:31 Miranda is an AI vulnerability
0:33 researcher and a trainer with
0:36 Maliva and she's the offensive security
0:39 team manager at malware security. At
0:41 malware security, she conducts
0:43 penetration testing for various sectors
0:45 including government and private
0:48 industry. Miranda has also worked on the
0:51 chips team within the ASD's
0:54 ACSC. Welcome to In the Black, Miranda.
0:55 Thank you. Thank you so much for having me.
0:56 me.
0:58 Look, we've got some questions lined up,
1:00 but before we start, I' I've just
1:03 rattled off a few acronyms there. The
1:08 CHIPS CHIPS team and the ASD and the
1:10 ACSC. Can you maybe explain for our
1:11 listeners who have no idea what I'm
1:14 talking about, what those are? Yeah,
1:16 absolutely. So ASD stands for the
1:19 Australian Signals Directorate and
1:21 they're an organization
1:22 organization
1:26 who work with foreign signals
1:28 intelligence, cyber security and
1:30 offensive cyber operations. So within
1:32 the ASD there's the ACSC which is the
1:36 Australian cyber security center and
1:38 specifically with the ACSC there's the
1:40 CHIPS team which was the cyber hygiene
1:44 improvement programs team and this team
1:47 is fairly public so what they do is is
1:49 allowed to be known. They're in charge
1:52 of performing enumeration and scanning
1:54 of government and critical
1:56 infrastructures attack surface and then
1:59 they provide quarterly reports to these
2:01 agencies on where their cyber security
2:03 posture is lacking. They have a really
2:06 really important role actually in making
2:08 sure that government's attack surface is
2:11 reduced as much as possible from the
2:14 internet facing aspects. So no potato
2:16 chips. No, no. Although they love the
2:19 chips acronym and they have a a section
2:21 called hot chips which is high priority
2:25 operational tasking and this is a
2:27 section where whenever a critical
2:30 vulnerability is notified they do
2:31 immediate scanning of government and
2:33 critical infrastructure to then notify
2:36 people who are exposed to the CVAs the
2:38 critical vulnerabilities. Going back to
2:40 the hot topic, you're involved in what's
2:43 known as adversarial machine learning.
2:46 Does that mean that you hack AI systems?
2:47 And how does that compare with
2:49 traditional cyber security like like
2:52 firewalls and penetration testing? Being
2:54 an AI hacker is is a really cool way to
2:56 put it. I would say I'm more of a
2:58 vulnerability researcher though I love
3:00 yeah performing AI hacks and and
3:02 learning about them too. So let's talk
3:06 about adversarial machine learning
3:08 quickly and then I'll compare AI systems
3:10 to IT systems so we kind of get a gist
3:12 of of what the difference in my work is
3:15 there. So adversarial machine learning
3:17 and I'm just going to call it AML from
3:19 now on because the whole acronym is hard
3:22 to say on and on. So, it's the study of
3:24 attacks on machine learning algorithms
3:27 designed to disrupt models, preventing
3:29 them from doing what they're meant to,
3:31 or deceive models into performing tasks
3:34 they're not meant to, or making models
3:36 disclose information that they aren't
3:38 meant to. So, at Maleva, we call these
3:43 the 3Ds. Disrupt, disclose, and deceive.
3:45 And we've made them as a sort of AI
3:48 equivalent to the CIA triad, which might
3:50 be familiar to listens. It's listeners.
3:55 It's a um a framework that is used to I
3:57 guess evaluate the impacts of
3:59 vulnerabilities through confidentiality,
4:02 integrity or availability. That's the
4:04 CIA and that's of people's data on
4:06 computer systems. Yeah. So that that's
4:08 used to measure the impact of
4:10 vulnerabilities on information security
4:13 systems. So the triple D disclose,
4:16 disrupt and deceive are a way to measure
4:19 the impact of AI or adversarial machine
4:21 learning attacks and
4:25 vulnerabilities. And in terms of how AI
4:27 an AI system is different to an IT
4:29 system, a few things that make it
4:32 different and which make it necessary to
4:35 differentiate AI security from the field
4:39 of cyber security and why risk
4:41 mitigation is really different for both
4:44 of them as well. So for example, IT
4:47 systems, they're deterministic and and
4:50 rule-based. They follow really strict
4:53 predefined and explicit logic or code.
4:55 And if an error occurs in one of these
4:57 types of systems, it can typically be
5:00 traced back to a specific line of code.
5:02 And for vulnerability management, that
5:06 means that you can often directly find
5:08 where a cyber security vulnerability
5:10 occurred and you can fix it with a
5:14 onetoone direct patch at the source of
5:15 the problem. And that might be just
5:17 through updating the code, configuring
5:20 settings or applying some other sort of
5:23 fix. But AI is is quite different from
5:26 that. The AI systems are inherently
5:29 probabilistic. And this comes down to
5:31 the underlying architecture that is
5:32 built off of mathematical and
5:35 statistical models. And that's that's a
5:37 whole talk for another time. But because
5:39 of that nature, there's rarely a
5:42 onetoone direct cause because AI systems
5:44 don't follow rigid rules or hard-coded
5:47 instructions. They generate outputs
5:49 based on these statistical
5:52 likelihoods. And that uncertainty is
5:54 what makes AI so powerful and so good at
5:56 what it does. Because of this
5:58 uncertainty, it can adapt and it can
6:00 infer and it can make generalizations
6:03 and really work with diverse data. But
6:05 it's also what makes it really
6:07 vulnerable because the uncertainty also
6:10 leads it being prone to errors and also
6:12 being prone to being biased,
6:14 unpredictable and
6:16 manipulatable. So yeah, AI vulnerability
6:18 management is really really difficult
6:21 because unlike cyber security and and
6:22 traditional software where you can just
6:25 patch it, with AI you can try and
6:27 optimize the architecture as much as
6:30 possible. You can try and fine-tune
6:33 models, which means align them and train
6:35 them closer to the purpose in which you
6:38 want them to perform. And you can add in
6:39 all these layers of internal and
6:43 external defenses. But because of this
6:45 likelihood in its output, there's always
6:48 a level left over where you just have to
6:50 accept that the model might be eronous
6:53 and produce mistakes. That's one aspect
6:56 and that was a lot. The second which is
6:58 I guess more simple to understand is
7:00 that it I it systems don't take
7:02 undefined inputs. They're really
7:04 structured. They're programmed to accept
7:07 one kind of input and output one type of
7:10 input. It might only intake database
7:13 queries or it might only intake language
7:15 when you're putting in your name in an
7:18 input field on a website, right? And if
7:19 you get it wrong, it will send you an
7:22 error. And errors around this are
7:24 usually due to people not having the
7:27 right protections in the backend code of
7:30 what's happening to that input. But AI
7:32 and people who have used chatbt for
7:35 example will know that you can you can
7:36 give it almost anything. You can give it
7:39 files, you can give it code, you can
7:41 give it mathematical questions, you can
7:43 give it language based questions. And
7:45 other systems also take in things like
7:48 sensory data from IoT devices and
7:51 things. And that just means that it's so
7:53 hard to secure that input because now
7:55 all of a sudden you have this
7:58 multimodal input and this huge attack
8:00 surface. It's it's really difficult to
8:02 secure. And what were those three Ds
8:05 again? Disrupt, deceive, and
8:08 disclose. Disrupt is denial of service,
8:09 preventing it from doing what it's meant
8:14 to. Deceive is about tricking the model
8:16 into doing something that it's not
8:19 usually allowed to do. For example, you
8:20 might have seen a lot of things called
8:22 jailbreaking or prompt injecting or
8:26 prompt engineering related to chatbt.
8:28 So where you might get it to talk about
8:29 a topic that it's not supposed to talk
8:32 about. Yeah, 100%. So that's deception.
8:34 You're deceiving the model into doing
8:36 that. And disclosure. So that would be
8:39 about getting the model to release
8:41 sensitive information, for example,
8:44 about other users. Is that because if
8:47 I'm using an AI system, what I'm typing
8:49 into the system is held somewhere in
8:51 memory and so somebody else might be
8:54 able to extract that from the memory. So
8:57 it could either be disclosing sensitive
8:59 user data if there's some sort of
9:02 problem where the AI can access data
9:05 about multiple users and then someone
9:06 might be able to pull your data across
9:09 into their session. Or it could be
9:11 disclosure of the proprietary
9:13 information from whoever has deployed
9:15 the AI and what the model has been
9:18 trained on. Yeah, absolutely. Or things
9:20 like the system prompt as well. So these
9:22 are this is a set of instructions that
9:26 is I guess a very fundamental piece of
9:28 how the AI knows how to perform its
9:30 task. And if you disclose that that
9:33 again is is a bit of a PI loss for the
9:35 company. So are all AI systems the same?
9:37 There's a few popular ones that are out
9:40 there that people will know of. Are they
9:44 all the same? So, all AI systems aren't
9:46 the same in terms of their purpose or
9:48 capability or even in their
9:50 architecture, but the processes that
9:53 that underpin them are the same. So, by
9:55 this I mean where they're different
9:58 could be in that the models can undergo
10:00 a variety of training types such as
10:02 supervised, unsupervised or
10:04 reinforcement learning. not worth
10:05 getting into those unless you're
10:08 actually wanting to design an AI system.
10:10 But those learning techniques can lead
10:12 to vastly different performance
10:14 outcomes. So people will choose one that
10:17 is is most optimal for their scenario.
10:20 Then models can also be fine-tuned which
10:22 means like I talked about earlier
10:24 aligning them to make them particularly
10:26 adept and good at doing one specific
10:28 thing. Or they could have entirely
10:31 different infrastructures. So you know
10:33 one that you will know of and probably
10:35 use day-to-day is a large language model
10:38 or LLM for example like chatbt or
10:42 deepseek or claude bard some of the
10:44 other ones and these reconstruct text
10:49 from human language or other inputs and
10:52 another type could be for example a
10:54 convolutional neural network or a CNN
10:56 and this provides computers with
10:58 vision-like abilities so it's referred
11:01 to as computer vision and it allows them
11:03 to be able to see differences in images
11:05 as a human would. So you would find
11:08 these types in um facial recognition
11:10 systems. But even though there are all
11:13 those differences, what is the same is
11:16 the underlying process which adversarial
11:18 machine learning exploits or AML
11:21 exploits used to target. So machine
11:23 learning models, whether they're an LLM
11:27 or a CNN or something else, they follow
11:29 the same life cycle of starting with
11:31 data gathering, data
11:34 prep-processing, model training, and
11:36 then finally deployment of the model and
11:38 inference, which is where it makes its
11:41 outputs. And all of these systems can
11:43 most definitely be exploited to access
11:45 sensitive data throughout any of the
11:48 stages in that life cycle. What type of
11:50 things have you encountered? If you've
11:51 got real world examples without
11:53 identifying anyone, of course, in my own
11:55 experience, there aren't many I can
11:58 share of disclosure processes that are,
12:00 you know, ongoing, etc. But one that I
12:03 can is a prompt injection and this is a
12:05 pretty accessible attack that's also
12:06 relatively easy to perform. So there's
12:09 lots of news about this. So it involves
12:12 targeting that deployment and inference
12:13 stage that I talked about where the
12:16 model is making its decisions. And
12:18 prompt injection involves crafting a
12:21 malicious prompt or a malicious input
12:23 that then elicits a dangerous response
12:25 from the model, bypassing their security
12:28 guard rails. So through these types of
12:29 attacks, people can confuse the model
12:32 into sharing data that shouldn't be
12:34 included either because it's malicious
12:36 or because it is sensitive information.
12:38 So it's either that deceive or disclose
12:41 or a mixture of both. The one that I can
12:45 share is I performed prompt injection on
12:47 a website to find some proprietary
12:50 technologies that an organization had in
12:52 use which would have been important PI
12:53 for them. So they had a chatbot on their
12:56 website which had too much access to
12:59 information about its own programming.
13:01 And after a few hours of me trying
13:03 various prompt injection techniques, I
13:06 could find out a system instructions or
13:07 the system prompts which I mentioned
13:10 before are important PI for the company
13:12 as it is the basis for their chatbot,
13:15 how it acts and how it performs as well
13:16 as being able to find information about
13:19 the model's architecture which is yeah
13:22 it was pretty huge. So unfortunately
13:23 it's very easy to achieve with most
13:26 language models. Chat bots are probably
13:28 an openf facing tool that a lot of
13:29 businesses might think are useful for
13:34 AI. Yeah, exactly. And they're often
13:35 Yeah, we'll talk about this later in the
13:37 pitfalls, but everyone wants one and no
13:39 one really thinks of the consequences.
13:41 But there are some really fun examples
13:44 that I've come across in my research of
13:46 very very interesting attacks if you
13:48 want to hear about them. I'm sure our
13:49 listeners would love to hear that, too.
13:52 Yeah. Awesome. So, this is a personal
13:55 favorite of mine, and it's about the
13:59 model deception stage. So, between 2020
14:02 and 2021, this guy called Eric
14:04 Jacklitch, I'm not sure how to say his
14:07 name, he successfully bypassed an
14:10 AIdriven identity verification system
14:12 and it allowed him to file fraudulent
14:14 unemployment claims in the state of
14:19 California. So basically this AI powered
14:21 facial recognition and document
14:23 verification system, it was used to
14:25 validate identities in government benefit
14:26 benefit
14:28 applications. So what it did was it
14:31 matched the image of someone's face in a
14:33 selfie that they took with their face on
14:36 their driver's license, but it missed
14:38 like a really crucial step where it
14:41 didn't correspond with any other sort of
14:44 database at all. So all it was doing was
14:46 matching that the driver's license
14:49 matched the selfie that was sent in, but
14:51 not any government records of what that
14:54 person actually looked like. So this
14:56 bloke, he went and he took a bunch of
14:59 stolen identities, like stolen names,
15:00 dates of birth, and social security
15:03 numbers, and he went and forged driver's
15:04 license with all these people, but then
15:07 replaced the real individual's photos
15:10 with his own, wearing a wig or some
15:12 other sort of disguise. And then he went
15:15 and created accounts on this system and
15:18 then uploaded the ID photo with the
15:20 photo of himself wearing a wig. And then
15:22 when he needed to do the confirmation of
15:24 identity, he put the wig on again and he
15:27 took a selfie. And the AI powered system
15:29 incorrectly was like, "Yeah, that's
15:32 that's the guy that's or the girl, I
15:34 don't know, that's Sarah." Because it
15:36 didn't check any other sort of database.
15:37 And with that identity verification
15:40 complete, he then filed fraudulent unemployment
15:41 unemployment
15:43 claims, directed the payments to his
15:44 account, and he just went to an ATM and
15:46 took them out. And I'm sure he got
15:48 himself in a lot of trouble for doing
15:50 this. Just a little bit. So that's an
15:53 error in the testing phase. Yeah. So I
15:54 definitely think there are a few
15:56 takeaways from that. A that in any sort
15:59 of critical decision-m system or any
16:02 system that has financial repercussions, etc.,
16:03 etc.,
16:05 humans should be involved in the process
16:08 of validating the AI outputs on mass. I
16:10 think AI is a really good use case
16:12 there. But a you need to make sure that
16:14 it's actually checking against some
16:18 other value that isn't based on a user's
16:19 input cuz that's where all problems
16:22 occur in every system AI or it is user
16:24 input, right? And having some sort of
16:26 human verifying that process whether
16:28 just tabbing through all of the
16:30 decisions that the AI made on mass or
16:33 picking a subset is important. And yeah,
16:34 of course that system could have
16:35 benefited from testing as well just
16:37 because knowing my own team of
16:39 pentesters like that's one of the first
16:40 scenarios we would have tested. It would
16:42 have been so fun. Are there any other
16:44 examples that might have some really
16:47 good takeaways? So there was this one
16:50 called the Morris 2 worm. So the Morris
16:52 worm was the first internet worm that
16:55 spread without user interaction. Right?
16:58 So last year researchers developed
17:00 Morris 2 which is a zeroclick worm
17:02 meaning it's a type of malware that
17:04 spreads automatically without requiring
17:07 user interaction. But this worm targeted
17:10 generative AI. So it used this technique
17:12 called adversarial self-replicating
17:14 prompt injection. So that prompt
17:16 injection that I talked about before,
17:18 but it was
17:21 self-perpetuating. And what they did was
17:23 they demonstrated a proof of concept of
17:26 this by attacking an AI powered email
17:30 assistant. It can send auto replies to
17:32 people. It can interpret emails that are
17:34 coming in. It can summarize to you
17:36 what's happening. All of that. But it
17:38 has access to your emails, which always
17:41 has security issues. So what they did
17:46 was they had attacker send an email to
17:49 users who used an AI powered assistant
17:51 and the incoming email would
17:53 automatically be processed and stored by
17:56 that AI assistant in its memory and then
17:59 the AI assistant would use it in the
18:01 reference with all the other emails like
18:02 within the context of all the other
18:05 emails to build its responses. that this
18:08 this adversarial email that they sent
18:11 first it included malicious instructions
18:13 for data leakage so such that the AI
18:15 assistant would respond to the original
18:19 email leaking sensitive information from
18:22 the target systems emails and then it
18:24 would also include this self-replication
18:26 aspect right where it's telling that AI
18:29 assistant to reinsert this malicious
18:31 prompt in future emails to all other
18:34 users so then in the case where any
18:36 other person you're emailing uses an AI
18:38 emailed assistant, they would receive
18:40 that malicious prompt. It would again be
18:43 stored in their AI assistance system,
18:45 cause leakage in their replies, like
18:47 data stealing in their replies, and then
18:49 it would self-replicate in their new
18:51 emails, and then it would just spread
18:53 from there. It's pretty scary prospect.
18:56 This Morris worm demonstration was was
18:59 really good to see like a again how
19:00 susceptible language models are to
19:03 surprise surprise language with all that
19:07 prompt injection. B how AI how different
19:09 AI systems can chain together to
19:11 perpetuate attacks. So that attack just
19:14 got carried on by AI to AI to AI. It
19:15 involves no human interaction. They they
19:18 did it themselves. And lastly, how AI
19:21 systems that store context and memory,
19:23 they introduce really bad persistent
19:25 risks because attackers can manipulate
19:27 the memory to achieve long-term effects
19:29 because now that that malicious email
19:32 prompt is stored in that person's email,
19:34 that the email assistant's memory, it
19:36 will continue being inserted into their
19:38 emails until they realize it's there.
19:40 So, what you're talking about is these
19:42 email assistants that might help you
19:45 rewrite your emails and reply to people
19:48 or might also be a system where a
19:50 business has an automatic reply system
19:52 that's using AI to reply to incoming
19:54 emails in inbox. Is that right? Yeah,
19:57 absolutely. That could target a system
19:59 that has any AI powered automation.
20:01 Yeah, it's a dangerous thought. So, you
20:04 mentioned that AI is being used for
20:06 fishing or vicing. I think some people
20:09 say as well. Can you maybe explain what
20:10 it's being used for in that context
20:11 because I think that that's something
20:13 that might be ending up in a lot of
20:16 inboxes. Yeah, absolutely. So to start
20:19 with fishing, how AI is being leveraged
20:21 there. Traditionally, fishing emails
20:23 were sort of sort of obvious if you knew
20:25 what to look for. You know, they were
20:28 really emotive in their language, trying
20:30 to get you to click on a link or
20:32 download something and respond to the
20:34 email immediately, interact with it in
20:36 some way. And also there are really
20:38 often I guess language barrier
20:41 differences, so spelling errors or bad
20:44 grammar. But with
20:47 LLMs, all of that is is reduced. People
20:49 can get these emails automatically
20:51 written in perfect English, so they
20:55 don't look too strange. And they're also
20:57 automating what we call staged fishing
21:00 campaigns. So instead of sending one
21:01 email to you with a link and being like,
21:03 "Please click now." They send you a
21:06 perfectly formatted email with no
21:09 dangerous link or attachment, they're
21:10 just trying to seek your engagement. And
21:12 then once you start talking to them as
21:14 if it's a normal conversation with a
21:16 human being, they automate replies from
21:19 an LLM to build rapport with you. And
21:20 then finally down the line, maybe in
21:23 your fifth correspondence, they'll send
21:25 the actual fishing attack, right? And by
21:26 this point, you think you're talking
21:28 with a legitimate client or customer or
21:30 someone from another organization, and
21:33 it's all been powered by an LLM in the
21:35 background. It's much harder to detect
21:37 than what people are used to looking for
21:41 in fishing emails. And then there's also
21:43 voice fishing, which is called vishing
21:46 for short. And people extract some level
21:49 of people's voices online, maybe your
21:52 voice or mine from the podcast, and then
21:54 they create a model that can mimic your
21:57 voice and get it to say whatever they
22:00 want it to say. And it might be CEO of
22:02 an organization, for example. And they
22:04 then call an employee playing back to
22:06 them the CEO's voice saying, "Hey, I
22:08 need you to make a transfer of this
22:10 amount to this bank account." And the
22:12 employee is like, "Yeah, that's Ben, my
22:14 CEO. No worries." And often times it
22:15 would be people in the financial
22:18 position that are being targeted.
22:20 Absolutely. So what about in the hiring
22:22 process? If I'm hiring somebody, is
22:25 there a chance that I could be duped by
22:27 some of these AI deep fakes? And is
22:29 there any examples where people have
22:33 been duped by this? Yeah, 100%. So an
22:36 example last year was that actually a
22:39 security company hired a North Korean
22:41 because as they were interviewing they
22:45 used an AI powered face changer and they
22:48 also used you know a deep fake generator
22:50 for all of their other photos on their
22:53 resume and things like that. So when
22:54 they were going through the interview
22:57 process and all stages of application,
23:00 they seemed like an American citizen and
23:01 they were successful in getting the role
23:04 because no one ever knew their real
23:06 identity. Going back to those pitfalls
23:08 that you mentioned, what are the
23:10 pitfalls in AI security that you're
23:12 seeing at the moment and what should
23:14 people who are listening to this podcast
23:15 think about if they want to mitigate the
23:17 risk of tools that they might be
23:20 planning on using?
23:23 Yeah, I guess the most common one that
23:25 has been around since AI became a
23:28 buzzword was around the AI hype and the
23:30 business use case often outruns the
23:32 security considerations because everyone
23:34 wants to capitalize on this AI hype,
23:37 right? So they they rush to pushing some
23:39 sort of AI system for their customers to
23:42 production, but they don't consider the
23:45 security implications of that or they
23:47 don't have the right security engineers
23:49 on the team. and they just have ML and
23:50 AI engineers who are wonderful at what
23:52 they do but they might not necessarily
23:57 be specialized in AI or ML security or
24:00 or cyber security in which there are a
24:03 lot of effects on AI systems as well. So
24:06 I think anyone looking to implement that
24:07 either internal use in their
24:09 organization or a chatbot on their
24:11 website etc you need to do the due
24:13 diligence that you would with any other
24:14 system. You need to get it tested and
24:17 assessed. You need to do risk profiling.
24:19 You need to make sure that the design is
24:21 secure from the outset, the coding of it
24:24 and that you're practicing dev sec ops
24:26 development security operations in your
24:29 processes. That's the core irk that we
24:30 have when we just see people being like,
24:33 "Ah, we pushed some AI model." The
24:35 second is relying blindly on the output
24:38 of AI models. So not validating their
24:41 outputs in decision-m contexts because
24:42 you know like we talked about earlier
24:44 they're statistical models and they're
24:47 prone to errors and bias and what this
24:49 could look like and is very commonly
24:52 happening is in a coding scenario lots
24:54 of junior developers are asking chatbt
24:56 to write code for them and they're just
24:59 copy pasting the output into
25:01 applications without performing due
25:03 diligence or understanding what the code
25:07 is saying. And in the case of say the
25:09 trading data of that model being
25:12 poisoned up in the supply chain, if the
25:15 poisoning included malicious malware to
25:17 be put in the code generation output,
25:19 then that developer might have just copy
25:21 pasted malware into their organization's
25:24 application and let it execute on a
25:26 sensitive system. That's a attack that
25:28 occurs in that data gathering and
25:30 pre-processing and training phase.
25:32 That's already hard baked into the type
25:34 of AI you've decided to use. Absolutely.
25:37 Yeah. It's also really hard to identify,
25:38 right? Because if you think about how
25:40 much training data goes into these
25:42 models, you only really need to affect a
25:44 small amount of that data to have huge
25:47 implications. So what we're expecting in
25:49 general is that there has been a lot of
25:50 training data poisoning in models that
25:53 are online today, but we just haven't
25:55 seen the effects of them yet. These
25:57 attacks might be waiting dormantly. It's
26:00 a bit of a concern, but it could be like
26:01 I said in that coding scenario, they
26:03 just copy paste some bad code and they
26:04 don't look at it and now the whole
26:06 system is compromised. Or it could be
26:09 things like relying blindly on the
26:10 output of that identity verification
26:13 system in that California case we talked
26:15 about where you don't check it. You just
26:17 assume that the AI is making the right
26:19 decision and you go from there. People
26:22 just think that AI systems are perfect
26:24 and they they do what humans can't do
26:26 and don't make mistakes. So they take
26:27 what they say for granted as well.
26:30 People often use AI systems to help them
26:31 with things that they're not sure about
26:33 themselves, right? They use it for
26:36 searching. So if you're not sure, you
26:38 often can't validate that what it's
26:40 saying to you is correct. You'll just
26:42 take it for granted. It's very dangerous
26:44 to do that. I guess the last one I would
26:45 warn about as well is sharing sensitive
26:48 information on publicly hosted models.
26:50 So if you know your organization's
26:53 running an internal only software, then
26:55 there's a bit of a lessened risk because
26:56 that information isn't going to the
26:59 cloud and it's not potentially publicly
27:02 accessible in some sort of data breach.
27:04 But in terms of just using chatbt
27:06 online, etc., you should definitely be
27:08 watching what you put in there in terms
27:10 of sensitive information.
27:12 So would you say that the first two
27:14 things would be policies for businesses
27:16 and then maybe professional advice if
27:18 any businesses are thinking about
27:20 implementing this type of technology?
27:23 Yeah, it never hurts to get some AI
27:26 subject matter expert advice on the case
27:30 and organizational policies for the use
27:33 of AI are really important, but they're
27:35 only as effective as how well people
27:37 understand them. So getting training for
27:41 your organization and your employees on
27:43 understanding AI risks is very
27:45 important. As these technologies evolve,
27:49 what emerging trends in AI security
27:52 should businesses and professionals keep
27:54 on their radar? So in terms of
27:57 vulnerabilities in AI systems, which is
27:59 what we've been talking about today, I
28:02 guess staying ahead of the trends and
28:03 understanding, you know, you don't need
28:05 to get technically deep into things. you
28:07 just just kept up to date with the news
28:09 in terms of what what's happening in AI
28:11 systems where they're at risk
28:13 particularly laws and regulations that
28:16 are coming out around AI use and
28:17 deployment because that will have
28:20 intense you know policy and governance
28:23 implications for organizations and
28:24 without doing a sales pitch part of my
28:28 work at Maleva is producing a
28:31 fortnightly newsletter which goes out to
28:32 whoever wants to subscribe there's a
28:34 TLDDR that's easy to understand and a
28:35 more technical explanation for those who
28:38 are really interested in the most recent
28:41 AI security news, vulnerabilities and
28:43 research with with implications. So
28:44 that's a good thing. We also do a
28:47 monthly industry briefing where security
28:50 professionals or their executives, they
28:52 can come in for like an hour and we'll
28:53 just talk about the takeaways of the
28:56 month. So yeah, I think staying up to
28:59 date is your best tool there. You also
29:01 mentioned regulatory risk there. Yeah.
29:06 So currently things like the the GDPR in
29:08 Europe, that's what's being used to
29:11 govern the use of AI and coming into
29:14 this year as well as the EU AI act. And
29:16 they're going to start finding people
29:19 for misuse and deployment that doesn't
29:21 have the privacy and information
29:22 security aspects that they're expecting
29:25 of organizations. So whilst Australia,
29:28 for example, hasn't implemented
29:30 something like that, they might seek to.
29:32 So it's important to stay up to date on
29:34 that. What they have said though is that
29:36 no one can use deepseek or organizations
29:38 and government etc. can't use deepseek.
29:41 So knowing what is coming into play at
29:43 what times it will very much help
29:46 organizations move through that space.
29:48 But I think people should really be
29:51 aware of how AI is being used by
29:54 adversaries potentially against them as
29:56 well. So, we've talked a lot about
29:58 vulnerabilities in AI systems, but it's
30:00 important to know and keep up to date
30:03 with how AI might be used to target you.
30:05 By that, I mean like fishing campaigns
30:08 or voice fishing campaigns or deep fakes
30:11 against CEOs and public figures in your
30:13 organization. So, staying up to date is
30:15 your best tool at the moment. Thanks for
30:17 your time and insights today. It's been
30:19 incredibly valuable and I'm sure that
30:21 our listeners know a lot more about AI
30:22 now than they did at the beginning of
30:24 our chat. So, thanks for joining us.
30:25 It's been great having you on the show.
30:27 Thank you for the opportunity. It's been
30:29 great speaking with you. And thank you
30:31 for listening to In the Black. Don't
30:33 forget to check our show notes for links
30:36 and resources from CBA Australia, as
30:38 well as other material from Miranda and
30:40 her teams at Maliva and Malware
30:42 Security. If you've enjoyed this show,
30:43 please share with your friends and
30:45 colleagues and hit the subscribe button
30:47 so you don't miss future episodes. Until
30:50 next time, thanks for listening.
30:52 If you've enjoyed this episode, help
30:54 others discover in the black by leaving
30:56 us a review and sharing this episode
30:59 with colleagues, clients, or anyone else
31:01 interested in leadership strategy and
31:03 business. To find out more about our
31:05 other podcasts, check out the show notes
31:07 for this episode. And we hope you can
31:09 join us again next time for another
31:12 episode of In the Black. [Music]
Chrome Extension