0:01 One thing we've seen is that there's
0:03 definitely more pull requests per
0:04 engineer. And what was really
0:06 interesting to us was that the average
0:08 size of a pull request is higher. The
0:10 other thing we've noticed is that now
0:12 20% of our code that is produced is
0:14 produced by AI. And we found that really
0:17 interesting. We also seen a resurgence
0:19 in people mentioning time for review
0:20 being a bottleneck because reviewing
0:23 pull requests coming from AI can be much
0:25 more complicated and long just to do
0:27 because there's just more code. As of
0:29 now, we have a new bottleneck and a new
0:31 problem to solve.
0:33 >> Welcome to another episode of
0:35 Engineering Enablement. I'm Laura Taco,
0:38 your host this week. My guest today is
0:40 Fabian Dei from Monzo Bank, a
0:42 Londonbased bank that serves over 13
0:44 million customers and is scaling fast.
0:46 Fabian oversees multiple platform
0:48 engineering teams at Monzo, who own
0:50 everything from CI/CD for backend and
0:52 mobile, mobile releases, feature flags,
0:54 and of course, AI tooling. Before he was
0:56 at Monzo, Fabian worked at Spotify on
0:58 backstage and developer experience.
1:00 Fabian, thank you so much for joining us today.
1:01 today.
1:02 >> Thank you very much. Thank you for
1:03 having me. Longtime listener of the
1:05 show, so very pleased to be here.
1:06 >> Yeah, well, we're we're pleased to have
1:09 you. Could you start by telling the
1:11 audience a little bit about Monzo for
1:12 those of them that haven't heard about
1:14 Monzo before?
1:16 >> Sure. So, Monzo is a digital bank in the
1:18 UK. We have more than 30 million
1:20 customers here and we're really proud of
1:22 what we're building, very new. Um and
1:25 yeah, we're we're coming to Europe now.
1:27 >> Very excited to hear that. Can you share
1:29 a little bit about your role? So, you're
1:31 responsible for several platform
1:33 engineering teams, but um you know, what
1:34 does the scope of your role look like?
1:36 >> Uh you look after four teams at the
1:38 moment, which are looking around what we
1:40 call developer experience, but there's
1:42 touching many different parts. So, for
1:44 example, we've got the mobile platform
1:46 which looks after our release and
1:48 everything to do with our release apps
1:52 and performance. Um we also have two
1:54 teams dedicated to developer experience
1:58 and teams about AI AI enablement. We
2:00 call it augmentation engineering.
2:02 >> Ah that's a good that's a good name for
2:04 it. AI augmented augmentation
2:06 engineering. To start our conversation
2:07 Fabian we're going to talk a lot about
2:10 AI. We'll talk about datadriven
2:12 evaluations of AI tools and talk about
2:14 some of the downstream impact that it's
2:16 having on your engineering organization
2:19 and your business. To start with, Monzo
2:22 is a bank. You It's a tech company that
2:24 also does banking. We could say you're a
2:25 modern bank, but a bank nonetheless. You
2:27 have a lot of sensitive information,
2:29 personal information. It's a highly
2:31 regulated industry. Can you share a
2:33 little bit about your approach to AI and
2:37 your AI strategy in a regulated industry
2:39 like banking where, you know, sometimes
2:41 those those things don't seem to go
2:43 together so well, innovation and and
2:45 regulation, but you've done it in a way
2:46 that's really working for you. Can can
2:47 you talk a bit about that?
2:50 >> I think what's important for us is that
2:52 us as being the platform group for all
2:54 the engineers at Monzo, we can provide
2:58 with provide tools to everyone. Um, and
3:02 by centralizing the kind of um making
3:04 these tools available, we can pro we can
3:05 make sure we have guardrails in place
3:09 when it comes to like PII, client data,
3:11 all of that. That's like top of mind for
3:13 all of us. So we're not going to make
3:14 all the tools available and going to
3:16 tell everyone you can use it on whatever
3:18 you want. Uh we're going to make sure
3:20 that it's only applicable to certain
3:22 domains to only certain people who've
3:25 been doing some training. That doesn't
3:26 mean we're not looking at expanding
3:28 these tools, but we want to make sure
3:31 that we have the things well in order
3:35 before expanding. Um, and we're also a
3:37 tech company as you said, so we also
3:38 want to make sure like we are not
3:40 falling behind the curve and we know all
3:42 of our competitors are looking at these
3:43 tools and we've done the same um
3:45 something like a year ago where we
3:47 looked at ourselves and wondered should
3:50 we just bring all those tools in or
3:53 should we wait a little bit? We decided
3:55 that the best approach was to get the
3:57 tools in and go through a trial with
4:00 some evaluation criteria and just be
4:01 very deliberate about like what are we
4:03 trying to achieve with these tools,
4:05 where are we going and which ones are
4:08 the one that we want to have. Um it's
4:10 also not about just looking at what
4:11 works for the others. It's about
4:13 understanding what works best for us
4:15 because we have a a very specific
4:18 context. We're as I you said a regulated
4:20 industry. We also have a monor repo. uh
4:23 we've we've got a few different language
4:26 uh go uh swift cotlin so all of these
4:31 things are kind of parameters for how AI
4:35 needs to work for us yeah I think that's
4:40 such um a an example of how actually
4:42 your regulation and the constraints that
4:43 you're operating in have forced you to
4:46 think very deliberately about every
4:48 choice that you're making with AI which
4:51 in my experience with companies that I
4:53 speak with has led to in a lot of cases
4:57 better outcomes because the AI tooling
4:59 and evaluations are a lot more targeted
5:01 at very specific problems and there's a
5:03 lot more change management around kind
5:05 of how they're being rolled out which
5:06 turns out to be an advantage in the long
5:08 term even if it feels like a
5:10 disadvantage in the short term because
5:13 you hear a lot uh on LinkedIn or in in
5:15 other headlines about how fast other
5:17 companies are moving but you know to
5:19 your point Fabian making sure it's the
5:22 right all for your circumstance and you
5:24 know every company is a little bit
5:26 unique. So I think that's a very good
5:29 grain of advice for for those listening.
5:31 Let's rewind
5:33 two years ago. Let's talk about where
5:34 you started with AI. I know it was with
5:37 Copilot. Can you talk us through kind of
5:40 the very early days of Monzo and AI
5:44 augmented engineering? So um I think
5:46 like many other companies our journey
5:48 with AI started with GitHub turning on
5:50 copilot for us and suddenly this landed
5:52 in our VS code and that was quite
5:54 interesting and nice and a lot of people
5:57 starting trying it but there was not a
5:59 lot of adoption not everybody was keen
6:01 and also they didn't really know what to
6:04 do with it um and I think from there we
6:07 started being more and more curious and
6:09 I think our leadership was really keen
6:13 to make sure that we are investing into
6:16 bringing these tools in um and we were
6:18 we were really empowered in the platform
6:20 uh collective to make sure that we can
6:22 bring these tools in but also make it in
6:24 a very structured way. So that's uh
6:25 that's what we did.
6:27 >> Let's talk about some of that structure
6:29 because datadriven evaluations for AI
6:32 tools are it's it's so critical. Can you
6:34 share why that was the approach that you
6:36 took? You know why was this structured
6:39 evaluation with really clear criteria so
6:41 important for your AI strategy?
6:44 >> We are like very data driven at Monzo
6:46 and any decision we make we need to make
6:49 sure that it is the right one and what
6:51 is best other than data to make this
6:54 decision. So we wanted to make sure that
6:56 even before we started any trial we had
6:59 a evolution criteria in mind so that we
7:00 could compare all the different tools
7:02 that we're going to try. So we've tried
7:05 three tools since then, cursor, wind
7:07 surf, and clo code. And we've applied a
7:08 set of evolution criteria for all of
7:11 these. Um, and this has been extremely
7:14 helpful because at the end of the trial,
7:16 we've got our evolution criteria, we've
7:17 got our values, our numbers, and we
7:19 could decide whether we want to continue
7:22 with it or not. And um, it was very easy
7:24 to convince anyone that could question
7:26 or that wanted to say, "Oh, this tool is
7:27 great. I've used it on my personal time.
7:29 We should definitely use it." And then
7:30 we look at the data and we can say, "Oh,
7:31 no, look at that. there's some things
7:33 that don't match up with what you're thinking.
7:34 thinking.
7:36 >> So having that evolution criteria has
7:38 been extremely helpful for us
7:40 >> and some of the criteria we've looked at
7:43 that was really helpful. So, um, we've
7:45 got a list of like 10 to 15 criterias
7:48 around adoption, uh, the weekly active
7:50 and the monthly active users, retention,
7:53 which is super important for me cuz I
7:55 see a lot of people wanted to try the
7:57 tool, then they'll be counted as part as
7:59 weekly active users, but then they're
8:01 just going to drop the tool cuz it's not
8:03 really helpful. So, having the retention
8:06 uh, criteria was really helpful.
8:07 We've been looking at the number of
8:09 lines of code written by the AI and the
8:12 percentage of code written by the AI.
8:14 We've been looking at the acceptance
8:17 rate of suggestions, the satisfaction of
8:19 our engineers. That has been really
8:22 really important. Um, we've looked at
8:24 the different use cases that it covers
8:26 and whether it's good at certain things
8:28 rather than others. So bug fixing,
8:30 documentation, large migration,
8:32 refactoring, developing new features
8:35 based on templates, all of these things.
8:37 We wanted to make sure we had detailed
8:40 feedback on all of these. Um, we also
8:43 looked at the company health and also
8:45 the just the cost of the tool because
8:46 some of these tools can be quite
8:48 expensive if you use them quite a lot.
8:50 So we want to make sure that there's a
8:52 good return on investment and we
8:53 understand where we're spending our
8:56 money. Yeah, thank you so much for
8:58 sharing that with with me and with the
8:59 audience out there. I know that
9:02 datadriven evaluations and understanding
9:05 how should I even be approaching making
9:06 it making a purchasing decision is
9:09 something so many leaders are in the
9:11 thick of right now really kind of
9:12 struggling to find an approach. So I
9:13 think your guidance will be really
9:16 helpful for them. I want to get into
9:19 specifics on some of these things. So
9:20 first let's start with the tools themselves.
9:22 themselves.
9:23 you so you're part of the platform
9:25 collective. You mentioned that you're
9:27 you're triing cloud code cursor and
9:30 windsurf. Who picked those tools? Was
9:31 that something the platform engineering
9:33 team or your platform engineering
9:35 collective was responsible for? Could
9:38 developers nominate tools for trials?
9:39 Like how did you land on those being the
9:41 ones that you were triing?
9:43 >> It's an interesting mix of both. So
9:45 we're listening to what people are
9:47 suggesting. We're looking at the market
9:49 and we're talking to some of our friends
9:52 and at other companies and trying to
9:54 understand where this is going.
9:56 Obviously, the market is moving really
9:59 quickly and I remember we started a
10:01 trial and suddenly there was another
10:03 tool that everybody wanted to try. Um,
10:05 but we also wanted to make sure we
10:07 sequence the trials not to have too much
10:10 overlap as to otherwise it could bias
10:11 some of our data.
10:14 So it's it's really a mix of what our
10:16 engineers and leaders have been asking
10:20 about uh but also us going more deeply
10:21 to try to understand if this is
10:24 something that could work for us in our
10:26 context. Uh we also want to make sure we
10:29 can cut for the largest use case. Uh so
10:32 we've looked at tools that could help
10:35 more specifically iOS engineer for
10:37 example. um which would be great because
10:39 they don't always have the best tooling
10:43 in their day-to-day life. But now might
10:45 not be the best time for us to invest in
10:48 a tool that can only serve 10% of our
10:50 engineering force. Uh when we look at
10:52 tools like cloud code, the great thing
10:54 with it is that it can integrate as many
10:58 different workflows and ids and that
11:01 makes it very easy to um like adopt and
11:04 start using. So there's also a question
11:08 of which tool and we've had to reject
11:10 some proposal of people wanting to do a
11:13 trial just because we believe that it's
11:15 not the best investment of our time.
11:17 We're not we don't have an unlimited
11:20 budget and a very large team that works
11:23 on like bringing these tools in, but we
11:24 are much more interested about making
11:27 sure we can bring tools that can help
11:30 the majority of our engineers and we can
11:32 invest in the tool because I think it's
11:34 one thing to bring a tool in, it's
11:35 another thing to make sure that the tool
11:38 is useful and understand all of the
11:41 context at Monzo. So we've in invested a
11:45 lot into tooling and code files and
11:48 workflows and and all of the markdown
11:51 files that really guide the models.
11:52 >> You know some organizations take the
11:54 approach that a developer just gets a
11:57 stipend and they can choose whatever
11:58 tool they want in which case you know
12:00 your iOS engineers could be picking
12:03 something different from your you know
12:07 your go engineers based on whatever you
12:08 know whatever preferences they have
12:11 there. their own workflows and and use
12:13 cases that they that they need to to
12:16 have covered. At Monzo though, because
12:19 you are, you know, operating at scale
12:21 and scaling, you're in a highly
12:23 regulated industry. When you talk about
12:25 bringing in a tool, you're not talking
12:27 about just developers being approved to
12:28 use it. You're really talking about like
12:30 an organizational
12:33 decision to support this tool, train
12:35 people, enable them how to use it. as
12:39 you said all the MD files and all the um
12:41 you know context gathering for this tool
12:43 to really integrate it in the workflow.
12:45 Um am I getting that right?
12:47 >> Yes, that's absolutely correct. Um
12:50 another example of that is how we use
12:53 MCP like it be it would be very easy for
12:56 us to just enable all of the MCP
12:58 integrations and get models to access
13:01 our BigQuery data and our notion data
13:03 and our Jira data and whatever tool you
13:06 want. But some of these tools might have
13:08 some PII or there might be things that
13:10 are confidential that we don't want to
13:12 share. So our approach by default is not
13:15 to enable that. But we're creating an
13:18 abstraction on top of all of these what
13:21 we call knowledge stores and we are just
13:23 allow listing the different things that
13:25 we want the modules to be able to
13:27 access. Uh so something that we've done
13:29 for example is we've got a lot of like
13:32 reference documentation as part of our
13:36 onboarding uh paved path or golden paths
13:37 however you want to call them and we
13:40 want to make sure that the um the models
13:44 know how to access them because it's by
13:47 ensuring that model can access the Monzo
13:49 context it's going to make it 10 times
13:52 better at doing our day-to-day job and
13:55 >> I think the first reaction of a lot of
13:56 engineers we've tried these tools is
13:59 like, oh, it's pretty good, but I had to
14:02 tweak the code to make it the Monzo way.
14:04 Um, and that's because the tool doesn't
14:06 know about it. So, it's not going to be
14:08 magic magically picking things up. And
14:10 because we are working on a monor repo,
14:12 there's quite a lot of context if you
14:14 were to push the entire monor repo
14:17 there. So for us having markdown files
14:20 that can guide models and having an MCP
14:25 that can open some like um rag workflow
14:26 for the models that's been extremely
14:28 important and that's been very
14:29 successful as well.
14:32 >> Yeah. So this is really about
14:34 transforming the ways that software gets
14:35 made at Monzo not just about an
14:38 individual developer having you know
14:40 three different AI tools that they can
14:42 use however they want for their own
14:45 individual tasks. This is really about
14:47 changing the way that your organization
14:50 works, which is why data is so important
14:52 and why you need the validation of the
14:54 use case and of the impact so that when
14:56 you do scale it, first of all, it's an
14:59 investment of money, but also time,
15:01 resources, training, um that you're
15:02 really making a solid decision.
15:05 >> Yeah, definitely. So important to make
15:06 sure that it's not just bring the tool
15:09 in and forget about it. it has to be
15:14 continuous um involvement and it's also
15:17 not just about one tool like if we can
15:19 work on tools and that's why MCB I think
15:22 is really great is it can interact with
15:23 all the different models and all the
15:27 different tools so that that's really a
15:28 great value for us
15:29 >> I want to get back to some of the
15:31 metrics that you mentioned before about
15:33 actually tactically how you are
15:35 evaluating these tools and get into some
15:37 specifics so you mentioned a couple
15:40 things like adoption rate, percentage of
15:42 code written with AI, percentage of PRs
15:45 written with AI, satisfaction, company
15:49 health. What are you actually using to
15:51 measure these things both during the
15:52 trial period but also beyond the trial
15:54 period? Are you using system data? Are
15:56 you using self-reported metrics? Like
15:58 what have you had to to put together in
16:00 order to get this to get a field of view
16:02 on the impact of these AI tools at
16:05 Monzo? some of these criteria that we
16:07 get the data straight from the tool
16:09 providers. So things like active users,
16:11 lines of code return, sometimes
16:13 acceptance rate as well, that's the kind
16:14 of thing that they like to put in front
16:17 of us and credit usage and cost as well.
16:19 That's very easy for us to gather. For
16:21 the other one, it's mainly uh survey
16:23 data. Um so we we send that to all the
16:25 users. We have the list of users. We we
16:27 send a short survey and we've used the
16:29 same template for all the different
16:31 tools again just to make sure we we're
16:32 doing this the right way. Yeah, it's
16:34 been it's been pretty good because we
16:37 can then dig into some details area like
16:38 I talk about satisfaction and we have
16:41 one satisfaction but we also dig into
16:44 the different use case and try to say um
16:47 how good is tool X for refactoring how
16:51 good is tool X for large migration how
16:54 did you find tool X at explaining code
16:56 so all of that help us dig into the
16:58 details and then um we're running
17:01 quarterly developer experience uh
17:03 surveys as And we've started adding some
17:06 like more AI taillet questions to that.
17:08 Example, one thing that was really
17:10 interesting is that we still only have
17:13 half of our engineers who feel confident
17:16 to use AI engineering tools on a
17:18 day-to-day bas basis. So it means it
17:20 shows us that we still have a long way
17:20 to go.
17:22 >> So even if the adoption rate and we can
17:24 see people are using it maybe weekly or
17:27 monthly, that's not the same as they
17:29 they feel confident, they feel like it's
17:30 really helping them, they trust the
17:33 output. Um and then from there you can
17:36 make strategic decisions about investing in
17:37 in
17:39 training enablement targeting other use
17:41 cases doing other things. So I can I
17:43 imagine that data is really helpful for
17:46 that continued continued support of the
17:49 tool and and helping people stay on
17:50 boarded as users.
17:53 >> Exactly. That's like it really points to
17:55 us where do we need to invest like do we
17:57 still need to invest in upskilling
17:59 people? Do we need to invest in more
18:01 tools? I think at the moment we're in a
18:04 position where we know that we have good
18:06 tools in but we just need the engineers
18:08 to feel more comfortable about it.
18:11 >> Yeah. So you mentioned uh doing
18:13 evaluations of a couple tools, cloud
18:17 code, cursor, windsurf for an individual
18:19 developer. Do they have access to
18:22 multiple trial tools at the same time or
18:24 do you segment who gets access to which
18:26 tools? So you're kind of comparing
18:28 different cohorts to different cohorts
18:30 or does does is there ever the the
18:32 experience of one developer has access
18:33 to a bunch of tools and then they pick
18:34 their favorite?
18:36 >> Yeah, that's a really great question.
18:38 And um what we've tried to do is first
18:40 sequence how we bring these tools in
18:43 just because we want to avoid uh like
18:45 overloading people with new things that
18:49 already the pace of change in AI is is
18:51 very high and we want to avoid people
18:53 feeling drowned or even having a fear of
18:55 missing out. That's also something we're
18:57 very conscious about. It's not because
18:59 we bring these tools in that everybody
19:01 needs to drop what they're doing and
19:03 jump on these things. We've had a good
19:05 learning story when we did these trials
19:08 cuz the the first one we did we
19:10 >> gave the tool to as many engineers as
19:14 possible and it was almost free for
19:16 anyone to use. Uh but we quickly
19:18 realized that there was a lot of people
19:21 who were interested but didn't have the
19:23 time to invest. couldn't give us any
19:26 good feedback and it felt a little bit
19:29 disappointing is that in that we spent
19:32 so much time trying to enable 400 450
19:35 engineers and only 100
19:39 150 would use it and then 50 would give
19:42 us feedback. Um so we decided then to
19:45 change our approach in future trials to
19:47 just start with a small cohort of people
19:51 that have a very welldefined use case.
19:53 So they might be working on a migration.
19:55 They might just be very interested. Um
19:57 we have a group called the AI
20:00 engineering champions which are just in
20:03 different parts of Monzo uh looking at
20:05 different part of the product. Having
20:08 couple of engineers who are very keen on
20:09 leveraging these tools but also be the
20:12 relay for the rest of their collective
20:14 to present their learnings. That was
20:17 very important to have them engage so
20:18 they can relay the information.
20:21 >> And starting with these group of like
20:23 smaller number of people who can really
20:25 push the boundaries of these tools
20:27 >> made us realize that some tools might
20:29 look great on the surface but when you
20:32 dig into actually using it on real
20:34 project for real use case there are some
20:36 limitations. Um so having this co this
20:40 cohort helping us evaluate and give us
20:43 better kind of feedback or more more
20:45 qualitative feedback was really helpful
20:47 for the subsequent trials that we've run.
20:48 run.
20:50 >> How are you getting that
20:52 information from those champions then
20:53 scaled across the rest of the
20:55 organization because I imagine that's
20:58 also like that advocacy part is part of
21:01 that responsibility of being an AI
21:03 champion. Yeah, it's some it's something
21:06 we haven't yet fully invested in. Um
21:07 because in our the way we look at
21:09 things, we believe that first we wanted
21:12 to bring some tools and and try them and
21:16 get our hands dirty. The second phase is
21:19 to invest and double down on these tools
21:22 by building all the tooling around it,
21:23 all the safety nets around regulation,
21:26 the guardrails that we want to have. And
21:28 then the last phase is we want to start
21:30 upskilling people. And we're just
21:33 getting there. But part of our strategy
21:36 is to communicate very often whether
21:38 it's like slack channels or all hands
21:41 announcements around what are the things
21:43 we're doing and what are the success
21:45 stories as well. I think it's really
21:47 motivating for people to hear that one
21:49 of their colleague tried the tool for a
21:51 particular use case and it really
21:53 worked. And what I see is that there's
21:56 there's a lot of people who are not part
21:58 of the champions but who just
22:00 >> did it just because they feel that it
22:02 was really productive and they just want
22:03 to share it with the rest of the
22:05 organization. And I find that really
22:07 great. But I also know that it will not
22:10 be will not be able to scale that model
22:12 for everyone. So we'll definitely bring
22:14 some more structured learning looking at
22:17 like prompting engineering for example.
22:18 I think that's something really
22:21 important. It's not really yet part of
22:23 our um like on boarding tutorials at
22:25 Monzo. So, it's not something we're
22:27 going to look at in the next six months
22:28 for sure.
22:29 >> Yeah. A lot of exploration. It sounds
22:31 like a lot of experimentation happening
22:35 trying to figure out where the high
22:39 leverage use cases are and then you know
22:40 nailing it really well and then scaling
22:42 it across your organization seems to be
22:43 your approach.
22:45 >> Definitely. And there's some good we've
22:47 got some good data around that because
22:50 when we look at AI usage and also
22:52 spending we see that we have a minority
22:56 of users who are the biggest spenders um
22:58 and the curve it start with a high peak
23:01 of like the top 10 users will spend
23:04 >> 50% or more of the of the money and then
23:06 it really decrease and then there's a
23:08 long tail
23:10 >> and what we want to see is that for that
23:11 curve to become a little bit more
23:14 linear. Uh, so yes, there always going
23:15 to be high spenders, people that are
23:18 using it more, but we want that critical
23:23 mass of the kind of 25 to 75 to be
23:25 spending a lot more and just being able
23:27 to be more comfortable with these tools.
23:29 And the great thing is we have that data already.
23:29 already.
23:31 >> Yeah. How and how are you getting that
23:33 data you're getting directly from the
23:35 tools themselves, right? The telemetry
23:37 has sort of matured a lot in the last 12
23:38 months when it comes to what you can get
23:40 from the tools. And I think usage and
23:42 cost spending data is one of the areas
23:45 that there's a lot more visibility into
23:47 than 12 months ago for example.
23:49 >> Yeah, definitely. And we also conducted
23:52 um a bunch of one-to-one interviews with
23:55 our most uh costly users or I'll call
23:56 them power users.
23:58 >> That's much nicer. It's really really
24:00 interesting to know what they're doing
24:02 because they're like very advanced in
24:04 the curve of what you can do with
24:07 engineering uh with AI in engineering.
24:09 There are also some people who just said
24:11 you know I I was just trying to compare
24:13 the different models. So yes there might
24:16 have been a lot of like spending for
24:17 that time but I think it's not going to
24:19 be always like that.
24:20 >> It was an experiment for a purpose right.
24:21 right.
24:23 >> Yeah but that person has so much
24:24 learning that we actually shared what
24:26 they've learned to the rest of the
24:27 company and I think that's really going
24:28 to help everyone.
24:30 >> I want to talk a little bit about the
24:31 spending and budget aspect. I mean,
24:33 Monzo is a bank, so it's probably not
24:36 unusual for you all to be thinking about
24:38 pounds and cents and and other things.
24:41 You know, cost control, I think, is
24:44 something that is on a lot of people's
24:45 minds. Maybe they're not actively doing
24:48 anything about it, but 2026 is not going
24:51 to see, I think, AI costs reduce, right?
24:53 We should be expecting to spend more.
24:55 Can you talk about how the cost piece of
24:58 it factors into your evaluation
25:01 criteria? like are you you have a budget
25:04 in mind for AI tool spend per developer?
25:06 Like how are you approaching bringing in
25:08 all of these tools, saving room for
25:11 experiment, but also not blowing the top
25:13 off of of the budget that you have for R&D?
25:14 R&D?
25:17 >> Yeah. Um that's something where we
25:19 started like a year ago. We had no idea
25:23 and we looked at what can what would be
25:25 sensible, how much would be would be
25:28 comfortable spending and we started with
25:29 that. But very quickly realized that
25:32 number was not really helpful. Um I
25:35 think what helped us to land on a figure
25:38 was to actually put the tool in the
25:40 hands of our engineers and seeing are
25:42 they satisfied with it and how much does
25:46 it cost. Um and we've landed on a figure
25:48 of um $1,000 per engineer per year at
25:50 the moment. And that's something we're
25:53 comfortable spending and we know we're
25:56 not there yet but we're close to that.
25:58 Um and that's something that has been
26:00 very transparent from the beginning up
26:02 to our our CTO for example we were
26:04 discussing with them and saying like
26:05 would you be comfortable if we spend
26:08 that much and uh they said yes and I
26:11 think we we had some rational behind it
26:13 it's not just a random number in the air
26:16 but we are thinking that a typical user
26:19 would use that many requests per day and
26:21 all these things
26:23 >> then agents are coming that's yet
26:27 another unknown and similar I think with
26:28 agents we're going to be spending more
26:30 but we'll take a similar approach of
26:34 saying we expect to see these gains from
26:37 agents and these gains will translate to
26:40 these different cost but I think until
26:43 we have real data it's difficult to project
26:43 project
26:46 >> and just for some kind of benchmarking
26:50 generally what we're seeing in the
26:51 conversations we're having with
26:54 engineering leaders as well as the a
26:56 poll we did during a recent webinar
26:58 about budgeting, we had about 200 plus
27:00 folks respond, how much money are you
27:02 spending per engineer? We had the
27:04 majority of people falling into that
27:08 500, a thousand and then, you know, ta
27:10 tailing off as we got above a thousand.
27:13 But that's also my general
27:15 general generic guidance is sticking
27:18 around that $1,000 mark per engineer per
27:19 year. You know, GitHub copilot
27:22 enterprise license is about $500 per
27:24 developer per year. And you know, I
27:27 think like Monzo, a lot of companies and
27:29 many companies that I talked to, I would
27:30 say most of them are taking a
27:33 multi-bender approach where there's a
27:35 lot of different use cases and tools are
27:36 specialized still. We don't have one
27:38 tool that's going to cover every single
27:40 use case. And so it's important to keep
27:43 keep uh room for experimentation,
27:45 challengers to incumbent tools, kind of
27:48 keeping options open, keeping options
27:51 open, especially as this ecosystem is
27:54 evolving so fast. Um just so fast it's
27:56 hard to keep keep track.
27:58 >> Yeah. Very tricky to know what's going
27:59 to happen in 3 months.
28:02 >> Absolutely. And one of the things about
28:04 cost which leads me into my next
28:06 question is about return on investment.
28:08 So we can look and say, okay, we're
28:10 comfortable spending up to $1,000 per
28:12 developer per year, but we also want to
28:13 see the results coming back to the
28:16 business. And so Fabian, I wanted to ask
28:18 you about, you know, at Monzo kind of
28:21 generally speaking, what are some of the
28:25 results that you're seeing from using AI
28:26 right now in terms of, you know,
28:28 throughput, speed, quality, those kinds
28:32 of engineering performance metrics? What
28:33 has been the the change or the shift
28:36 with AI, augmented engineering?
28:37 collected some data very recently on
28:39 that. So I'm very happy to be able to
28:42 talk about them. Um one thing we've seen
28:43 is that there's definitely more pull
28:46 requests per engineer. Not not 100% more
28:49 but something around the 10 10 to 20%
28:52 more PRs per engineers. And what was
28:54 really interesting to us was that the
28:57 average size of the pull request is
29:02 higher. Um so it's like 20% more um
29:05 larger pull requests in terms of size.
29:07 Uh which I think was really really
29:08 interesting. The other thing we've
29:11 noticed is that now 20% of our code that
29:14 is produced according to um quantit
29:18 qualitative data is produced by AI. Um,
29:20 and that we found that really
29:22 interesting. And I think on our latest
29:24 developer experience survey, we also
29:26 seen a resurgence in people complaining
29:29 or mentioning time for review being a bottleneck.
29:31 bottleneck.
29:34 >> Okay. Because reviewing AI or pull
29:39 requests coming from AI can be much more
29:43 I mean complicated and long just to to
29:46 do because there's just more code. Uh so
29:47 we've we found that now we have a new
29:49 bottleneck and a new new problem to solve.
29:50 solve.
29:52 >> Yeah. So you're having not not just more
29:54 PRs per engineer but also the size of
29:59 the PRs per engineer is growing. And so
30:02 that's you know kind of two two things
30:04 um that lead to an increased bottleneck
30:07 in in code review. Are you using AI
30:09 tools for code review? Right now
30:10 >> we're using it but we're using it as
30:13 optin. Um it's interesting because at
30:16 some point we tried um having it like a
30:18 blanket thing on all of our pro requests
30:21 we would have an AI review and we we did
30:23 some survey again after two weeks and
30:24 realized that a lot of people thought it
30:27 was like not a great investment was
30:29 really good at certain things but also
30:32 created created a lot of noise so we did
30:34 decided to make it optin.
30:36 >> Talking about bottlenecks we mentioned
30:38 code review being a bottleneck. One of
30:40 the things you mentioned as well was
30:43 developers trying to get their changes
30:45 into a dev environment for them to test
30:48 and now we have more code more
30:50 frequently. This was an existing
30:52 bottleneck that you had and AI is just
30:54 exacerbating the problem making it
30:56 worse. But you came up with a solution
30:57 which is these preview environments.
30:59 You're calling them tendencies. Can you
31:01 talk a little bit about how that project
31:03 came to be and you know how it's helping
31:07 developers get more valuable work done?
31:08 Yeah, that's been a very interesting
31:11 journey that started roughly at the same
31:13 time when we started to adopt AI. So the
31:15 bottleneck was already there and we saw
31:17 that through our developer experience
31:20 surveys and we knew it was a problem and
31:22 we spent a lot of time thinking about
31:24 what would be the right solution because
31:26 at Monzo at the moment we have a dev
31:27 environment and a production
31:28 environment. We don't have anything in between.
31:29 between.
31:32 >> Mhm. And then we could have just said oh
31:35 let's create another environment like a
31:38 staging integration environment or
31:40 something like that but that would just
31:41 move the bottleneck somewhere else I feel
31:42 feel
31:44 >> um and we really went back to our first
31:46 principles and trying to understand what
31:47 are the engineering try to achieve
31:49 basically they want to test their code
31:52 in an environment
31:54 um but also don't want to go through the
31:56 process of to emerge and wait for the
31:59 deploy and and and do all of that so
32:01 they would just straight deploy without
32:03 reviews or anything which would create a
32:05 lot of instability in our dev
32:09 environment. So what we did there was um
32:11 as you said created preview environments
32:14 which allows us to route some of our
32:17 request through our dev environment to
32:22 this tenency which would be um per user
32:24 and then the the user would deploy their
32:27 service to that tenency and then we
32:30 would route the traffic straight to that
32:33 service. And that allows us to scale
32:35 really really well because now we have
32:37 one per user and our dev environment is
32:40 a lot less stretched. And some of the
32:42 side effect we saw around that was we
32:44 had a lot of flaky tests for example
32:46 because we were running tests on that
32:48 environment that was constantly changing
32:51 and that state was mutating a lot and
32:52 and so we've seen a lot of improvements
32:55 on that area already and we're very glad
32:57 that we're building that now and we've
33:00 got that available because with AI being
33:03 here and soon AI agents being able to
33:06 test their changes on these environments
33:08 that's going to be fantastic. So you
33:10 know AI can exacerbate and really
33:12 highlight some of the existing
33:13 bottlenecks as you said you had this
33:16 bottleneck even before AI came and
33:17 thankfully you know you have such
33:20 platform principles at Monzo thinking
33:22 about scaling this and addressing the
33:24 bottleneck which is also helping you
33:27 accelerate AI. This is now groundwork
33:30 preparation for agents being able to
33:31 create and then test and verify their
33:33 own changes in these sort of ephemeral
33:35 environments and you already have the
33:37 scaling built in. So that's a huge uh a
33:39 huge accelerant multiplier for you all
33:41 by having that really good plumbing and
33:44 that platform uh platform mindset to
33:45 start with that will be very cool to see
33:47 in the next 12 months how this all unfolds.
33:48 unfolds.
33:50 >> Yeah, exactly. That's very exciting
33:52 future. I wanted to ask you as we start
33:54 to wrap up here is not just developers
33:58 but also other folks are using AI um and
34:01 we're sort of expanding the definition
34:04 maybe of like who maybe not who you know
34:06 who is a developer but you know where
34:08 where the work gets done who where it
34:10 gets handed off. You mentioned before
34:14 about product managers and designers now
34:16 having access to more access to AI tools
34:19 in order to get a bit further along in
34:21 prototyping and validation before that
34:23 work would maybe be transitioned to a
34:25 developer team. Can you talk a little
34:27 bit about that approach because it's um
34:30 I think it's a really great illustration
34:32 of using AI not just to speed up
34:35 individual coding but to really change
34:38 the way that software gets made at Monzo
34:41 in general. It's been a fantastic
34:43 thing to observe this change because it
34:46 came organically from our designers, our
34:48 product managers, and other disciplines
34:51 really at Monzo who just reach out to us
34:54 saying, "Hey, I know you're um in charge
34:57 of tool X or tool Y. Could you help us
34:59 like get on board because we have these
35:01 great things we want to achieve." Um,
35:03 and we've seen some fantastic results
35:05 where all of our designers have been
35:08 able to try the tool and prototype
35:11 things directly. And the feedback we got
35:13 from them is fantastic because they say
35:16 that the the prototype they create is
35:18 much more much better quality and it's
35:22 much much faster to get it out. Um but
35:24 in interestingly and similarly to what
35:27 we've discussed before, we need to make
35:30 sure that we give the tool the or we
35:33 give the LLMs the right tools to work
35:36 well at Monzo. So for our designers, we
35:38 have a design system with with building
35:40 blocks for what does a button look like,
35:42 what are the different tools in terms of
35:44 spacing look like and everything and all
35:47 of that is available to the tools. So
35:49 when they create the prototypes, all of
35:52 that is pretty much instantly created.
35:54 Um, and that that that was fantastic to
35:56 to see and to observe. But I think the
35:59 best aspect for me as an engineer is to
36:01 see that we're blurring the line between
36:04 an engineer and other discipline and
36:07 where do we create the value of like
36:11 these products. Um, I think the pass
36:15 from an ID to a prototype to something
36:17 that can be passed on to end engineer
36:21 has widely changed in the past few
36:23 months and I think it's going to speed
36:25 up the way we can uh develop and ship
36:26 features at Monzo for sure.
36:28 >> Yeah, that's exciting to hear. It's
36:31 exciting to hear. Two things to ask you
36:34 as we close out our conversation. first
36:36 for companies that are maybe feeling
36:38 stuck. They're making these tools
36:39 available and they're noticing like you
36:41 did in early days, adoption is just not
36:44 what they expected. The the value or
36:47 impact is isn't what they expected. What
36:49 is your advice to companies who are
36:52 trying to use AI as a way to really
36:53 change the way work gets done, move the
36:56 needle for the business, and not just be
36:59 a fun, enjoyable thing for engineers to
37:02 to do? What advice do you have for them?
37:05 Yeah, I'd say there's multiple advice
37:06 because it's quite a complex thing. I'd
37:10 say don't don't limit yourself to just
37:12 making these tools available. There's
37:14 going to be a lot of things that comes
37:15 with it. You you'll need to share
37:18 examples that work. You need to build a
37:20 foundation to make sure people can
37:22 leverage it and it it is efficient for
37:25 their context. You need to share the
37:27 prompts that you've used successfully.
37:29 You need to share success stories and
37:31 you need to build communities as well. I
37:32 think it's very important to be able to
37:34 have relays in different part of your
37:36 organization who understand what you're
37:38 building and can relay your message when
37:40 you're saying that there's a new tool
37:43 available, you've built an MCP for them
37:45 or there's something else that that is
37:47 coming, new features, new models that
37:50 are really good at. Um, and I think
37:53 related to that, bringing a tool in your
37:57 company is is quite an investment. It's
37:59 not something that you just bring it,
38:01 put it on the shelf, and people can just
38:02 self-s serve. You're going to have to
38:04 invest in it to make sure that it work.
38:07 So, don't try to bring all the tools at
38:08 once. I don't think that's going to work
38:11 well. Make sure you're very like make
38:16 sure you have a evaluation criteras or um
38:18 um
38:20 you you have ways to understand what's
38:22 going to work for you, what's not going
38:24 to work and and be, you know, just happy
38:26 to say no to certain tools if it doesn't
38:29 meet the threshold that you've defined.
38:32 Um invest where you think it's best. Um
38:34 and yeah, good luck because uh it keeps
38:37 changing. So it's a it's a tricky domain
38:39 to be in for sure, but super exciting.
38:40 >> Yeah, really exciting. Things are
38:42 changing really fast. When it comes to
38:45 AI and the tooling ecosystem, what do
38:46 you think will be the biggest changes
38:49 that we'll see in the next 12 months?
38:51 >> I think for me there's something around
38:54 the agents and how engineer are going to
38:57 use and leverage agents in the future.
39:01 We definitely can picture a word where
39:04 as an engineer you're actually leading a
39:06 team of agents that are specialized to
39:08 do very different task. So you might
39:11 have your unit test engineer or your
39:14 unit test model or tool or you and your
39:16 refactoring one and you have your
39:19 architecture one and you might have your
39:22 Monzo X domain expert one which knows
39:24 all about finance and all these things.
39:28 And for me the big challenge and where I
39:29 haven't seen any tools around that yet
39:31 that does it greatly is the
39:34 orchestration of all of that. Um I
39:35 believe that's definitely something
39:37 that's going to be very powerful because
39:39 these tools are going to be here. Um and
39:41 they are going to be even better at what
39:45 they're doing. But for engineers or
39:47 developers to leverage that to their
39:49 maximum, I think we'll need something in
39:54 between where you won't have to give all
39:55 of that context and say, "Oh, look at
39:57 that file here and this is the
40:00 instruction for unit test." So all of
40:02 that kind of manual linking that
40:04 engineers still have to do sometimes. I
40:07 think there's an orchestration tool here
40:09 that can really help with with that and
40:11 >> make it very easy for everyone.
40:13 I bet there is a platform engineering
40:16 team out there working on tooling like
40:18 this that will I don't know open source
40:21 it or create it make it available
40:22 somehow but I think you're right it's
40:25 that orchestration layer we just want to
40:27 declare what we want and get it and I
40:29 think at the heart that's really that's
40:30 really the role of platform engineering
40:32 right and developer productivity
40:34 engineering devx engineering is
40:36 connecting the dots you know
40:38 centralizing the the complexity so that
40:40 we can abstract it away from the
40:43 developer in in a sensible way. But uh
40:45 that will be really interesting. Um
40:47 wonderful Fabian, thanks so much for
40:51 sharing your your guidance on datadriven
40:54 AI tooling evaluations, how that's
40:57 looked for Monzo, some of the the some
40:58 of the impact that you're seeing
41:00 already, and then some advice to
41:01 engineering leaders. I think it's been
41:03 really helpful for those listening out
41:05 there. And thank you very much for for
41:06 joining us.
