Listen to an insightful dive into the future of AI in internal audit in the ninth episode of Inside the Auditorium with our guest Shehryar Humayun, audit director for models, data and applied sciences at Lloyds Banking Group. 

This episode explores: 

• Navigating the AI Landscape: Shehryar sheds light on the widespread adoption of AI, particularly Gen.AI and LLMs in organisations. He discusses the need for adept auditors with an understanding of these concepts to ensure organisations implement AI with the right guardrails. 
• Apprenticeship Programmes: Discover more about Lloyd’s award-winning apprenticeship programmes, where data science graduates actively participate in rotation, gaining exposure to multiple areas of the business, including internal audit. Shehryar highlights the value of this experience, noting the recognition and impact the internal audit function has. 
• Broadening Exposure in Internal Audit: Gaining broader exposure is important for internal auditors, opening opportunities for diverse experiences. Shehryah’s perspective challenges the notion that moving out of internal audit is detrimental, emphasising there is a mutual benefit between internal audit and other business areas. 

Don’t miss this engaging conversation that explores the exciting intersection of AI, internal audit apprenticeships, award-winning schemes, and the importance of diverse experiences in retaining top talent. 

Enjoy! 

Note: The views expressed by Shehryar are his own and do not necessarily reflect those of his employer. 

Good morning, Shreya. Welcome to our episode of Inside the Auditorium. Perhaps you could just give an introduction on yourself and a little bit about your career.

Morning Hazel, thanks for having me on the podcast. So I'm actually a software engineer by qualification. So I started off after my computer science undergrad, I started off working for a Swiss bank in Dubai. And then I moved to the UK working for Deloitte and then for a couple of other banks,

that include MUFG and Deutsche. And now for the last few years, I've been with Lloyds, where I made the move from being a software engineer to an auditor, to be more precise, an IT auditor. I was in Dubai initially when, after working in IT for some time, I got an opportunity to move across and I took it up.

So that's a very quick brief on my background and my career. What I do now is that I'm the audit director for models, data, and applied sciences at Lloyds Bank and Grow. And that includes, as the name suggests, that basically includes oversight of three teams. One is the model risk team, which actually provides thematic coverage of models across the group.

Lloyds Banking Group. The second team is the Data Risk Team and that team actually looks at the thematic coverage of data risk, looking at things such as data strategy, data quality, remediation and so on and so forth. And the third team, which is the biggest amongst the three, is the Data Science Team. And that team is actually involved in providing support for audits.

through the use of data analytics and data science techniques. And it provides coaching on data analytics to the rest of the function. And it works on developing repeatable tools and techniques for auditors to use on their own. So that basically is what I do and what my career has been.

Not a lot then. So can you just tell us in terms of model risk, can you just give us a bit of an idiot's guide on what a model risk is for people that perhaps don't know anything about it?

Yeah, so even models in this day and age, models are used across all organization, all parts of the organization. They'd be used in credit risk for regulatory, for calculation, for instance, calculation of regulatory capital. They'd be used for credit decisioning. They'd be used for market risk. For instance, you've got the bar model as well. They will be used, models will be used.

and creating as well, for pricing as well. And then moving out of these areas, there are also operational models, for instance, for fraud detection, for other areas, such as the risk areas as well. And now moving on, a lot of organizations are developing and rolling out AI and machine learning models.

So that are actually adding to the mix as well. So all in all, the use of models is almost pervasive across the entire organization and it's becoming more and more important. Now, what the team does is the team looks at or all model risk audit teams, they actually look at the control framework.

of around these models, how they have been, for instance, how they have been inventorized, how the controls over these models are being implemented, how that's at a management level, how the oversight over that is provided, what the governance is. And then, you know, it could, for instance, the way we operate in the model space is we get at it through...

three different mechanisms. Firstly, we look at the governance in the model risk framework, i.e. how is the management comfortable that they've got the right controls and they're operating over their suite of models with the risk proportionality. Secondly, we do model deep dives, which means that, for instance, if they're looking at the mortgage pricing model,

the team is looking at the mortgage pricing process, the audit team is looking at the mortgage pricing process, the model risk team would actually provide SME support whereby they'd look at the conceptual soundness of the model, they'd look at how it's being tested, they'd look at how the ongoing monitoring is being done. And then lastly, as I said, we do the deep tech and lastly, there would be a lot of regulatory

audits that we need to do in terms of providing attestation for models that are being submitted to the regulators. So at a very high level, this is how we actually operate in the model risk space.

Sure. So when I've been, because model risk, it is quite new in so many, you know, two, three years, quite new. And I've seen that most people that get into model risk actually come from an actuarial or a quantitative type background. Would you say then that...

Anybody that comes to work with in model re-schooled yet has to come from that type of background.

Yeah, that's interesting. I think in terms of the models being around, they've been around for a very long time. For instance, since we've been actually wanting to, an example is since we've been wanting to calculate the regulatory capital RWA, they've always been used for that. I think there's been a lot more focus on the controls around them.

Yeah.

To your point on the skill set, so we at Lloyds are actually, from the exposure perspective, quite lucky because we've got an insurance arm as well. We've got the Scottish Widows, which is the insurance business, and then we've got, of course, the banking business. So on the insurance side, we do look for actuaries being part of the models risk team.

to support the audit of models in the insurance space. And on the banking side, and more generally, we do need to have a certain number of people who worked in the first and second line, either as quantitative analysts in terms of modern development or in the second line if they worked in the function of model validation. So these skill sets are very much

thought after and the reason is it's essentially the technicality of the subject. In fact, you find people who've been working in the market risk model space for a very long time and that's their specialty. The same happens with create risk as well. So we do need those specialist skills. However, as you would do with any good well-formed team.

sure.

you need to have a mix of all skill sets, right? So for instance, you need people who've worked in audit or who have covered models from audit perspective in the past. So as I've built up the team, I've actually used the sort of backgrounds I've looked for are actuaries, quantitative analysts in terms of backgrounds, as well as...

auditors because that brings in a really nice holistic perspective for the entire team to be working on in terms of approach and how they move forward.

Sure. Okay. I meant sort of more in terms that there's now sort of banks have become siloed their model risk teams, whereas I believe that previously, although that you were doing models, there wasn't particularly separate teams that there are now. And with regards to sort of somebody working as a quantum analyst, let's say that we are now trying to get into these teams.

or why would that be of an interest to somebody like that to come into internal all debt?

I think the real value that I could actually tell people about, and that's what I've talked about as I've recently built up the models team as well, it's the view and the exposure you get working within audit. Firstly, as I just said, people would be working in the great risk model space for a very long time.

once they get a very deep understanding, they won't necessarily get the breadth of exposure that they could get working in audit. When you're in the audit team, you actually get to look at great risk models, you look at market risk models, you look at operational models, you look at trading models. And for instance, within our environment, as I said, we'd be lucky that we've got the insurance side of the business as well. So you could also look at the insurance models.

So it's the breadth of exposure that you get in an audit team that you won't necessarily get in the second or the third line. The other piece is actually the holistic view you get. You take a step back, and you're able to see the end-to-end, working with other auditors, which actually helps. You get that holistic view when you're looking at models and the risk.

they have or the lack of controls around them poses to the organization. And I believe it's the other, for instance, in terms of exposure, I've talked about variety, right? And then I've talked about the holistic view you get as well. But the other thing I think that you could really,

sort of take something of a pride in is the value you're able to add in terms of the work that you do. An audit typically is about three months or less, any typical audit, and the same applies to model audit as well. And the recommendations you make or the findings you come up with, you could actually see the impact of that on the organization.

I personally feel having worked outside of audit and then within audit, I personally feel the recognition that we get within audit because of our proximity to the senior and executive management is far more than what you get working in some of the other functions within the organization.

Sure. And just going back to yourself, I mean, as you said earlier on, you started as a software engineer and then you moved to IT audit. And now obviously you're doing models, data, et cetera. How did you get there in your career? Or did you have to retrain at all?

That's a really interesting question actually. At times I wonder if I got there. But I didn't actually. I think I didn't retrain. Initially when I made the move it was because of my technical skills, because I was a software engineer and they wanted in that organization, they wanted someone who had a good knowledge of IT to be able to identify

assess and identify and assess IT risk, but also someone who had the technical depth as well. And of course, data analytics. So that helped. And then I did CESA as I moved into audit or IT audit to upscale myself in that regard. And then I've moved into doing business audits as well as looking at algorithmic trading back at Deutsche Bank.

of course, data science, aerotroids, and then models.

Most of the learning that you get is on the job. What you learn, all you need to have is a curious mindset. And of course, you need to have certain foundational knowledge and background. And the rest you could build as you move across your career. You need to have that curious mindset. You need to have that focus on what you want to learn and how you want to progress your career.

And that helps immensely. For me, the reason to move to audit was clearly the fact that I could clearly see that being in audit, I'd get more exposure to senior management. At that point, there was a lot of travel involved as well. So I was looking forward to the global exposure as well. And it was also the variety.

that actually enticed me to the work that I've mentioned, even for the modern space. And all of these things put together really piqued my interest and got me interested in all the different things I've since done, for instance, initially IT audit, then algorithmic grading audit, then data science has been, or data analytics has been a part of my career throughout. It's been a common thread and then more than just.

And in terms of the models then, do you think the algorithmic trading helped you with that? I mean, so if somebody was going to look to get into model risk, do you think that they're better off doing some sort of quantitative qualification or?

I think looking at for someone who wants to get into the model audit team at a lower level, I think they should have quantitative background. Again, it depends on, as I said, not everybody in the team would exactly be the same. But that does help on the insurance side, actual background on the banking side, some sort of quantitative background.

that does help, especially as you get into the depths of models and when you're looking at the conceptual soundness and when you're looking at the testing of models and the calibration data and so on and so forth. But I feel someone who's actually done wider auditing and they want to move into

data science or models, they could actually do that by on the job learning as well. In every team, you'd have some very technical people, and then you'd have people who would be able to look at the high level risk and governance. And it depends on where you wanna sit within that spectrum.

Because just moving now to your data science team, I know that you've got some very strong programmers within your team. And I do find that when I've been recruiting just programmers within internal audit, I tend to find that they don't stay there for that long because they...

make it do a programme and then there isn't really, you know, they just keep doing the same thing over again. Why is that so different in your team?

I think that's, again, an excellent question, Hesil. I believe anyone moving, coming into internal audit, at some point or moving out of internal audit should have, should try to get broader exposure, right? For instance, within the data science team, I've hired, we've hired people from the business who've come and worked with us in the data science team again.

people who were auditors and we upscale them on data analytics and data science. Now they're working in the business. So we've both been, we've both been an area that attracts talent, but we've also been providing a talent pool to the business as well. So moving out of internal audit into other areas isn't necessarily a bad thing as it is.

for other areas to come and work in audit. But in terms of how we actually attract and retain people within audit is, again, as I said, variety, especially for techie within data science, variety is absolutely one thing that they love, right? One day you could be working on a consumer audit, looking, running some analytics,

certain data science techniques, such as natural language processing on sort of consumer data and complaints. The next day you could be working on a finance audit, running something totally different. The third day you may actually be doing some outlier detection in insurance space. So variety absolutely keeps people engaged.

Sure. And in terms of the areas that you do, or that you're managing, should I say, is really that you've got some very, very good technical people. I know you're head of Model Risk. She's actually from a market risk background. Generally, in the banking world, let's say,

in a communicative role. Do you think now just generally, if he was working as a market risk quant or what have you, that they would now have to be more communicative anyway? So it's easier for people to come into internal audit because it's more of a consultative role now, internal audit, right?

I think internal audit is an excellent area to upscale yourself on communication, on influencing skills because at the end of the day, when you go and find issues around controls, it's not as black and white as that. You need to be able to take your stakeholders on a journey. You need to be able to influence them and you need to be able to articulate the risk as well.

Internal audit provides an excellent training ground for that. And as you grow more senior, let's say initially, if you're working on audits as an auditor, then you become an engagement lead, then you become a head of audit. You start off with dealing with more technical stakeholders, but as you move up, you're able to see the big picture and you should be able to articulate the big picture in terms of the risks.

that those control deficiencies pose as well. So it is a great area to upskill your skills on communication skills, but also on influencing skills, so to speak. And people who are looking to build those skills, this is a great area for them to come into. I found it myself, Farsi.

Sure. And just going back to yourself, obviously going from IT all day and then into data science and now Model Res. Did you actually have a career plan for that? Or why is it that you've taken on these new challenges?

I never tried to be honest, I never planned to be in audit when I did my software engineering. But as you move through your career, you get exposure to different things you may not even have thought about. So I think it's best to keep an open mind and as you move through your career, as you move through life and then...

You see certain opportunities, you need to look at the pros and cons of it. You should think about what value you can add in those opportunities, but also what those opportunities provide to you as part of your career. So I think that's the mindset I've actually always kept. I've always had one thing that really intrigues me and engages me is the technical streak. Something very technical I love getting into that probably talks to.

the portfolios I've held over the past few years, like algorithmic trading audit, models audit, data science, as well as data risk and certain parts of IT audit as well. So I've always been fascinated by more technical stuff, but it's always been the point that I've kept an open mind to opportunities that have come my way and what I've noticed is if I genuinely follow

My passion to learn and develop, the rest of it follows, the title, the portfolio and the rest of it.

Sure, sure. So if you had any advice for your younger self with an internal audit, what would that be?

That's a good one.

My advice to my younger self would be to perhaps be more bold. More bold about taking up opportunities, more bold about taking up risks. I think after working in audit for a while, you do tend to, because that's the nature of the job, right? You're there to identify risk and then talk about how you mitigate it. So at times, it becomes too risky.

I would actually want people to be bolder, to experiment more, which I've started to do more and more of as my career has progressed, and do away with the fear of failure. Because the fear of failure is one thing that holds you back, and there isn't anyone in any senior position of significance who hasn't failed. So that's what my advice would be.

Yeah, I think when you're younger anyway, you're just so much more, you worry more about putting yourself out there and making how you feel making yourself look, look silly, right? I think once we get a bit older, we just don't care. To a degree. Rather, you know, you've just got to put yourself out there and do the best, hence the podcast. So

That's true. That is true.

Yeah.

I think, so I won't go into the entire risk scenario, but in the context of where we are at the moment, for instance, from a banking perspective and specifically in the context of models, I just talked about a supervisory statement, SF123, that's been published by the PRA last May, and then it goes into...

sort of control framework around models in general. I think apart from that, one thing that I talked about was the use of AI and machine learning models, right? Now, every organization, every firm I've talked to is actually ramping up their use of AI and specifically Gen.AI as well, right? With the use of LLMs as...

commonly, the most common term people use for it because it's one instance of an LLMS, charge GPT. Every organization is trying to see how they can benefit from it. Now, whilst that brings opportunities, there is a lot of risk involved with the use of it, as well, in a lot of ways, in terms of the data that's used, in terms of...

data privacy in terms of some of the risks being called out are hallucinations, data toxicity, and so on and so forth. So it would be very important to make sure we have the right guardrails as we go around implementing or ramping up on the use of AI and machine learning models across our organizations. And we do need people

within audit who understand these concepts, who understand the risks, who have the ability on the technical side to be able to see what risks lie underneath a model that may on the face of it seem really, really benign.

Sure, I mean, I was recording another podcast a couple of days ago and we were talking about AI and around sort of relationship managers and you know how they can tell which is a good complaint, how that was followed up all through AI. Do you think eventually then AI will be...

will be taking over and reducing internal audit teams or do you think there will still be such a big need for internal auditors?

I think it is an absolute sort of opportunity for internal audits, for internal auditors to leverage AI as opposed to seeing it as a threat. There are two things. Firstly, as I said, you need to be able to see the risks around the use of AI by the organization. But then we, at Lloyds, we use...

AI machine learning is part of our audience. And that doesn't mean that the auditors, we need to sort of reduce the number of auditors. That actually, the way we put it is, you let machine do the ordinary so our people can do the extraordinary. There are a lot of tasks, especially in terms of exploring data, in terms of driving insights out of data.

that you could use machine learning and AI to inform you on. So then you could actually get those insights and apply human judgment to it to really add value to your organization and to the control landscape. I think it won't be, as I said, it won't be a matter of, you know, or it is doing less. It would be a matter of

auditors working differently, not less, but differently, whereby they need to know the art of the possible, they need to know how they leverage AI and ML to their benefit. Because bear in mind, the data that's piling up in organizations, digital data is exploding. And with our old techniques of using a sample of...

30, 100, whatever that might be, we can never provide the level of assurance that our stakeholders need from us. So it almost becomes a necessity to be using analytics, to be using data science, to provide the right kind of assurance in this highly digitized data heavy environment.

Sure. And do you think then that data analytics just in terms, you know, I remember years ago that I was talking to somebody and before data analytics, it was just as data analytics was coming in and they were talking about credit risk and that each year, you know, in the audit plan, it was right, you have to do this credit risk audit and auditor went done the credit risk audit, fight have done, you know, realize what the fact

were report and then never done anything about it. Then the next year, done the same audit. And really the audit didn't really need to be done again. And with the use of data analytics, now obviously that doesn't happen because you can see, you know, right, that audit needs to be done at this particular time. And you know, where are the findings and what's happened.

So do you think data analytics is calling, you know, calling internal auditors out now to do a better job?

I think it is, I think clearly if you apply or if you target the analytics right, you could really, you know, the level of depth you can get into is huge. For instance, we had an instance whereby we went through the population size was three quarters of a million and we found one exception.

And that exception was significant. That didn't matter. So clearly, it offers you the power that you wouldn't otherwise have.

Sure.

But again, to your point, it's not necessarily calling out where auditors aren't doing a good job, but it does offer you that power, which incrementally you could use to do a better job, is how I'd call it.

Thank you.

Yeah, I'm not as grammatically correct as you, so you brought that back for me very well. All right, great stuff. And in terms of sort of going forward, who do you think sort of been most instrumental or a great mentor for your career and why?

Hahaha

That's a brilliant question actually.

There was a leader, well, to be honest, if I look across my career, there have been so many people who've influenced me.

in a way or end who's inspired me in a way that I am a function of what I've learned from them. I would not be here without their support, without their help, without their guidance and stare. There are too many of them to mention, but even currently the leaders within the function, I look up to, or chief internal auditor as well.

you think about how they, at times, you know, you're great in certain areas, but then there are always other areas those people would help you out with. I think there, I really, but one person I'd probably call out before I, that was before I joined Low Edge, was the person that helped me sort of move from being a total techie to someone.

who would create wider impact and come across as such as well. And he was an MD I worked for, and he actually is someone I looked up to in that regard, because he really helped me move up from being a techie to being someone who would actually deliver strong messages and take a holistic view, not only about work.

but also in terms of career, also in terms of building resilience, also in terms of building influencing skills. So that's the person that comes to mind.

Great stuff and I know lawyers have a fantastic graduate program and especially within the audit space. Can you just sort of tell me a little bit about that? Do you find that this has been a good route because there's generally that people just normally try to find auditors out of the big four, you know, yeah please tell me a little bit more about the graduate scheme that you guys do there.

and freshest fruit.

It was Karen Court from internal audit at Loox. And the other one was about perhaps the program, the apprenticeship program that we ran to develop apprentices. So I'll quickly talk about that. And that was preceded by the grad program. We used to have the grad program. Then we moved to the apprenticeship program. I have personally found the apprentices and previously grads to be so full of energy.

to be so driven, motivated, focused, they would come up to speed on things way quicker than some of the other colleagues, some of the experienced colleagues as well. So, and they're hungry for more, they're hungry for information, for knowledge. So that's been very, very useful. So that's the apprenticeship program that we run. The way it works is they do rotations across a number of teams.

And we make sure they do rotation within some of the technical teams, for instance, the IT audit team and also the data analytics team. In fact, the data analytics team, almost all of them do at least one rotation in. And they also do a rotation in the business. So it's not that they only see what's happening in audit. They go and see probably for six months, at times I think probably even more, what's happening in the business. And they get to take, they get to actually have

preferences in terms of what business they go to. I think that gives them a brilliant opportunity that once they graduate out of that apprenticeship program, you could actually see the value they're already adding and the people they can out to be. Now, interestingly, I talked about the internal audit apprenticeship program, right? Now, but within the data science team, we're part of the rotation program.

for the group's data science graduate program as well. So the data scientists that are the data science graduates that the group hires, they get to do a rotation in internal audit as well. And that says volumes about the standing and recognition that internal audit gets across the group, whereby the group's happy for its data science grads

to work in audit because they know they're going to learn being in audit. And again, they have multiple rotations across a number of businesses as well. And we're actually quite proud to be hosting that as well.

Yeah, you should be. Yeah, that's fantastic. And I'm assuming once people have done this apprenticeship, they come back toward it and work, is it in an analyst role?

Yeah, they do actually. Once they've completed their program, they actually, the role they've finally landed within audit as an analyst. And the same happened previously with the graduate program as well. And we're actually quite proud to host them. I think we see them as real talent and of course...

we wouldn't have it any other way because the amount of time we spend invest with them and the amount of time they invest within the function that goes, that works out perfectly well for both.

Fantastic. So we're coming to the end. So if I can just sort of do a quick fire questions, just to ask her what book are you currently reading at the moment?

There is a book, I think, I don't know whether I should mention it or not, some of it is controversial but it's good. It's 48 Laws of Power. I found it fascinating because it doesn't talk about present day but it goes all the way into BC and the Greeks and the Roman Empire and all. So that's the one I'm reading, 48 Laws of Power.

And what's the best country you've visited and why?

That is hard to pick.

I, it's Switzerland actually. And I was totally impressed by the landscape and the natural beauty. I went to Interlaken and the areas around it. And it was simply sublime, more from the landscape perspective. If I think about the cultural stuff, then that would probably be Italy because Rome and Venice were.

Brilliant.

All right, great. And is there a company, it doesn't have to be, but you know, a company that you admire the most?

Now, your rapid-fire questions really are pretty deep, aren't they? Maybe I should have spoken to you ahead of the podcast to figure out what they were. I think in terms of the company that I admire the most, I'll just talk about one that I've recently been reading a lot on. And that's Microsoft. And I'll tell you why I say Microsoft.

laughter

I'm sorry.

Yep.

not because of the products, not because of what they do, but the cultural transformation they've been able to bring about in the company over the last few years. And that especially in our management courses, we do end up reading a lot about that. So it's the cultural transformation of the company for which I believe Microsoft is a brilliant example as of late.

Well look, thank you very much for your time today. It's been very, yeah, it's just been fantastic and very knowledgeable and I look forward to speaking to you soon.

Thanks a lot for having me, Hazel. Great talking to you. Thank you.

Thank you.