Episode 6 Jason Rhoades - Attribution, Tags & Labels . . . OH MY!

It's the episode about tags, labels, account structures - how you attribute your cloud usage and spend to information that is important to you! Learn how to start, evolve and mature your attribution strategy with insights from Jason Rhoades, Development Manager at Intuit.

• Jason Rhoades | LinkedIn
• Cost Allocation (Metadata & Hierarchy)

Learn more and join the FinOps Foundation at www.FinOps.org.

[00:00:14] Joe: And I'm Joe Daly. Stacy,

[00:00:53] Joe: Lucky luggage gets a weekend away in Germany.

[00:01:04] Joe: So we are under the gun. You have 20% battery left on your laptop

[00:01:09] Joe: to get this recorded.

[00:01:12] Joe: well, I want to point out the reviews are in Stacy, about FinOpsPod,

[00:01:19] Joe: Thursday at the May FinOps Summit during the pre-show you were walking around KubeCon showing everyone the expo hall and during that time, J.R. Storment, Executive Director of FinOps Foundation, promoted FinOpsPod and said it's "oddly good".

[00:01:41] Joe: ugly. Good.

[00:01:45] Joe: Oddly. Good. Surprising. You

[00:01:48] Joe: You expect something like this to not be good, but this is odd. Odd.

[00:01:58] Joe: Good-ish.

[00:02:01] Joe: Actually did receive an email this weekend Ian Brisson, who new member of the foundation, who said that he's listened to every episode and he really appreciates it because there's not much podcast content in the space like this. So thanks for listening. Ian Brisson, he is a FinOptinaut. And for those of you who don't know what, a FinOptinaut is maybe finish an episode. I talk about them at the end, for completionist listeners who listen all the way to me thanking everybody at the end, they are FinOptinauts.

Anyway, this week we've got Jason Rhodes from Intuit on the podcast.

[00:02:53] Joe: You are correct.

[00:02:56] Joe: He's talking about data ingestion and normalization. That's data do you pull in

[00:03:02] Joe: you it? Which means make it similar so that you can compare it apples to apples together. One of those nuts and bolts that you absolutely have to do in order to do finops well,

[00:03:22] Joe: Noel and Deana interview Jason about tagging and labeling and they kind of start interchanging all the terms together. So we do have to give some forgiveness here, basically it's attribution. How do you attribute your cloud spend to whatever it is that you want to attach to it?

Whatever metadata that you want attached to. A lot of people call it, tagging. Some people call it, labeling. Some people call it account structures. So it's all about that. It's incredibly thoughtful.

Hey folks, this is Joe in post-production. I just wanted to point out that we lost an extra 5% on Stacy's laptop battery. While we waited for her to stop laughing at her own joke. All right back to the podcast

[00:05:20] Stacy: no, I found it in my head and it sounded so funny, but it was so bad. I couldn't even say it out loud. Okay. Go on. Keep

[00:05:50] Stacy: The.

It's going to be a valuable one for all listeners. Folks will be going back to listen to different parts about this later again, I'll add the transcript in the show notes so that it'll be easy for folks to review and grab information from for this one, because this is a very good practical information, sharing session that

[00:06:16] Joe: so let's get into it.

[00:06:37] Jason: Sure. Yeah, I was in Intuit it for a few years before we really got started. And I had just actually changed teams inside Intuit we, I was not pleased with how Intuit was handling its data center accounting. So every month there was big headaches around who owed how much of the data center costs and everybody was upset and surprised, pulling their hair out.

Do you think we could save some money there? And I never heard I had, I'd used AWS a bit where it was before Intuit, but you know, small, tiny scale stuff. So I downloaded our DBR. You know, this was the predecessor to the CUR for those who remember. And I had to add a hard drive to my laptop cause I didn't have enough space to hold it. But I mean, again, coming from small companies the amount of money that I saw, we were leaving on the table cause we had basically no coverage of our fleet and we were about 75% of our spend was EC2, at that time it was just totally uncovered. I couldn't believe how much money we were not saving by not having been at the wheel of this.

And so we, that really just, that was the fire that ignited our program pretty much overnight. And from there, it was a rush to, to build the sort of data apparatus to, to manage our prepay investments on an ongoing basis, sort of clear the pathways with finance and accounting and everybody about what we wanted to do to start investing in these.

And that was for us, that was the genesis of our FinOps program. We probably should have started a year earlier than we did. We're spending now more than 10 X, what we were then, but even one 10th of what we spend today is still quite a bit. So, it was in a way it was good though, because there was nobody who argued like, oh no, this is a waste of time, we shouldn't do this. It was like, no, this is big opportunity right now. Let's go for it.

[00:08:43] Jason: Yeah, by 2017, 2018 we were really getting heavier into our migration out of the data center to the cloud. And we invested heavily in automation of what we call our allocations process.

[00:09:47] Noel: How do you look at tagging? Do you look at it as in tag everything, tag it to death, tags, 15 tags, or are you on the other side of it

[00:09:57] Noel: tagging that has a piece of information that can be referenced from other places. How do you look at it?

So in regards to tag everything or tag nothing I'm interested in how can we get to that place where we reliably can answer these important questions in a way that's sustainable, reliable, but also minimizes the distraction to the company software developers, right? Who have other priorities, like getting tomorrow's blockbuster, feature out the door or patching the new zero day vulnerability or, you know, whatever other pressure they have that isn't some cost or finance-related tagging initiative.

So sometimes tagging is the answer. I think sometimes it's not, it just depends on what sort of data you need and what the right way is to reliably be able to get it.

[00:11:42] Jason: Possibly. I mean, I think it's definitely the case that you may not be the first sort of stakeholder to the tagging party. Perhaps your organization's security or compliance groups got there first or maybe somebody else. So, you shouldn't expect necessarily to have a Greenfield , flexibility in designing how it is that your organization is going to go about doing tags.

I think what we found works pretty well is that each of the different domains have their own what we think of as a namespace. Maybe you would tag something with, you know, Intuit colon domain and then the tag key, you know, within that domain. So there might be, you know, a billing domain for things related to FinOps, but there might also be, security domain, compliance domain, other operational domains, like whatever it is, but you want to be able to not worry about, Hey, like that person is calling their thing, business unit. What if I have one called business unit, as long as it's clear, like what the namespace that those things are in, you can keep the purpose and the function of those tags, separate and clear to the people that have to actually maintain them.

[00:12:51] Jason: Yeah. I mean, some of the there's some practical, I mean, in reality, we have thousands of tags in use and Intuit tends to be remarkably laissez-faire and how they allow developers and teams to choose the technology that works best for them, and implement it in a way that works best for their customers.

But certainly when it comes time to collapse all that into a cost and usage report, you need to be specific about the set of tags that you're going to ingest and then maybe process and do things with downstream. Anybody can create and use any tag they want but what we've done is, to enable certain rich supplementary functionality, you need to be using these specific tags in these specific ways. And if you do that now you'll be able to slice and dice your spend by that in all of our different tools are you be able to move your costs around. So it's more of a carrot versus a stick approach that we've taken. And that if you want to be able to do this cool new stuff, that your peers are doing. Then you've got to adhere to this tagging standard.

[00:15:09] Jason: Yeah. I'm trying to think of, you know, we have had tagging initiatives you might say, but only a couple. One thing just quick Intuit history is that as we embarked in our move to the cloud initially we took an approach. We're very concerned about the blast radius of if we were to have some kind of security breach.

And so what we wanted to do was make sure that. You know, a bad guy able to compromise a single AWS account of ours that we by design had constrained the amount of damage they could do. And as a result of that, we took an approach where we actually have many thousands of AWS accounts. So really the first few years for us tagging was almost a non-issue because any sort of fence around a set of things that you might want to make happen with tags, we were already doing with the account boundary, which is a wonderful cost boundary because even the things that you can't tagged are necessarily , nestled into different accounts.

And it's only as sort of multitenancy inside accounts start to come to the fore that the need to tag , also rose because any sort of attribute that you might seek to understand. Say a resource, you could just, you could apply that at the account level.

And we had a nice, good business workflow around account creation and lifecycle management. you know, who owns it, who pays for it, and who's responsible for operating at all the different questions you might seek to answer. We just maintain that information at the account level. And so again it's only maybe over the last few years as account multitenancy has risen that we've needed to tag, but the interesting other thing, and you guys all do as well as the other thing that's happened around the same time is things like Kubernetes.

We also moved our call center into AWS, which nothing, none of that is taggable, there's no taggable resource there. So we simultaneously are dealing with sub resource level multitenancy or just large bodies of spend that, that have no sort of naturally taggable or attributable resources.

So in all these cases. Just had to choose a solution that, that fits the best for that one.

[00:17:52] Jason: One thing I'll say is that I don't think that anybody in the finops seat necessarily think that their first sort of design of how they're going to do go about tags is going to be the thing remains perfect and perpetuity. Certainly like I'd say a pain point we've had a few times at Intuit was that there were groups and different groups are very different levels of maybe operational maturity around managing their cloud business. Some groups migrated earlier somewhat just to have more expertise there. So I'm trying to do our best to get everybody up to the same high level but some have realized the need for certain things earlier than others. And so we would work with that team you know, to do something with tagging and labeling to enable that new functionality they wanted. And so they would adopt it and they'd have a lot of success with it, but then maybe a year or two down the road, the rest of Intuit and sort of the main architects over everything would say, Hey, you know, we need to do this. And it's something that's sort of redundant to what that sort of advanced team had already pioneered and figured out. so now the teams that were those early adopters have to go back and adopt what has now become an enterprise standard. So they sort of pay a price of having to do it twice. But I do think there's still value in the learnings and the experience of those groups, having, blaze that trail that everybody else can go from.

But if you're the one that's trying to steer the ship of tags and things, then it's just natural that the standard is going to continue to evolve as the company learns and grows and changes. .

[00:19:23] Noel: I like the way you said that the standard will evolve. I know what else, no more than anybody else. We went to the cloud first. Hey yeah, we're going to have 20 tags and everybody's going to put these 20 tags on everything. And very quickly we realized that a lot of the tags were telling us nothing or people were putting stuff into them that made them irrelevant and what we discovered is the vendors put tags on things automatically as well.

And we actually pulled our tagging requirements right back to one mandatory tag. And then we have another tool that we use. We can reference everything based off that. As you say you evolve the, how many tags are you up to now? Tags labels, whichever that you're using.

There's others, I'd say there's two other classes of tag though, that we have one is sort of just a free for all. So I think of them as channel one through five. So, Hey, you can use these five tags and we'll make sure that in every sort of BI tool that we have where you're slicing and dicing your fully allocated costs that you'll be able to do so by whatever you put in here.

Right? So there don't think of them that that it has to be like the business unit or the deployment ID or anything like that. Just use them however you want. Each group is free to use them. So those are sort of like the free for all tags meant to contain human readable values and things like that.

The third class of tag though, are things that. Functionality that people might want. So these are things like, oh, I want to be cross charging these resources to some other group within Intuit. We have dozens of centrally developed services that are consumed by the TurboTax group and the QuickBooks group. There's a lot of costs moving around between groups all the time. And to enable that, there's a supplementary agreement workflow that has to happen where a contracts put in place. But once that's in place, if you're the one that's developing and running that centralized service, if you want your customer to be paying for it, you have to follow this tagging process. Otherwise you're going to be paying those costs.

So there's a number of similar workflows like that, that are tagged powered, so to speak where you have to follow it, but it's not mandatory, but if you don't want to be paying it, it is.

[00:22:06] Jason: Yeah,

[00:22:09] Jason: Exactly. And you're incentivized to be on top of it and make sure it's right. And that your own development workflows or ascertaining who the customer is each time you're standing up new resources and things so that the costs go to the right place.

[00:22:33] Jason: you Always have to have a fallback. So yeah you have to fall back to maybe who's who is the catchall for the account, something like that gets recorded at the account creation time. So that if anything, that it doesn't meet any other workflow to move the costs anywhere else. It's going to end up with the party who's the catchall for the account.

Has that, is that something that you've tried in your organization? Has it worked? Where has it not worked?

Once you're there, then you can help teams focus on that which remains untagged but is taggable within their operation. And so you could look at it by, what percentage of resources or even what percentage of spend. Because if you'd like to get to a place where you can look at the entirety of your cloud by that tag dimension and right now you've got, 40% of my cloud spend is just this big question mark black box that I don't know. What are the best sort of juiciest opportunities that remained in that untagged arena that I should be going after first.

[00:24:20] Jason: uh,

[00:24:28] Jason: Well, all taggable costs in my experience, maybe 80% of your costs are taggable and it depends again on the technologies that you use. But get to that place. 99% of taggable cost would be a really good place to

be.

We put in automation to catch instances and catch anything out there that isn't tagged to put tickets in the people's queues. Go, Hey, go tag us. Or we're going to stop us in non production environments. Have you gone down that road or do you go down that road or do you think that's a good or a bad idea?

[00:25:48] Deana: I'm glad that you describe it that way because in an organization that seems to have achieved a high level of maturity, it's still not as rigid as someone might expect for a level of maturity. Not myself personally. From my experience, scaling back on the number of required tags or scaling back on the level of compliance and looking at key indicators has been a better use of time especially for small delivery teams or small finops teams. What factor do you think being mature on automation and having DevOps maturity in terms of culture and accountability helped your tagging maturity?

We did have a lot of experience with those things prior to AWS. And so a lot of the things I've talked about around, you know, using a really large numbers of accounts, but we had a really rich workflow around lifecycle management. So everything that you like, every question you might ever answer with a tag, or we thought we might ever need to answer, maybe via tags, we attach all of that data to the account part of that workflow. And the same is true today. Actually with Kubernetes, we have our own custom Kubernetes platform. And there too, we have a really rich workflow around onboarding new namespaces and pods. And so we capture all of that critical metadata that you might care about to make choices and figure out how to run things down the line as part of that process. So there's no need to try to go and just fix all of these things to the actual namespace itself, because we have a data system that already has that information captured at namespace create time. We've leaned heavily on those sorts of processes and that's really minimized the set of problems that we've sought to solve through fixing labels to running technical things.

Because I've had conversations with developers where they're like, well, I'll go back and unchange tag on that instance that was there a week ago with the thinking that it's going to be updated in the billing file or whatever you're using. And tried to explain that. Have you ever found a good way to explain that, the tag is as good as right now? It's only good now.

But hours before that, I mean, certain things, maybe if needed, we can automate some kind of backdating and things like that. One of the challenges we have is org changes are a constant. If we're looking back 12 or more months in arrears at our organization to look at how the spend of larger groups of the company have maybe changed over time, it can be tricky as things break apart or get merged together in different places.

So I think there's places where you have to maybe apply some care to historical looks at spend so that it makes sense to people. There's really no one right way. Do you apply the current structure historically or do you keep those historical months under the structure that they were at the time?

Different people have different perspectives on what the right answer is there? There's a little bit of white glove and customer service that I think is important.

[00:30:06] Jason: Oh, yeah, that's good. I mean, I think maybe into, it was a little bit unique and so my experiences heavily forged by that again, in that are very security centric posture in the early days that the dictated this very high number of small accounts that was beautiful from an attribution point of view, because we had workflows around account lifecycle from day one. And just everything you would want to know about, you know, what app, what application are these costs a part of and who's paying for it and who's running it and who's securing it. And who should we say is responsible for the waste? All of those questions we can answer at the account level and all of the pain and complexity of sub-account level attribution really only came after we'd been at it for a few years. And so we were more ready. We had already started dipping our toes in the water of tags to enable these sort of extra credit, type billing related functionalities. So for people that are out there today that are maybe just getting started, I just would say that don't assume that you're going to be the one that's entirely at the wheel of how your organization does tags. And you've got to be a cooperative and constructive member of the committee that may include your peers in the security and compliance and operational and other organizations.

And you've got to try to come together with an approach that's gonna work for everybody, but still solves of the problems that you're being tasked with and the finops world of who's paying for this and who should I say is responsible for the waste? Cause sometimes it might, there might be three different people that have a stake one resource, right. That have different sort of responsibility profiles over that resource. And you need some way of maintaining that information, but also that's in line with your organization's broader priorities around innovation and other things, so don't expect it to be perfect.

[00:32:04] Jason: Yeah, it's definitely the case. Ultimately what you want to affect is changed behavior on the parts of developers and development leadership, and. You have to be corralling everything in a way that produces that behavior change. And if people feel that they are free to behave, however they wish with no repercussions, no consequence, then it's unlikely that they're going to really want to change their behavior.

But when you can get to a place where you have that solid attribution, and you can say, Hey here's a hotspot of waste in our cloud. And by the way, it's got this person's name on it. Now they're incentivized to potentially go address that issue. Same true, even just from a forecasting, budgeting and against your actual sort of thing. I know some companies have one group that sort of pays for the whole cloud at the center and forecasts at the center. We have a distributed approach. So there's hundreds of development managers across Intuit who all have a hosting budget who all have to forecast their spend and who are all individually held accountable for adherence to their plan.

So having that cost attribution that incorporates all of your business rules, all the chargebacks and things so that people can't fight and say, oh, well, no, your costs are wrong. That's not what I really owe. When you have an airtight thing there, you can really hold people accountable to both their waste and their spend.

So that's what we've seen drive the best sort of behavioral changes is when we've taken something that was sort of nebulous and we weren't really sure who owned it, but it's not getting any better. We want it to get better. It's not getting better. As soon as we can break it apart and attach the pieces to people. Now that we see traction. Now we see movement. Now we see the behavior change.

[00:34:03] Jason: Well, that's a good question. I know that there's a lot of places where we aren't at utopia. One of the things we do regularly, as we assess, you know, if we had infinite resources and five years of time to work on stuff, where would we like to be in our FinOps program? What are all the outstanding things that we don't have, that we'd like to have or things we'd like to be able to do that we can't. And I think that's a healthy thing to do regularly as part of your program. And maybe in the early days, the answer is everything. But after you've been at it awhile, like, okay, we've got a few things pretty well handled, but we know there's some outstanding pieces.

And so better attribution or lower cost attribution. If you think about it that way, in terms of what are the sort of mental cycles that we're taking away from people who are out there building revenue generating, or important things for the business to do, essentially what feels like an administrative task?

How can we get to a place where we're in a more in a utopian state there. And I just look at all the different opportunities as just areas of investment for the program, as a FinOps leader, whether it's just yourself or a big team, you have resources to invest at going after capabilities.

You just have to be able to prioritize that and be able to quantify, like, what's the value add of saving, 10,000 developers, five minutes a month versus whatever other sort of thing you might have before you, and just use your judgment to, to make the call there.

I think just constant reevaluation of everything you're doing to make sure you're working on the most important things is healthy. And is part of that.

[00:35:49] Deana: Yeah.

[00:35:59] Jason: Probably, it depends on who I'm talking to. I do have some tech backed family members, but in general, I'd say that the story I tell to say people that I'm interviewing for the team who don't know what FinOps is: in the old days, companies like Intuit used to run a data center and we had to pay at our scale hundreds of people to, to operate that data center. We had to pay for power and cooling and network connections. And we had to run around and replacing hard drives all day long. And today we don't do any of that. We just pay our cloud provider a big check every month, but things can get wildly out of hand with somebody just writing a bad loop.

And so my job is to make sure that Intuit is getting the most out of its relationship with the cloud provider that it's achieving the business outcomes that seeks and that yeah, it's just maximizing our use of the cloud.

[00:36:55] Joe: All right, FinOptinauts, you've made it through another complete episode of FinOpsPod. I could tell you who didn't make it through the episode. That Stacy Case because her laptop battery is totally dead.

That was a great episode. A big takeaway. I got out of this, is that a mature attribution strategy is one that is minimal enough and flexible enough, to be able to work in any scenario that you come across. We all know new problems come at us every day. It's pretty impressive. How Jason and Intuit team have adapted and matured their attribution strategy and how they've gone with it. We're going to post this and a lot of the information in it in the FinOps framework section of the finops.org website. Go to finops.org, click on the framework. I'm going to put a link to the cost attribution capability in the show notes. Folks, we're an open source community and we're looking for your best practices and stories. Go to that website, contribute your knowledge and your best practices, your experiences. So the community can continue to improve and evolve. As new problems, new technologies, new scenarios come our way.

Big, thank you to our guest, Jason Rhoades, and our interviewers, Deana Solis and Noel Crowley. Fantastic job you three. Thank you so much. Thank you to Stacy Case and her laptop.

And that's it for today. We'll see you next time on finops pod.