Goose is the highly scalable load testing framework preferred by Tag1. In this series of Team Talks on Goose, we’ll take a look at how Goose scales on a single server, while distributed, and CEO Jeremy Andrews - the creator of Goose, VP of Software Engineering Fabian Franz, and Managing Director Michael Meyers walk through a demonstration of Goose load testing. 

In this team talk, our load test runs in AWS and we're testing against Drupal 9 with the default Unami install against a Tag1 Kubernetes cluster fronted by the Fastly CDN. These tests start with a limited set of user functions, and we’ll ramp up from 1,000 users up to 8,000 simultaneous users and show you how performance changes.

For more Goose content, see Goose Podcasts, Blogs, Presentations, & more!

Hello, and welcome to another Tag1 Team Talk, the podcast

and blog ofTag1 Consulting.

Today, we're going to be talking about how to load test with Goose.

It's going to be running in AWS and we're going to test against Drupal 9 With the

default Unami install against a Tag1 Kubernetes cluster fronted by the Fastly

CDN.

I'm Michael Meyers and managing director of Tag1 Consulting.

And I'm joined today by Jeremy Andrews, the Founder and CEO of Tag1, along with

Fabian Franz, our VP of Technology.

Let's jump right in.

I'm excited to see how Goose works.

Jeremy, welcome.

Fabian, welcome.

Thank you guys so much for walking us through this.

Thank you.

Yeah, I'm going to start off by sharing my screen.

So if you're watching this on a video, you'll be able to

see what we're doing as we go.

, to start off quickly looking at Goose itself Goose is written in Rust.

And if you go to Goose.rs, it'll take you to , the GitHub page.

and inside there you'll find a pretty normal looking Rust code base.

For today.

The part we care about is within the examples folder and we've written

several examples Primarily to demonstrate how Goose can be used.

the first one was just called simple and all that does, is it loads an

endpoint and shows you how to write the most simple of load tests.

It uses a few features like wait times.

So it'll load a page weight randomly between five and 15 seconds.

and then load it index page and about page and whatnot.

It does simple stuff.

It logs in.

then there's another version of that, where it was written to use

closures, a fancy feature of Rust, and Fabian wrote that version.

I find closures more difficult to understand, but they're

very, very flexible for programmatically doing things.

So it's, it's a super cool example to have in there.

So essentially what you're doing with the closure is don't even know what a

closure is, but what you can see is.

There's a pass array at the top or vector, how it's called in Rust.

And you can essentially just extend this vector, whatever pages you want.

You can draw that from a CSV, from wherever, and you can have a

programmatic load test, essentially.

I mean, even if you don't know too, too much about Rust, just, just looking

at these, you know, you can go to this examples, directory, cut and

paste, make some modifications and, you know, simple modifications and use

this to test your own Drupal website in a pretty sophisticated fashion.

This one is not as as helpful yet.

conceptually it helps you with with closures, but then there's

one called Drupal load test.

And that was literally written to load test every release of the memcache module.

Originally, we were doing it in JMeter and then we replaced it with Locust and

now we use Goose and every time it's a huge step forward, but this one has a

much more Drupal examples and much more.

You can cut and paste.

That said I believe it was Fabian who suggested we should

load test against Umami.

So if, if your goal is to write a load test for a Drupal 8 or Drupal

9 website, this is absolutely the, example to look at it's broken into

useful pieces, common.rs are functions that you can use, you just call

directly from, from your own load test.

And it, does a lot of things that aren't generic to Drupal websites.

Cause you can see here, it's defining a list of all nodes and whatnot in Umami.

but in any case, the, the load test we used when we've written a load

test for customers we've borrowed heavily on this code base because

it is a great starting place.

I actually love that you, you, you finished this Umami load

test because one of the things.

The Drupal community has been looking for since at least.

2015 is essentially a profile which we can, can do for performance,

regression, testing and core.

And this now would essentially allow that to just spin it up somewhere, load test

it, put some New Relic on it and then.

Essentially we could look at what's slow, what's fast, did we regress, et cetera.

So it's not only useful for memcache, but with Umami being finally complex

enough to resemble a real site I think we could get some meaningful data out of

it for future performance work in core.

So that's really cool.

It's funny you should mention that.

I spun this up to test earlier this week.

And I found that there are certain paths that are struggling under

the amount of load that we're putting against our little install.

And so there is some functionality that was being tested that I disabled on the

load test to not, to not load those paths.

for example I disabled searching , uh, the load test does searching in English and in

Spanish, and both are just disabled in our current test filling out the contact form.

And I also disable the user that logs in and basically just logs in edits a

article and saves it flushing the cache.

Those are the things it'd be great at some point.

And maybe on this series, we'll, we'll do that.

We could dig into why we had to disable them, how to optimize and fix it so

that we don't have to, and everything

can be tested.

Essentially what we are testing today and what we're seeing today, as we

are doing the big event scenario.

But essentially you get featured on Slashdot, on Reddit, on wherever, like,

and people are just crashing your site and all coming in, like, like angry Geese.

and and essentially they are all wanting to get the recipes from Umami

because they are so hot right now.

I mean, they're, they're perfect.

And because of that, and we cannot afford at this stage

essentially to allow people to do.

Things that are like really complex.

So what often happens in the real world on a scenario when a big day like Black

Friday comes in, is that you actually disable or rate limit functionality that

is very costly on the site like searching.

And you'll also tell your editors, Hey, please prepare all content before

this date and do as less content changes while it's going on, because

it always needs to clear some caches.

And obviously that can lead to.

A worse performance but that's pretty cool.

So, this is the big events scenario.

We have set up here,

so let's give it a run.

what we're going to see is, well, first of all, Goose is gonna

run from An instance in AWS.

we just spun it up here about 15 minutes ago.

we'll spin up some other instances as we test to demonstrate how different

hardware can perform different tests.

but a high-level summary Goose makes really, really good use

of available CPU power and.

Ultimately it consistently tends to bottleneck on bandwidth

because it can generate as much load as your uplink supports.

so that's, you know, that's a good thing that that's what we're trying to do

here and you can of course control it.

You can you don't always have to just flood things with

as much traffic as possible.

but for today, that's what we're most interested in is just seeing what we

can do, what traffic we can throw at it.

High level specs on this box.

Like how many cores is this thing?

this is an eight core server.

it's Amazon's CFA two X large, which should answer all your questions.

the two X large means it's got eight CPU's and the CF line

is pretty powerful with CPU.

I actually chose it.

This particular one, because eight cores is what I generally use

when I'm, when I'm load testing.

And I wanted something with a reasonably fast uplink and this one has, I think

it says up to, I think it was 10 gigabits or maybe it's five gigabits.

but it's enough to put some significant load and to show that, you know,

Goose is actually bottle-necking on uplink, not on, available.

CPU resources.

So the public IP, this is up and running.

so I already SSH into this And the, in the Goose directory on this server is

just the Git checkout of the current head of Goose the, the, the main branch.

And I'm going to see if I can quickly find in my history, the, commands that I run.

So we'll start off with just a simple thousand users.

this is an SSH into this AWS instance and we're going to just launch a

very small Goose load test a Goose attack against Umami website that

we're running to quickly talk through the, the flags, cargo run is because

instead of compiling a binary and then running the binary I'm using cargo to

manage that, which allows me to tweak the code and whatnot and recompile.

The dash dash release is important to optimize away debug symbols.

any load tests you're running, you're going to want to use dash dash release.

dash dash example is because we're just running an existing example

that we've provided the Umami one.

and then the dash dash space dash dash host is where we actually

start configuring Goose itself.

We're we're no longer in cargo, but we're in Goose.

we're we're load testing against umami dot tag one.io.

the dash V will give us a little bit of verbosity.

you don't want to go past.

One dash V, because then it gets too verbose.

It starts showing verbosity from all the libraries and whatnot, and

that, that will impact your results because it's a lot of overhead.

but one, one minus V is great.

Uh, dash dash log file basically takes everything in dash V and dumps it to a

log file , um, which we're defining as.

Goose.log.

And that's because of the dash G that follows, then we have a dash dash

debug file to a debug log because if something goes wrong Goose will capture

everything all the headers, everything that was requested and everything

that was returned and gives us a ton of insight into what's going wrong.

whether there's something's bottle-necking or there's something

wrong on the server or, or who knows.

but we've found that unbelievably helpful in actually troubleshooting

with , uh, with customers.

Dash T one M means that this is only gonna run for one minute and

then it's going to stop itself.

minus U 100 is we're gonna simulate 100 users and the minus r 3.4 means

that we're starting 3.4 users.

per second.

So we'll go ahead and give that a try and See what happens?

There's a couple of warnings.

I've commented some code.

So it starting is showing users launching in English and Spanish.

So now if we go to Fastly momentarily, we should start seeing some

traffic and we would expect it.

The server is actually located on the West coast of the United States.

So we'd expect to see traffic there.

It obviously saw something because it flushed it.

There we go.

So now it's seeing right here traffic coming from Seattle and We're currently

doing 140, 160 requests per second.

It's still ramping up.

down below you can see that it's ramping up to 95 megabits a second.

So there's traffic happening.

if we look on, on the server itself, we can see that Goose knows it's running.

it has successfully ramped everything up.

And so it's going to run for 60 seconds.

Just generating a little load 60 seconds.

is only enough for this demo.

but you can see the things generally stabilize and , uh, and you know,

it's pulling stuff out of the cache.

What's really important here.

This hit rate going up, this is the cache warming up and that's going to

be essential to, to, to scale up this test and put serious load against it.

You can see here that you know, we're capping out at about 350, 380 requests

per second with the a hundred users that we're simulating , um, there's

more requests than users, obviously, because there's a, it's not just re

requesting the index file, but it's also requesting elements from the page.

It's the CSS and the images and whatnot.

So all of that's loading the load test finished There's a delay in Fastly.

Uh, you saw it was a delay to start up.

It's also a delay to go down.

if, if we look at the results, you can see that everything ran.

It prints the final statistics there's, lots of statistics, there's flags.

You can use to change what it shows you.

it breaks it down per task, showing you that, you know, we've

been loading the front page.

A basic page, different articles, different recipes, different nodes by

node ID shows you how many times each was done, shows you the zero failures.

it was a total of 39.8 tasks per second.

and then you can also see how long each took so you can kind of get a

sense of which tasks are taking longer.

the recipe page is taking about a second to load, which is, which is fairly slow

for a webpage, especially coming out of a CDN any ideas on why it's that

slow, Fabian, would that be because of the amount of content on the page?

No, it's probably it might've been the first request, like on a cold cache?.

We've got a minimum and a max.

The minimum is still 972 milliseconds.

you've probably already exhausted the bandwidth of the Amazon instance.

Definitely not.

we'll we're gonna ramp this up considerably.

we can get you about 2000 users before we cap it out,

then it doesn't make sense.

So my thought process is that it's Hmm.

Okay.

Well, let's see per requests, so sorry.

So that's per task.

Okay.

Actually here it is what I was thinking because , per requests for

requesting slash it was let's see here.

Yeah, it's super, super fast.

absolute maximum of 148 milliseconds where it gets.

The, the reason it's slower is because making so many requests in those tasks.

And even though it's fast that's going to probably include sleeps and that's

going to include while it's giving up to let another user use the box.

so that's a combination of that user doing the entire page load

task and it would be slower.

The actual requests themselves are very, very fast, like we would expect anyway.

it's not very exciting.

But it gives you a quick demonstration of what this does.

So the general idea here then is, you know, now that now that we've seen what

it can do, let's Let's increase it.

So all we're going to do is we're going to go from a hundred users to a thousand

users, a 10 X speed up, and we're going to go instead of starting 3.4 a second, we're

going to go ahead and start 34 second.

important to note that you know, the, the resources it takes to start that up it's

still has it has no trouble with that.

That reminds me one thing that I should be doing here.

That's useful.

I'm, I'm exiting out just so that I can cut and paste this string and

connect in with a second window.

And the reason is on that second window, what's helpful is if we run a SAR command.

So like every 15 seconds, when you create a snapshot of what's going on

that'll give us a little more insight into what's going on in the box.

for that we're only going to be using.

The CPU stats.

but that was what I found was the most telling.

So we'll go in and do this again with a thousand starting at 34 seconds.

It would help if I was in the Goose directory.

All right.

The first thing it does is it builds a bunch of stuff in memory.

now that it's built, it's launching them.

if we switched to this view, we can see, we'll see that it starts using

more and more CPU as, as this spins up.

it only does a snapshot every 15 seconds and soon we should start

seeing more considerable traffic hitting the end point here.

All right.

There we go.

Traffic starting to hit.

It's going to ramp up considerably faster.

And what's one thing I noticed when I was testing this earlier, when

you throw 10 times as much, 10 times as many users at a server, Goose

throws 10 times as much traffic.

so it's, it's pretty simple math.

You're going to see 10 times or slightly more requests per second and you know,

10 times as much bandwidth happening.

Uh, what's really important for me here is that you are, that we can actually

see like a line going up for this because this line is so important for

me when I'm doing performance testing, it's what I call the scalability test.

Because, and we can do that in another session, if you combine this

test with New Relic, essentially, and we then do a scalability test.

There's a very slow ramp up.

You can actually see where a site starts to get keeping trouble, until

when it performs, because that is when your performance goes from linear.

Or even parallel , um, not impacted to exponential, and

that is when you get problems.

So that's a great way to do it.

That's why we have this ramp up variable because it's so, so

useful in testing performance.

So the load test just finished.

it's sustained 1.3, 1.4 gigabits per second.

it didn't, you know, those eight CPU's you know, they're 60% idle, 55, 60% idle.

So the server has way, way more, capacity to continue generating more load.

but the, the sheer amount of traffic is, is going to be our, our breakdown point.

Uh, when we try to keep ramping up the load test.

let's go ahead and quickly go back and look the per request metrics.

Um, you can see that at this point We were making 3,300 requests per second.

none of them failed The majority of them are static assets which, you know,

we're loading off all the pages that we're loading And then the actual page

load times continued to be quite fast.

an average of a little over eight milliseconds.

They stay this fast because they're coming out of Fastly.

They're not coming out of out of the end point.

But yeah the load test is continuing to look quite good.

So the next thing we'll do is we'll double the ramp up again to 64, and we'll

go from a thousand users to 2000 users.

and see what happens when we double the load.

As before it takes a moment while it's creating user states and it takes a little

longer because it's creating twice as many, it's an area I want to optimize.

and now it's ramping them up.

And again, the server has no problem.

even though we're doing twice as many per second again, the server can handle this.

and soon we'll start seeing traffic hitting on Fastly

and boom, there it goes.

there's a delay obviously for the reporting side of things, but once again,

you're going to see that nice ramp up and it's going to ramp up even further.

you can also see it's using more CPU this time as expected.

we're down to 13% idle and it's still ramping up.

But that's, that's great.

and then down here, sure enough, we're up to a 2.6 gigabits and still climbing.

so as noted, it's capable of doubling as we double how much we throw at it.

so Goose is more or less linearly scaling what's available to it.

which is great to see.

it's using about 90% of the CPU resources sustaining great load.

and of course, you know, what's, I guess what's also worth seeing is

that this website is continuing.

It should continue load because Fastly is actually taking most of the traffic it's

not actually hitting the back server.

So even though I'm logged in I can still use the website even under this much load.

With that all said and done.

See here, it did finish.

we were seeing up to 7,000 requests per second, which added up to, or

yes, and, and sure enough well, 6,600 requests per second is what what

Goose believed that it was doing.

Again, there were zero failures, which is good to see.

Fastly is doing a great job.

when it says zero failures, it's important to note that Goose is.

It, it's not guessing on that for every single request it makes for,

you know, 6,600 requests a second, it checks the status code to make

sure that it's what it expects.

So at 200, but then it's also analyzing the page that's returned.

And it's looking for key words, it's looking for, you know, the, whether

it's the title of the page or whether it's a specific search term and making

sure that really does exist on the page.

and it does it for 100% of the pages returned, not just a subset.

Rust is, incredibly efficient and a fantastic platform for

doing a load test like this.

do you have any questions at this point, Michael, or Fabian?

You said that it's doing

a hundred percent of the pages and checking.

Are you implying that other tools and systems

don't or?

Yeah.

What happens at scale is to be able to do to, to be able to scale up a

load to this size or even larger, like we're going to do you start having to

use performance tricks, where you look at a percentage of them and you have

a pretty reasonable assumption that things are working you know, check 25%.

If they all passed, you probably are okay.

but we don't have to do that.

And we're able to scale up.

I haven't found an uplink big enough that we can't handle that

with the available CPU power.

Essentially Michael, a people could also create this much load just by using AB

tools or HTP perf, et cetera, which, which is why people could say this is

nothing special, but what Jeremy is.

Telling us that this load test, not only is it programmed in Goose , but

it does a lot more than just sending traffic and also analyzes the

traffic and analyze all the pages.

And it's insane how efficient it is using the CPU.

Is like usually if you want to create this much traffic you would need several boxes.

Several AWS boxes to, to even do it, or do some parallelzation,

et cetera, with this other tools.

So it's really fantastic.

One Goose, one instance.

You're good to go.

No complexity.

That's great.

So you know, the logical thing to do is to double everything

again and see what happens.

At least that's how it seems logical to me.

So we'll go from a 2000 to 4,000 users and we'll create 128 a second.

I'm going to restart this SAR because I'm not sure it's going

to run long enough otherwise.

Um, yeah.

And the thing to see here is, again, it can.

Launch this many users this fast without problems.

Okay.

Yes.

So it can start them which is great news.

what we're going to find is that we're going to hit a different

bottleneck this time though.

and it's not the server CPU.

It's going to be our uplink.

We stalled for a little while, creating user states, getting

everything ready to go, and then boom, all of the user threads start.

there's not much to watch here.

It's just going to scroll through all the users it creates.

So I'll keep it open on our CPU chart to see how much CPU it's using.

if there's errors it'll become pretty apparent in various places.

So for now, we'll just keep an eye on, on on this website.

I love it.

I love it to, to just watch this in real time.

See the traffic coming in.

It's always my favorite part of when Tag1 is doing load tests for various clients.

And when we are doing this much load and then watching New Relic, like,

like going into insane numbers.

Watching Fastly, going to insane numbers.

It's like,

It's fun.

it's disappointing here.

We're not, we'll have the four 4k a request per second on, on Fastly.

So we seem to have capped out here at 6,600 requests a second.

And sure enough, interestingly, we, we have the same, even though

we're have twice as many users we're seeing the same amount of traffic.

One thing.

That's interesting.

If you compare this bandwidth chart to the previous bandwidth chart, it's flatter.

and so what's happening is the CPU has sufficient power.

It's able to make all of these requests but the way that async

works, they asynchronous requests.

Whenever something stalled, it just goes for another request.

And so it's able to flatten out how fast these requests are working.

so that load test finished nicely has a relatively flat line

because the CPU had enough power to, to keep generating requests.

even though the bandwidth was limited.

what that shows is that if, if we could start up a server with more power,

we could do more interesting things with it, like generate even more load.

So let's do exactly that.

I'm going to exit out of here.

I went ahead and started another instance.

this one is an R five in eight X large and the eight X large means it has.

32 cores.

and I believe it's supposed to have like a 25 gigabit uplink.

and that's great because we are needing more bandwidth.

So we'll connect into the server.

And as before we'll connect into it twice so that we can see how much,

how much we're stressing it out.

and there we go.

So inside the Goose directory we can spin up another load test.

now that we're not capped by bandwidth, we hope let's rerun our 4,000 user tests.

make sure there's no typos in this looks good.

Goose debug file one minute, 4,000 users.

All right.

So , um,

Yup.

It's gonna start allocating memory.

And that takes a bit of CPU power.

We will be able to see on Fastly once this traffic starts hitting.

And, and the goal here is that because we have more uplink that

we're able to generate more traffic.

the expectation absolutely is that that should work.

Let's see start hitting Fastly.

the server doesn't have to work very hard because again, it has 32 cores.

So even though it's generating quite a bit of users there's plenty of power there.

So that's not much of an issue now.

We've got the requests starting to come in.

And already you can see traffic on the Seattle PoP.

Uh, whoops.

There was a spike, which means there's errors and sure enough.

So what we have going on here is a perfect storm of too many

requests and the server's down.

So there's no point in running any further.

what's happened is this was able to ramp up so fast and it was able to

hit enough pages that were not yet in the cache that it caused some errors.

Um, It's worth taking a quick look and see here that we've got a, we've

got data in our, in our debug log.

It has 19 megs of , uh, of data there.

but if you look, you can get some sense of what's happening and sure

enough here, what you can see is that we're not getting anything back.

there's just the, the servers not responding.

Oh, And actually I'm wrong.

Okay.

It wasn't what I thought it was.

The specific problem here is DNS stopper.

We killed DNS locally.

the solution fortunately is quite easy.

if we ping umami.tag1.io and we grab this IP address, we can

add it to our host file.

And I did have it.

Oh, it is there.

Okay.

Next theory.

Too many open files.

That's our problem.

All right.

So the other issue is that you need to increase your, u limit, and I go crazy

large that we don't run into this.

I forgot we've spun up a new server.

So I had, I didn't, I didn't make that permanent, so I have to set the limit.

Now, if we run again, everything should work a lot better.

And luckily that's, that should be all we need to do to to be

able to run this test properly.

So the two things that I've I've ran into when I was testing, this was DNS

dying which is for me, the quickest solution was added as the host and not

having enough file handles, which is why we increase it with U limit with those,

it makes perfect sense.

if you if you want to load test it's can make sense to just.

Put in the IP address.

however one needs to be careful when doing that with a distributed load test

because then else you would be hitting the Fastly PoP of Seattle from China.

That's not what we want to do, obviously.

So yeah, but then we also don't want to load test DNS, so it

makes sense to cache it locally.

I was pinging from each server and then using the response to

the ping to set up the host.

So whatever resolves for that user, I just cache it in as the host, essentially.

and that works pretty well.

So now we have what we like to see.

We can see a proper ramp up happening and it's continuing

to ramp up fairly aggressively.

that's a lot of requests happening per second, that Goose is making

and every single one is validated.

and it's the real deal.

plus we can see in the global PoP traffic that, you know,

we we've hit a new, new level.

We went from where it was the topping out at 4k.

Now we're talking about 8 K so.

As before we were able to double it.

and sure enough bandwidth is up at 4.8 to five gigabytes of

traffic pretty consistently.

Can we go even higher?

Yes.

In fact we can , um, the thing to do, I, I just like doubling

and seeing how far we can get.

So let's double it again.

I'll go ahead and cancel this.

There's no need to let it run to completion at this point.

Um, I will let us do a clean shutdown.

actually, you know what, let's just force it to quit.

So it's quicker.

128, we'll go to 256 and we'll double 4,000 8,000.

So now we're gonna simulate 8,000 users actively loading

the page very aggressively and.

Yeah, Let's start that.

we'll see whether or not our uplink can handle 8,000 users.

go

ahead.

While, we're waiting for that.

Could you add like DNS caching to Goose itself like that?

it's this only one request

or.

Yes.

I mean, it might get complicated cause there's nothing to prevent your load

tests from hitting multiple domains.

But that's interesting.

We should look into that.

You want to file

Do you want to

take a

request live on stage.

Nice.

I liked the idea.

Uh, it'd be a good question for Narayan to whether or not he would like that

feature, but it seems great to me.

Whereas before we were using 50% of our our server to generate traffic,

we'll see how much, how much we end up using for this much traffic.

But our, our Fastly stats should start, start ramping up pretty quickly here

and there we go.

There are, we can see there's no errors because ramping up cleanly like that.

It's catching up.

we're already surpassing the original tests.

Now we're at eight K requests per second.

It's still going up.

12 K 16.

K sure enough.

We were able to double it again.

so we're doing 20 to 23,000 requests a second and sustaining 9.2 gigabytes.

nine gigabytes of traffic a second with a nice steady line, I should add, you

know, it's not, it's not spiky traffic.

It's very, very consistent traffic which is fantastic when you're trying to do a

load test and the, and the server, you know, it's, it's using a CPU's there's

only 4% idle but it's, it's generating some impressive some impressive load.

This was, this was really cool to see.

Thank you guys so much for walking us through this.

we're going to do some follow-ups to this.

we're going to do some how tos on testing with authenticated traffic.

We're going to show you how to spin up Gaggles and do.

Distributed load testing with Goose and Fabian.

I'm going to take you up on your scalability webinar, show

people how all that works.

so please stay tuned for more.

we'll put all of the links that Jeremy and Fabian mentioned in the show notes.

Do you have any questions for us?

Please reach out to us at tag1teamtalks@tag1.com.

We'd love your feedback on this, questions about Goose, or even better

hit us up in the issue queues on GitHub.

and we love feature suggestions.

You know, what, what should we cover in the future on this?

You can see our past Tag1 Team Talks at tag1.com/TTT for Tag1 Team Talks,

and you can go to Tag1.com/goose to get a lot of links and information

about Goose Jeremy, Fabian.

Again, a huge thank you for walking us through this today and to our listeners.

Thanks for tuning in for anotherTag1 Team talk..