In this second part of our team talk series on live load testing with Goose, we focus on demonstrating load testing using a Gaggle. A Gaggle is a distributed load test using running Goose from one or more servers. Here, we’re testing with 20,000 users using ten Workers and a Manager process on services spun up using Terraform. 

CEO Jeremy Andrews, the creator of Goose; Fabian Franz, VP of Software Engineering;  CTO Narayan Newton, and Managing Director Michael Meyers demonstrate running a Goose Gaggle and discuss how variations on these load test change testing results, and what you can expect from a Gaggle. Our goal is to prove to you that Goose is both the most scalable load testing framework currently available, and the easiest to scale.

Narayan is a key member of the Drupal.org infrastructure team, responsible for ensuring the site stays up under load or attack. Load testing is a critical part of ensuring Drupal.org, or any website’s continued success.

For more Goose content, see Goose Podcasts, Blogs, Presentations, & more!

Hello, and welcome to Tag1 Team Talks, the podcast and blog of Tag1 Consulting.

Today, we're going to be doing distributed load testing how to: a deep dive into

running a Gaggle with Tag1's, open source Goose load testing framework.

Our goal is to prove to you that Goose is both the most scalable load

testing framework currently available.

And the easiest to scale.

We're going to show you how to run a distributed load, test yourself.

And we're going to provide you with lots of code and examples to make it easy and

possible for you to do this on your own.

I'm Michael Meyers, the managing director at Tag1, and joining

me today is a star-studded cast.

We have Jeremy Andrews, the founder and CEO of Tag1.

Who's also the original creator of Goose, Fabian Franz, our VP of Technology.

Who's made major contributions to Goose, especially around

performance and scalability.

And Narayan Newton, our CTO who has set up and put together all the

infrastructure that we're going to be using to run these load tests.

Jeremy, why don't you take it away?

Give us an overview of what we're going to be covering and let's jump into it.

Yeah.

So last time we were exploring with setting up a load test from a single

server and confirmed that Goose makes great use of that server.

it leverages all the CPU's and ultimately tends to get as far as it

can until the uplink slows it down.

So today what we're going to do is.

Use a feature of Goose called a Gaggle which is a distributed load test.

If you're familiar with Locust, it is like a swarm.

and the way it, the way that this works with Goose you have a manager

process that you kick off and you say, I want to simulate 20,000 users and I'm

expecting 10 workers to do this load.

the manager process prepares things and, and all the workers then connect

in through a TCP port and it sends each of them a batch of users to run.

And then each of them the manager coordinates a start, each of the

workers start at the same time.

And And then they send their statistics back to the managers so that you can

actually see what happened in the end.

what this nicely solves is if you're uplink can only do so much traffic,

or if you want traffic coming from multiple regions around the world you

could let Goose manage that for you in all of these different servers.

So today Narayan has set up a pretty cool test where we're going to

be spinning up a lot of workers.

and he could talk about how many each one is not going to be working too hard.

They'll run maybe a thousand users per server.

which means it'll be at least 50% idle.

It won't be maxing out the uplink on any given server.

but in spite of that, we're going to show that working together in a Gaggle

we can generate a huge amount of load.

so now Narayan, if you can talk about what you've set up here.

Sure.

so what I built today is basically a simplistic Terraform tree.

what is interesting about this is that we wanted to distribute the load between

different regions and for those people that have used Terraform in the past

that can be slightly odd in that you can only set one region for each AWS provider

that Terraform uses to spin things up.

So how we've done, this is defined multiple providers, one

for each region and a module that spins up our region workers.

And we basically initialize multiple versions of the module

passing each a different region.

So in the default test, we spin up 10 worker nodes in various regions.

the Western part of the United States, the Eastern part of the

United States, Ireland Frankfurt.

India and Japan with how the test currently works.

It's the load testing truss, which is what we decided to call it.

it's a little limited because once you start it, you can't really

interact with the workers themselves.

They start up, they pull down Goose and they run the test.

then next revision of this would be something that has a clustering agent

between the workers to, so that you can actually interact with the workers

after they start it gets very annoying to have to run Terraform, to stand up

these VMs all over the world, and then you want to make a change to them.

You have to destroy all of them and then relaunch them which isn't terrible.

But as a testing, sequence.

It adds a lot of time, just because it takes time to

destroy and recreate these VMs.

so the next revision of this would be something other than Goose,

creating a cluster of these VMs.

how it currently works is that we're using Fedora CoreOS so that we have

a consistent base at each location.

And so I could only send it a single file for initialization.

And then Fedora CoreOS pulls down a container that has the Goose load test

and a container that has a logging agent so that we can monitor the workers

and send all the logs from the Goose agents back to a central location

I had a quick question, so Narayan , um, the basic setup is that we have EC2

instances, like on AWS, and then we run containers like normal Kubernetes

like on them, or how is it working?

It's using Docker.

So that is the big thing that I want to improve.

And I almost got there before today.

what would be nicer is if we could use one of the IOT distributions or Kubernetes

at the edge distributions to run a very slim version of Kubernetes on each

worker node so that we get a few things.

One is cluster access, so we can actually interact with the clusters spread

load, run, multiple instances of Goose.

it would be interesting to pack multiple instances of Goose on things

like the higher end and also be able to actually edit the cluster after

it's up and not have to destroy it and recreate it each time.

The other thing is to get containerd and not Docker.

just because there are some issues that you can hit with that.

as it stands right now, CoreOS ships with Docker running, and that's how

you interact with it for the most part is a systemctl Docker, but you could

also use Podman but I ran into issues with that for redirecting the logs.

So we are actually using Docker itself and Docker is just running

the container as you would in a local development environment.

So what we are missing from standard kubernetes deployment thing that

we would normally have is the ability to deploy a new container.

You were saying that if I want to deploy a new container versus

simplistic infrastructure right now, I need to shut down the EC2

instance and then start them up again.

Okay.

So that's, that's the, so like what I did when before this test, Jeremy released

a new branch with some changes to make this load test faster as on startup.

what I did to deploy that is run Terraform destroy, wait for it, to kill

all their VMs across the world, and then Terraform apply and wait for it to

recreate all those VMs across the world.

And like that is.

management style, honestly, but in this specific case, because we're

doing sometimes micro iterations, it can get really annoying.

Yeah, for sure.

No, no, that makes perfect sense.

I just want to understand, because I was like, in this container world, you

can just deploy a new container, but obviously you need a manager for that.

Yes.

Yes.

I could totally deploy a new container.

So what I could do is have Terraform output the list of IPs, and then I can SSH

to each of them and pull a new container.

But at that point,

But seriously, there's another Git repository that I have started.

The version of this that uses a distribution of Kubernetes is called

K3s that is designed for CI systems and IOT and deployments to the edge.

And it's a It's a single binary version of Kubernetes where everything is

wrapped into a single binary and starts on edge nodes and then can connect

them all together and so we could have a multi-region global cluster

of this little Kubernetes agents.

And then we could spin up Gooses on that.

And that I think will actually work.

You totally blew my mind.

So now you've just signed up for follow up to show that

because that's, I mean, that's, that's what you want actually, but now I'm

really curious, how does this Terraform configuration actually look, can,

can you share a little bit about it?

So this is the current tree.

If everyone can see that it's pretty simplistic.

So this is the main file that gets loaded.

And then for everyone, there's a module that is named after its region.

They're all hitting that same actual module is just different

revisions of this module.

And then they'll take a worker count and their region and their provider

and the provider is what is actually separating them into regions.

And then if you look at the region worker, which is where most of these

things are happening, There's a variables thing, which is interesting

because I have to define an AMI map because every region has a different

AMI because the regions are disparate.

Like there's no, there's no consensus building between these regions for images.

So one of the reasons I picked CoreOS is because it exists in each of these regions

and can handle a single start-up file.

when when we do the K3s version of this K3s kind of run on Ubuntu

of, and Ubuntu of obviously exists in all these regions as well.

but I'll still have to do something like this, or there's another way I can do it,

but this was the way to do it for CoreOS.

And then we set, instance type, this is this a default.

And then the main version of this is very simple.

we initialized our key pair, cause I want to be able to SSH into these instances at

some point and upload it to each region.

We initialize a simple security group that allows me to SSH in to each region.

And then a simple instance that doesn't really have much because it's, it

doesn't even have a large root device cause we're not using it at all.

Basically we're just spinning up a single container and then pushing the logs to

Datadog, which is our central log agent.

So even the logs aren't being written locally on that we

associated a public IP address.

We spin up the AMI, we look up which AMI we should use based on our region.

and then we output the worker address.

So the other part of this is the manager.

The only real difference in this, we basically spent out the exact same

way is we also allow the Goose ports, which is 5115, and we spin up a DNS

record that points to our manager because that DNS record is what all the

region workers are going to point at.

Um, and we make use of the fact that they're all using route 53.

So this update propagates really quickly.

And that's basically that it's pretty simple.

each VM is running.

Sorry, go ahead.

where do you actually put in the Goose part?

Because I've seen the VM.

Yep.

So each CoreOS VM it can take a ignition file.

The idea behind CoreOS is it was a project to simplify infrastructure

that was based on containers.

It became an underlying part of a lot of Kubernetes deployments

because it's basically read only in essence on a configuration level.

It even can auto update itself.

it's a very interesting way of dealing with an operating system.

It its entire concept is you don't really interact with it outside of containers.

It's just a stable base for containers that remain secure can auto update is

basically read only in its essence and it takes these ignition files that define

how it should set itself up on first boot.

So if we look at one of these ignition files,

okay.

we can see that it's basically YAML.

And we defined the SSH key.

We want to get pushed.

We define an /etc/hosts file to push.

We then define some systemd units, which include turning off SELinux

because we don't want to deal with that on short-lived workers.

And then we define the Goose service, which pulls down the image.

And right here actually starts Goose.

this is mostly defining the log driver, which ships logs back to

Datadog, the log driver, the actual logging agent is started here.

but then like, this is one of the workers.

So we pulled the temp, umami branch of Goose.

We start it up, set it to worker.

Point it to the manager host set it to be somewhat verbose set the log

driver to be Datadog startup data dogs that we get metrics in the logs.

And then that's just how it runs.

And this will restart over and over and over again.

So you can actually run multiple tests with the same infrastructure.

You just have to restart Goose on the manager and then the workers will

kill themselves and then restart.

And so you get this plan, where it shows you where all the

instances it's going to spin up.

It's actually fairly long just because there are a lot of params

for each EC2 instance that we're spinning up 11 of them, 10 plus

the manager, you say that's fine.

And it goes,

and I will stop sharing my screen now is this is going to take a bit.

So is this already doing something now.

Yes.

And this is, you're probably going to see one of the quirks.

and this is another thing I dislike about this.

Because we're using CoreOS . These are all coming up on an outdated AMI and

they're all going to reboot right there

because they come up, they start pulling the Goose container and

then they start the update process and they're not doing anything.

So at that point, they think it's safe to update.

And so they update and reboot.

it's somewhat cool that that has no impact on anything like the entire infrastructure

comes up, updates itself, reboots.

Then it continues on with what it's doing, but it's another

little annoyance that I just don't.

You spin up this infrastructure and you don't really have

a ton of control over it.

And so this is the logs of the manager process of Goose, and it's just

waiting for its workers to connect.

They're all, they've all updated, rebooted and Goose is starting on them.

As you can see, eight of them had completed that process.

is the, you know, all of this, the stuff that you put together here is

this going to be available open source for folks to download and leverage?

Yep.

Awesome.

It's all online.

On our Tag1 Consulting GitHub organization and the K3s version will be as well.

And that's the one I'd recommend you use.

This

one's real annoying.

I know I keep going on about it, but like, this is how it skunkworks projects work.

You make the first revision and you hate it and then you

decide to never do that again.

And then you make the second revision.

Okay.

This is starting.

Now I'm going to switch my screen over to the web browser

so we can see what it's doing.

Sure.

Great.

the logs that we're seeing there are they coming from the Datadog

or just the manager director.

And so

that was a direct connection to the manager.

If we go over to Datadog here, there, these are going to be the logs.

As you can see, the host is just like what an EC2 host name looks

like, and they're all changing, but we're getting logs from every agent.

As well as the worker.

You can see they're launching.

If we go back to Fastly, we can see that they're getting global traffic.

So we're getting traffic on the West coast, the East

coast, Ireland, Frankfurt, and

Mumbai,

and the bandwidth will just keep ramping up from here

For the Datadog is that way to also filter by the manager, like,

sure.

this is the live tail.

We'll go to the past 15 minutes and then you can go service Goose.

And then we have worker and manager, so I can do all my worker.

And that's sorry, only manager.

The manager is pretty quiet.

The workers are not.

You must've disabled.

displaying metrics regularly.

Cause I would have expected on the server to see that.

if I did, I did not intend to, but I probably did.

Can we, is it easy to quickly see what command you passed in or not to go back

there from where you're at right now?

It's in Terraform, I think.

It is all set here.

So it

must be interesting.

I have to figure out why you're not getting statistics on the

manager because you should be getting statistics on the manager.

Is this the log you're tailing or is this what's verbosly put out to the screen?

This is

what is put out to the screen.

Yeah.

Interesting.

Okay.

I would have expected statistics every 30

seconds.

So what's kind of interesting is you can expand this in Fastly and see we're

doing significantly less traffic in Asia Pacific, but that makes sense.

Considering we're only hitting one of the PoPs and then Europe and North

America tends to be about the same, but you can even drill down further,

one quick question.

I saw you hard-code the IP address of the end point in the Terraform.

how does Fastly is still know essentially to which PoP to route

and they're doing it through magic.

Um, you mean I'd put the IP the same IP address everywhere in /etc/hosts?.

Yep.

Yeah.

It's because of how they're doing traffic.

So it is the same IP address everywhere, but they the IP

address points to different things.

Basically.

It's cool.

A lot of CDNs do it that way.

so instead of different IP addresses, it's basically routing tricks.

We seem

to have maxed out.

Can you look at the

Yeah, this should be about it.

It should be all started at this point.

Yeah.

So we've launched a thousand users, we've entered Goose attack.

So we have evened out at 14.5 gigabits per second, which is, I think what we got

on one server with 10,000 users as well.

this is more, this is more than a single server single server.

I think we max out at nine gigabit.

Awesome.

Thank you guys.

All for joining us.

It was really cool to see that in action.

all the links we mentioned are going to be posted in the video summary and the

blog posts to that correlates with this.

Be sure to check out Tag1.com/goose that's tag the number one.com.

that's where we have all of our talks, documentation links to GitHub.

There's some really great blog posts there that will show you

step-by-step with the code, how to do everything that we covered today.

So be sure to check that out.

If you have any questions about Goose, please post them to the Goose issue queues

that we can share them with the community.

Of course, if you like this talk, please remember to upvote

subscribe and share it out.

You can check out our past Tag1 TeamTalks on a wide variety of topics from open

source and funding, getting funding on your open source projects to things like

decoupled systems and architectures for web applications at tag1.com/tag1teamtalks

as always we'd love your feedback and input on both this episode, as

well as ideas for future topics.

You can email us at ttt@tag1.com Again, a huge thank you, Jeremy, Fabian and

Narayan for walking us through this and to everyone who tuned in today.

Really appreciate you joining us.