aws-samples / distributed-load-testing-with-locust-on-ecs

Deploy Locust load testing framework on Amazon ECS

Distributed Load Testing with Locust on Amazon ECS

This sample shows you how to deploy Locust, a modern load testing framework, to Amazon Elastic Container Service (ECS).

• ✅ From small to massive-scale load test with AWS serverless technologies
• ✅ Highly cost-efficient with Fargate spot capcity
• ✅ Instant deployment using AWS CDK

It leverages a serverless compute engine Fargate with spot capacity, which allows you to run massive-scale load test without managing infrastructure and with relatively low cost (70% cheaper than using on-demand capacity).

How it works

Below is the architecture diagram of this sample.

We deploy Locust with distributed mode, hence two ECS services - Locust master and worker service.

The Locust master consists of a single Fargate task, and its Web GUI can be accessed via Application Load Balancer.

Unlike master node, there can be N Locust worker nodes, which is usually the dominant factor of load test infrastructure cost. We use Fargate spot capacity for Locust workers, allowing you to run load tests at most 70% cheaper than on-demand capacity.

Note that all the access from Locust workers go through NAT Gateway, which makes it easy to restrict access by IP addresses on load test target servers, because all the Locust workers shares the same outbound IP address.

Deploy

To deploy this sample to your own AWS account, please follow the steps below.

1. Prerequisites

Before you deploy, make sure you install the following tools in your local environment.

• Docker
• Node.js (v14 or newer)
• AWS CLI

Also you need Administorator IAM policy to deploy this sample.

2. Set parameters

You need to set several parameters to configure the system.

Please open bin/load_test.ts and find property named allowedCidrs. This property specifies the CIDRs which can access the Locust web UI ALB. You should set this as narrrow as possible because otherwise unwanted users can access your Locust control panel.

To change the AWS region to deploy this samples to, please replace us-west-2 in env.region property to your desired region.

For additional security, you can set certificateArn and env.account to protect ALB with TLS. By default Locust Web GUI can be accessed with HTTP. You can make it HTTPS by those properties.

You can also enable basic authentication for Locust web UI by specifying webUsername and webPassword properties.

3. Setup CDK

After confirming the parameters, you can proceed to CDK deployment.

First, you need to setup CDK environment by the following command:

npm ci
npx cdk bootstrap

You only need those commands once for your AWS environment (pair of account ID and region.)

5. Run cdk deploy

Now you can deploy your CDK project.

npx cdk deploy

Deployment usually takes less than 10 minutes.

After a successful deployment, you can find some stack outputs like below in your terminal:

 ✅  LoadTestStack

✨  Deployment time: 109.55s

Outputs:
LoadTestStack.EcsClusterArn = arn:aws:ecs:ap-northeast-1:123456789012:cluster/LoadTestStack-ClusterEB0386A7-xxxxxx
LoadTestStack.MasterServiceLoadBalancerDNSD23C49A7 = LoadT-Maste-1MZQ6IPOOJ262-xxxxxxx.ap-northeast-1.elb.amazonaws.com
LoadTestStack.MasterServiceServiceURL66A06FCF = http://LoadT-Maste-1MZQ6IPOOJ262-xxxxxxx.ap-northeast-1.elb.amazonaws.com
LoadTestStack.WorkerServiceName = LoadTestStack-WorkerService3F60922A-xxxxxxx

You need the value of EcsClusterArn and WorkerServiceName in later steps, so it is recommended that you take a note of those values.

After that, please make sure that you can open the URL in LocustMasterServiceURLCA3E9210 and view Locust Web GUI like below image.

Now the deployment is completed! You can start to use Locust load tester.

Tips

There are a few things you need to know to use this sample effectively.

Adjust the number of Locust worker tasks

Depending on the amount of load you want to generate, you may need to increase Locust worker capacity.

It can be adjusted with the following command:

aws ecs update-service --cluster <EcsClusterArn> --service <WorkerServiceName> --desired-count <the number of workers>

Please replace <EcsClusterArn> and <WorkerServiceName> with the value you took a note when deployment, and <the number of workers> with any integer you desire for the worker count.

Please also be aware that your default quota for the number of Fargate tasks is 1000. If you need more tasks, you can request a limit increase from Service Quotas console. You can read further detail here.

When Fargate spot is out of capacity

It is expected that sometimes Fargate spot fails to allocate the required capacity for your Locust workers because of insufficient capacity. That issue should resolve if you wait for certain time. However, if it continues for unacceptable time, you can always add on-demand capacity to fill your desired task count.

Please open lib/constructs/locust_worker_service.ts and find the lines below:

      capacityProviderStrategies: [
        {
          capacityProvider: 'FARGATE_SPOT',
          weight: 1,
        },
        {
          capacityProvider: 'FARGATE',
          weight: 0,
        },
      ],

You can specify the ratio of spot (FARGATE_SPOT) vs on-demand (FARGATE) by the weight properties. The default is to use spot 100% (1:0).

Modify Locust scenario

Default locustfile is placed on ./app/locustfile.py.

You can modify this file as you like. To deploy your change, run cdk deploy again.

Also, you can build and run Locust locally by the following commands:

cd app
docker build . -t locust
docker run -p 8089:8089 locust 

Clean up

To avoid incurring future charges, clean up the resources you created.

You can remove all the AWS resources deployed by this sample running the following command:

npx cdk destroy --force

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

distributed-load-testing-with-locust-on-ecs's Issues

enable basic auth for Locust web UI

Using --web-auth WEB_AUTH option, we can enable basic authentication for Locust web UI.

c.f. https://docs.locust.io/en/stable/configuration.html

Standalone vs master/worker mode

Hello,

I am currently reviewing the Dockerfile in this repository and need some clarification regarding the commented-out code sections related to configuring Locust in different modes.

Specifically, I am trying to understand the implications of uncommenting the various lines for standalone, worker, and master modes. The current comments indicate that Locust is set up for standalone mode, but I am unclear about the exact meaning and implications of this setup.

Could you please provide more details on:

1. What the standalone mode entails in the context of this Dockerfile? Does it imply that only the master component is operational?
2. If uncommenting the worker or master sections is necessary, how would these changes affect the behavior of Locust? Would it mean that both master and worker operate as separate, unconnected instances?
3. In case I need to configure a distributed setup with both master and worker roles, what steps should I follow to ensure proper connectivity and coordination between these components?

I have included the relevant section of the Dockerfile for reference:

FROM --platform=linux/amd64 locustio/locust:2.8.3
COPY . ./

# for standalone
ENTRYPOINT ["locust", "-f", "./locustfile.py"]
# for worker
# CMD [ "--worker", "--master-host", "MASTER_HOST_NAME"]
# for master
# CMD ["--master"]

Any insights or guidance you can provide on this matter would be greatly appreciated.

Thank you for your time and assistance.

Use Locust modern auth

The current basic auth will be deprecated soon. We need to switch to the modern auth way: locustio/locust#2538

Error when running npx cdk bootstrap

I executed npm ci without any issues. When I run npx cdk bootstrap I get the error below.

distributed-load-testing-with-locust-on-ecs\node_modules\ts-node\src\index.ts:513
return new TSError(diagnosticText, diagnosticCodes)
^
TSError: ⨯ Unable to compile TypeScript:
bin/load_test.ts:27:1 - error TS2304: Cannot find name 'n'.

27 n;
~

at createTSError (C:\Users\nbrisard\GIT\distributed-load-testing-with-locust-on-ecs\node_modules\ts-node\src\index.ts:513:12)
at reportTSError (C:\Users\nbrisard\GIT\distributed-load-testing-with-locust-on-ecs\node_modules\ts-node\src\index.ts:517:19)
at getOutput (C:\Users\nbrisard\GIT\distributed-load-testing-with-locust-on-ecs\node_modules\ts-node\src\index.ts:752:36)
at Object.compile (C:\Users\nbrisard\GIT\distributed-load-testing-with-locust-on-ecs\node_modules\ts-node\src\index.ts:968:32)
at Module.m._compile (C:\Users\nbrisard\GIT\distributed-load-testing-with-locust-on-ecs\node_modules\ts-node\src\index.ts:1056:42)
at Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
at Object.require.extensions.<computed> [as .ts] (C:\Users\nbrisard\GIT\distributed-load-testing-with-locust-on-ecs\node_modules\ts-node\src\index.ts:1059:12)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)