pwrdrvr / lambda-dispatch Goto Github PK

View Code? Open in Web Editor NEW

25.0 25.0 2.0 2.87 MB

Lambda Dispatch for AWS Lambda - Avoid cold starts, save up to 80%!

Shell 0.33% C# 58.69% JavaScript 5.10% Rust 35.88%

aws aws-lambda cold-start csharp dotnet http2 rust-lang

lambda-dispatch's People

Contributors

Stargazers

Watchers

Forkers

ajjahn cesine

lambda-dispatch's Issues

Router - Get cert private key from AWS Secrets Manager

Motivations

Do not ship any cert in the docker image
Get the cert from AWS Secrets Manager on startup of the ECS containers

General - Add support for WebSockets

Motivations

WebSockets on Lambda, serving multiple users from one Lambda instance, is an often requested feature for Lambda
Should be buildable

Acceptance Criteria

Research what needs to be done

Add proxying to contained application

Overview

Startup the app
Check the health check route
Check the startup ready route
Proxy the request to the app port
Proxy the response back from the app port

Extension - Contained app crashes and Extension does not recover

Motivations

If the contained app refuses connections then we need to throw out of the Lambda handler
We're going to lose some requests in this case because the contained app may crash when it already has requests sent to it

Acceptance Criteria

Detect the exception
Throw an exception out of the Lambda handler so the exec env will be recreated
Fix the error in this specific case in the demo-app

LambaLB Logs

=> LambdaId: 3170030e-384d-46ea-8353-8199743d7e04 => TaskNumber: 8 => ChannelId: 09e65c5b-c262-469d-a791-430e8cbfc552
--
Exception caught in task
System.Net.Http.HttpRequestException: Connection refused (localhost:3000)
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError, CancellationToken) + 0x4c
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16) + 0x60
at System.Net.Sockets.Socket.<<ConnectAsync>g__WaitForConnectWithCancellation\|285_0>d.MoveNext() + 0x1bc
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
at System.Net.Http.HttpConnectionPool.<ConnectToTcpHostAsync>d__104.MoveNext() + 0x714
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.<ConnectToTcpHostAsync>d__104.MoveNext() + 0x79c
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
at System.Net.Http.HttpConnectionPool.<ConnectAsync>d__103.MoveNext() + 0x1dc
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
at System.Net.Http.HttpConnectionPool.<CreateHttp11ConnectionAsync>d__105.MoveNext() + 0x1b8
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
at System.Net.Http.HttpConnectionPool.<AddHttp11ConnectionAsync>d__79.MoveNext() + 0x350
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.<WaitWithCancellationAsync>d__1.MoveNext() + 0x178
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
at System.Net.Http.HttpConnectionPool.<SendWithVersionDetectionAndRetryAsync>d__89.MoveNext() + 0xac0
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
at System.Net.Http.RedirectHandler.<SendAsync>d__4.MoveNext() + 0x184
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
at System.Net.Http.HttpClient.<<SendAsync>g__Core\|83_0>d.MoveNext() + 0x3d8
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
at PwrDrvr.LambdaDispatch.LambdaLB.Function.<>c__DisplayClass9_1.<<FunctionHandler>b__1>d.MoveNext() + 0x740
01:01:25.580 warn: PwrDrvr.LambdaDispatch.LambdaLB.Function[0]
=> LambdaId: 3170030e-384d-46ea-8353-8199743d7e04 => TaskNumber: 8 => ChannelId: 09e65c5b-c262-469d-a791-430e8cbfc552
Connection refused

Contained App Logs

Starting task
--
/app/dist/app.cjs:476
throw new TypeError("invalid media type");
^
TypeError: invalid media type
at Object.parse2 [as parse] (/app/dist/app.cjs:476:15)
at setCharset (/app/dist/app.cjs:20086:32)
at ServerResponse.send2 [as send] (/app/dist/app.cjs:21472:36)
at ServerResponse.json (/app/dist/app.cjs:21536:19)
at ServerResponse.send2 [as send] (/app/dist/app.cjs:21464:25)
at /app/dist/app.cjs:71216:9
at Layer.handle [as handle_request] (/app/dist/app.cjs:17246:9)
at next (/app/dist/app.cjs:17415:17)
at rawParser (/app/dist/app.cjs:14424:11)
at Layer.handle [as handle_request] (/app/dist/app.cjs:17246:9)
Node.js v18.18.2
01:01:25.580 fail: PwrDrvr.LambdaDispatch.LambdaLB.Function[0]

Router - Use IO Pipelines to Read/Write Request/Response

Motivations

Pipelines are more efficient than Streams because they use a buffer manager

Acceptance Criteria

Remove all usage of the Stream interfaces
Convert to usage of the IO Pipelines

Docs

HttpContext, HttpRequest, HttpResponse

IO Pipelines

LambdaLB - Use statically linked openssl

Motivations

Different versions of libssl and libcrypto result in an error of missing symbols when https connections get established for the first time
- Cannot get required symbol EVP_rc2_cbc from libssl
Dynamically linking ssl/crypto stinks as it makes distributing the app much more difficult

Acceptance Criteria

Read the docs: https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/docs/compiling.md#using-statically-linked-openssl
Attempt to statically link openssl libs
Test

Minimize the size of the LambdaLB image

Motivations

We want this as small as possible
"Make it look like Rust" 🤣

ToDo

Remove the accidentally copied in non-Native AoT files from the router build (update .dockerignore) (100+ MB)
Remove the libicu files (about 40 MB)
Test and confirm this works

Extension - Fix Errors on `oha` Connection Terminations

Motivations

When oha stops with a time boundary it aborts in-progress requests
The router catches this and aborts the request/response to the extension
The extension (rust) is throwing a few errors in this case
Eventually the Lambda becomes non-responsive

Acceptance Criteria

Reproduce locally
Identify cause
Fix

Reproducing Locally

The below will cause the open sockets count to go up and up.

Lambda has a 1024 socket descriptor limit so that is probably being exceeded at some point.

# Count sockets for the node.js app
watch "netstat -a -n -p tcp | grep 3001 | wc -l"

# Repeatedly Allow `oha` to Terminate Sockets
while true; do oha --no-tui -m POST -c 10 -z 1s -D ./oha -T "application/octet-stream" http://localhost:5001/echo ; done

General - Add support for gRPC

Motivations

gRPC should be supportable

Router / Extension - Add auth data to ensure only invoked lambdas are allowed to serve requests

Motivations

Some level of security is nice
Prevents accidentally taking traffic if an extension connects to the wrong router (e.g. because of IP reuse and some sort of delay/hang/replayed request)

Acceptance Criteria

Generate secure random data or signature
Send data in the invoke
Echo the data back in the connections
Reject requests that do not have the data or have the wrong value

Router - `/` Path Does Not Pass to Lambda

Motivations

The / root path should be passable to an app
The control interface and the incoming request interface should be completely isolated if possible

Acceptance Criteria

See if we can create two Kestrel instances instead of 1
Check for any other options
Confirm that / passes through to apps

Router - Dispatch pools via HTTP headers with Lambda ARNs

Motivations

Routing to several apps should not require several ALBs
Routing to several apps should not require several ECS deployments
Supports the use case of MicroApps with loose coupling - the OriginRequest will still lookup the Lambda ARN and just write it as a header for the single Lambda Dispatch ALB

Acceptance Criteria

Restructure Dispatcher, Queue, and LambdaInstanceManager to be created in a set instead of as a singleton
When LAMBDA_DISPATCH_FunctionName env var is set, setup a default dispatch group
Allow enabling or disabling the header usage via a config option
When a Lambda ARN is passed in the header, lookup the ARN and see if there is already a dispatch group
Add a parameter to the Router <-> Extension protocol that has the dispatch group GUID
Have the Extension echo back the dispatch group GUID
When extension connections arrive, lookup their dispatch group and hand them off to that group
Test
Deploy

Router - Reduce lambda count as no longer needed

Motivations

Reducing the lambda count does not immediately reduce the number of "warm" execution environments
Reducing the lambda count DOES reduce our costs, immediately

Acceptance Criteria

Modify the function that increases capacity to also reduce capacity
Check that the min duration of the lambdas is no longer 45 seconds

Router - Config options for insecure http2

Motivations

Should not require recompile to allow insecure http2

Acceptance Criteria

Add option to allow insecure http2 (this determines whether the insecure port is open at all for http2)
- LAMBDA_DISPATCH_AllowInsecureControlChannel=true
Add option to specify http/https preference for http2 for extensions to contact back on (useful for migrations to enable to or disable)
- LAMBDA_DISPATCH_PreferredControlChannelScheme=http

Router - 5 Second Delay in 1st Lambda Invoke

Overview

The Router appears to be unable to invoke a lambda (to even make the call) for 5 seconds after receiving the first incoming request
The example below shows a ~3 second delay on invoking the Lambda

LambdaTestTools - First invoke never executes

Log - Request Queued - Did start lambda - 03:11:21.131

LambdaTestTool appears to have failed to send this message

lambda-dispatch-router-1    | 03:11:20.266 info: PwrDrvr.LambdaDispatch.Router.LambdaInstance[0]
lambda-dispatch-router-1    |       Starting Lambda Instance de5b44d3-19f3-4795-98cb-112d4bbf5905

lambda-dispatch-router-1    | 03:11:21.131 info: PwrDrvr.LambdaDispatch.Router.LoggerMetricsReporter[0]
lambda-dispatch-router-1    |       Metrics:
lambda-dispatch-router-1    |       LambdaInstanceStartingCount: 1 items
lambda-dispatch-router-1    |       report_success: 2 items
lambda-dispatch-router-1    |       LambdaInstanceCount: 1 items
lambda-dispatch-router-1    |       QueuedRequests: 1 req
lambda-dispatch-router-1    |       RequestCount: 1 req
lambda-dispatch-router-1    |       IncomingRequestTimer: 0 count 0 last 0 mean 0 min 0 max ms

Log - Choice to Start Lambda - 4 Seconds Later

lambda-dispatch-router-1    | 03:11:25.421 info: PwrDrvr.LambdaDispatch.Router.LambdaInstance[0]
lambda-dispatch-router-1    |       Starting Lambda Instance 5d540d52-a9bc-448d-9ff4-51409f40ecc0

HttpClient - Augment the `ProtocolError` with specific info as to what happened

Motivations

Debugging this was near impossible
Others will have no idea what is happening
That's not cool

Acceptance Criteria

Check if the Trace logging is logging what happened in detail
- If it isn't, consider adding it
Check how exceptions in HttpClient are augmented with additional info, such as with HttpRequestException (which ProtocolError might actually be a part of already)
Make an issue and/or draft PRs with proposed changes

Router - Start Lambda while prior Lambda is stopping

Motivations

True graceful shutdown means we are going to be waiting up to, say, a minute for some longer-lived Lambdas to complete all of their in-flight requests when it is time to shutdown
This means that the Router will be down 1 instance for up to 1 minute for each Lambda that shuts down during a given 1 minute period
This is VERY noticeable with small quantities of Lambdas
This shows up as 5 second delays on requests in the best case when 2 of 2 Lambdas both gracefully exit at the same time
Starting another Lambda while the prior Lambda is still stopping will:
- Init a few more execution environments which will be helpful during traffic bursts
- Allow RPS rates to remain steady when invokes are being rotated

To-Do

Every time a Lambda is invoked, invoke 2 with 1 set to "init-only"?
Return after init for the lambda marked as init-only
If the lambda is already init'd, return immediately
Do not count the init-only lambda as a starting or running instance
Drop the closing lambda from the lambda invoke count as soon as the close initiates

Kestrel - Check how Kestrel handles response body write before StartAsync call

Motivations

This may be a bug or lack of warning log in Kestrel
It took 3 days and deep magic to diagnose what was happening on the HttpClient side, given that the ProtocolError exception said nothing about receiving DATA before HEADERS
Printing a warning or throwing an exception in Kestrel may be the correct approach since this is not valid (at least, according to HttpClient)

Acceptance Criteria

Create a self-contained reproducible example
Submit an issue and draft PR (or PRs if there are a few options)

Extension (rust) - Breaks with Next.js apps on URL passed in HTTP request line

Motivations

Next.js is examining the entire string in the 2nd place in the request line: VERB /path HTTP/1.1
The extension was passing http://localhost:3001/ as the path
Next.js saw this as a path with double // chars and redirects to remove it, resulting in a URL turned into a broken path

Acceptance Criteria

Fix the bug (pass only the path)

LambdaLB - Support arbitrary content types on the request/response - Do not assume text

Overview

Currently there is debugging code that interprets the data as strings and logs it
This needs to be removed
May need the header parsing strategy from HttpClient

Router - Enable more than 10 outstanding requests per Lambda

Overview

~~[ ] Finish the prime-based bucketing in the LeastOutstandingQueue~~
Allow less than 10 outstanding requests per Lambda
Allow more than 10 outstanding requests per Lambda
Allow the number of pooled connections per Lambda to be completely independent of the number of outstanding requests per Lambda - Fixed in #130

Router - Dispatching stalls for 1 concurrent incoming connection and 1 extension connection

Motivations

oha -c 1 and LAMBDA_DISPATCH_MaxConcurrentCount=1 for the router causes 1-2 second pauses every couple of seconds
This is bad

Acceptance Criteria

Use debugger to figure out where this is stopping
- This is happening because of a race condition
  - Incoming request looks for connections and finds none
  - Incoming connection looks for queued requests and finds none
  - Incoming request gets added to pending queue
  - Incoming connection gets added to connection queue
  - This should be waking up the background dispatcher but it appears it either is not or it is going back to slow mode before dispatching
    - FIX: The issue was that the background dispatcher was awoken before the pending request count was incremented, causing the background dispatcher to go back to sleep for 1 second
Fix

DirectLambda - Giving 502 from ALB on some requests

Problem

The ALB was generating a 502 on routes that gave 404
The issue was that the content-length response header was a number instead of a string
The ALB will not accept a number in the content-length header

Upstream Issue

CodeGenieApp/serverless-express#563

Upstream Solution

H4ad/serverless-adapter#74

Had to switch from serverless-express to serverless-adapter to get the fix.

Allow configuration

FunctionName = "lambda-dispatch-lambdalb"
Port numbers (5002 and 5003)
Max concurrent connections
Have the Extension follow the scheme and port in the waiter message
Reconfigure all the ports to make sense

Router/LambdaLB - Confirm first bytes received by client immediately

Motivations

Confirm that the first response bytes are not delayed in being sent to the caller
Having no delay on the first response bytes is a big feature

Acceptance Criteria

Add a route to the Node.js app that simulates this
Test to see when the first bytes are received
Fix if necessary

Results

After

hey -c 10 -n 10000 http://127.0.0.1:5002/public/silly-test-image.jpg


Summary:
  Total:        3.2683 secs
  Slowest:      0.0671 secs
  Fastest:      0.0007 secs
  Average:      0.0033 secs
  Requests/sec: 3059.6653
  
  Total data:   1681610000 bytes
  Size/request: 168161 bytes

Response time histogram:
  0.001 [1]     |
  0.007 [9854]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.014 [131]   |■
  0.021 [4]     |
  0.027 [0]     |
  0.034 [0]     |
  0.041 [0]     |
  0.047 [0]     |
  0.054 [0]     |
  0.060 [0]     |
  0.067 [10]    |


Latency distribution:
  10% in 0.0020 secs
  25% in 0.0024 secs
  50% in 0.0029 secs
  75% in 0.0038 secs
  90% in 0.0047 secs
  95% in 0.0055 secs
  99% in 0.0080 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0000 secs, 0.0007 secs, 0.0671 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0000 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0013 secs
  resp wait:    0.0021 secs, 0.0004 secs, 0.0644 secs
  resp read:    0.0011 secs, 0.0000 secs, 0.0080 secs

Status code distribution:
  [200] 10000 responses

Before

hey -c 10 -n 10000 http://127.0.0.1:5002/public/silly-test-image.jpg


Summary:
  Total:        5.0350 secs
  Slowest:      0.1223 secs
  Fastest:      0.0009 secs
  Average:      0.0050 secs
  Requests/sec: 1986.0920
  
  Total data:   1681610000 bytes
  Size/request: 168161 bytes

Response time histogram:
  0.001 [1]     |
  0.013 [9967]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.025 [22]    |
  0.037 [0]     |
  0.049 [0]     |
  0.062 [0]     |
  0.074 [0]     |
  0.086 [0]     |
  0.098 [0]     |
  0.110 [0]     |
  0.122 [10]    |


Latency distribution:
  10% in 0.0022 secs
  25% in 0.0030 secs
  50% in 0.0043 secs
  75% in 0.0069 secs
  90% in 0.0080 secs
  95% in 0.0087 secs
  99% in 0.0109 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0000 secs, 0.0009 secs, 0.1223 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0000 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0007 secs
  resp wait:    0.0037 secs, 0.0006 secs, 0.1183 secs
  resp read:    0.0013 secs, 0.0000 secs, 0.0091 secs

Status code distribution:
  [200] 10000 responses

Motivations

Unpredictable issues can cause hundreds of thousands of Lambda invokes, which can be very costly
#154
- Demonstrates a case of hundreds of thousands of invokes
#182
- Added information on the Extension response to indicate whether the Router needs to back off

Acceptance Criteria

Need to put the requests to start instances into a queue
Need configurable limits to the number of lambdas to run concurrently
This will still get into a tight Lambda Invoke loop if the lambda is invoked correctly but cannot connect back to the Router
- Need to backoff retry on that so we don't cause huge AWS bills
- Could consider throwing an exception out of the Lambda, which would then cause AWS's SDK to perform the backoff

Router - If a Lambda exits early, all requests bound to it should be aborted

Motivations

An incoming request can hang for 60 seconds, before Kestrel times out, if the Lambda the request was sent to exits
Maybe???

Acceptance Criteria

Add tests for this
Fix issue if there is one

Extension (rust) - Panic on odd status code of real application

Motivations

Panic observed on real application
It appears that this is on this line: app_parts.status.canonical_reason().unwrap()
This is most likely on an invented status code (e.g. 519)

Message

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/run.rs:344:59

thread 'main' panicked at 'LambdaId: 484618c5-509b-446c-9b93-924c33f67688 - run - Error in futures::future::try_join_all', src/run.rs:464:17

ERROR Lambda runtime invoke{requestId="abcfa9d6-4f62-4952-a925-b44b4ec060d1" xrayTraceId="Root=1-65a8969d-4779d0686ed209ef72fd30f8;Parent=7360266b34a4c915;Sampled=0;Lineage=5847b82f:0"}: Any { .. }

Router / Extension - Await either of the response or request body - Do not deadlock

Motivations

If a LambdaLB connected to the Router but sent a post body before reading the response body, the Router would deadlock on that request
Deadlocking is... not great
This is a security issue, resource usage issue, and crash issue all rolled into one

Acceptance Criteria

Await either of the request or response bodies and act accordingly (which can be to close down the stream since it's not the protocol we expect)

Router / Extension - Hanging on `/echo` duplex POST request/response on Lambda (throttling?)

Motivations

Only happens when deployed to Lambda
At 45 seconds, when the extension gracefully exits, a few requests either get stuck or are already stuck and cause the Lambda to timeout at 60 seconds
The Lambda does not return from the invoke so it cannot be immediately replaced

To-Do

Add unit tests for the extension
Add units tests for the router
Try testing with Docker or Raspberry Pi to shift the timing around in an attempt to trigger the problem locally

LambdaLB - Content headers on response are not copied back

Motivations

Missing headers on responses
Content-Type
Content-Length
Last-Modified

Acceptance Criteria

Copy the headers back

Router - Allow `https` / `http2` from the ALB to the Router

Motivations

This will be important for some
CoPilot claims that the ALB will accept basically any cert for the targets, including self-signed, expired, and wrong names

Acceptance Criteria

Add an https / http2 listening port for incoming requests from the ALB
Connect the ALB to this new port
Confirm everything works

General - Publish assets / docker images

Motivations

Assets need to be published so that others can utilize the project

Acceptance Criteria

Publish the extension as a docker image to AWS ECR Public Repository
Publish the router docker image to AWS ECR Public Repository
Test images and confirm the arm64 and amd64 images have the appropriate binary format
Remove the demo app from the extension image (possibly publish as another image)
Router: Get the lambda name from an env var
Update docs with the image names

Docs

Images

public.ecr.aws/pwrdrvr/lambda-dispatch-extension:main
- https://gallery.ecr.aws/pwrdrvr/lambda-dispatch-extension
public.ecr.aws/pwrdrvr/lambda-dispatch-router:main
- https://gallery.ecr.aws/pwrdrvr/lambda-dispatch-router
public.ecr.aws/pwrdrvr/lambda-dispatch-demo-app:main
- https://gallery.ecr.aws/pwrdrvr/lambda-dispatch-demo-app

Running Public Images with `platform: linux/amd64`

# docker-compose -f docker-compose-public.yml up --build

# docker-compose -f docker-compose-public.yml cp lambdalb:/app/bootstrap bootstrap
[+] Copying 1/0
 ✔ lambda-dispatch-lambdalb-1 copy lambda-dispatch-lambdalb-1:/app/bootstrap to bootstrap Copied                                                 
 0.1s 

# file bootstrap
bootstrap: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=18a5f37f3196919e19489fcbc5cfdf77029258c2, for GNU/Linux 3.2.0, stripped

Running Public Images with `platform: linux/arm64`

# docker-compose -f docker-compose-public.yml up --build

# docker-compose -f docker-compose-public.yml cp lambdalb:/app/bootstrap bootstrap
[+] Copying 1/0
 ✔ lambda-dispatch-lambdalb-1 copy lambda-dispatch-lambdalb-1:/app/bootstrap to bootstrap Copied                                                 
 0.1s 

# file bootstrap
bootstrap: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=0aeb353ae33f2fd42d1ae87fd6267e1ac3f8de73, for GNU/Linux 3.7.0, stripped

Comparison - Add Direct ALB to Lambda Deploy

Motivations

Need the same Node.js app deployed to Lambda and directly invoked by the ALB
Need this to do cold start response time comparisons

Acceptance Criteria

Add a handler entrypoint for the node.js app
Use vendia or a similar library to translate the ALB payload into something Express can route
Add to the CloudFormation template
Add a route for the host lambdadirect.ghpublic.pwrdrvr.com that goes directly to the Lambda target group
Run some performance comparisons

Add throttling with backoff to lambda instance starting

Overview

Currently a failure to start a Lambda will lead to a tight loop that will start that lambda millions of times
This can cause runaway billing problems

Lambda - Fix too long init case

Motivations

When lambda init takes > 10 seconds it gets killed and re-run as init in first invoke (paid)
The bootstrap is not detecting that it didn't init and that first contained app request will cause an init
The lambda then connects back and picks up requests before the init is truly finished
This causes 10 second delays on first requests when Lambda CPU is low enough to cause slow init

Acceptance Criteria

Detect when init has not happened and do not connect to Router until it has been re-performed

Router - Pre-warming of Lambda

Motivations

Demos are terrible when an application gets no hits
- The first page load for some apps can take 8-18 seconds (if the time goes just over 10 seconds then it repeats all the work)
Pre-warming is very inexpensive as it just spends a few ms every couple of minutes to keep the exec envs around

Acceptance Criteria

Send init-only requests
Send requests in parallel
Send requests every couple minutes
Allow configuration

The background dispatcher should be a last resort

Motivations

The background dispatcher for pending requests is a last resort
The background dispatcher can get used in this race condition:
- 10 incoming requests see that there are no connections
- Just a moment later, 10 incoming connections see that there are no pending requests
- The incoming connections get added to their queue
- The incoming requests get added to their queue
- Nothing happens until the background dispatcher notices this
We should be able to wake up the background dispatcher to make it poll more frequently when it is known that requests are in the queue

After - No more 1 second response times

hey -c 20 -n 100000 http://127.0.0.1:5001/fact

Summary:
  Total:	8.9211 secs
  Slowest:	0.0271 secs
  Fastest:	0.0004 secs
  Average:	0.0018 secs
  Requests/sec:	11209.4022
  

Response time histogram:
  0.000 [1]	|
  0.003 [96749]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.006 [2242]	|■
  0.008 [237]	|
  0.011 [304]	|
  0.014 [288]	|
  0.016 [111]	|
  0.019 [20]	|
  0.022 [6]	|
  0.024 [2]	|
  0.027 [40]	|


Latency distribution:
  10% in 0.0011 secs
  25% in 0.0013 secs
  50% in 0.0016 secs
  75% in 0.0020 secs
  90% in 0.0024 secs
  95% in 0.0028 secs
  99% in 0.0058 secs

Before - Instance where background dispatcher was used 60 times (1 second delay)

hey -c 20 -n 10000 http://127.0.0.1:5001/fact

Summary:
  Total:	6.2497 secs
  Slowest:	1.0220 secs
  Fastest:	0.0004 secs
  Average:	0.0125 secs
  Requests/sec:	1600.0795
  

Response time histogram:
  0.000 [1]	|
  0.103 [9879]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.205 [0]	|
  0.307 [0]	|
  0.409 [0]	|
  0.511 [20]	|
  0.613 [0]	|
  0.716 [1]	|
  0.818 [39]	|
  0.920 [0]	|
  1.022 [60]	|

LambdaLB - Add example that reads binary file from S3

Motivations

Reading from S3 is slower than reading from a local file
This may reduce the performance benefits

Acceptance Criteria

Create S3 bucket
Allow LambdaLB to read from the S3 bucket
Add a route that reads the binary file from S3
Test performance

Router - Dispatcher - Exception on StartAsync call - object already disposed

Exception Summary

Microsoft.AspNetCore.Http.DefaultHttpResponse.StartAsync(CancellationToken cancellationToken)
at PwrDrvr.LambdaDispatch.Router.Dispatcher.AddConnectionForLambda(HttpRequest request, HttpResponse response, String lambdaId, String channelId)
at PwrDrvr.LambdaDispatch.Router.ChunkedController.Post(String instanceId) in /app/src/PwrDrvr.LambdaDispatch.Router/ChunkedController.cs:line 126

Exception in Router

lambda-dispatch-router-1    | 16:41:51.543 fail: Microsoft.AspNetCore.Server.Kestrel[13] Connection id "0HMVUAN471FMI", Request id "0HMVUAN471FMI:000509E5": An unhandled exception was thrown by the application. System.NullReferenceException: Object reference not set to an instance of an object.    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<FireOnStarting>g__ProcessEvents|241_0(HttpProtocol protocol, Stack`1 events)
lambda-dispatch-router-1    | 16:41:51.569 fail: PwrDrvr.LambdaDispatch.Router.ChunkedController[0] Router.ChunkedController.Post - Exception System.ObjectDisposedException: The response has been aborted due to an unhandled application exception.  ---> System.NullReferenceException: Object reference not set to an instance of an object.    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<FireOnStarting>g__ProcessEvents|241_0(HttpProtocol protocol, Stack`1 events)    --- End of inner exception stack trace ---    at Microsoft.AspNetCore.Http.DefaultHttpResponse.StartAsync(CancellationToken cancellationToken)    at PwrDrvr.LambdaDispatch.Router.Dispatcher.AddConnectionForLambda(HttpRequest request, HttpResponse response, String lambdaId, String channelId)    at PwrDrvr.LambdaDispatch.Router.ChunkedController.Post(String instanceId) in /app/src/PwrDrvr.LambdaDispatch.Router/ChunkedController.cs:line 126
lambda-dispatch-router-1    | 16:41:51.574 fail: Microsoft.AspNetCore.Server.Kestrel[13] Connection id "0HMVUAN471FMI", Request id "0HMVUAN471FMI:000509E5": An unhandled exception was thrown by the application. System.ObjectDisposedException: The response has been aborted due to an unhandled application exception.  ---> System.NullReferenceException: Object reference not set to an instance of an object.    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<FireOnStarting>g__ProcessEvents|241_0(HttpProtocol protocol, Stack`1 events)    --- End of inner exception stack trace ---    at Microsoft.AspNetCore.Http.DefaultHttpResponse.StartAsync(CancellationToken cancellationToken)    at PwrDrvr.LambdaDispatch.Router.Dispatcher.AddConnectionForLambda(HttpRequest request, HttpResponse response, String lambdaId, String channelId)    at PwrDrvr.LambdaDispatch.Router.ChunkedController.Post(String instanceId) in /app/src/PwrDrvr.LambdaDispatch.Router/ChunkedController.cs:line 126    at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location ---    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Extension - Allow gzip compression of responses

Motivations

Compressing response bytes should be optional
Compressing response bytes in the Lambda minimizes the number of bytes transmitted across AZs and handled by the Router (reducing CPU usage and ENI bandwidth consumption)

References

Based on PR for aws-lambda-web-adapter:
- awslabs/aws-lambda-web-adapter#157
aws-lambda-web-adapter switched to using the Tower Compression layer which appears to support streaming:
- https://docs.rs/tower-http/latest/tower_http/compression/

Extension - Startup as Extension, register with Extension API, then register with Runtime API

Motivations

Lambda Web Adapter does this
This allows the app to start themselves by specifying CMD in the Dockerfile or ENTRYPOINT if it's an AWS Lambda base image
This removes the need for child process management

Acceptance Criteria

Add routes to LambdaWebTestTool
Add extension starting
Make app starting optional
Test deployed on Lambda

Closing message causes exception

Overview

The closing messages cause an exception because we've already set the status to 200
See if we can use a trailer (header at end of HTTP2 message) to indicate that it's actually a 409
Discarding connection for X-Lambda-Id

Error Message

04:58:44.217 fail: PwrDrvr.LambdaDispatch.LambdaLB.HttpReverseRequester[0]
      => LambdaId: e57820b8-cb54-43f5-be21-8a126db443d9 => TaskNumber: 1 => ChannelId: fd6ffbf2-a04b-4dce-88ca-a2ec7f5b378f
      Error reading request from response
      System.Exception: Invalid request line: Discarding connection for X-Lambda-Id: e57820b8-cb54-43f5-be21-8a126db443d9, X-Channel-Id: fd6ffbf2-a04b-4dce-88ca-a2ec7f5b378f, closing
         at PwrDrvr.LambdaDispatch.LambdaLB.HttpReverseRequester.<GetRequest>d__8.MoveNext() + 0xb94
04:58:44.217 info: PwrDrvr.LambdaDispatch.LambdaLB.Function[0]
      => LambdaId: e57820b8-cb54-43f5-be21-8a126db443d9 => TaskNumber: 1
      Exiting task

Router - Local Dev Fast Ctrl-C Handling

Motivations

Currently the router hangs on Ctrl-C for up to a minute
There is no cancellation token to cancel all long-running tasks gracefully
Note: graceful exit under ECS works fine because ECS is deregistering tasks with the ALB before shutting down the containers

Overview

Add signal handlers that signal a cancellation token when other processes have wrapped up
Start failing health checks after receiving SIGTERM
Wait a bit then signal the cancel token to break out of the scheduled loops like metrics, queue dispatcher, and rebalancer

Fix 10-15% CPU usage in queued request dispatch

Motivations

The Router has been observed to use 10-15% CPU steadily with no requests
The issue does not happen on startup, but only after a request
The loop is in the queued request dispatch code in the part that checks every 10 ms to see if it can dispatch the request that should be in the queue

Acceptance Criteria

Stop the looping every 10 ms

LambdaLB - Exit before the Lambda runs out of time

Motivations

The Lambda knows how long it has before it will be terminated for timing out
We don't want any terminated requests because that terminates the execution env and we not only lose requests we also pay for another cold start

Acceptance Criteria

Check how much time we have before termination and send the close request at the offsets to that time below
Log a warning if under 1 minute timeout on startup
< 1 minute: 5 seconds
1 minute exactly: 15 seconds
<= 10 minutes: 30 seconds
<= 15 minutes: 60 seconds

Debugger.Break - Package up the Debugger.Break changes

Motivations

It appears that Debugger.Break just is not supported on Mac/Linux
Figuring out how to set this up is not something everyone should have to repeat

Acceptance Criteria

Package up the changes
Confirm that the changes are useful and that the existing functionality is not sufficient
Provide 3 options: press any key if attached to console, spin until debugger attached (if debugger attached logic is working), or potentially display a popup if able
Write up an issue report
Submit a draft PR

Code

#if false
            // Insert a call to raise(SIGTRAP) here
            var sw = new System.Diagnostics.Stopwatch();

            // Debugger.IsAttached returns immediately on Mac OS X
            while (!Debugger.IsAttached)
            {
                sw.Restart();
                System.Console.WriteLine("Waiting for debugger to attach...");
                while (sw.ElapsedMilliseconds < 1000); // Spin for 1 second
            }
#else
            System.Console.WriteLine("Press any key once the debugger is attached...");
            System.Console.ReadKey();
            System.Console.WriteLine("Trying Debugger.Break()...");
            System.Diagnostics.Debugger.Break();
            System.Console.WriteLine("Break here if the debugger didn't break...");
#endif
            // coreclr debugger cannot catch SIGTRAP (5) on Mac OS X
            // DebugHelper.raise(5);

Router/LambdaLB - Fix Very High CPU Usage

Motivations

Both the Router and the LambdaLB were using high CPU
- 400-500% CPU each
- Node.js "server" was using ~50% CPU
The flame charts showed the time was being spent in System.Private.CoreLib!System.Threading.Tasks.Task.SpinThenBlockingWait(int32,value class System.Threading.CancellationToken)

Acceptance Criteria

Find a way to reduce the CPU usage of the Router and LambdaLB when processing a high rate of requests

Issue Report and Work-Around

dotnet/runtime#72153 (comment)

Set DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0 to a value such as 5 or 10, much lower than the default of ~50. This cuts the CPU usage from 400-500% down to 100%.

Router - Close Extension Channel when Incoming Request Abnormally Ends

Motivations

The incoming request into the Router can throw an exception
When that incoming request throws (typically on Read of Request or Write of Response bytes), there is nothing that will automatically abort the request up to the extension
If the request to the extension is not aborted then the extension will hang
When the extension hangs it prevents new requests from being dispatched to it and it prevents it from being replaced with a good instance until it times out
This was observed primarily with the Rust extension, but the issue is actually in the Router
This is typically observed when oha aborts the requests when using the -z 10s parameter

Acceptance Criteria

Catch exception on incoming request/response in Router and abort LambdaConnection
Reduce log verbosity in the case of an incoming request being aborted

Extension - Rewrite in `rust`, test performance, release

Motivations

The extension within the Lambda is the most performance sensitive: it's using CPU that the Lambda could use to perform tasks
The 400% CPU usage of dotnet due to inefficient workers looking for work is having a negative impact on performance
A simple demo showed that Node.js can be pinned at 110% CPU with Rust at < 30% CPU for sending the requests

To-Do

Create debug feature in Router to allow an extension to connect and have its LambdaInstance lazily created
Write a demo to confirm that duplex recv/send works on HTTP2 requests for Rust/hyper
Handle GET requests without Lambda Invoke (use debug feature above)
Run a performance comparison
Implement POST requests
Gracefully handle errors
- This is mostly complete
Implement duplex requests for caller requests (or new ticket)
Degenerate case of /ping with 1 channel from Router to Extension but 10 concurrent requests into the Router from oha, is running at 1/4 to 1/3rd the speed of the dotnet extension (e.g. 1k to 1.3k RPS vs 3k RPS)
- This is fixed in #84

Results

Test Case	oha `-c`	Channels Router <--> Extension	Rust RPS	Rust CPU	Rust Node.js CPU	DotNet RPS	DotNet CPU	DotNet Node.js CPU	Direct Node.js RPS	Direct Node.js CPU
`/ping`	10	10	19,007	90%	113%	14,652	330-350%	95%	20,035	115%
`/ping`	10	1	4,934	27%	36%	3,603	115%	30%	^^^	^^^
`/ping`	1	10	4,997	29%	37%	3,451	111%	27%	15,975	97%
`/echo` 9.3 MB	10	1	81	76%	113%	N/A	N/A	N/A	228	179%
`/echo-slow` 9.3 MB	10	1	54	52%	150%	51	154%	147%	117	158%
`/echo-slow` 9.3 MB	1	1	54	52%	150%	50	152%	150%	96	215%

After - Rust Extension

After - `/ping` - 10 Concurrent Requests - 10 Channels from Router to Extension

✅ Extension CPU is ~80-90% (vs Node.js at 113% or dotnet extension at ~350%) - note that the extension is doing 2x more work than Node.js in this case (it's receiving and proxying the request and response while Node.js is only receiving the request and immediately writing back a static response - this is the worst case scenario where the contained app is using no CPU at all and only shuffling requests).

Note: Tokio set to current thread

oha -c 10 -z 10s http://localhost:5001/ping
Summary:
  Success rate: 100.00%
  Total:        10.0009 secs
  Slowest:      0.0047 secs
  Fastest:      0.0002 secs
  Average:      0.0005 secs
  Requests/sec: 19007.2403

  Total data:   742.54 KiB
  Size/request: 4 B
  Size/sec:     74.25 KiB

Response time histogram:
  0.000 [1]      |
  0.001 [165746] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.001 [19340]  |■■■
  0.002 [2468]   |
  0.002 [1803]   |
  0.002 [511]    |
  0.003 [182]    |
  0.003 [19]     |
  0.004 [7]      |
  0.004 [8]      |
  0.005 [5]      |

Response time distribution:
  10.00% in 0.0004 secs
  25.00% in 0.0004 secs
  50.00% in 0.0005 secs
  75.00% in 0.0006 secs
  90.00% in 0.0006 secs
  95.00% in 0.0007 secs
  99.00% in 0.0016 secs
  99.90% in 0.0025 secs
  99.99% in 0.0034 secs


Details (average, fastest, slowest):
  DNS+dialup:   0.0019 secs, 0.0014 secs, 0.0022 secs
  DNS-lookup:   0.0005 secs, 0.0004 secs, 0.0006 secs

Status code distribution:
  [200] 190090 responses

After - `/ping` - 10 Concurrent Requests - 1 Channel from Router to Extension

✅ Extension CPU is ~27% with current thread Tokio runtime config

oha -c 10 -z 10s http://localhost:5001/ping
Summary:
  Success rate: 100.00%
  Total:        10.0005 secs
  Slowest:      0.0212 secs
  Fastest:      0.0013 secs
  Average:      0.0020 secs
  Requests/sec: 4934.2600

  Total data:   192.75 KiB
  Size/request: 4 B
  Size/sec:     19.27 KiB

Response time histogram:
  0.001 [1]     |
  0.003 [46522] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.005 [882]   |
  0.007 [557]   |
  0.009 [530]   |
  0.011 [580]   |
  0.013 [239]   |
  0.015 [12]    |
  0.017 [9]     |
  0.019 [0]     |
  0.021 [13]    |

Response time distribution:
  10.00% in 0.0016 secs
  25.00% in 0.0016 secs
  50.00% in 0.0017 secs
  75.00% in 0.0017 secs
  90.00% in 0.0020 secs
  95.00% in 0.0039 secs
  99.00% in 0.0105 secs
  99.90% in 0.0126 secs
  99.99% in 0.0199 secs


Details (average, fastest, slowest):
  DNS+dialup:   0.0008 secs, 0.0006 secs, 0.0010 secs
  DNS-lookup:   0.0002 secs, 0.0001 secs, 0.0003 secs

Status code distribution:
  [200] 49345 responses

After - `/echo` - 10 Concurrent Requests - 1 Channel from Router to Extension

Note: /echo sends/receives in duplex while /echo-slow is all that has been implemented in the dotnet Extension; this is not the fault of dotnet (as the Router handles duplex) it's just that this was not built in the Extension. This is provided for reference as an upper bound on the throughput of the Rust extension.

oha -c 10 -z 10s -m POST -T application/octet-stream -D ./oha http://localhost:5001/echo    
Summary:
  Success rate: 100.00%
  Total:        10.0018 secs
  Slowest:      0.1470 secs
  Fastest:      0.0319 secs
  Average:      0.1215 secs
  Requests/sec: 81.7855

  Total data:   7.44 GiB
  Size/request: 9.32 MiB
  Size/sec:     762.18 MiB

Response time histogram:
  0.032 [1]   |
  0.043 [0]   |
  0.055 [1]   |
  0.066 [1]   |
  0.078 [1]   |
  0.089 [1]   |
  0.101 [1]   |
  0.112 [1]   |
  0.124 [731] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.135 [79]  |■■■
  0.147 [1]   |

Response time distribution:
  10.00% in 0.1201 secs
  25.00% in 0.1208 secs
  50.00% in 0.1217 secs
  75.00% in 0.1226 secs
  90.00% in 0.1239 secs
  95.00% in 0.1247 secs
  99.00% in 0.1285 secs
  99.90% in 0.1470 secs
  99.99% in 0.1470 secs


Details (average, fastest, slowest):
  DNS+dialup:   0.0021 secs, 0.0017 secs, 0.0023 secs
  DNS-lookup:   0.0009 secs, 0.0008 secs, 0.0009 secs

Status code distribution:
  [200] 818 responses

After - `/echo-slow` - 10 Concurrent Requests - 1 Channel from Router to Extension

oha -c 10 -z 10s -m POST -T application/octet-stream -D ./oha http://localhost:5001/echo-slow
Summary:
  Success rate: 100.00%
  Total:        10.0004 secs
  Slowest:      0.2075 secs
  Fastest:      0.0421 secs
  Average:      0.1833 secs
  Requests/sec: 53.9977

  Total data:   4.91 GiB
  Size/request: 9.32 MiB
  Size/sec:     503.22 MiB

Response time histogram:
  0.042 [1]   |
  0.059 [0]   |
  0.075 [1]   |
  0.092 [1]   |
  0.108 [1]   |
  0.125 [1]   |
  0.141 [1]   |
  0.158 [1]   |
  0.174 [1]   |
  0.191 [491] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.208 [41]  |■■

Response time distribution:
  10.00% in 0.1808 secs
  25.00% in 0.1817 secs
  50.00% in 0.1828 secs
  75.00% in 0.1853 secs
  90.00% in 0.1899 secs
  95.00% in 0.1944 secs
  99.00% in 0.2033 secs
  99.90% in 0.2075 secs
  99.99% in 0.2075 secs


Details (average, fastest, slowest):
  DNS+dialup:   0.0017 secs, 0.0012 secs, 0.0021 secs
  DNS-lookup:   0.0005 secs, 0.0004 secs, 0.0006 secs

Status code distribution:
  [200] 540 responses

After - `/echo-slow` - 1 Concurrent Requests - 1 Channel from Router to Extension

oha -c 1 -z 10s -m POST -T application/octet-stream -D ./oha http://localhost:5001/echo-slow 
Summary:
  Success rate: 100.00%
  Total:        10.0002 secs
  Slowest:      0.0319 secs
  Fastest:      0.0170 secs
  Average:      0.0184 secs
  Requests/sec: 54.2988

  Total data:   4.94 GiB
  Size/request: 9.32 MiB
  Size/sec:     506.02 MiB

Response time histogram:
  0.017 [1]   |
  0.018 [393] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.020 [117] |■■■■■■■■■
  0.021 [14]  |■
  0.023 [6]   |
  0.024 [4]   |
  0.026 [3]   |
  0.027 [2]   |
  0.029 [0]   |
  0.030 [0]   |
  0.032 [3]   |

Response time distribution:
  10.00% in 0.0175 secs
  25.00% in 0.0177 secs
  50.00% in 0.0180 secs
  75.00% in 0.0185 secs
  90.00% in 0.0193 secs
  95.00% in 0.0201 secs
  99.00% in 0.0258 secs
  99.90% in 0.0319 secs
  99.99% in 0.0319 secs


Details (average, fastest, slowest):
  DNS+dialup:   0.0007 secs, 0.0007 secs, 0.0007 secs
  DNS-lookup:   0.0002 secs, 0.0002 secs, 0.0002 secs

Status code distribution:
  [200] 543 responses

Before - dotnet Extension

Before - `/ping` - 10 Concurrent Requests - 10 Channels from Router to Extension

🔴 Extension CPU is 330-350% (vs Node.js at 97%)

oha -c 10 -z 10s http://localhost:5001/ping 
Summary:
  Success rate: 100.00%
  Total:        10.0006 secs
  Slowest:      0.0984 secs
  Fastest:      0.0002 secs
  Average:      0.0007 secs
  Requests/sec: 14652.3652

  Total data:   572.39 KiB
  Size/request: 4 B
  Size/sec:     57.24 KiB

Response time histogram:
  0.000 [1]      |
  0.010 [146522] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.020 [0]      |
  0.030 [0]      |
  0.039 [0]      |
  0.049 [0]      |
  0.059 [0]      |
  0.069 [0]      |
  0.079 [0]      |
  0.089 [0]      |
  0.098 [10]     |

Response time distribution:
  10.00% in 0.0004 secs
  25.00% in 0.0005 secs
  50.00% in 0.0005 secs
  75.00% in 0.0007 secs
  90.00% in 0.0009 secs
  95.00% in 0.0013 secs
  99.00% in 0.0037 secs
  99.90% in 0.0045 secs
  99.99% in 0.0079 secs


Details (average, fastest, slowest):
  DNS+dialup:   0.0019 secs, 0.0017 secs, 0.0021 secs
  DNS-lookup:   0.0009 secs, 0.0008 secs, 0.0010 secs

Status code distribution:
  [200] 146533 responses

Before - `/ping` - 10 Concurrent Requests - 1 Channel from Router to Extension

🔴 Extension CPU is ~90%

oha -c 10 -z 10s http://localhost:5001/ping
Summary:
  Success rate: 100.00%
  Total:        10.0008 secs
  Slowest:      0.0257 secs
  Fastest:      0.0017 secs
  Average:      0.0028 secs
  Requests/sec: 3603.8977

  Total data:   140.79 KiB
  Size/request: 4 B
  Size/sec:     14.08 KiB

Response time histogram:
  0.002 [1]     |
  0.004 [34294] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.006 [1463]  |■
  0.009 [152]   |
  0.011 [66]    |
  0.014 [53]    |
  0.016 [3]     |
  0.018 [0]     |
  0.021 [2]     |
  0.023 [2]     |
  0.026 [6]     |

Response time distribution:
  10.00% in 0.0022 secs
  25.00% in 0.0023 secs
  50.00% in 0.0025 secs
  75.00% in 0.0030 secs
  90.00% in 0.0035 secs
  95.00% in 0.0040 secs
  99.00% in 0.0062 secs
  99.90% in 0.0121 secs
  99.99% in 0.0251 secs


Details (average, fastest, slowest):
  DNS+dialup:   0.0014 secs, 0.0011 secs, 0.0018 secs
  DNS-lookup:   0.0005 secs, 0.0003 secs, 0.0007 secs

Status code distribution:
  [200] 36042 responses

Before - `/echo-slow` - 10 Concurrent Requests - 1 Channel from Router to Extension

oha -c 10 -z 10s -m POST -T application/octet-stream -D ./oha http://localhost:5001/echo-slow
Summary:
  Success rate: 100.00%
  Total:        10.0009 secs
  Slowest:      0.2978 secs
  Fastest:      0.1180 secs
  Average:      0.1925 secs
  Requests/sec: 51.3954

  Total data:   4.68 GiB
  Size/request: 9.32 MiB
  Size/sec:     478.97 MiB

Response time histogram:
  0.118 [1]   |
  0.136 [0]   |
  0.154 [1]   |
  0.172 [1]   |
  0.190 [250] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.208 [234] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.226 [16]  |■■
  0.244 [8]   |■
  0.262 [1]   |
  0.280 [1]   |
  0.298 [1]   |

Response time distribution:
  10.00% in 0.1875 secs
  25.00% in 0.1886 secs
  50.00% in 0.1900 secs
  75.00% in 0.1934 secs
  90.00% in 0.1981 secs
  95.00% in 0.2112 secs
  99.00% in 0.2340 secs
  99.90% in 0.2978 secs
  99.99% in 0.2978 secs


Details (average, fastest, slowest):
  DNS+dialup:   0.0017 secs, 0.0012 secs, 0.0023 secs
  DNS-lookup:   0.0005 secs, 0.0003 secs, 0.0006 secs

Status code distribution:
  [200] 514 responses

Before - `/echo-slow` - 1 Concurrent Request - 1 Channel from Router to Extension

oha -c 1 -z 10s -m POST -T application/octet-stream -D ./oha http://localhost:5001/echo-slow
Summary:
  Success rate: 100.00%
  Total:        10.0006 secs
  Slowest:      0.0773 secs
  Fastest:      0.0175 secs
  Average:      0.0193 secs
  Requests/sec: 51.6968

  Total data:   4.71 GiB
  Size/request: 9.32 MiB
  Size/sec:     481.78 MiB

Response time histogram:
  0.017 [1]   |
  0.023 [502] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.029 [8]   |
  0.035 [4]   |
  0.041 [1]   |
  0.047 [0]   |
  0.053 [0]   |
  0.059 [0]   |
  0.065 [0]   |
  0.071 [0]   |
  0.077 [1]   |

Response time distribution:
  10.00% in 0.0182 secs
  25.00% in 0.0185 secs
  50.00% in 0.0188 secs
  75.00% in 0.0193 secs
  90.00% in 0.0199 secs
  95.00% in 0.0210 secs
  99.00% in 0.0299 secs
  99.90% in 0.0773 secs
  99.99% in 0.0773 secs


Details (average, fastest, slowest):
  DNS+dialup:   0.0005 secs, 0.0005 secs, 0.0005 secs
  DNS-lookup:   0.0001 secs, 0.0001 secs, 0.0001 secs

Status code distribution:
  [200] 517 responses

pwrdrvr / lambda-dispatch Goto Github PK

lambda-dispatch's People

Contributors

Stargazers

Watchers

Forkers

lambda-dispatch's Issues

Motivations

Motivations

Acceptance Criteria

Overview

Motivations

Acceptance Criteria

LambaLB Logs

Contained App Logs

Motivations

Acceptance Criteria

Docs

Motivations

Acceptance Criteria

Motivations

ToDo

Motivations

Acceptance Criteria

Reproducing Locally

Motivations

Motivations

Acceptance Criteria

Motivations

Acceptance Criteria

Motivations

Acceptance Criteria

Motivations

Acceptance Criteria

Motivations

Acceptance Criteria

Overview

LambdaTestTools - First invoke never executes

Log - Request Queued - Did start lambda - 03:11:21.131

Log - Choice to Start Lambda - 4 Seconds Later

Motivations

Acceptance Criteria

Motivations

To-Do

Motivations

Acceptance Criteria

Motivations

Acceptance Criteria

Overview

Overview

Motivations

Acceptance Criteria

Problem

Upstream Issue

Upstream Solution

Motivations

Acceptance Criteria

Results

After

Before

Motivations

Acceptance Criteria

Motivations

Acceptance Criteria

Motivations

Message

Motivations

Acceptance Criteria

Motivations

To-Do

Motivations

Acceptance Criteria

Motivations

Acceptance Criteria

Motivations

Acceptance Criteria

Docs

Images

Running Public Images with platform: linux/amd64

Running Public Images with platform: linux/arm64

Running Public Images with `platform: linux/amd64`

Running Public Images with `platform: linux/arm64`