Handle processors larger than 256mb on AWS lambda about artillery

[bernardobridge]

Hi @jaxoncreed 👋

That's peculiar, the processor isn't being sent through SQS, no. A couple of questions:

• Can you share what the run-lambda command initial output was? It should have had something saying Test bundle contents: detailing which files were being bundled.
• Any chance you could show your processor file and contents of your YAML? I'd like to try to replicate it.
• Have you tried going into the Cloudwatch logs for the function and seeing if there's an error there?

Note: I did try running a processor file of 2.6MB, and it ran fine. My suspicion is it doesn't have to do with the size of the file itself, but possibly some error is happening that is being sent through SQS and exceeding the message limit. I would need to be able to replicate this.

[jaxoncreed]

Can you share what the run-lambda command initial output was? It should have had something saying Test bundle contents: detailing which files were being bundled.

Here's the full output from the run-lambda command:

> artillery run-lambda --region us-east-1 --memory-size 3000 test.yml

Test run id: tbb3g_g4mra3w5yweywxt9eje7abb7qgnhn_pzkf
λ Preparing AWS Lambda function...
Test bundle contents:
┌───────────────────┬──────┬───────┐
│ Name              │ Type │ Notes │
├───────────────────┼──────┼───────┤
│ test.yml          │ file │       │
├───────────────────┼──────┤       │
│ bundle/bundle.js  │ file │       │
├───────────────────┼──────┤       │
│ package.json      │ file │       │
├───────────────────┼──────┤       │
│ package-lock.json │ file │       │
└───────────────────┴──────┴───────┘

 - Lambda function: artilleryio-v2-0-15-arm64-2d06d78a397d293c09f46482dbd993b4
 - Region: us-east-1
 - AWS account: 993748562375
[92057351-867a-45bb-9f42-ecdd09b4bffe]: A Lambda function has exited with an error. Reason: StartupFailure
{
  err: {
    message: 'One or more parameters are invalid. Reason: Message must be shorter than 262144 bytes.',
    stack: 'InvalidParameterValueException: One or more parameters are invalid. Reason: Message must be shorter than 262144 bytes.\n' +
      '    at Request.extractError (/artillery/node_modules/aws-sdk/lib/protocol/json.js:80:27)\n' +
      '    at Request.callListeners (/artillery/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n' +
      '    at Request.emit (/artillery/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n' +
      '    at Request.emit (/artillery/node_modules/aws-sdk/lib/request.js:686:14)\n' +
      '    at Request.transition (/artillery/node_modules/aws-sdk/lib/request.js:22:10)\n' +
      '    at AcceptorStateMachine.runTo (/artillery/node_modules/aws-sdk/lib/state_machine.js:14:12)\n' +
      '    at /artillery/node_modules/aws-sdk/lib/state_machine.js:26:10\n' +
      '    at Request.<anonymous> (/artillery/node_modules/aws-sdk/lib/request.js:38:9)\n' +
      '    at Request.<anonymous> (/artillery/node_modules/aws-sdk/lib/request.js:688:12)\n' +
      '    at Request.callListeners (/artillery/node_modules/aws-sdk/lib/sequential_executor.js:116:18)'
  }
}
--------------------------------
Summary report @ 11:43:12(-0400)
--------------------------------

No measurements recorded during this period

Estimated AWS Lambda cost for this test: $0.0005

---

Any chance you could show your processor file and contents of your YAML? I'd like to try to replicate it.

Sure thing! I created a temp branch for the entire project here: https://github.com/digitalcredentials/dcc-server-tester/tree/test-upload. You can find the YML here (https://github.com/digitalcredentials/dcc-server-tester/blob/test-upload/test.yml), the processor bundle here (https://github.com/digitalcredentials/dcc-server-tester/blob/test-upload/bundle/bundle.js) and the unbundled code here (https://github.com/digitalcredentials/dcc-server-tester/blob/test-upload/src/customFunctions.ts)

One thing to note is that I've also attempted to do an AWS upload with unbundled code (just using node modules) and I got the following error. So, I attempted to bundle it to reduce the memory footprint.

[b8ab3e7d-1759-4a5b-ac5c-39fd65941f65]: A Lambda function has exited with an error. Reason: TestDataSyncFailure
{
  err: {
    message: "ENOSPC: no space left on device, mkdir '/tmp/test_data/twa5r_hch4b55ppaprzjbe5c644dwkxrf6p_ehk4'",
    stack: "Error: ENOSPC: no space left on device, mkdir '/tmp/test_data/twa5r_hch4b55ppaprzjbe5c644dwkxrf6p_ehk4'\n" +
      '    at Object.mkdirSync (node:fs:1372:26)\n' +
      '    at syncTestData (/artillery/a9-handler-dependencies.js:11:8)\n' +
      '    at Runtime.handler (/artillery/a9-handler-index.js:70:11)\n' +
      '    at Runtime.handleOnceNonStreaming (file:///artillery/node_modules/aws-lambda-ric/dist/index.mjs:1289:29)'
  }
}

---

Have you tried going into the Cloudwatch logs for the function and seeing if there's an error there?

I did not. Now that I check it it looks like there are error messages that try to include the entire raw code of the bundle:

It would seem that your theory was correct. An error was thrown somewhere in the bundle (the test works locally but there might be something about the AWS environment that causes a problem), then because the bundle is all one line, it will include the entire line in the error message pointing out that error.

[bernardobridge]

Thanks for the details @jaxoncreed !

Sure thing! I created a temp branch for the entire project here:

That's great! However - I don't think the project is public, I don't seem to be able to access to it.

---

One question - in the package.json that's being used, do you have artillery installed there? It is possible that that is the reason you're running out of space in the Lambda container. Ideally, have a separate package.json (in e.g. a folder just for the tests) with only the dependencies the tests themselves need, not artillery.

---

It would seem that your theory was correct. An error was thrown somewhere in the bundle (the test works locally but there might be something about the AWS environment that causes a problem), then because the bundle is all one line, it will include the entire line in the error message pointing out that error.

Yes, it's possible we need to handle a couple of things better here. Once you give me access to your repo, I can try to replicate it!

[jaxoncreed]

oops sorry about that. Use these links:

• Repo: https://github.com/jaxoncreed/dcc-test-server
• YML: https://github.com/jaxoncreed/dcc-test-server/blob/test-upload/test.yml
• Bundle: https://github.com/jaxoncreed/dcc-test-server/blob/test-upload/bundle/bundle.js
• Raw Code: https://github.com/jaxoncreed/dcc-test-server/blob/test-upload/src/customFunctions.ts

One question - in the package.json that's being used, do you have artillery installed there? It is possible that that is the reason you're running out of space in the Lambda container. Ideally, have a separate package.json (in e.g. a folder just for the tests) with only the dependencies the tests themselves need, not artillery.

No, I manually removed all the dependencies from the package.json because I saw it was trying to upload those dependencies. Even the dev dependencies. If I'm using a bundle, it shouldn't require dependencies.

[bernardobridge]

Hey @jaxoncreed 👋

I usually try to run things locally first (before moving to Lambda), and I mistakenly ran the test without providing the npm_config_emailed_url environment variable. Sure enough, it errored where expected. However, I did notice that it errored with a very similar thing to the logs you showed above (shows the whole bundle, and then only after do we see the actual issue).

Could it be that you aren't passing that npm_config_emailed_url to the Lambda? Have you tried using the --dotenv flag with a .env file to do it, perhaps?

[jaxoncreed]

Thanks for your help. Using the --dotenv flag worked!