kong / lua-resty-aws

AWS SDK for OpenResty

lua-resty-aws

Overview

AWS SDK for OpenResty. The SDK is generated from the original AWS JavaScript repository details.

The documentation will mostly cover the specifics for this library, the actual services invoked are documented by AWS.

For a quick start on how to use this library checkout the examples of the AWS class.

---

Status

Not everything has been implemented, and testing is hard since it requires access to AWS resources and not just regular CI.

---

Example

See the example in the documentation.

---

Usage IMPORTANT!!

attempt to yield across C-call boundary error

This typically happens when initializing from within a require call. See Global settings below on how to initialize properly.

---

TLS and certificate failures

The http client defaults to tls name verification. For this to work, the CA store must be set. With OpenResty this is done through the lua_ssl_trusted_certificate directive. However; the compatibility module used, lua-resty-luasocket, cannot automatically read that setting, hence you have to set it manually, see the docs.

---

Global settings

This library depends on global settings. Especially the core services for authentication and metadata. Many of those can (also) be specified as environment variables. Environment variables can only be accessed during the OpenResty init phase. Thus, to ensure correct configuration from environment variables, the resty.aws.config module must be required on the top-level of the module using this library:

local aws_config = require("resty.aws.config")

The .global property of the aws_config variable can then be used as the global configuration. Note that when .global is first accessed, automatic region detection through the AWS metadata service is performed. Thus, it is not advisable to access it on the module level unless to avoid startup delays in non-AWS environment, caused by the requests to the metadata service timing out.

---

EC2 metadata

The endpoint for EC2 metadata can block (until timeout) if the SDK is used on a non-EC2 machine. In that case you might want to set the AWS_EC2_METADATA_DISABLED to a value different from false (which is the default).

        export AWS_EC2_METADATA_DISABLED=true

---

Installation

Installation is easiest using LuaRocks:

luarocks install lua-resty-aws

To install from the git repo:

git clone https://github.com/Kong/lua-resty-aws.git
cd lua-resty-aws
make install

Troubleshooting

MacOS has a known issue that the libexpat header file 'expat_config.h' is missing. If you run into that issue, install libexpat manually (eg. brew install expat). And then include the libexpat location when installing; luarocks install lua-resty-aws EXPAT_DIR=/path/to/expat

Details: lunarmodules/luaexpat#32

---

Development

To update the SDK version being used edit the version tag in update_api_files.sh and then run:

make dev

Make sure to run make dev to pull in the generated files. Documentation can be generated using ldoc by running:

make docs

Note that distribution is a little more complex than desired. This is because the repo does not contain all the json files pulled in from the JS sdk. This in turn means that luarocks upload cannot build a rock from the repo (because it is incomplete after just being pulled).

To work around this the make pack command actually builds a .rock file that is compatible with LuaRocks. The make upload target will upload the generated rock.

See the detailed release instructions at History.

---

Testing

Tests are executed using Busted and LuaCheck:

busted
luacheck .

or run

make test

---

To do

• Implement the request/response objects (more AWS like, currently Lua modules)
• Implement additional signatures (only V4 currently)
• Implement retries from the global config
• Additional tests for other services

---

Copyright and license

Copyright: (c) 2020-2024 Kong, Inc.

Author: Thijs Schreijer

License: Apache 2.0

---

History

Versioning is strictly based on Semantic Versioning

Release process:

1. create a release branch VERSION=x.y.z && git checkout main && git pull && git checkout -b release/$VERSION
2. update the changelog below
3. run make clean && make dev && make test && make docs
4. commit as release x.y.z
5. push the branch, create a PR and get it merged.
6. tag the release commit with the version VERSION=x.y.z && git checkout main && git pull && git tag $VERSION
7. push the tag
8. run VERSION=x.y.z make pack
9. test the created .rock file VERSION=x.y.z && luarocks install lua-resty-aws-$VERSION-1.src.rock
10. upload using: VERSION=x.y.z APIKEY=abc... make upload
11. test installing the rock from LuaRocks

1.5.3 (02-Aug-2024)

• fix: build the request body based on payload field 126

1.5.2 (29-Jul-2024)

• fix: fix sts regional endpoint injection under several cases 123

1.5.1 (20-Jun-2024)

• fix: when a "blob" type has no location specified, then use it as the body, same as with other types. 120

1.5.0 (20-May-2024)

• feat: decode AWS api response json body with array metatable 114

• fix: do not inject region info for sts service with VPC endpoint hostname 113

1.4.1 (19-Apr-2024)

• fix: patch expanduser function to be more friendly to OpenResty environment 111

1.4.0 (20-Mar-2024)

• fix: aws configuration cannot be loaded due to pl.path cannot resolve the path started with ~ 94
• fix: fix the bug of missing boolean type with a value of false in the generated request body 100
• security: remove the documentation entry that contains a sample access key from AWS SDK. This avoids false postive vulnerability report. 102
• feat: container credential provider now supports using auth token defined in AWS_CONTAINER_AUTHORIZATION_TOKEN and AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE. 107
• fix: operations without inputs (eg, some S3 ones) would cause errors to be thrown 108

1.3.6 (25-Dec-2023)

• fix: validator failure for some of the field types 95

1.3.5 (19-Sep-2023)

• fix: lazily initialize structures to avoid c-boundary errors on require 87

1.3.4 (13-Sep-2023)

• fix: remove more module-level uses of config.global 83

1.3.3 (13-Sep-2023)

• fix: don't invoke region detection code on the module toplevel and advise against trying to. 81

1.3.2 (13-Sep-2023)

• fix: unsigned request should support network related config option 79

1.3.1 (17-Aug-2023)

• fix: fix v4 signing request should correctly canonicalized query table as well 76

1.3.0 (15-Aug-2023)

• fix: fix AWS_CONTAINER_CREDENTIALS_FULL_URI parsing. #65
• feat: support configure timeout on service request. #67
• feat: support configure keepalive idle time on service request connection. #67
• feat: support configure ssl verify on service request. #67
• feat: add http/https proxy support for service request #69
• fix: fix proxy-related global config var name to lowercase. #70
• feat: EC2 metadata credential provider support IMDSv2 #71

1.2.3 (20-Jul-2023)

• fix: fix assumeRole function name on STS. #59
• fix: fix STS regional endpoint injection in build_request #62
• fix: replace deprecated pl.xml with luaexpat; fix STS assume role logic. #61

1.2.2 (2-May-2023)

• fix: add the SharedFileCredentials into rockspec so it can be packed and used correctly. #53
• fix: the field idempotencyToken should be allowed and remain unvalidated as an opaque string. #52

1.2.1 (24-Apr-2023)

• fix: fix the rds signer cannot be used in init phase. #50

1.2.0 (1-Mar-2023)

• IMPORTANT-IMPORTANT-IMPORTANT feat: enable TLS name verification. This might break if your CA store is not the default system one. See usage notes. #47
• fix: STS regional endpoints woudl re-inject the region on every authentication (after a token expired), causing bad hostnames to be used #45
• Feat: add RDS.Signer to generate tokens for RDS DB access #44

1.1.2 (7-Dec-2022)

• fix: auto detection scheme and default to tls #42

1.1.1 (21-Nov-2022)

• fix: port is repeated when port is not standard #39

1.1.0 (18-Nov-2022)

• fix: template handling of query string #36
• fix: blob param should be in raw body #36
• feat: support for credential from file #36
• fix: escaping for param in uri #36
• fix: handling raw body conflict with body param #36
• fix: crash when no type check designated #36
• fix: support for "headers" location in API template #36
• fix: support new API format (bucket in host) for S3 #36

1.0.1 (20-Oct-2022)

• fix: for some method incorrect URL is generates because of incorrect handling of "+" in URL template #34

1.0.0 (13-Oct-2022)

• fix: latest doesn't indicate the most recent service version #28

0.5.5 (26-Sep-2022)

• fix: variable names for ECS Conatiner Metatdata were missing an '_' #26

0.5.4 (19-Aug-2022)

• chore: remove error message when no region is found during config initialization #24

0.5.3 (19-Aug-2022)

• feat: lazy load API modules #23

0.5.2 (12-Jul-2022)

• fix: relax validation to not validate some generic metadata fields. Encountered while trying to use Lambda #21
• fix: better error handling when credential providers fail to load #22

0.5.1 (01-Jun-2022)

• feat: socket compatibility; overriding luasocket use in phases now returns the existing setting

0.5.0 (01-Jun-2022)

• feat: enable use of regional STS endpoints
• deps: bumped the lua-resty-http dependency to 0.16 to disable the warnings and use the better connection building logic.
• fix: added sock:settimeouts to the socket compatibility layer.
• feat: implement a config object based on AWS CLI configuration.
  • for most use cases it will now suffice to load the config in the init phase since it caches al predefined environment variables.
  • BREAKING: getting EC2 credentials will now honor AWS_EC2_METADATA_DISABLED. Behaviour might change, but is expected to be very rare.
  • BREAKING: The TokenFileWebIdentityCredentials will honor the role_session_name setting (file or env) as default name. Behaviour might change, but is expected to be very rare.

0.4.0 (06-Dec-2021)

• feat: added TokenFileWebIdentityCredentials. This adds default IAM credentials to be picked up on EKS. The default AWS instance creates a CredentialProviderChain which includes TokenFileWebIdentity. So on EKS it will now pick up container based credentials instead of falling back to the underlying (more coarse) EC2 credentials.
• fix: for 'query' type calls, add target action and version, which are required
• fix: allow for unsigned requests for services requiring that (STS)
• fix: do not validate patterns as regexes are incompatible

0.3 (02-Sep-2021)

• feat: capability to fetch metadata for ECS tasks (EC2 & Fargate), versions 2, 3, and 4
• feat: capability to fetch IMDS metadata (EC2 & EKS), versions 1, and 2
• feat: automatic region detection, check the docs for details (utils module)
• fix: EC2MetadataCredentials no longer reuses the http-client to prevent issues with the underlying compatibility layer.

0.2 (05-Aug-2021)

• fix: rockspec, add Penlight dependency
• fix: add proper json Content-Type header from meta-data
• fix: use proper signingName for the signature

0.1 (03-Feb-2021) Initial released version

lua-resty-aws's Issues

Repeated region in STS Regional Endpoints for TokenFileWebIdentity

Summary

When using the TokenFileWebIdentity, the data plane will successfully initialize with a token, but when a request is sent to refresh the token, an additional region is added to the hostname (Ex: sts.us-west-2.us-west-2.amazonaws.com). Rapid succession requests will continue to add regions to the host name as well (Ex: sts.us-west-2.us-west-2.us-west-2.us-west-2.us-west-2.amazonaws.com). This causes a DNS resolution error and the data plane is no longer authenticated with AWS.

The generate_service_methods will add additional regions if the region currently exists in the hostname, but I'm not certain that is causing the issue given that the function is called in init.lua and the class is only initialized once with no config passed as an argument.

Steps to Reproduce

1. Initialize plugin with TokenFileWebIdentity
2. Wait for token to expire (about an hour)
3. Send requests to routes using the AWS SDK
4. DNS Resolution error will appear with the additional regions in the hostname

Dynamodb get item nil

Im trying to fetch from dynamodb, but data keeps coming back as nil.

locat host = "example.com"
local result, err = dynamoClient:getItem({
    Key = {
        id = {
            S = host
        }
    },
    TableName = "tablename"
})
ngx.log(ngx.ERR, "err: ", err)
ngx.log(ngx.ERR, "result response: ", result.Item)

Its hard to debug as there is no error, but there is no data returned, so perhaps in the input is incorrect?

Im not familiar with this version of the dynamodb API, or lua, here is how i tested the data was present in Go

package main

import (
	"context"
	"fmt"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/dynamodb"
	"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)

func main() {
	ctx := context.Background()
	cfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		panic(err)
	}

	svc := dynamodb.NewFromConfig(cfg)

	resp, err := svc.GetItem(ctx, &dynamodb.GetItemInput{
		Key: map[string]types.AttributeValue{
			"id": &types.AttributeValueMemberS{
				Value: "example.com",
			},
		},
		TableName: aws.String("tablename"),
	})

	if err != nil {
		panic(err)
	}

	fmt.Println(resp.Item["some_field"].(*types.AttributeValueMemberS).Value)
}

Is there somewhere i can look at debug logs from the API request to dynamo so i can see if the input is wrong?

How to use with OpenResty?

I've been trying to install lua-resty-aws into a docker container using the latest openresty/openresty:focal image and then import into an nginx.conf file to test that I can an AWS service with no avail.

Dockerfile

FROM openresty/openresty:focal

RUN apt-get update \
    && apt-get install -y \
       sudo \
    && luarocks install lua-resty-aws

Dockerfile build log

Building nginx
Step 1/5 : FROM openresty/openresty:focal
 ---> 5bb3b7099d7a
Step 2/5 : RUN apt-get update     && apt-get install -y        sudo     && luarocks install lua-resty-aws
 ---> Running in c1358eec8f10
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1356 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [777 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1032 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [411 kB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [925 kB]
Fetched 4829 kB in 3s (1437 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  sudo
0 upgraded, 1 newly installed, 0 to remove and 6 not upgraded.
Need to get 514 kB of archives.
After this operation, 2257 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 sudo amd64 1.8.31-1ubuntu1.2 [514 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 514 kB in 1s (350 kB/s)
Selecting previously unselected package sudo.
(Reading database ... 14766 files and directories currently installed.)
Preparing to unpack .../sudo_1.8.31-1ubuntu1.2_amd64.deb ...
Unpacking sudo (1.8.31-1ubuntu1.2) ...
Setting up sudo (1.8.31-1ubuntu1.2) ...
Installing https://luarocks.org/lua-resty-aws-0.1.0-1.src.rock

No existing manifest. Attempting to rebuild...
lua-resty-aws 0.1.0-1 is now installed in /usr/local/openresty/luajit (license: Apache 2.0)

nginx.conf

    init_by_lua_block {
        aws = require('resty.aws')
    }

Error

nginx_1          | 2021/06/29 07:30:38 [error] 1#1: init_by_lua error: ...sty/luajit/share/lua/5.1/resty/aws/request/http/http.lua:14: module 'resty.http' not found:
nginx_1          | 	no field package.preload['resty.http']
nginx_1          | 	no file '/usr/local/openresty/site/lualib/resty/http.ljbc'
nginx_1          | 	no file '/usr/local/openresty/site/lualib/resty/http/init.ljbc'
nginx_1          | 	no file '/usr/local/openresty/lualib/resty/http.ljbc'
nginx_1          | 	no file '/usr/local/openresty/lualib/resty/http/init.ljbc'
nginx_1          | 	no file '/usr/local/openresty/site/lualib/resty/http.lua'
nginx_1          | 	no file '/usr/local/openresty/site/lualib/resty/http/init.lua'
nginx_1          | 	no file '/usr/local/openresty/lualib/resty/http.lua'
nginx_1          | 	no file '/usr/local/openresty/lualib/resty/http/init.lua'
nginx_1          | 	no file '/usr/local/openresty/site/lualib/resty/http.ljbc'
nginx_1          | 	no file '/usr/local/openresty/site/lualib/resty/http/init.ljbc'
nginx_1          | 	no file '/usr/local/openresty/lualib/resty/http.ljbc'
nginx_1          | 	no file '/usr/local/openresty/lualib/resty/http/init.ljbc'
nginx_1          | 	no file '/usr/local/openresty/site/lualib/resty/http.lua'
nginx_1          | 	no file '/usr/local/openresty/site/lualib/resty/http/init.lua'
nginx_1          | 	no file '/usr/local/openresty/lualib/resty/http.lua'
nginx_1          | 	no file '/usr/local/openresty/lualib/resty/http/init.lua'
nginx_1          | 	no file './resty/http.lua'
nginx_1          | 	no file '/usr/local/openresty/luajit/share/luajit-2.1.0-beta3/resty/http.lua'
nginx_1          | 	no file '/usr/local/share/lua/5.1/resty/http.lua'
nginx_1          | 	no file '/usr/local/share/lua/5.1/resty/http/init.lua'
nginx_1          | 	no file '/usr/local/openresty/luajit/share/lua/5.1/resty/http.lua'
nginx_1          | 	no file '/usr/local/openresty/luajit/share/lua/5.1/resty/http/init.lua'
nginx_1          | 	no file '/usr/local/openresty/site/lualib/resty/http.so'
nginx_1          | 	no file '/usr/local/openresty/lualib/resty/http.so'
nginx_1          | 	no file '/usr/local/openresty/site/lualib/resty/http.so'
nginx_1          | 	no file '/usr/local/openresty/lualib/resty/http.so'
nginx_1          | 	no file './resty/http.so'
nginx_1          | 	no file '/usr/local/lib/lua/5.1/resty/http.so'
nginx_1          | 	no file '/usr/local/openresty/luajit/lib/lua/5.1/resty/http.so'
nginx_1          | 	no file '/usr/local/lib/lua/5.1/loadall.so'
nginx_1          | 	no file '/usr/local/openresty/luajit/lib/lua/5.1/resty/http.so'
nginx_1          | 	no file '/usr/local/openresty/site/lualib/resty.so'
nginx_1          | 	no file '/usr/local/openresty/lualib/resty.so'
nginx_1          | 	no file '/usr/local/openresty/site/lualib/resty.so'
nginx_1          | 	no file '/usr/local/openresty/lualib/resty.so'
nginx_1          | 	no file './resty.so'
nginx_1          | 	no file '/usr/local/lib/lua/5.1/resty.so'
nginx_1          | 	no file '/usr/local/openresty/luajit/lib/lua/5.1/resty.so'
nginx_1          | 	no file '/usr/local/lib/lua/5.1/loadall.so'
nginx_1          | 	no file '/usr/local/openresty/luajit/lib/lua/5.1/resty.so'
nginx_1          | stack traceback:
nginx_1          | 	[C]: in function 'require'
nginx_1          | 	...sty/luajit/share/lua/5.1/resty/aws/request/http/http.lua:14: in main chunk
nginx_1          | 	[C]: in function 'require'
nginx_1          | 	...resty/luajit/share/lua/5.1/resty/aws/request/execute.lua:1: in main chunk
nginx_1          | 	[C]: in function 'require'
nginx_1          | 	.../local/openresty/luajit/share/lua/5.1/resty/aws/init.lua:7: in main chunk
nginx_1          | 	[C]: in function 'require'
nginx_1          | 	init_by_lua:2: in main chunk

openresty use ssm service Increased memory

openretsy: 1.21.4.1

When we use kong to initialize the ssm service object and call the getparameters interface, the more parameters, the greater the memory consumption. code is as follows:

local ssm_kong -- Define global SSM variable

local function init_ssm_by_kong()
if ssm_kong then
logger:error("Reusing the existing AWS SSM")
return ssm_kong
end
logger:error("Initializing AWS SSM")
local AWS_global_config = resty_aws_config.global
local config_aws = { region = AWS_global_config.region }
local aws_result, aws = pcall(resty_aws, config_aws)
if not aws_result then
logger:error("Failed to initialize AWS: %s", aws)
return aws, nil
end
local ssm_result, new_ssm = pcall(aws.SSM, aws, { region = AWS_global_config.region })
if not ssm_result then
logger:error("Failed to initialize AWS SSM: %s", new_ssm)
return new_ssm, nil
end
ssm_kong = new_ssm -- Save the newly initialized SSM to global variable
logger:error("Successfully initialized AWS SSM")
return ssm_kong
end

local function get_ssm_parameters_by_kong(names, with_decryption)
local local_ssm = init_ssm_by_kong()
local response, err = local_ssm:getParameters{
Names = names,
WithDecryption = with_decryption,
}
return {
response = response,
err = err,
}
end

but the original aws-sdk method will not affect the memory. code is as follows:

local function get_ssm_parameters(region, endpoint, names, with_decryption)
-- protect ENVs with older IMGs
if not LOADED_LIBS["aws-sdk.ssm"] then
logger:error("Failed to load aws-sdk.ssm, reason %s", sdk_conf)
return
end
sdk_ssm.init{ region = region, endpoint_override = endpoint }
local request = sdk_ssm.GetParametersRequest{
Names = names,
WithDecryption = with_decryption,
}
local _response, _error_type, _error_message
sdk_ssm.GetParametersAsync(request, function (response, error_type, error_message)
logger:debug ("%s", {response, error_type, error_message})
_response = response
_error_type = error_type
_error_message = error_message
end)
return {
response = _response,
error_type = _error_type,
error_message = _error_message,
}
end

I don’t know why. Can you help me?

Can you provide an example of getting the aws s3 object using the TokenFileWebIdentityCredentials approach?

Can you provide an example of getting the aws s3 object using the TokenFileWebIdentityCredentials approach?

HostPrefix issue

Some endpoints have hostPrefix and it's necessary to include it into host such as SFN:startSyncExecution.

"endpoint": {
        "hostPrefix": "sync-"
}

Nice work.

This looks really impressive.

Can you give an proper example for any service method

regional-sts branch ssl issue

I was digging a bit deeper into this issue and I wanted to share what I had found:
https://github.com/Kong/lua-resty-aws/blob/regional-sts/src/resty/aws/request/execute.lua#L25

verified I was getting the same error with ssl and lua-resty-aws and lua-resty-secrets:

AWS_REGION=eu-west-1 KONG_PASSWORD=password /usr/local/bin/kong migrations bootstrap -c /etc/kong/kong.conf  --vv
....
....
....
2022/05/26 14:33:35 [debug] lua_ssl_trusted_certificate = {"/etc/pki/tls/certs/ca-bundle.crt"}
....
....
....
2022/05/26 14:33:35 [verbose] generating trusted certs combined file in
2022/05/26 14:33:36 [warn] [Penlight 1.12.0] the contents of module 'pl.xml' has been deprecated, please use a more specialized library instead (deprecated after 1.11.0, scheduled for removal in 2.0.0)
2022/05/26 14:33:36 [error] 21988#0: *2 [lua] init.lua:285: update(): [Secrets aws-sm] failed to fetch secrets: failed to retrieve secret 'arn:aws:secretsmanager:eu-west-1:412431539555:secret:postgres_credentials_secret-fun20r-S0MVRL' (stage 'AWSCURRENT'): SecretsManager:getSecretValue() failed to connect to 'https://secretsmanager.eu-west-1.amazonaws.com:443': 20: unable to get local issuer certificate, context: ngx.timer
Error:
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:227: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: ["failed to retrieve secret 'arn:aws:secretsmanager:eu-west-1:412431539555:secret:postgres_credentials_secret-fun20r-S0MVRL' (stage 'AWSCURRENT'): SecretsManager:getSecretValue() failed to connect to 'https:\/\/secretsmanager.eu-west-1.amazonaws.com:443': 20: unable to get local issuer certificate"]
stack traceback:
        [C]: in function 'assert'
        /usr/local/share/lua/5.1/kong/cmd/migrations.lua:227: in function 'cmd_exec'
        /usr/local/share/lua/5.1/kong/cmd/init.lua:97: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:97>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/kong/cmd/init.lua:97: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:54>
        /usr/local/bin/kong:9: in function 'file_gen'
        init_worker_by_lua:50: in function <init_worker_by_lua:48>
        [C]: in function 'xpcall'
        init_worker_by_lua:57: in function <init_worker_by_lua:55>

I then confirmed that all the system certificates in on my test servers / containers could be used to call the https://secretsmanager.eu-west-1.amazonaws.com endpoint without causing an ssl error

curl --cacert /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem -i https://secretsmanager.eu-west-1.amazonaws.com
HTTP/1.1 404 Not Found
x-amzn-RequestId: b31461a4-9bb9-4ba1-8eb5-ad98ebb8fa46
Content-Length: 29
Date: Thu, 26 May 2022 09:03:49 GMT

<UnknownOperationException/>

I then fixed the control plane by hard coding the database password and username in the kong.conf, now it bootstrapped fine

and I could log in, this was interesting as by logging in I was using the more secret references stored in aws secrets manager for the oidc plugin

admin_gui_auth_conf      = {"verify_parameters":false,"auth_methods":["authorization_code","password"],"authenticated_groups_claim":["roles"],"client_id":["ext:{\"type\":\"aws-sm\",\"key\":\"client_id\",\"config\":{\"SecretId\":\"arn:aws:secretsmanager:***************\"}}"],"client_secret":["ext:{\"type\":\"aws-sm\",\"key\":\"client_secret\",\"config\":{\"SecretId\":\"arn:aws:secretsmanager:********************************\"}}"],"consumer_by":["username"],"admin_claim":"name","issuer":"https://login.microsoftonline.com/43fa6b04-e14c-4e06-be5b-758ec24308d5/v2.0/.well-known/openid-configuration","login_redirect_uri":["https://kong-steveb-wpb-aws-fun20r.kong-cx.com/"],"logout_methods":["GET","DELETE","POST"],"logout_query_arg":"logout","logout_redirect_uri":["https://kong-steveb-wpb-aws-fun20r.kong-cx.com/"],"redirect_uri":["https://kong-steveb-wpb-aws-fun20r.kong-cx.com/"],"scopes":["openid","profile","email","ext:{\"type\":\"aws-sm\",\"key\":\"scopes\",\"config\":{\"SecretId\":\"arn:aws:secretsmanager:*********************************************\"}}"],"ssl_verify":false,"http_proxy":"http://ip-10-99-0-236.eu-west-1.compute.internal:3128","https_proxy":"http://ip-10-99-0-236.eu-west-1.compute.internal:3128"}

this means that after the bootstrap there are no ssl issues when retrieving other control plane secrets.

I also then tested the data planes, these use the aws secrets manager for splunk tokens - the data planes all work without issue.

The only time this SSL issue occurs is during the bootstrap of Kong.

hopefully this helps somewhat in trying to get to the bottom of the issue.

could you provide an example to execute a lambda

Hello,

could you provide an example on how to execute a lambda and return the result of this one ?

thanks

New bug in v1.4.0-1, path.lua:501: attempt to concatenate a nil value

/usr/local/share/lua/5.1/pl/path.lua:501: attempt to concatenate a nil value
stack traceback:
/usr/local/share/lua/5.1/pl/path.lua:501: in function 'expanduser'
/usr/local/share/lua/5.1/resty/aws/config.lua:240: in function 'load_config'
/usr/local/share/lua/5.1/resty/aws/config.lua:289: in function 'get_config'
/usr/local/share/lua/5.1/resty/aws/config.lua:350: in function '__index'

Hi this is a new issue we have encountered in your recent update 1.4.0. Seems to be caused by the change at resty/aws/config.lua:240, where you call pl_path.expanduser(env_vars.AWS_CONFIG_FILE.value).

Regards
Rushikesh

Error when issuing a KMS sign request.

I am trying to issue a KMS sign request but I am unable get it to work. The request looks like so

  local sign_response, sign_error = kms:sign({
    KeyId = conf.signing_kms_key_id,
    Message = table.concat(jwt_contents, "."),
    SigningAlgorithm = signing_algorithm,
    MessageType = "RAW",
  })

The error returned is as follows

[kong] init.lua:359 [capabilities] /usr/local/share/lua/5.1/resty/aws/request/build.lua:213: raw body set while parameters need to be encoded in body, client: 172.21.0.1, server: kong, request: "GET /debug HTTP/1.1", host: "localhost:8000"

As far as I can tell, the problem is because Message parameter has the shape of PlainTextType which has a type of blob.

The build_request function checks the type and sets the body to the value of message, because of the type of "blob" here. The other parameters are added to the body_tbl here to be encoded as json.

The assert found here then fails because there's a value in the body & parameters to still be serialized as json.

Is there some other way to encode these parameters so it works? or am I off base on what seems to be the issue?

Does this module supports general AWS API or ParameterStore?

Does this module supports general AWS API or ParameterStore?
Thank you.

SDK_VERSION_TAG

SDK_VERSION_TAG variable hardcoded in update_api_files and it's old ! I think you need to change it with environment variable or get it from rockspec

Dependency on deprecated Penlight XML (pl.xml) module

This module brings in pl.xml in order to parse an XML response from an STS call.

Since it's only this one code path, I wonder if we can just drop the XML parse operation instead of trying to find another suitable XML library. It's been so long since I've played with any of the AWS APIs directly though, and it's a tough one to google for--any idea if the STS API is capable of returning a json response?

I am trying to use this library in Kong custom plugin

I am trying to use this library in Kong custom plugin, it is not working, AWS class itself cannot be instantiated, I am getting a not yielding error in the constructor of the class.

Lua(): /usr/local/share/lua/5.1/resty/aws/init.lua:498: attempt to yield across C-call boundary
stack traceback:
	[C]: in function 'connect'
	/usr/local/share/lua/5.1/resty/http_connect.lua:232: in function 'connect'
	/usr/local/share/lua/5.1/resty/http.lua:927: in function 'request_uri'
	/usr/local/share/lua/5.1/resty/aws/utils.lua:29: in function 'make_request'
	/usr/local/share/lua/5.1/resty/aws/utils.lua:77: in function 'getIDMSMetadata'
	/usr/local/share/lua/5.1/resty/aws/utils.lua:221: in function 'detect_region'
	/usr/local/share/lua/5.1/resty/aws/utils.lua:265: in function 'detect_region_locked'
	/usr/local/share/lua/5.1/resty/aws/utils.lua:295: in function 'getCurrentRegion'
	/usr/local/share/lua/5.1/resty/aws/config.lua:336: in function '__index'
	...ua/5.1/resty/aws/credentials/CredentialProviderChain.lua:11: in main chunk
	[C]: at 0xffff95e810a8
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/resty/aws/init.lua:474: in function 'AWS'
	/kong-plugin/kong/plugins/kong-cert-manager/handler.lua:52: in function </kong-plugin/kong/plugins/kong-cert-manager/handler.lua:50>
	[C]: in function 'pcall'
	/usr/local/share/lua/5.1/resty/timerng/job.lua:274: in function 'execute'
	/usr/local/share/lua/5.1/resty/timerng/thread/worker.lua:169: in function </usr/local/share/lua/5.1/resty/timerng/thread/worker.lua:153>
	[C]: in function 'pcall'
	/usr/local/share/lua/5.1/resty/timerng/thread/loop.lua:101: in function 'phase_handler_wrapper'
	/usr/local/share/lua/5.1/resty/timerng/thread/loop.lua:144: in function 'do_phase_handler'
	/usr/local/share/lua/5.1/resty/timerng/thread/loop.lua:170: in function </usr/local/share/lua/5.1/resty/timerng/thread/loop.lua:162>, context: ngx.timer

What I understood is that this library uses xpcall, and it will not yield, I checked few posts online, people are suggesting to move to pcall. Please let me know how to resolve this issue, I can contribute code for any fix required.

localhost endpoint doesn't work for AWS:SQS

I'm trying to connect to SQS in local development by localstack, so I successfully setup a local SQS and it can send/receive message by command but it got some problem when I connect to SQS via lua-resty-aws

local instance = aws:SQS({
  apiVersion="2012-11-05",
  accessKeyId="test",
  secretAccessKey="test",
  region="eu-central-1",
  endpoint="http://localhost:4566"
})

local results, err = instance:getQueueUrl({
    QueueName = "test-queue"
})

print(results) // got nil
print(err) // got SQS:getQueueUrl() failed to connect to 'http://localhost:4566:80': no resolver defined to resolve "localhost:4566"

Look above error message, somehow lua-resty-aws can't resolve endpoint properly? Above similar code is work well in Javascript version:

const SQS = require('aws-sdk/clients/sqs');
const sqs = new SQS({
   apiVersion: "2012-11-05",
   accessKeyId: "test",
   secretAccessKey: "test",
   region: "eu-central-1",
   endpoint: "http://localhost:4566"
});

sqs.getQueueUrl({
   QueueName: "test-queue"
}, (error, data) => {
   if (error) {
     console.error(error.message);
   }

   const queueUrl = data['QueueUrl'];
   console.log(queueUrl)
 });

Notes: because lua-resty-aws is built by v2.751.0, I also downgrade my JS SDK version to v2.751.0 and it still work to connect the local SQS

Did I did something wrong? or is there any better way to custom the endpoint? Seems this lib doesn't support the AWS.Endpoint service which isn't like JS version, so I can't find any other ways to solve it.

thanks 🙇

Can i use `prepare_awsv4_request` in nginx?

I want to use prepare_awsv4_request func in Nginx configuration like this library(https://github.com/jobteaser/lua-resty-aws-signature < It does not work 😢 )
How to build config and request_data?

How to use it in openresty

Hi guys,

I installed it using luarocks and tried to just use it in openresty, but got the following error:

2023/02/23 17:17:21 [error] 29783#29783: *15 lua entry thread aborted: runtime error: /usr/local/openresty/nginx/conf/vhost.d/aws_resty.lua:7: key 'S3' not found
stack traceback:
coroutine 0:
	[C]: in function 'error'
	/usr/local/share/lua/5.1/resty/aws/init.lua:24: in function '__index'
	/usr/local/openresty/nginx/conf/vhost.d/aws_resty.lua:7: in main chunk, client: 10.8.2.139, server: , request: "GET / HTTP/1.1", host: "10.8.0.224:18888"

Here is my lua script:

local AWS_global_config = require("resty.aws.config").global
local AWS = require("resty.aws")

local config = { region = AWS_global_config.region }
local aws =  AWS(config)

aws:S3()

And here is my nginx config:

init_by_lua 'local _ = require("resty.aws.config").global';

server {
    listen  18888;

    default_type text/html;

    location / {
        lua_code_cache off;
        content_by_lua_file conf/vhost.d/aws_resty.lua;
    }
}

Input is not valid error when I call POST API

I encountered an issue when calling the API of cloudfront. I called a POST API and got a 400 http error code. I print the log of request body in the execute.lua:

<?xml version="1.0" encoding="UTF-8"?>
<input>
  <ResponseHeadersPolicyConfig xmlns="http://cloudfront.amazonaws.com/doc/2020-05-31/">
    <Name>cdn_test</Name>
    <CorsConfig>
      <AccessControlAllowMethods>
        <Items>
          <Method>GET</Method>
          <Method>POST</Method>
        </Items>
        <Quantity>2</Quantity>
      </AccessControlAllowMethods>
      <AccessControlAllowCredentials>false</AccessControlAllowCredentials>
      <AccessControlMaxAgeSec>600</AccessControlMaxAgeSec>
      <AccessControlAllowHeaders>
        <Items>
          <Header>*</Header>
        </Items>
        <Quantity>1</Quantity>
      </AccessControlAllowHeaders>
      <OriginOverride>true</OriginOverride>
      <AccessControlAllowOrigins>
        <Items>
          <Origin>*</Origin>
        </Items>
        <Quantity>1</Quantity>
      </AccessControlAllowOrigins>
    </CorsConfig>
    <CustomHeadersConfig>
      <Items>
        <ResponseHeadersPolicyCustomHeader>
          <Override>true</Override>
          <Value>rcy</Value>
          <Header>name</Header>
        </ResponseHeadersPolicyCustomHeader>
      </Items>
      <Quantity>1</Quantity>
    </CustomHeadersConfig>
  </ResponseHeadersPolicyConfig>
</input>

And the response body returned by AWS server:

<?xml version="1.0"?>
<ErrorResponse xmlns="http://cloudfront.amazonaws.com/doc/2020-05-31/">
    <Error>
        <Type>Sender</Type>
        <Code>MalformedInput</Code>
        <Message>input is not valid, expected ResponseHeadersPolicyConfig</Message>
    </Error>
    <RequestId>cd10f090-22eb-41b3-94df-4264f9d45402</RequestId>
</ErrorResponse>

Then I checked the AWS REST API document, just found that request body shouldn't have the root input tag.
The input tag come from the raw-api api-description files. After I changed the code to remove the input tag, the call succeed.

build.lua#232
    -- encode rest of the body data here
    -- poor_mans_xml_encoding(xml_data, operation.input, "input", body_tbl)

    for name, member in pairs(operation.input.members) do
      if body_tbl[name] then
        poor_mans_xml_encoding(xml_data, member, name, body_tbl[name], 0)
      end
    end

But I'm not sure if I missed something else.

Incorrect URI generated for s3:PutObject

{KEY+} is not replaced.

The problem probably comes from:

lua-resty-aws/src/resty/aws/request/build.lua

Line 103 in bd1d16b

local place_holder = "{" .. locationName .. "}"

Ref:
https://github.com/aws/aws-sdk-js/blob/c14da5afad7fa81c6889d2663743204b694f657c/dist/aws-sdk-react-native.js#L3717
The file is too large, so here is the line:

		      var regex = new RegExp('\\{' + member.name + '(\\+)?\\}');

tls config requirement

I was using version 1.0.1 to retrieve secrets successfully but then when I built a new image with 1.1.0 it started to fail with a timeout trying to reach the secretsmanager endpoint over http:

failed to connect to 'http://secretsmanager.us-west-2.amazonaws.com:80'

I had not changed any configuration at all. I went looking in your unit tests and saw the extra change:

config.tls = true

If this is required now you might want to update your public examples to reflect this requirement.

Thanks for the great library.

QUESTION: Using IAM role for Credentials

I want to run the lua script in a container ( ECS Fargate ) with IAM role. How it's possible to implement the credential section to use IAM attached to the container ?

local AWS = require("resty.aws")
local AWS_global_config = require("resty.aws.config").global
local config = { region = AWS_global_config.region }

local aws = AWS(config)

local my_creds = aws:RemoteCredentials{ }
aws.config.credentials = my_creds
    local lambda = aws:Lambda { region = "eu-west-2"}

Since it's ECS Fargate I have environment variables for AWS_CONTAINER_CREDENTIALS_RELATIVE_URI and AWS_CONTAINER_CREDENTIALS_FULL_URI but module doesn't read any environment variable !

2023/08/15 21:57:32 [error] 5561#5561: *50310593 [lua] RemoteCredentials.lua:71: Failed to construct RemoteCredentials url: Environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI or AWS_CONTAINER_CREDENTIALS_FULL_URI must be set to use AWS.RemoteCredentials., context: init_worker_by_lua* | 2023/08/15 21:57:32 [error] 5561#5561: *50310593 [lua] RemoteCredentials.lua:71: Failed to construct RemoteCredentials url: Environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI or AWS_CONTAINER_CREDENTIALS_FULL_URI must be set to use AWS.RemoteCredentials., context: init_worker_by_lua*
``

Missing `Lambda.invokeWithResponseStream` in the generated `lambda-2015-03-31.lua`

Even in the latest release (1.5.0-1) the Lambda.invokeWithResponseStream is missing.
It is provided in the aws-js-sdk, see https://github.com/aws/aws-sdk-js/blob/d5f58a06ed7e551658293f7fb555543945ed783b/apis/lambda-2015-03-31.normal.json#L1133

unable to get local issuer certificate

This is a business pod in my k8s,
My code structure is as follows
│ │ ├── src
│ │ │ -├── lib
│ │ │ -- ├── aws
│ │ │ ---- ├── AWS.lua
│ │ │ -- ├── operator
│ │ │ ---- ├── DynamicConfig.lua
│ │ │ -├── nginx
│ │ │ -- ├── nginx_init.lua
│ │ │ -├── service
│ │ │ -- ├── cc
│ │ │ ---- ├── CcManager.lua
DynamicConfig.lua is initialized in nginx_init.lua, AWS.lua is introduced in DynamicConfig.lua and the get_credentials() method is called, returning successfully

local function get_credentials()
    local AWS_global_config = resty_aws_config.global
    local config_aws = { region = AWS_global_config.region }
    local aws = resty_aws(config_aws)

    local creds, err = aws:TokenFileWebIdentityCredentials()
    logger:error(" to get AWS credentials due to '%s', '%s'",creds, err)

    local success, accessKeyId, secretAccessKey, sessionToken =
        creds:get()
    logger:error("TokenFileWebIdentityCredentials %s", { success, accessKeyId, secretAccessKey, sessionToken })

    return {
        access_key = accessKeyId,
        secret_key = secretAccessKey,
        security_token = sessionToken
    }
end

response success ,The log is as follows：
2023/07/31 02:06:30 [error] 1#0: [lua] NgxLogSink.lua:43: write(): [/mnt/lua/libs/aws/AWS.lua:40] TokenFileWebIdentityCredentials [true,"AS*****IT","I10t2s9*****81Se2o","FwoGZXIvYXdzECwaDGZZKm9yNb******68scObxG7ybiqlh114oA=="]

But when I use http to request my CcManager.lua, it refers to AWS.lua and also calls get_credentials(), but there are error as follows：

2023/07/31 02:07:33 [error] 11#0: *15 [lua] NgxLogSink.lua:43: write(): [ERROR] [:03fcbd76***8cc664e7e0][/mnt/lua/libs/aws/AWS.lua:40] [user_id: 06***e8fd69cdb5edf] TokenFileWebIdentityCredentials [null,"Request for token data failed: STS:assumeRoleWithWebIdentity() failed to connect to 'https://sts.amazonaws.com:443': 20: unable to get local issuer certificate"], client: 10...2**, server: , request: "POST /api/v3/cc/add HTTP/1.1", host: "***.svc.cluster.local", referrer: "https://***.com/"

I don’t understand why the same method called under the same process. It can be obtained normally during initialization, but an error will be reported when other modules are called later. Do you have any professional advice? I need your help, thank you very much

DynamoDB - AWS signature version 'v4' does not exist or hasn't been implemented

How can I query DynamoDB? It's giving an issue with signature v4 version not being implemented.

Example Call and Query

    aws_sdk = require('resty.aws')

    config = {
        region = "ap-southeast-1",
        credentials = {
            accessKeyId = "ACCESS KEY ID",
            secretAccessKey = "SECRET ACCESS KEY",
        }
    }
    local aws = aws_sdk:new(config)
    
    local dynamodb = aws:DynamoDB()
    
    local results, err = dynamodb:getItem {
        Key = {
            COLUMN = {
                S = "QUERY EXAMPLE"
            }
        },
        TableName = "TABLE NAME"
    }

Returns the following error

nginx_1          | 2021/06/29 14:33:10 [error] 7#7: *1 lua entry thread aborted: runtime error: ...penresty/luajit/share/lua/5.1/resty/aws/request/sign.lua:21: AWS signature version 'v4' does not exist or hasn't been implemented
nginx_1          | stack traceback:
nginx_1          | coroutine 0:
nginx_1          | 	[C]: in function '__index'
nginx_1          | 	...penresty/luajit/share/lua/5.1/resty/aws/request/sign.lua:21: in function 'sign_request'
nginx_1          | 	.../local/openresty/luajit/share/lua/5.1/resty/aws/init.lua:260: in function 'getItem'
nginx_1          | 	access_by_lua(nginx.conf:42):11: in main chunk, client: 192.168.1.1, server: , request: "GET / HTTP/1.1", host: "domain"
nginx_1          | 2021/06/29 14:33:10 [error] 7#7: *2 lua entry thread aborted: runtime error: ...penresty/luajit/share/lua/5.1/resty/aws/request/sign.lua:21: attempt to call a number value
nginx_1          | stack traceback:
nginx_1          | coroutine 0:
nginx_1          | 	...penresty/luajit/share/lua/5.1/resty/aws/request/sign.lua: in function 'sign_request'
nginx_1          | 	.../local/openresty/luajit/share/lua/5.1/resty/aws/init.lua:260: in function 'getItem'

Any guidance on a simple DynamoDB API Call/Query setup?

luarocks install lua-resty-aws failed on Mac OS

hello guys, I'm trying to install this package in my Mac OS(M2) but encountered an issue:

luaexpat 1.5.1-1 depends on lua >= 5.1 (5.4-1 provided by VM)
env MACOSX_DEPLOYMENT_TARGET=11.0 gcc -O2 -fPIC -I/opt/homebrew/opt/lua/include/lua5.4 -c src/lxplib.c -o src/lxplib.o -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -Isrc/
src/lxplib.c:12:10: fatal error: 'expat_config.h' file not found
#include "expat_config.h"
         ^~~~~~~~~~~~~~~~
1 error generated.

Error: Failed installing dependency: https://luarocks.org/luaexpat-1.5.1-1.src.rock - Build error: Failed compiling object src/lxplib.o

it is probably a known issue from luaexpat ?! -> lunarmodules/luaexpat#32
but seems not solution yet.

Can someone help on this ? thank you 🙇

Could not parse secret value

I'm following the example code to get the secret value from secret manager. The results table has status200 but i couldn't figure out how to parse the secret value. Can you please provide an example?

Thanks

add support for proxies

see https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-proxy.html

Error in secretsmanager client: don't know how to validate operator 'idempotencyToken' of type 'string'

I think this is a blind spot in the request validation code. The following yields an error:

setmetatable(_G, nil)

package.path = "./src/?.lua;./src/?/init.lua;" .. package.path

local aws = require("resty.aws")
local EnvironmentCredentials = require "resty.aws.credentials.EnvironmentCredentials"

local AWS = aws({ credentials = EnvironmentCredentials.new() })
local sm = assert(AWS:SecretsManager({ region = "us-east-1" }))

local res, err = sm:putSecretValue({
  SecretId = "my-secret",
  SecretString = "my-secret-value",
  ClientRequestToken = "1df6ced9-83b2-4393-9405-4fad519b1ee9",
})

print("error: ", err)
print("status code: ", res and res.status)
print("body: ", res and res.body)

$ resty repro.lua
ERROR: ./src/resty/aws/request/validate.lua:207: don't know how to validate operator 'idempotencyToken' of type 'string'
stack traceback:
        ./src/resty/aws/request/validate.lua:9: in function '__index'
        ./src/resty/aws/request/validate.lua:207: in function 'validate'
        ./src/resty/aws/request/validate.lua:179: in function <./src/resty/aws/request/validate.lua:176>
        ./src/resty/aws/request/validate.lua:318: in function 'validate_input'
        ./src/resty/aws/init.lua:310: in function 'putSecretValue'
        repro.lua:11: in function 'file_gen'
        init_worker_by_lua:45: in function <init_worker_by_lua:43>
        [C]: in function 'xpcall'
        init_worker_by_lua:52: in function <init_worker_by_lua:50>

Looking at the docs for ClientRequestToken it looks like there's not much we need to do. The parameter definition declares that ClientRequestToken is a ClientRequestTokenType, which also has its own string validation schema, so I think everything is already covered.

dependency on Penlight

rockspec has no dependencies, yet there is usage;

• pl.stringx is used in src/resty/aws/request/signatures/v4.lua
• pl.utils is used in src/resty/aws/init.lua