rrahul963 / serverless-create-global-dynamodb-table Goto Github PK

First, thanks for releasing this awesome library. I have a feature request for stack name used in

In my serverless.yml file, I am not using the stack name as serviceName-stageName. Below is my configuration:

provider:
  name: aws
  runtime: nodejs12.x
  region: ${opt:region}
  stackName: ${opt:stage}-${self:service.name}

If I use above configuration, I will get an error Failed to setup global table. Error Stack with id dbs-api-dev does not exist. dev is my stage name.
I think this is because the code doesn't read stack name from my serverless. Can you make it read from serverless configuration?

fixes #3 to deploy even if backups are already created

With createStack: true, I'm running into an error in my second region informing me that the custom role already exists. Reason is because it is already deployed in the first region. I've tried adding region name but it still picks up the first region's name. Any workaround on this? Thanks

Trying to deploy a staging env with this plugin I get the following error: Failed to setup global table. Error One or more parameter values were invalid: DynamoDB global tables version 2017.11.29 is not supported in this region 'eu-west-3'

While the error is pretty straightforward, is it possible to use this plugin with the current version of global tables?

Error

CreateGlobalTable: Starting setting up global tables...
CreateGlobalTable: Global table doesn't exist...
CreateGlobalTable: Creating new table devtable in us-east-2 region...
CreateGlobalTable: Creating new table devtable in us-west-2 region...
CreateGlobalTable: Failed to setup global table. Error There were 2 validation errors:
* UnexpectedParameter: Unexpected key 'NumberOfDecreasesToday' found in params.GlobalSecondaryIndexes[0].ProvisionedThroughput
* UnexpectedParameter: Unexpected key 'NumberOfDecreasesToday' found in params.GlobalSecondaryIndexes[1].ProvisionedThroughput

These tables use bill per request and have no ProvisionedThroughput on the table or the GSI

Table Example:

Table:
      Type: AWS::DynamoDB::Table
      Properties:
        PointInTimeRecoverySpecification:
          PointInTimeRecoveryEnabled: true
        StreamSpecification:
          StreamViewType: NEW_AND_OLD_IMAGES
        TableName: ${self:custom.serviceName}
        AttributeDefinitions:
          - AttributeName: Id
            AttributeType: S
          - AttributeName: AnotherId
            AttributeType: S
          - AttributeName: CreatedDateTime
            AttributeType: S
          - AttributeName: YetAnotherId
            AttributeType: S
          - AttributeName: somethingelsesId
            AttributeType: S
          - AttributeName: fakeId
            AttributeType: S
        KeySchema:
          - AttributeName: AnotherId
            KeyType: HASH
          - AttributeName: CreatedDateTime
            KeyType: RANGE
        BillingMode: PAY_PER_REQUEST
        GlobalSecondaryIndexes:
          - IndexName: Id_AnotherId_IDX
            KeySchema:
              - AttributeName: AnotherId_YetAnotherId
                KeyType: HASH
              - AttributeName: CreatedDateTime
                KeyType: RANGE
            Projection:
              ProjectionType: 'ALL'
          - IndexName: AnotherId_YetAnotherId_IDX
            KeySchema:
              - AttributeName: Id
                KeyType: HASH
              - AttributeName: FakeId
                KeyType: RANGE
            Projection:
              ProjectionType: 'ALL'

Obviously the above field names and everything going along with it has been made up but the actual implementation works when the table is deployed without this plugin.

I am also able to create Global tables from the aws Console without an issue.

Please assist.

When you have a large stack the plugin is not always able to fetch the DynamoDB tables.

Cause: describeStackResources from AWS-SDK only returns the first 100 resources.

Code:

Possible Solution: use listStackResources and use NextToken in response to cope with large Stacks

serverless remove should delete all the backup tables and global table connection. I'm working on the patch now using the

      'remove:remove': function () {
        deleteGlobalAndBackupTables(serverless)
      },

hook. PS love the plugin, there is really limited support for deploying global tables without cloudformation.

Hey I have just set this plugin up and it looks to be failing silently.

When I fun a sls deploy -v I am getting the following put:

CreateGlobalTable: Starting setting up global tables...

And then dropping back into the terminal with no more messages and then in the AWS console it looks like nothing has happened at all.

Please advise best way to debug.
Thanks

I have multiple dynamodb tables under Resources section and I only want to create global table for one dynamodb table. How can I specify the table name under globalTables?

I have blow configuration:

globalTables:
    version: v2
    regions:
      - ap-southeast-1
      - ap-southeast-2
    createStack: false

but when run sls deploy in ap-southeast-2, it gives me below error

CreateGlobalTable: Failed to setup global table. Error Cannot add, delete, or update the local region through ReplicaUpdates. Use CreateTable, DeleteTable, or UpdateTable as required.

what could be the reason causing this issue?

I've got following error:

CreateGlobalTable: Failed to setup global table. Error 2 validation errors detected: Value '0' at 'provisionedThroughput.writeCapacityUnits' failed to satisfy constraint: Member must have value greater than or equal to 1; Value '0' at 'provisionedThroughput.readCapacityUnits' failed to satisfy constraint: Member must have value greater than or equal to 1

Because I don't use provisioned throughput. Instead, I use:

          "BillingMode": "PAY_PER_REQUEST",

Can you add support for on-demand capacity mode?

Update the package to add Remove hook so that the tables removed when running sls remove command

I tried to setup a multi region serverless app which has a Lambda function that serves api requests and uses DynamoDB.

While deploying thru CI/CD tool for one of the region,

1. Global Table scripts started execution after the completion of stack updates for one of the desired regions.
2. Next, script started creating/updating stack of other region without uploading the artifacts/deployment package to next region.
3. The stack creation failed saying that the S3Key(path to package) as NoSuckKey and the Lambda deployment is failing causing the stack to rollback.

Please find the screenshots below for the stack updates verbose and the aws CFN error event from my sample project. Here, I tried to deploy to the default region but has two regions under custom node.

Didn't face any issue if there is no lambda to be created in the stack as the Lambda creation has a constraint that the deployment package should be in the same region.

Due to my delivery deadlines, had to search if there is a way and found another package for creating global table which just creates a table and create a replication for the same.

Any updates or any suggestions related to this issue is much appreciated

serverless.yml:

service: my-auth-service-globaltable-issue

plugins:
      - serverless-create-global-dynamodb-table

custom:
  globalTables:
    regions: # list of regions in which you want to set up global tables
      - us-east-1
      - us-west-2
    createStack: true
    tags: # List of tags that needs to applied to the new table (optional)
      - Key: tag-key
        Value: tag-value
      - Key: tag-key-2
        Value: tag-value-2

provider:
  name: aws
  runtime: python3.8
  
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:DescribeStream
        - dynamodb:GetRecords
        - dynamodb:GetShardIterator
        - dynamodb:ListStreams
      Resource: arn:aws:dynamodb:*:*:table/usersTable-final
      # you can overwrite defaults here
  stage: dev
  region: us-east-1

functions:
  hello-final:
    handler: handler.hello

resources: # CloudFormation template syntax
  Resources:
    usersTable:
      Type: AWS::DynamoDB::Table      
      Properties:
        TableName: usersTable-globaltable-issue
        AttributeDefinitions:
          - AttributeName: email
            AttributeType: S
        KeySchema:
          - AttributeName: email
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 3
          WriteCapacityUnits: 3
        StreamSpecification:
          StreamViewType: NEW_AND_OLD_IMAGES

When the plugin fails to setup global table relation between the tables due to some error, we need to make sure that the table(s)/stack created in the new region is cleanup.

Plugin doesn't support global tables (creation/replication) with KMSMasterKeyId. Could you please add it (here)?

Can you please show a sample serverless.yml that creates a global table in us-east-1 and replica in us-west-2 (or any other regions) using version v2 with createStack true. I have not been able to get it to work. The message i keep getting is:
Failed to setup global table. Error Failed to create a the new replica of table because one or more replicas already existed as tables.

I have createStack: true with two regions. In my second region's cloudformation I am getting "Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist. I believe it is skipping the step of placing the lambda function into the second region.

I am running into a similar issue that was posted in #18.

Same error.

CreateGlobalTable: Failed to setup global table. Error There were 2 validation errors:
* UnexpectedParameter: Unexpected key 'NumberOfDecreasesToday' found in params.GlobalSecondaryIndexes[0].ProvisionedThroughput
* UnexpectedParameter: Unexpected key 'NumberOfDecreasesToday' found in params.GlobalSecondaryIndexes[1].ProvisionedThroughpu

The dynamoDb.createTable() will fail since the dynamoDb.describeTable() returns properties in ProvisionedThroughput that are not taken as input parameters in createTable(). The ProvisionedThroughput property includes the NumberOfDecreasesToday property which is not taken as an input.

eg. GlobalSecondaryIndexes from describeTable()

{
    IndexName: 'test-index',
    KeySchema: [ [Object], [Object] ],
    Projection: { ProjectionType: 'INCLUDE', NonKeyAttributes: [Array] },
    IndexStatus: 'ACTIVE',
    ProvisionedThroughput: {
      NumberOfDecreasesToday: 0,
      ReadCapacityUnits: 0,
      WriteCapacityUnits: 0
    },
    IndexSizeBytes: 0,
    ItemCount: 0,
    IndexArn: '...'
  },

The logic also has to take into consideration if ProvisionedThroughput is not defined in the GlobalSecondaryIndexes index since it is an optional parameter https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-dynamodb-table-globalsecondaryindex.html. The example above was taken from a table where the GlobalSecondaryIndexes did not have a ProvisionedThroughput defined however the ProvisionedThroughput property is still returned from the describe.