channl / dynamodb-lambda-autoscale Goto Github PK

Setting the minimum write and read capacity has no effect.
Both get set to 1 if too little activity is detected no matter the setting.

{
"ReadCapacity": {
"Min": 5,
"Max": 500,
"Increment": {
"When": {
"UtilisationIsAbovePercent": 80
},
"By": {
"Units": 10
},
"To": {
"ConsumedPercent": 120
}
},
"Decrement": {
"When": {
"UtilisationIsBelowPercent": 30,
"AfterLastIncrementMinutes": 60,
"AfterLastDecrementMinutes": 60,
"UnitAdjustmentGreaterThan": 1
},
"To": {
"ConsumedPercent": 100
}
}
},
"WriteCapacity": {
"Min": 5,
"Max": 250,
"Increment": {
"When": {
"UtilisationIsAbovePercent": 80
},
"By": {
"Units": 10
},
"To": {
"ConsumedPercent": 120
}
},
"Decrement": {
"When": {
"UtilisationIsBelowPercent": 30,
"AfterLastIncrementMinutes": 60,
"AfterLastDecrementMinutes": 60,
"UnitAdjustmentGreaterThan": 1
},
"To": {
"ConsumedPercent": 100
}
}
}
}

I think there is a logical problem with the behavior of default config. I have tried to deploy this code on system with no traffic at this point. The reaction of default config.js was to do nothing since all metrics returned 0 reads and 0 writes.

Maybe I am doing something wrong, but it seems like I cannot make this code to react on such system. I configured one table with read and writes values equals 10 and created 2 additional ones with default value (5 for both). Still, nothing...

I see a constant in RateLimitedDecrement.js that I could change from 4 to the desired value I suspect but wasn't sure if this could be done via config.

Since we have to fork to deploy anyway I can just change the constant.

We have a challenge where we scale reads way up on a table, dump to S3, and then scale them back down. We need to reserve at least one decrement window in the 24 hour window for this process. Reducing the 4 to a 3 seems like it will solve this need.

Thanks!

Hello,

I'd like to scale a low number of dynamo-DB tables in a completely relative manner only setting boundaries by Min/Max Thresholds for read and write capacity.
The relative increment is working fine, but the relative decrement wouldn't simply decrement.
In the following I'd like to show u this just by changing definition of read-strategy.

Can you verify the issue or give me a hint if I am doing something wrong?

In (1) you can find the provisioner.json I intend to use. A run of it leads to the following result (recognize change wanted and allowed, but doing nothing):

node ./scripts/start.js
*** LAMBDA INIT ***
*** LAMBDA START ***
Getting table names
Getting table details
Getting table description testtable
Getting table consumed capacity description testtable
Getting table update request testtable
testtable is consuming 0 of 410 (0%) read capacity units
testtable is consuming 0 of 410 (0%) read capacity units so a decrement is WANTED and is ALLOWED
testtable is consuming 0 of 200 (0%) write capacity units
testtable is consuming 0 of 200 (0%) write capacity units
Getting required table update requests
No table updates required
{
"Index.handler": {
"mean": 242.57998418807983
},
"DynamoDB.listTablesAsync": {
"mean": 105.22508716583252
},
"DynamoDB.describeTableAsync": {
"mean": 73.80222463607788
},
"DynamoDB.describeTableConsumedCapacityAsync": {
"mean": 46.67149496078491
},
"CloudWatch.getMetricStatisticsAsync": {
"mean": 40.9902560710907
},
"TableUpdates": {
"count": 0
},
"TotalProvisionedThroughput": {
"ReadCapacityUnits": 410,
"WriteCapacityUnits": 200
}
"TotalMonthlyEstimatedCost": 131.976
}
*** LAMBDA FINISH ***

If I extend the provisioner.json from (1) with an Decrement.To to:

... "Decrement": {
"When": {
"UtilisationIsBelowPercent": 50,
"AfterLastIncrementMinutes": 60,
"AfterLastDecrementMinutes": 60
},
"By": {
"ConsumedPercent": 50
},
"To": {
"Units": 200
}

I get an successful update on the table as you can see in the following.
The Decrement done is a decrement to the value from the Decrement.To(200) and not a relative (should be 205).
To be complete, if i remove the Decrement.By only having Decrement.To the run is also successful.

node ./scripts/start.js
*** LAMBDA INIT ***
*** LAMBDA START ***
Getting table names
Getting table details
Getting table description testtable
Getting table consumed capacity description testtable
Getting table update request testtable
testtable is consuming 0 of 410 (0%) read capacity units
testtable is consuming 0 of 410 (0%) read capacity units so a decrement is WANTED and is ALLOWED
testtable is consuming 0 of 200 (0%) write capacity units
testtable is consuming 0 of 200 (0%) write capacity units
Getting required table update requests
Updating tables
Updating table testtable
Updated table testtable
Updated tables
{
"Index.handler": {
"mean": 336.56548500061035
},
"DynamoDB.listTablesAsync": {
"mean": 114.78861999511719
},
"DynamoDB.describeTableAsync": {
"mean": 66.52075910568237
},
"DynamoDB.describeTableConsumedCapacityAsync": {
"mean": 56.809733867645264
},
"CloudWatch.getMetricStatisticsAsync": {
"mean": 48.26799559593201
},
"TableUpdates": {
"count": 1
},
"TotalProvisionedThroughput": {
"ReadCapacityUnits": 410,
"WriteCapacityUnits": 200
},
"TotalMonthlyEstimatedCost": 131.976
}

(1)

{
"ReadCapacity": {
"Min": 200,
"Max": 1000,
"Increment": {
"When": {
"UtilisationIsAbovePercent": 70
},
"By": {
"ConsumedPercent": 20
}
},
"Decrement": {
"When": {
"UtilisationIsBelowPercent": 50,
"AfterLastIncrementMinutes": 60,
"AfterLastDecrementMinutes": 60
},
"By": {
"ConsumedPercent": 50
}
}
},
"WriteCapacity": {
"Min": 200,
"Max": 500,
"Increment": {
"When": {
"UtilisationIsAbovePercent": 70
},
"By": {
"ConsumedPercent": 20
}
},
"Decrement": {
"When": {
"UtilisationIsBelowPercent": 30,
"AfterLastIncrementMinutes": 60,
"AfterLastDecrementMinutes": 60
},
"By": {
"ConsumedPercent": 50
}
}
}

Hi there - We've started using this app in production but the amount of tests over it is starting to worry me. I think I'm going to spend some time writing some tests for this @tmitchel2. Just checking that you have not already begun writing tests.

Assuming you haven't, do you have any preference for tools or frameworks to use? I was thinking of writing some unit tests using Mocha/Chai and see where I end up.

would it be possible to log both a before and after TotalMonthlyEstimatedCost? From what I can tell, the TotalMonthlyEstimatedCost that's logged currently appears to be "current state"

could we also assemble a projected state (since we can't query it post update since updates dont apply immediately) and estimate a cost off of that?

Seems like it would be useful to see the projected cost before/after any particular update.

Getting below message

"errorMessage": "RequestId: c544d7ed-704f-11e7-8aae-a5fadaa86aee Process exited before completing request"

@tmitchel2 Any pointers?

Hi there, I have a question around how you envisage people scaling only a set of tables. I have a number of dynamo tables and I just want a subset of them to use autoscaling. I've had a look through the code and I think the right place to filter the list of tables to scale is the DynamoDB object.

Is this correct, or is there some other built in way that I could filter tables?

Thanks in advance,
Ryan

This is more of a question, but potentially an issue. Does/should the config.env file be required when you deploy to a lambda? Isn't that running as a role, and therefore could inherit its permissions based on that?

In cases of rapid changes in utilization, it can be slow to respond as it is limited to a multiplier of the provisioned capacity. In cases where the consumed capacity significantly exceeds the provisioned capacity, it can take several steps to meet demand.

I just ran npm install and I got these warnings.

npm WARN deprecated [email protected]: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
npm WARN deprecated [email protected]: lodash@<3.0.0 is no longer maintained. Upgrade to lodash@^4.0.0.
npm WARN deprecated [email protected]: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.

Hi @tmitchel2 First of all I would like to thank you for the great work. It's awesome!

I just configured this in my systems and I have few questions:

1. I have multiple envs(dev/qa/stg/uat) in one account and I want to run this on specific env rather than on all the tables in this account. What's the place to make those changes?
2. It's setting the value to 1, where can I modify those values?

I have multiple groups of tables each of which need a different set of configuration settings.

These tables all have similar stems. Is the best way to do this by modifying Provisioner.js to look for those specific tables and load configs for each?

So that I could test what my current configuration would do against the current table metrics without actually making changes...

Hi,

at first thank's for your great project. I have already tried it a lot using different scaling policies.
Today the write capacity was not decreased although 0 of 100 provisioned write capacity units have been used. This is the CloudWatch log:
...is consuming 0 of 100 (0%) write capacity units and is below minimum threshold of 20% so a decrement is WANTED but is DISALLOWED
At exactly this time the read capacity has been decreased. Thus there have been enough decreases left for this calendar day. This is really confusing, so I started to debug.
I ended in the file src/Provisioner.js containing this line:

let isAdjustmentAllowed = isAfterLastDecreaseGracePeriod && isAfterLastIncreaseGracePeriod && isReadDecrementAllowed;

Why is only ReadDecrement considered? Why is there no "isWriteDecrementAllowed" using the function calculateDecrementedWriteCapacityValue?
Maybe it's bug and this explains why decreasing read capacity has been allowed whereas decreasing write capacity was denied.

You find my config DefaultProvisioner.js as an attachment.

Thank's for your help.

Best regards,
Chris
DefaultProvisioner.txt

I'm getting the following types of cloudwatch logs and inability to increase Read/Write Capacity past 100:

{{tablename}} is consuming 96.725 of 100 (96.725%) read capacity units and is above maximum threshold of 75% so an increment is not required

Maybe I'm just reading this incorrectly but why does this condition not require an increment? I'm thinking that possibly the config I'm using has a 100 unit ceiling, but that's not quite reflected in this message.

npm WARN notsup Not compatible with your operating system or architecture: [email protected]

running Linux 4.5.0-2-amd64 #1 SMP Debian 4.5.5-1 (2016-05-29) x86_64 GNU/Linux

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#default-limits-capacity-units-provisioned-throughput

"A dial down is allowed up to four times any time per day. A day is defined according to the GMT time zone. Additionally, if there was no decrease in the past four hours, an additional dial down is allowed, effectively bringing maximum number of decreases in a day to nine times (4 decreases in the first 4 hours, and 1 decrease for each of the subsequent 4 hour windows in a day)."

Right now only 4 decreases work, not the full potential of 9.

This is my configure

{
  "ReadCapacity": {
    "Min": 2,
    "Max": 20,
    "Increment": {
      "When": {
        "UtilisationIsAbovePercent": 90
      },
      "By": {
        "Units": 2
      },
      "To": {
        "ConsumedPercent": 110
      }
    },
    "Decrement": {
      "When": {
        "UtilisationIsBelowPercent": 30,
        "AfterLastIncrementMinutes": 60,
        "AfterLastDecrementMinutes": 60,
        "UnitAdjustmentGreaterThan": 2
      },
      "To": {
        "ConsumedPercent": 50
      }
    }
  }
}

When I set table in Dynamodb is 25. It will scale down 20. But from 20 it doesn't scale down to 2. I want to autoscale to min value if the table in Dynamo doesn't read any units. How to configure?

I followed the install instructions and that attached dist.zip was produced. This however fails on run with the following:

"errorMessage": "Cannot find module 'winston'",

dist.zip

I use dynamodb-lambda-autoscale but when table and global secondary indexes allow INCREMENT. But I see only global secondary indexes INCREMENT and capacity of table NOT INCREMENT. The occur I get from log

Table IOPS are currently being updated

May I can INCREMENT table and global secondary indexes in this case.

getDimensions(tableName: string, globalSecondaryIndexName: ?string): Dimension[] {
if (globalSecondaryIndexName) {
return [
{ Name: 'TableName', Value: tableName},
{ Name: 'GlobalSecondaryIndex', Value: globalSecondaryIndexName}
];
}

return [ { Name: 'TableName', Value: tableName} ];

}
}

Name for the GSI Dimension is slightly off, should be GlobalSecondaryIndexName instead of GlobalSecondaryIndex. I haven't merged in your latest, but this bit is unchanged, and it was fixed when I changed it in my fork.

Issue with 10 or more tables to be updated. Tried to use something like mapLimit for

let capacityTasks = tableNames
      .map(async tableName => {

but i'm not familiar with the babeljs stuff.

Thinking then I could just tag tables that should be included in scale decisions and not have to redeploy the lambda

Create a command npm run deploy which:

• Build a CloudFormation template
• Deploys template (create or update) with a predefined stack name into predefined region

I guess that you run this on mac, since by default HFS+ is case insensitive.
Since I'm running this on debian, ./scripts/start.js and ./scripts/Start.js are not the same thing.

And just a couple days after I forked your project and deployed it...

https://aws.amazon.com/blogs/aws/new-auto-scaling-for-amazon-dynamodb/

Hi Guys,

We have occasional spikes that are only picked up after a minute. During this time we see failures due to underprovisioned capacity. Would there be any issues in using CloudWatch events to trigger the lambda script additionally to a time interval?

Enrico

If dynamodb-lambda-autoscale had the ability to fire off an SNS notification for every increase or decrease of capacity, it could help with building some tooling around this. For instance, i'd love to create a simple Slackbot that just notifies when capacity increases/decreases, so that I can keep an eye on what is happening and when. Thoughts?

Raised in response to #2

Looks like dotenv using a .env file is not supported on a windows OS as per motdotla/dotenv#83

In order to accommodate multiple strategies for different tables/indexes I am proposing the following strategy schema update for fixed values:

ReadCapacity.Fixed: Number
WriteCapacity.Fixed: Number

The Provisioner.js file would have to be adapted, since there is quite some conditional logic on Read/WriteCapacity.Increment and Read/WriteCapacity.Increment.When.

Currently the following strategy would have to be set when using fixed values for specific tables/indexes without causing any errors in Provisioner.js:

{
  "ReadCapacity": {
    "Min": 1,
    "Max": 1,
    "Increment":{
      "To": {
        "Units": 1
      },
      "When":{
      }
    },
    "Decrement":{
      "To": {
        "Units": 1
      },
      "When":{
      }
    }
  },
  "WriteCapacity": {
    "Min": 1,
    "Max": 1,
    "Increment":{
      "To": {
        "Units": 1
      },
      "When":{
      }
    },
    "Decrement":{
      "To": {
        "Units": 1
      },
      "When":{
      }
    }
  }
}

I'd be happy to add a pull request should this functionality be approved by more people/maintainer.

as a non-developer - what's the right approach to extend getTableNamesAsync to treat it as config instead of code?

It seems like updating it in src/Provisioner.js gets the job done - but will make merging in upstream changes painful through the lifetime of support.

If there's a config oriented mechanism for extending this i'd love to hear about it and see some examples of it. exploring forks of the project mostly seems to point to just changing the code in Provisioner.js is the common solution (for branches based off something like the current version, anyway)

I'm running in eu-west-1, and after debugging the getTableUpdate I wondered why I got 0 consumed write capacity units while I was writing to it at quite a high rate.
I'm not observing any automatic increase and I have two tables.

"tableConsumedCapacityDescription": {
  "Table": {
    "TableName": "test",
    "ConsumedThroughput": {
      "ReadCapacityUnits": 0,
      "WriteCapacityUnits": 0
    },
    "GlobalSecondaryIndexes": []
  }
}

Hi

I'm getting a socket timeout when running this with a large amount of dynamo tables. I suspect its because of the large amount of connections needed at once.

Maybe a thread limit could help? Along with #46 possibly.

I'm considering using this, but put off by the need to fork / clone. What do people think about publishing a reusable versioned artifact to npm?

Following the instructions in the readme, when carrying out
npm run build
I'm getting this error
npm ERR! missing script: build
It looks like the package.json requires a build entry in the script section

Hi again.

Since I pulled down the latest changes to get the config per table I noticed that my tables have not been scaling up as I had expected so I did some digging and found the following on my DynamoDb metrics page.

If I'm reading the CapacityCalculatorBase correctly then it's getting the Average count which is always returning as 0.5 for me even when my consumed capacity is much higher.

After this I went into Cloudwatch to try and explain what was going on and found that it was reporting consistent stats to that which the lambda was getting:

I then went through the process of trying to work with the cloudwatch metrics and the formula talked about on the DynamoDb metrics page to turn the ConsumedReadCapacityUnits into something similar to what was being reported by the DynamoDb metrics. I found that if I changed what I queried in cloudwatch to Sum then it started making a bit more sense:

1,752/60 = 29.05

And that marries up quite nicely with this number from my DynamoDb metrics:

One thing I don't understand is that I'm sure that the lambda was scaling my tables prior to pulling in the latest changes but from what I can see in your most recent commits, nothing around the cloudwatch metrics gathering changed.

So I think what I'm asking is, is this 🐛 or am I doing something silly?

I'm happy to put together a fix and open a PR if this is in fact a bug, let me know.

Thanks,
Ryan

On the screenshot, the increase was imposed manually, all the parameters are default except for the region (eu-west-1), and the max value (100).
I know the script is working to some extent because it scales down to 1, but it does not scale up properly.

Thank you for this excellent project!

I have only one big issue when i use this package:

Do you have any idea how I can get around this issue?

Thanks!

Hi there I've started using this and it works fantastically well. I thought I'd offer some insights as to how I'm using it and a potential feature.

I'm scaling a subset of my tables with this function. The way I'm doing that is by maintaining a tables-to-scale dynamo table that holds the names of the tables I want to scale. Our config then selects that list in the getTableNamesAsync function. This works well in that I don't need to update the code in order to add/remove tables we want to scale.

What I have found though is that I want to scale the tables with different configuration. For example, I have 1 table that has enough load to warrant >1000 read throughput and another that barely reaches 5 read throughput. Ideally I would like to define a different scaling policy for each.

If I were to implement this, then I think I'd try and keep the config next to the tablenames in the dynamo table I mentioned earlier. I wanted to check in with you before doing that though. Is this a usecase that you have thought about implementing and if so do you have any idea as to how you would do it?

Thanks again,
Ryan