Comments (3)
Any update on this? I need this too 😄
from vogels.
+1
from vogels.
Do to the way DynamoDB works this would be difficult to implement correctly in a generic way. I'll give an example:
// find all users named bob
Account.scan().where('name').equals('bob').limit(500);
Lets say there are 500,000 accounts total and 800 are named 'bob'.
First issue is say all the accounts named bob are towards the end of the scan iterator. You are going to scan over all 500K items before you reach 500 Bobs and potentially use up all your provisioned throughput. If you enabled loadAll()
on your scan, vogels will scan 500 items at a time, but will eventually iterate over the entire table. limit
with loadAll
is probably a bit confusing for new users when combining them. Limit with dynamodb really just provides a way to limit the number of read throughput used per request, not really a way to limit the number of items returned.
The second issue is say in the first scan request 100 Bobs are returned and then a subsequent request is made again in order to attempt to fully load 500 Bobs, on this next request 450 Bobs get returned (because many bobs were located near this iterator). Should vogels just 500 users, or should it return 550 (total found). Data isn't sorted on scans so it would have to just randomly return 500 users and ignore the 50 others. The next iterator returned from DynamoDB wouldn't be valid if you then attempted to load up the next 500 Bobs, because you didn't return 50 of them. The same issue applies on queries to secondary indexes.
I'm open to suggestions on how to make this as user friendly as possible. We need to be able to work with the limitations DynamoDB and Id want to make it explicit as possible to developers that they might be consuming lots of throughput when executing certain functions.
from vogels.
Related Issues (20)
- Setting StreamSpecification option during table create/update? HOT 1
- ConfigError: Missing region in config
- Secondary Index - ValidationException: The provided starting key is invalid HOT 2
- Vogels is getting stuck when Dynamodb call timeouts HOT 2
- Is $add atomic? (no doc reference) HOT 2
- How can i get all the records using scan ... now i am getting only 780 items .. I need to scan all 50000 items HOT 4
- Dynamodb Error: Nodejs App with dynamodb does not work in docker but works without it HOT 6
- Using eachSeries causes slow table creation
- Problem writing tests using mocha with vogels HOT 12
- Multiple Batch Get Items with different Tables.Handling table references
- scan().limit() return 0 Count and items HOT 1
- How to handle array of objects in models HOT 1
- conditional get an item using get() HOT 1
- Empty object in Dynamo DB when using conditional validation
- Retry logic will indefinitely retry
- strongly consistent reads in scan operation
- Support for Amazon Trust Services (ATS)
- Enable encryption (SSESpecification) when creating table
- Abandoned project?
- Adding extra 's' after the table name HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vogels.