Comments (3)
I just dealt with this by doing the following:
const queryDatabase = async (query, parameters = {}) => {
try {
const { records } = await db.query(query, parameters)
return records
} catch (error) {
console.error(error.message)
if (error.message.includes('Database returned more than the allowed response size limit')) {
const getCount = async (_query) => {
const query = _query.replace(/SELECT\s(.+?)\s(FROM|WHERE)/is, 'SELECT COUNT(*) $2')
const {
records: [{ count }],
} = await db.query(query, parameters)
return count
}
let result = []
// Get result count
const count = await getCount(query)
// Define limit based on your average record size
const limit = 50
// Use limits and offsets to page through the query
for (let page = 1; page <= Math.ceil(count / limit); page++) {
const limitQuery = query.replace(/;?$/, `\nLIMIT ${limit} OFFSET ${(page - 1) * limit};`)
const { records } = await db.query(limitQuery, parameters)
result = [...result, ...records]
}
return result
}
}
}
It you're doing a batch query that passes parameters as nested arrays, then you will need to handle that for each page query.
from data-api-client.
thanks @gnestor! one small note in case someone else ends up here:
my query contained a GROUP BY
clause, which doesn't return the correct number of rows with the snippet above, so i changed the following:
const getCount = async (_query) => {
// old: const query = _query.replace(/SELECT\s(.+?)\s(FROM|WHERE)/is, 'SELECT COUNT(*) $2')
const query = `SELECT COUNT(*) FROM (${_query}) sub`; // wrap the original query and count the number of results
const {
records: [{ count }],
} = await db.query(query, parameters)
return count
}
other than that, it works great!
from data-api-client.
I would love to see this abstracted as well! +1
from data-api-client.
Related Issues (20)
- suggestion for bulk delete HOT 3
- Problem when calling create database.
- Handling multiple statements HOT 1
- datetime(3) fields sometimes parsed UTC, sometimes LocalTime.
- IS THIS REPOSITORY STILL MAINTAINED? HOT 5
- Error tracing issues HOT 2
- Can't get this to work with Next.js api routes
- Type defintions not aligned to js code
- Cast Injection
- How to run a sql file with data api
- Updating Docs For Sending in Arrays HOT 1
- some of string data that was inserted into database had a whitespaces. In order to get the specific data I need to remove the whitespace first, can you guys fix it? Thank you.
- Lambda on NodeJS v18 HOT 13
- Alternative to using Secret Manager for secretArn?
- Node 18 error HOT 3
- How can I pass a variable to a SQL variable?
- Error - Transaction not found
- Feature Request: Support for returning uuid from batch INSERT
- error updating "name" column HOT 4
- Is connection takes long time HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from data-api-client.