Comments (2)
It won't be portable across platforms. Why do you need it though? You can just not pass any callback if you prefer to, it will save file nevertheless.
from jbinary.
I load in a point cloud(60-200MB binary final of int16's). Then query ~10 million position and apply the positions to the point cloud. The positions come in a stream from mysql server. Then at certain intervals write out the point cloud. I need to stop the mysql stream in order to output the file if being written async to ensure the point cloud isnt modified during output. When sync i can maintain the stream synchronously. I can do a pause in the stream, then pass in call back to resume when outputs the last file, but I think its really messed compared to sync write and doesnt save any time since I need to pause the database stream.
CONNECTION_POOL = mysql.createConnection({
connectionLimit: 10,
host: server_name,
user: user_name,
password: password,
database: schema,
connectTimeout: 99999999999,
multipleStatements: true,
.........
..........
});
var myObjects, myObject = {};
myObject.header = module.loadUsingJBin_and_other_stuff('up_to_200MbBinaryFile');
myObject.myMassiveArray = module.loadUsingArrayBuffer_and_other_stuff('up_to_200MbBinaryFile');
var positionStream = CONNECTION_POOL.query(myQuery)
positionStream.on('result', function(currentPositionRow) {
// this includes spawning multiple myObject's all with their own myMassiveArray
myObjects = doStuff(myObject, currentPositionRow);
// if time is write we need to output all the point clouds made
if (currentPositionRow.time > outputTime && previousPositionRow.time < outputTime) {
// We could add a puase here
//CONNECTION_POOL.Pause();
for (var i = 1; i < myObjects.length; i++) {
// Conenction pool is paused, do i really want to deal with callback hell for async file write ?
// callBack = (last) ? function (CONNECTION_POOL) { CONNECTION_POOL.Pause() } : 'NOT_TIME';
module.WriteUsingJBin(outputTime, myObjects[i] /*, callBack*/ );
}
}
})
I have tried cloning off the object to write but I get memory overflow, because the database stream and math is way faster then the write and garbage collection. Open for any suggestions.
from jbinary.
Related Issues (20)
- Dispatch to typedef based on context? HOT 3
- ios write blob to file HOT 3
- Type Caching and Memory Use HOT 3
- Question: How to convert the "uint16" value to half-precision float (i.e. "float16")? HOT 1
- Dynamic Types? HOT 2
- jBinary::readAll does not mutate the internal pointer HOT 3
- writeAll() method documentation and functionality HOT 3
- corsproxy.com redirects to dubious web sites HOT 1
- BROWSER vs NODE differences HOT 3
- Change Endianness (question) HOT 4
- "index out of range" with dynamic array size HOT 5
- Just don't work, maybe it caused by electron? :-{ HOT 1
- Local build is failing HOT 3
- Remove old polyfills
- Switch to ESM+Rollup
- How to read a string HOT 8
- reproducable out-of-memory errors using 'array'
- Strings ignores length on writing
- disregard
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jbinary.