audreyt / node-webworker-threads Goto Github PK
View Code? Open in Web Editor NEWLightweight Web Worker API implementation with native threads
Home Page: https://npmjs.org/package/webworker-threads
License: Other
Lightweight Web Worker API implementation with native threads
Home Page: https://npmjs.org/package/webworker-threads
License: Other
new Worker(new Function("console.log('workerTest')")) doesn't work.
I'm getting "Error opening the file [object Object]."
I imagine for some reason its treating the argument as a file instead of as a function(){}?
I want to use the function constructor as I'm creating the worker function dynamically. For now as a work around I'm using
eval('var evalWorker=function(){'+workerText+'}');
worker = new Worker(evalWorker);
Thanks.
It would be great to have the onError() callback implemented to catch exceptions on the worker.
See http://www.w3.org/TR/workers/#runtime-script-errors
the build is failed when using latest io.js and node.js
> node-gyp rebuild
child_process: customFds option is deprecated, use stdio instead.
CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
../src/WebWorkerThreads.cc:291:5: error: no matching member function for call to 'Set'
JSObjFn(fs_obj, "readFileSync", readFileSync_);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/jslib.cc:25:7: note: expanded from macro 'JSObjFn'
obj->Set(NanNew<String>(name), NanNew<FunctionTemplate>(fnname)->GetFunction(), attribute_ro_dd);
~~~~~^~~
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2112:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
bool Set(Handle<Value> key, Handle<Value> value);
^
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2114:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
bool Set(uint32_t index, Handle<Value> value);
^
../src/WebWorkerThreads.cc:292:13: error: no matching member function for call to 'Set'
global->Set(NanNew<String>("native_fs_"), fs_obj, attribute_ro_dd);
~~~~~~~~^~~
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2112:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
bool Set(Handle<Value> key, Handle<Value> value);
^
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2114:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
bool Set(uint32_t index, Handle<Value> value);
^
../src/WebWorkerThreads.cc:295:5: error: no matching member function for call to 'Set'
JSObjFn(console_obj, "log", console_log);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/jslib.cc:25:7: note: expanded from macro 'JSObjFn'
obj->Set(NanNew<String>(name), NanNew<FunctionTemplate>(fnname)->GetFunction(), attribute_ro_dd);
~~~~~^~~
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2112:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
bool Set(Handle<Value> key, Handle<Value> value);
^
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2114:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
bool Set(uint32_t index, Handle<Value> value);
^
../src/WebWorkerThreads.cc:296:5: error: no matching member function for call to 'Set'
JSObjFn(console_obj, "error", console_error);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/jslib.cc:25:7: note: expanded from macro 'JSObjFn'
obj->Set(NanNew<String>(name), NanNew<FunctionTemplate>(fnname)->GetFunction(), attribute_ro_dd);
~~~~~^~~
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2112:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
bool Set(Handle<Value> key, Handle<Value> value);
^
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2114:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
bool Set(uint32_t index, Handle<Value> value);
^
../src/WebWorkerThreads.cc:297:13: error: no matching member function for call to 'Set'
global->Set(NanNew<String>("console"), console_obj, attribute_ro_dd);
~~~~~~~~^~~
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2112:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
bool Set(Handle<Value> key, Handle<Value> value);
^
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2114:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
bool Set(uint32_t index, Handle<Value> value);
^
../src/WebWorkerThreads.cc:345:17: error: no member named 'IdleNotification' in 'v8::V8'
V8::IdleNotification();
~~~~^
../src/WebWorkerThreads.cc:430:17: error: no member named 'IdleNotification' in 'v8::V8'
V8::IdleNotification();
~~~~^
7 errors generated.
make: *** [Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/Users/gitawego/.nvm/v0.11.16/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack at ChildProcess.emit (events.js:110:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:1067:12)
gyp ERR! System Darwin 14.1.0
gyp ERR! command "node" "/Users/gitawego/.nvm/v0.11.16/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/gitawego/workspace/audit-server/node_modules/webworker-threads
gyp ERR! node -v v0.11.16
gyp ERR! node-gyp -v v1.0.2
gyp ERR! not ok
I have a question about webworker-threads usages
When I use .eval(function) and make function thread
function's param object is impossible?
callback function is impossible?
Workers don't have access to setTimeout, setInterval, etc.
I have a small file threads.js
based on this example the content of the file is:
var threads = require('webworker-threads'),
sleep = require('sleep').sleep;
var random = function(low, high) {
return Math.floor(Math.random() * (high - low + 1) + low);
};
function fibo(n, low, high) {
var rnd = random(low, high);
sleep(rnd);
return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
}
var threadOne = threads.create(),
threadTwo = threads.create(),
threadThree = threads.create();
threadOne.eval('fibo(10, 60, 120)', function(err, result) {
if (err) { console.error(err); }
console.log('[ thread-1 ] result: %s', result);
});
threadTwo.eval('fibo(10, 60, 120)', function(err, result) {
if (err) { console.error(err); }
console.log('[ thread-2 ] result: %s', result);
});
threadThree.eval('fibo(10, 60, 120)', function(err, result) {
if (err) { console.error(err); }
console.log('[ thread-3 ] result: %s', result);
});
When I execute node threads.js
:
vagrant@vagrant-ubuntu-trusty-64:~/nodejs$ node threads.js
[Error: ReferenceError: fibo is not defined]
[ thread-2 ] result: null
[Error: ReferenceError: fibo is not defined]
[ thread-3 ] result: null
[Error: ReferenceError: fibo is not defined]
[ thread-1 ] result: null
For every thread I receive the error [Error: ReferenceError: fibo is not defined]
but as we can see I have the function declared as the example and for an experiment, I changed to var fibo = function(n, low, high) { ... }
still did not worked.
I'm working on load testing CPU intensive functions with both synchronous calls as well as web workers on a multi-core VM, and am running into SegFaults when using WebWorker-Threads. It runs about 20-30 requests then dumps the core.
The project is up here: https://github.com/WakeskaterX/NodeThreading
And I have a question on Stack Overflow:
http://stackoverflow.com/questions/26812646/segmentation-fault-during-high-load-concurrency-test-with-webworker-threads
I've tried running it with out the heavy process and it still faults after 20-30 requests.
I'm using Oracle VM Virtual Box 4.3.18 with a 32 bit Ubuntu 14.04 VM with 4 Cores and 4 Gigs of RAM.
The load testing command that breaks it (when the web worker section is uncommented) is:
loadtest -c 4 -t 20 http://localhost:3030/fib?num=30
Is there data: URI support or could it be added?
Hi,
I have an application which spawns several Workers at a time and posting "init" messages from the main thread right away. Then the worker responds "initdone".
I see that main thread is not getting some responses ("initdone") from Workers (I see thru console.log that Workers are receiving the main's thread message "init" and using the postMessage to respond back) . Is it possible that the queue is not working properly?
Thanks
Jose
PS: trying to isolate an example with no luck yet. The app where I see this behavior is complex.
npm http GET https://registry.npmjs.org/webworker-threads
npm http 304 https://registry.npmjs.org/webworker-threads
[email protected] install /Users/vivekanandan/Source/spawn/node_modules/webworker-threads
node-gyp rebuild
CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
In file included from ../src/WebWorkerThreads.cc:36:
../src/bson.cc:199:18: error: use of undeclared identifier 'NanPersistentToLocal'
if(object->Has(NanPersistentToLocal(bson->_bsontypeString)))
^
../src/bson.cc:202:7: error: use of undeclared identifier 'NanPersistentToLocal'
if(NanPersistentToLocal(bson->longString)->StrictEquals(constructorString))
^
../src/bson.cc:205:30: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteInt32(object, NanPersistentToLocal(bson->_longLowString));
^
../src/bson.cc:206:30: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteInt32(object, NanPersistentToLocal(bson->_longHighString));
^
../src/bson.cc:208:12: error: use of undeclared identifier 'NanPersistentToLocal'
else if(NanPersistentToLocal(bson->timestampString)->StrictEquals(constructorString))
^
../src/bson.cc:211:30: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteInt32(object, NanPersistentToLocal(bson->_longLowString));
^
../src/bson.cc:212:30: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteInt32(object, NanPersistentToLocal(bson->_longHighString));
^
../src/bson.cc:214:12: error: use of undeclared identifier 'NanPersistentToLocal'
else if(NanPersistentToLocal(bson->objectIDString)->StrictEquals(constructorString))
^
../src/bson.cc:217:33: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteObjectId(object, NanPersistentToLocal(bson->_objectIDidString));
^
../src/bson.cc:219:12: error: use of undeclared identifier 'NanPersistentToLocal'
else if(NanPersistentToLocal(bson->binaryString)->StrictEquals(constructorString))
^
../src/bson.cc:223:35: error: use of undeclared identifier 'NanPersistentToLocal'
uint32_t length = object->Get(NanPersistentToLocal(bson->_binaryPositionString))->Uint32Value();
^
../src/bson.cc:224:43: error: use of undeclared identifier 'NanPersistentToLocal'
Local bufferObj = object->Get(NanPersistentToLocal(bson->_binaryBufferString))->ToObject();
^
../src/bson.cc:227:29: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteByte(object, NanPersistentToLocal(bson->_binarySubTypeString)); // write subtype
^
../src/bson.cc:229:20: error: use of undeclared identifier 'NanPersistentToLocal'
if(object->Get(NanPersistentToLocal(bson->_binarySubTypeString))->Int32Value() == 0x02) {
^
../src/bson.cc:235:12: error: use of undeclared identifier 'NanPersistentToLocal'
else if(NanPersistentToLocal(bson->doubleString)->StrictEquals(constructorString))
^
../src/bson.cc:238:31: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteDouble(object, NanPersistentToLocal(bson->_doubleValueString));
^
../src/bson.cc:240:12: error: use of undeclared identifier 'NanPersistentToLocal'
else if(NanPersistentToLocal(bson->symbolString)->StrictEquals(constructorString))
^
../src/bson.cc:243:49: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteLengthPrefixedString(object->Get(NanPersistentToLocal(bson->_symbolValueString))->...
^
../src/bson.cc:245:12: error: use of undeclared identifier 'NanPersistentToLocal'
else if(NanPersistentToLocal(bson->codeString)->StrictEquals(constructorString))
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: make
failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:797:12)
gyp ERR! System Darwin 13.1.0
gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/vivekanandan/Source/spawn/node_modules/webworker-threads
gyp ERR! node -v v0.10.26
gyp ERR! node-gyp -v v0.12.2
gyp ERR! not ok
npm ERR! [email protected] install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the webworker-threads package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls webworker-threads
npm ERR! There is likely additional logging output above.
npm ERR! System Darwin 13.1.0
npm ERR! command "node" "/usr/local/bin/npm" "install" "webworker-threads"
npm ERR! cwd /Users/vivekanandan/Source/spawn
npm ERR! node -v v0.10.26
npm ERR! npm -v 1.4.3
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
@springmeyer mentioned it's a viable option now, over at sindresorhus/awesome-nodejs#117 (comment) — URLs to relevant documentations & pullreqs welcome!
Greetings!
I'm running the my code:
main.js:
var Worker = require('webworker-threads').Worker;
var worker = new Worker('worker.js');
worker.onmessage = function (event) {
console.log(event.data);
};
worker.postMessage('John');
worker.js:
onmessage = function(event) {
setInterval(function(){
postMessage('Hi, ' + event.data);
}, 3000);
};
but the code does not work.
What am I doing wrong?
Thanks in advance for help!
e.x. new Worker(function() {}).postMessage(new Buffer(""))
It would be nice if it handled Buffers more gracefully, at a minimum not crashing, and possibly automagically converting to ArrayBuffer (maybe dependent on #18)?
PID 587 received SIGSEGV for address: 0x0
0 segfault-handler.node 0x0000000100fe990f _ZL16segfault_handleriP9__siginfoPv + 287
1 libsystem_platform.dylib 0x00007fff8af3ff1a _sigtramp + 26
2 ??? 0x00000001054806e0 0x0 + 4383573728
3 node 0x00000001005cb828 _ZN4node6Buffer3NewEPN2v87IsolateEPKcm + 58
4 WebWorkerThreads.node 0x0000000103bc9e9b _ZN16BSONDeserializer16DeserializeValueE8BsonTypeb + 375
5 WebWorkerThreads.node 0x0000000103bc9bba _ZN16BSONDeserializer27DeserializeDocumentInternalEb + 154
6 WebWorkerThreads.node 0x0000000103bc9b00 _ZN16BSONDeserializer19DeserializeDocumentEb + 66
7 WebWorkerThreads.node 0x0000000103bc9e4c _ZN16BSONDeserializer16DeserializeValueE8BsonTypeb + 296
8 WebWorkerThreads.node 0x0000000103bc9bba _ZN16BSONDeserializer27DeserializeDocumentInternalEb + 154
9 WebWorkerThreads.node 0x0000000103bc9b00 _ZN16BSONDeserializer19DeserializeDocumentEb + 66
10 WebWorkerThreads.node 0x0000000103bd114a _ZL9eventLoopP10typeThread + 1963
11 WebWorkerThreads.node 0x0000000103bd035b _ZL7aThreadPv + 113
12 libsystem_pthread.dylib 0x00007fff91c8a268 _pthread_body + 131
13 libsystem_pthread.dylib 0x00007fff91c8a1e5 _pthread_body + 0
14 libsystem_pthread.dylib 0x00007fff91c8841d thread_start + 13
I want to parse HTML in child-thread (need jsdom) because it is much slower than the speed fetching web pages, if I parse in main thread, it will fill up the event loop, cause socket timeout and finally ruin the program.
But jsdom is nodejs package, I can't use global variable jsdom
, or declare new variable var jsdom = require('jsdom');
as I do in main-thread,
I also try to pass it to child-thread via parameter, just like worker.postMessage([jsdom, html]);
and of course, failed. Actually I get the jsdom in child-thread. console.log(event.data[0]);
have some output, but It's not the same with main-thread.
I have a test on serveral nodejs-defined variable, such as console.log(typeof(require)); console.log(typeof(global));
and I get undefined or very different.
I think the child-thread using a completely different context with main-thread, so is it possible to use nodejs package in child-thread?
Or, what's the right way to do this? (Fork, spawn, cluster? or anything better? These three things will make the program much more complex, and what I want to do is just a simple spider.)
There's a small error in your new binding.gyp file that breaks my npm install process (node 8.20, ubuntu):
'-fno-exceptions -DV8_USE_UNSAFE_HANDLES' should be '-fno-exceptions', '-DV8_USE_UNSAFE_HANDLES'
(sorry, no pull request, seemed a bit too much effort for this small thing)
Hi. The following code works on Node 0.10 but segfaults on Node 0.12:
$ node -v
v0.12.7
$ cat 1.js
var Worker = require("webworker-threads").Worker;
new Worker(function() {}).postMessage(new Uint8Array(65));
Here is the gdb backtrace with debug symbols:
$ gdb node
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from node...Reading symbols from /usr/lib/debug//usr/bin/nodejs...done.
done.
(gdb) run 1.js
Starting program: /usr/bin/node 1.js
Traceback (most recent call last):
File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff6bca700 (LWP 1779)]
[New Thread 0x7ffff7fe7700 (LWP 1780)]
[New Thread 0x7ffff7fd6700 (LWP 1781)]
[New Thread 0x7ffff7fc5700 (LWP 1782)]
[New Thread 0x7ffff6199700 (LWP 1783)]
[New Thread 0x7ffff5998700 (LWP 1784)]
[New Thread 0x7ffff7fb4700 (LWP 1785)]
[New Thread 0x7ffff7fa3700 (LWP 1786)]
[New Thread 0x7ffff7f92700 (LWP 1787)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff6199700 (LWP 1783)]
0x00000000007981e3 in v8::EscapableHandleScope::EscapableHandleScope(v8::Isolate*) ()
(gdb) bt full
#0 0x00000000007981e3 in v8::EscapableHandleScope::EscapableHandleScope(v8::Isolate*) ()
No symbol table info available.
#1 0x0000000000bb0f25 in node::Buffer::New(v8::Isolate*, char const*, unsigned long) ()
No symbol table info available.
#2 0x00007ffff61b94ea in NanNewBufferHandle (data=0x12d5a46 "", size=65)
at ../node_modules/nan/nan.h:672
No locals.
#3 0x00007ffff61aef0e in BSONDeserializer::DeserializeValue (this=0x7ffff6198a00,
type=BSON_TYPE_BINARY, promoteLongs=true) at ../src/bson.cc:539
length = 65
subType = 0
buffer = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff61999c0}, <No data fields>}
argv = {{val_ = 0x7ffff6198900}, {val_ = 0x7ffff61b843c <v8::Value::IsNull() const+24>}}
#4 0x00007ffff61ae545 in BSONDeserializer::DeserializeDocumentInternal (this=0x7ffff6198a00,
promoteLongs=true) at ../src/bson.cc:427
type = BSON_TYPE_BINARY
name = @0x7ffff6198940: {val_ = 0x7ffff0044a38}
value = @0x0: <error reading variable>
returnObject = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff0044a30}, <No data fields>}
#5 0x00007ffff61ae491 in BSONDeserializer::DeserializeDocument (this=0x7ffff6198c40,
promoteLongs=true) at ../src/bson.cc:414
length = 81
documentDeserializer = {bson = 0x7ffff004bdb0, pStart = 0x12d5a3b "\005data",
p = 0x12d5a46 "", pEnd = 0x12d5a87 ""}
#6 0x00007ffff61af17f in BSONDeserializer::DeserializeValue (this=0x7ffff6198c40,
type=BSON_TYPE_OBJECT, promoteLongs=true) at ../src/bson.cc:576
No locals.
#7 0x00007ffff61ae545 in BSONDeserializer::DeserializeDocumentInternal (this=0x7ffff6198c40,
promoteLongs=true) at ../src/bson.cc:427
type = BSON_TYPE_OBJECT
name = @0x7ffff6198b80: {val_ = 0x7ffff0044a18}
value = @0x7ffff6198e10: {val_ = 0x7ffff004bdb0}
returnObject = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff0044a10}, <No data fields>}
#8 0x00007ffff61ae491 in BSONDeserializer::DeserializeDocument (this=0x7ffff6198e10,
promoteLongs=true) at ../src/bson.cc:414
length = 89
documentDeserializer = {bson = 0x7ffff004bdb0, pStart = 0x12d5a34 "\003\060",
p = 0x12d5a88 "", pEnd = 0x12d5a88 ""}
#9 0x00007ffff61b4ca6 in eventLoop (thread=0x11cc6e0) at ../src/WebWorkerThreads.cc:418
bson = 0x7ffff004bdb0
data = 0x12d5a30 "Y"
i = -166097600
size = 89
deserializer = {bson = 0x7ffff004bdb0, pStart = 0x12d5a30 "Y", p = 0x12d5a89 "",
pEnd = 0x12d5a88 ""}
result = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff0044800}, <No data fields>}
args = {{<v8::Handle<v8::Value>> = {val_ = 0x7ffff00448f8}, <No data fields>},
{<v8::Handle<v8::Value>> = {val_ = 0x7ffff0044908}, <No data fields>}}
len = 1
array = {<v8::Handle<v8::Array>> = {val_ = 0x7ffff0044908}, <No data fields>}
onError = {isolate_ = 0x7ffff00008c0, next_ = 0x0, exception_ = 0x2a4a76804181,
message_obj_ = 0x2a4a76804181, message_script_ = 0x2a4a76804181,
js_stack_comparable_address_ = 0x7ffff6198c48, message_start_pos_ = 0,
message_end_pos_ = 0, is_verbose_ = false, can_continue_ = true, capture_message_ = true,
rethrow_ = false, has_terminated_ = false}
str = 0x12d59f0
resultado = {<v8::Handle<v8::Value>> = {val_ = 0x7ffff00448f0}, <No data fields>}
source = {<v8::Handle<v8::String>> = {val_ = 0x7ffff00448b8}, <No data fields>}
script = {<v8::Handle<v8::Script>> = {val_ = 0x7ffff00448e0}, <No data fields>}
job = 0x12d59b0
qitem = 0x11cfc30
scope = {isolate_ = 0x7ffff00008c0, prev_next_ = 0x0, prev_limit_ = 0x0}
ctx = {<v8::Handle<v8::Context>> = {val_ = 0x7ffff00443c8}, <No data fields>}
ctr = 2
ftmpl = {<v8::Handle<v8::FunctionTemplate>> = {val_ = 0x7ffff00443b0}, <No data fields>}
otmpl = {<v8::Handle<v8::ObjectTemplate>> = {val_ = 0x7ffff00443c0}, <No data fields>}
fs_obj = {val_ = 0x7ffff00443f0}
---Type <return> to continue, or q <return> to quit---
console_obj = {val_ = 0x7ffff0044480}
dispatchEvents = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff0044818}, <No data fields>}
dispatchNextTicks = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff0044858}, <No data fields>}
_ntq = 0x7ffff0044870
nextTickQueueLength = 0
extensions = {name_count_ = 0, names_ = 0x0}
global = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff00443d8}, <No data fields>}
threadObject = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff0044748}, <No data fields>}
isolate_scope = {isolate_ = 0x7ffff00008c0}
#10 0x00007ffff61b3acf in aThread (arg=0x11cc6e0) at ../src/WebWorkerThreads.cc:243
myLocker = {has_lock_ = true, top_level_ = true, isolate_ = 0x7ffff00008c0,
static active_ = true}
dummy = 0
thread = 0x11cc6e0
#11 0x00007ffff6f98182 in start_thread (arg=0x7ffff6199700) at pthread_create.c:312
__res = <optimized out>
pd = 0x7ffff6199700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737322260224, 6434267845187848142, 1, 4,
140737322260928, 140737322260224, -6434253325479983154, -6434252500698019890},
mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0,
cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#12 0x00007ffff6cc547d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
Hi,
first, thankyou for your module, it has the best interface ever.. but I have a problem, in my code I have this (inside a foreach):
.........
if(l !== null){
var worker = new Worker(function(){
this.onmessage = function(event){
var level = event.data.level;
console.log(level);
var p = dowork(level); //dowork defined first
console.log(p);
postMessage(s);
console.log("Processing level: "+level);
self.close();
}});
worker.onmessage = function(event) {
console.log("I'm here");
pile.push(event.data);
};
worker.postMessage({'level':l.level});
}
......
worker.onmessage is not receiving the message from worker on the Master thread, but all the logging tell me that worker performed it's task correctly (receives data from main thread and so on).. and closes.
Is there a bug, or am I missing something?
From a couple of short tests it seems that delegation / subworkers are not working at the moment, eg, this example: http://www.whatwg.org/specs/web-apps/current-work/multipage/workers.html#delegation fails silently.
Thus, the first question is: is this example supposed to work?
$ npm install --save webworker-threads --msvs_version=2012
npm WARN package.json [email protected] No repository field.
> [email protected] install c:\Users\amcwatters\Documents\Repositories\api
-v1\node_modules\webworker-threads
> node-gyp rebuild
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-threads
>node "c:\nodejs\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\
bin\node-gyp.js" rebuild
Building the projects in this solution one at a time. To enable parallel build,
please add the "/m" switch.
WebWorkerThreads.cc
C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\include\xlocale(336): wa
rning C4530: C++ exception handler used, but unwind semantics are not enabled.
Specify /EHsc [c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\w
ebworker-threads\build\WebWorkerThreads.vcxproj]
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-thread
s\node_modules\nan\nan.h(1185): warning C4267: '=' : conversion from 'size_t' t
o 'int', possible loss of data [c:\Users\amcwatters\Documents\Repositories\api-
v1\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-thread
s\node_modules\nan\nan.h(1202): warning C4267: '=' : conversion from 'size_t' t
o 'int', possible loss of data [c:\Users\amcwatters\Documents\Repositories\api-
v1\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-thread
s\node_modules\nan\nan.h(1215): warning C4267: 'initializing' : conversion from
'size_t' to 'int', possible loss of data [c:\Users\amcwatters\Documents\Reposi
tories\api-v1\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-thread
s\node_modules\nan\nan.h(1229): warning C4267: 'initializing' : conversion from
'size_t' to 'int', possible loss of data [c:\Users\amcwatters\Documents\Reposi
tories\api-v1\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-thread
s\node_modules\nan\nan.h(1318): warning C4244: 'return' : conversion from 'intp
tr_t' to 'int', possible loss of data [c:\Users\amcwatters\Documents\Repositori
es\api-v1\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
c:\users\amcwatters\documents\repositories\api-v1\node_modules\webworker-thread
s\src\bson.cc(395): warning C4344: behavior change: use of explicit template ar
guments results in call to 'v8::Local<T> NanNew<v8::String,char*>(P,int)' [c:\U
sers\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-threads\bu
ild\WebWorkerThreads.vcxproj]
with
[
T=v8::String,
P=char *
]
but the regular function 'v8::Local<T> NanNew(const char *,int)' is a
better match
with
[
T=v8::String
]
if you expect 'v8::Local<T> NanNew(const char *,int)' to be called th
en you need to make it an explicit specialization
with
[
T=v8::String
]
c:\users\amcwatters\documents\repositories\api-v1\node_modules\webworker-thread
s\src\bson.cc(405): warning C4344: behavior change: use of explicit template ar
guments results in call to 'v8::Local<T> NanNew<v8::String,uint16_t*>(P,int)' [
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-thread
s\build\WebWorkerThreads.vcxproj]
with
[
T=v8::String,
P=uint16_t *
]
but the regular function 'v8::Local<T> NanNew(const uint16_t *,int)'
is a better match
with
[
T=v8::String
]
if you expect 'v8::Local<T> NanNew(const uint16_t *,int)' to be calle
d then you need to make it an explicit specialization
with
[
T=v8::String
]
c:\users\amcwatters\documents\repositories\api-v1\node_modules\webworker-thread
s\src\bson.cc(891): warning C4267: 'argument' : conversion from 'size_t' to 'ui
nt32_t', possible loss of data [c:\Users\amcwatters\Documents\Repositories\api-
v1\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
c:\users\amcwatters\documents\repositories\api-v1\node_modules\webworker-thread
s\src\jslib.cc(79): warning C4344: behavior change: use of explicit template ar
guments results in call to 'v8::Local<T> NanNew<v8::String,char*>(P)' [c:\Users
\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-threads\build\
WebWorkerThreads.vcxproj]
with
[
T=v8::String,
P=char *
]
but the regular function 'v8::Local<T> NanNew(const char *,int)' is a
better match
with
[
T=v8::String
]
if you expect 'v8::Local<T> NanNew(const char *,int)' to be called th
en you need to make it an explicit specialization
with
[
T=v8::String
]
..\src\WebWorkerThreads.cc(226): warning C4101: 'dummy' : unreferenced local va
riable [c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworke
r-threads\build\WebWorkerThreads.vcxproj]
..\src\WebWorkerThreads.cc(943): error C2601: 'Init' : local function definitio
ns are illegal [c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\
webworker-threads\build\WebWorkerThreads.vcxproj]
..\src\WebWorkerThreads.cc(940): this line contains a '{' which has n
ot yet been matched
..\src\WebWorkerThreads.cc(975): error C2598: linkage specification must be at
global scope [c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\we
bworker-threads\build\WebWorkerThreads.vcxproj]
..\src\WebWorkerThreads.cc(975): error C2071: 'WebWorkerThreads_module' : illeg
al storage class [c:\Users\amcwatters\Documents\Repositories\api-v1\node_module
s\webworker-threads\build\WebWorkerThreads.vcxproj]
..\src\WebWorkerThreads.cc(980): fatal error C1075: end of file found before th
e left brace '{' at '..\src\WebWorkerThreads.cc(940)' was matched [c:\Users\amc
watters\Documents\Repositories\api-v1\node_modules\webworker-threads\build\WebW
orkerThreads.vcxproj]
gyp ERR! build error
gyp ERR! stack Error: `C:\windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe
` failed with exit code: 1
gyp ERR! stack at ChildProcess.onExit (c:\nodejs\node_modules\npm\node_modul
es\node-gyp\lib\build.js:267:23)
gyp ERR! stack at ChildProcess.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:810:
12)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "node" "c:\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\
bin\\node-gyp.js" "rebuild"
gyp ERR! cwd c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webw
orker-threads
gyp ERR! node -v v0.10.32
gyp ERR! node-gyp -v v1.0.1
gyp ERR! not ok
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the webworker-threads package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls webworker-threads
npm ERR! There is likely additional logging output above.
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "c:\\nodejs\\node.exe" "c:\\nodejs\\node_modules\\npm\\bin\\npm
-cli.js" "install" "--save" "webworker-threads" "--msvs_version=2012"
npm ERR! cwd c:\Users\amcwatters\Documents\Repositories\api-v1
npm ERR! node -v v0.10.32
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! not ok code 0
Hi, I am running the example code listed but with fibo(30) instead of fibo(40) and using a valid port number:
var Worker = require('webworker-threads').Worker;
require('http').createServer(function (req,res) {
var fibo = new Worker(function() {
function fibo (n) {
return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
}
this.onmessage = function (event) {
postMessage(fibo(event.data));
}
});
fibo.onmessage = function (event) {
res.end('fib(30) = ' + event.data);
};
fibo.postMessage(30);
}).listen(8080);
Steps to reproduce:
Although this Worker api is not explicitly configured using a thread pool, I do expect the thread count to start off with some kind of default pool size and remain there. Both the size of the thread count increase is unexpected and as is the behavior of a never-ending increase in count.
This is just a basic example that serves to show the api; however this behavior really precludes the use of this lib in a long-lived application. It looks like there is fine-grained control over threading using some of the other apis. I am interested to see a better usage of the Worker api, that prevents this behavior using the steps listed, together with an explanation for what I am observing.
Thanks!
Hello
This is more of a question than an issue, and may become a feature request.
How can I make use of node modules in workers? Especially node modules that rely on other modules which make use of native C++ code? ex: https://github.com/brianc/node-postgres
Thanks
When I tried to require some native modules in threads like this:
// Generated by CoffeeScript 1.6.2
var assert, equal;
assert = require('assert');
equal = assert.equal;
self.onmessage = function(e) {
return self.postMessage({
test: 'QAQ'
});
};
and worker.postMessage doesn't work
// Generated by CoffeeScript 1.6.2
var Worker;
Worker = require('webworker-threads').Worker;
setInterval(function() {
var worker;
worker = new Worker('client.js');
worker.onmessage = function() {//this function has never called since the thread requires modules
return console.dir(arguments);
};
return worker.postMessage({
foo: 'bar'
});
}, 1000);
When i use
threadpool.any.emit('string', object);
I get the following error, but if i use threadpool.all.emit, it works.
Could anybody provide any info? Thanks
undefined:5
tion(e,d){var f;nextJob(t);f=job.cbOrData;if(f){return job.cbOrData.call(t,e,d
^
TypeError: Object # has no method 'call'
at null. (:5:170)
Greetings
I'm running the following example:
var Worker = require('webworker-threads').Worker;
var fibo = new Worker(function() {
function fibo (n) {
return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
}
onmessage = function (event) {
postMessage(fibo(event.data));
}
});
fibo.onmessage = function (event) {
res.end('fib(40) = ' + event.data);
};
fibo.postMessage(40);
The Worker is created, but postMessage mostly does nothing, i.e. the onmessage is not being executed, for whatever reasons.
I've tried many of the Threads examples though, and they work fine.
What am I doing wrong?
Thanks in advance for some help!
I am getting a the receiver must be a thread object. What excalty does it mean?
Using Node.js 0.10.29 built from source, NPM 1.4.14, and [email protected], I don't get the expected output from examples/quickIntro_oneThreadEvented.js, or an even simpler example:
{{{
The gyp build fails on node 0.11.x
. 0.10.9
builds fine.
> [email protected] install /tmp/wwtest/node_modules/webworker-threads
> node-gyp rebuild
make: Entering directory `/tmp/wwtest/node_modules/webworker-threads/build'
CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
[...]
../src/WebWorkerThreads.cc: In function ‘typeThread* isAThread(v8::Handle<v8::Object>)’:
../src/WebWorkerThreads.cc:152:39: error: ‘class v8::Object’ has no member named ‘GetPointerFromInternalField’
[...]
For the full log see this gist.
On OS X using 0.4.3 every file I try to load gives me Error reading the file some_file.js
This appears to be caused by the most recent commit because running git checkout HEAD~1 && npm install
and then ln -s
that into the repo, it loads perfectly fine.
I suspect an off-by-one error, here, and will make another PR if I can fix it, but I wouldn't mind a simple revert since the OS failing to provide the entire file means the server is probably going to fail, anyways, and this sort of check is overkill for a dynamic language where the developer has no control over memory usage.
Also, in the future could you increment the version number and keep the old one in NPM, though, even if you think its broken? Starting today I want to use webworker-threads in production, but having what I'm installing change after I've specified a hardwired version number is not fun. (Caught this by dumping my node_modules
directory and re-installing with --production
from scratch before merging my internal PR.)
I'd rather stick with a version with known issues that I know isn't affecting my code and then upgrade to the better version after testing than be automatically updated and not know about it. :)
Thanks again,
David
Hi
Ive setup a worker thread to run that roughly takes around 3 seconds to process.
In a timer ive got another process that every 500milliseconds runs and determines how many threads are available and if there are any available repeat the first step again.
After the first iteration of the 500minisecond timer, while the first worker is still running my pool shows 1 idle thread even though it is still executing its code (using thread.nextTick so the application doesnt lock)
Am i looking at this wrong or is there an issue with that?
Thanks
Regards
Craig
When using a thread to execute a function that returns a promise,
on the function's immediate return the thread is considered to be terminated,
therefore the threads' pool is initializing more parallel threads then it is supposed to :(
Besides, I fail to get the promise returned, it appears as " 'undefined' "
Solution? workaround?
THANKS
Using Node.js 0.10.29 built from source, NPM 1.4.14, and [email protected], on Ubuntu 12.0.4.4 LTS (x86_64), I don't get the expected output from examples/quickIntro_oneThreadEvented.js, or an even simpler example:
var fs = require('fs');
var Worker = require('webworker-threads').Worker;
var w = new Worker(function() {
var self = this;
self.onmessage(function(event) {
self.postMessage(event.data+1);
});
});
w.onmessage = function callback(event) {
process.stdout.write(event.data);
fs.writeFileSync("myoutput.txt", event.data, {'encoding': 'utf-8'});
}
w.postMessage(5);
process.stdout.write("Did anything happen?\n");
/* Expect 6 to be written to stdout eventually */
I just see "Did anything happen?", and the program hangs forever without output (no file is written either).
I can't tell if this is a problem with node-gyp, or node-webworker-threads.
Here is the relevant log:
> [email protected] install /Users/will/Code/happy/node_modules/webworker-threads
> node-gyp rebuild
WARN install got an error, rolling back install
gyp ERR! configure error
gyp ERR! stack Error: 404 status code downloading tarball
gyp ERR! stack at Request.<anonymous> (/Users/will/.nvm/versions/io.js/v1.8.1/lib/node_modules/npm/node_modules/node-gyp/lib/install.js:246:14)
gyp ERR! stack at emitOne (events.js:82:20)
gyp ERR! stack at Request.emit (events.js:166:7)
gyp ERR! stack at Request.onRequestResponse (/Users/will/.nvm/versions/io.js/v1.8.1/lib/node_modules/npm/node_modules/request/request.js:1156:10)
gyp ERR! stack at emitOne (events.js:77:13)
gyp ERR! stack at ClientRequest.emit (events.js:166:7)
gyp ERR! stack at HTTPParser.parserOnIncomingClient (_http_client.js:412:21)
gyp ERR! stack at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)
gyp ERR! stack at Socket.socketOnData (_http_client.js:302:20)
gyp ERR! stack at emitOne (events.js:77:13)
gyp ERR! System Darwin 14.3.0
gyp ERR! command "/Users/will/.nvm/versions/io.js/v1.8.1/bin/iojs" "/Users/will/.nvm/versions/io.js/v1.8.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/will/Code/happy/node_modules/webworker-threads
gyp ERR! node -v v1.8.1
gyp ERR! node-gyp -v v1.0.3
gyp ERR! not ok
npm ERR! Darwin 14.3.0
npm ERR! argv "/Users/will/.nvm/versions/io.js/v1.8.1/bin/iojs" "/Users/will/.nvm/versions/io.js/v1.8.1/bin/npm" "install" "webworker-threads"
npm ERR! node v1.8.1
npm ERR! npm v2.8.4
npm ERR! code ELIFECYCLE
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script 'node-gyp rebuild'.
npm ERR! This is most likely a problem with the webworker-threads package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls webworker-threads
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /Users/will/Code/happy/npm-debug.log
My googling has shown that this might be an issue with the latest versions of node and io and node-gyp, although I have seen other packages that have submitted fixes for similar issues.
I cannot use ArrayBuffer from a worker:
worker.js
:
console.log(typeof ArrayBuffer);
main.js
:
console.log(typeof ArrayBuffer);
Worker = require('webworker-threads').Worker;
new Worker('./worker.js');
Then:
> nodejs main.js
prints
function
undefined
NodeJS v0.10.17, webworker-threads v0.4.7
Hello,
I'm using this module and I'm having an extreme level of difficulty trying to figure out why I'm getting segmentation faults. I'm trying to use this module to be able to execute a regular expression, but with a timeout. The regular expressions are provided by users and thus we need to protect against users who put in a malicious regex that takes a long time to execute.
No matter what I do and how I modify the code, it always seems to crash with exactly the same segmentation fault error:
PID 27647 received SIGSEGV for address: 0x7e68
/home/bradley/sensibill-api/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x10e5)[0x7f63535fd0e5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f635a50c340]
/usr/local/bin/node[0x6c16ef]
/usr/local/bin/node(ZN2v85Utils16ReportApiFailureEPKcS2+0x12)[0x6c2302]
/usr/local/bin/node(_ZN2v87Isolate7DisposeEv+0x25)[0x6d8ac5]
/home/bradley/sensibill-api/node_modules/webworker-threads/build/Debug/WebWorkerThreads.node(+0x17377)[0x7f63531db377]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f635a504182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f635a23147d]
The address is always EXACTLY the same - 0x7e68. It never changes no matter what I do to the code, its always segfaulting on that address.
Code looks like this:
var Threads = require('webworker-threads'),
Worker = Threads.Worker;
var globalWorker = null;
/**
@param regex
@param str
@param callback
*/
module.exports.match = function (regex, str, callback)
{
var error = null;
var match = null;
if (!regex)
{
callback(error, match);
return;
}
var timeoutID = null;
globalWorker = new Worker(matchWorker);
globalWorker.onmessage = function (evt)
{
// When the worker says its loaded start a timer.
if (evt.data == "onload")
{
timeoutID = setTimeout(function ()
{
globalWorker.terminate();
callback("timeout", match);
}, 250);
}
else
{
match = evt.data.match;
error = evt.data.error;
clearTimeout(timeoutID);
globalWorker.terminate();
callback(error, match);
}
};
globalWorker.postMessage({regex: regex, str: str});
};
function matchWorker ()
{
var self = this;
this.onmessage = function (evt)
{
"use strict";
var regex = new RegExp(evt.data.regex);
var str = evt.data.str;
var error = null;
var match;
// Let the callee know we're loaded
self.postMessage("onload");
match = regex.exec(str);
self.postMessage({error: error, match: match});
self.close();
}
}
node-gyp rebuild
child_process: customFds option is deprecated, use stdio instead.
make: Entering directory `/home/marco.minetti/Desktop/test/node_modules/webworker-threads/build'
CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
In file included from ../src/WebWorkerThreads.cc:6:0:
/home/marco.minetti/.node-gyp/0.11.14/src/node.h:377:43: error: overloaded function with no contextual type information
(node::addon_register_func) (regfunc),
^
/home/marco.minetti/.node-gyp/0.11.14/src/node.h:408:3: note: in expansion of macro ‘NODE_MODULE_X’
NODE_MODULE_X(modname, regfunc, NULL, 0)
^
../src/WebWorkerThreads.cc:971:1: note: in expansion of macro ‘NODE_MODULE’
NODE_MODULE(WebWorkerThreads, Init)
^
make: *** [Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o] Error 1
Using npm install webworker-threads will not work unless you have make and cc (as that is what node-gyp requires). On OSX, one way to do this is:
install XCode from Apple (https://itunes.apple.com/au/app/xcode/id497799835) and then use it to install command line tools (under Preferences -> Downloads).
It would be nice to have the install instructions in the README updated to reflect this.
I get a failure during node-gyp rebuild:
CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
../src/WebWorkerThreads.cc:943:35: error: function definition is not allowed
here
void Init (Handle<Object> target) {
^
../src/WebWorkerThreads.cc:975:1: error: expected unqualified-id
NODE_MODULE(WebWorkerThreads, Init)
^
/Users/Matthew/.node-gyp/0.10.13/src/node.h:224:10: note: expanded from macro
'NODE_MODULE'
extern "C" { \
^
../src/WebWorkerThreads.cc:979:3: error: expected '}'
*/
^
../src/WebWorkerThreads.cc:940:57: note: to match this '{'
void Init (Handle<Object> target, Handle<Value> module) {
^
3 errors generated.
make: *** [Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:789:12)
gyp ERR! System Darwin 14.0.0
gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
...etc
importScripts currently only supports laoding a file from disk.
On the W3C standard (http://www.w3.org/TR/workers/#importing-scripts-and-libraries), it support http:// & https:// urls
Would be great if supported
Installing webworker-threads fails when using io.js 2.0.0:
> [email protected] install /home/travis/build/andywer/threadpool-js/node_modules/webworker-threads
> node-gyp rebuild
make: Entering directory `/home/travis/build/andywer/threadpool-js/node_modules/webworker-threads/build'
CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
In file included from ../node_modules/nan/nan_new.h:190:0,
from ../node_modules/nan/nan.h:74,
from ../src/WebWorkerThreads.cc:13:
../node_modules/nan/nan_implementation_12_inl.h: In static member function ‘static NanIntern::FactoryBase<v8::Signature>::return_t NanIntern::Factory<v8::Signature>::New(NanIntern::Factory<v8::Signature>::FTH, int, NanIntern::Factory<v8::Signature>::FTH*)’:
../node_modules/nan/nan_implementation_12_inl.h:181:76: error: no matching function for call to ‘v8::Signature::New(v8::Isolate*, NanIntern::Factory<v8::Signature>::FTH&, int&, NanIntern::Factory<v8::Signature>::FTH*&)’
../node_modules/nan/nan_implementation_12_inl.h:181:76: note: candidate is:
/home/travis/.node-gyp/2.0.0/deps/v8/include/v8.h:4188:27: note: static v8::Local<v8::Signature> v8::Signature::New(v8::Isolate*, v8::Handle<v8::FunctionTemplate>)
/home/travis/.node-gyp/2.0.0/deps/v8/include/v8.h:4188:27: note: candidate expects 2 arguments, 4 provided
In file included from ../src/WebWorkerThreads.cc:38:0:
../src/bson.cc: In static member function ‘static void BSON::BSONDeserialize(const v8::FunctionCallbackInfo<v8::Value>&)’:
../src/bson.cc:796:45: warning: ‘ssize_t node::DecodeBytes(v8::Handle<v8::Value>, node::encoding)’ is deprecated (declared at /home/travis/.node-gyp/2.0.0/src/node.h:297): Use DecodeBytes(isolate, ...) [-Wdeprecated-declarations]
../src/bson.cc:803:41: warning: ‘ssize_t node::DecodeWrite(char*, size_t, v8::Handle<v8::Value>, node::encoding)’ is deprecated (declared at /home/travis/.node-gyp/2.0.0/src/node.h:310): Use DecodeWrite(isolate, ...) [-Wdeprecated-declarations]
In file included from ../src/WebWorkerThreads.cc:38:0:
../src/bson.cc: In static member function ‘static void BSON::BSONSerialize(const v8::FunctionCallbackInfo<v8::Value>&)’:
../src/bson.cc:897:73: warning: ‘v8::Local<v8::Value> node::Encode(const void*, size_t, node::encoding)’ is deprecated (declared at /home/travis/.node-gyp/2.0.0/src/node.h:278): Use Encode(isolate, ...) [-Wdeprecated-declarations]
../src/WebWorkerThreads.cc: In function ‘void Callback(uv_async_t*, int)’:
../src/WebWorkerThreads.cc:540:37: warning: ‘void node::FatalException(const v8::TryCatch&)’ is deprecated (declared at /home/travis/.node-gyp/2.0.0/src/node.h:262): Use FatalException(isolate, ...) [-Wdeprecated-declarations]
make: *** [Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o] Error 1
See travis log.
Hint: The failure seems NOT to be related to #55. At least not on first glance ;)
Does thread.destroy() actually destroys the thread? Thanks.
hi,
i'm trying to access to a sqlite3 db inside the web worker, i use this sqlite3 library:
https://github.com/mapbox/node-sqlite3
this is what i do:
var Worker = require('webworker-threads').Worker;
var worker = new Worker(function(){
postMessage("I'm working before postMessage('ali').");
this.onmessage = function(event) {
console.log("before sqlite3");
var sqlite3 = require('sqlite3').verbose();
var db = new sqlite3.Database('MyDB.db');
console.log("after sqlite3");
};
});
worker.postMessage('test');
this is the result:
before sqlite3
and then block, how i can solve the problem? i do something wrong?
I've been doing some tests with workers and I realize that several async functions (e.g. setInterval, setTimer, xhr requests, etc) are not working inside threads.
I understand that this happens because of the lack of a "true" JS eventLoop in th thread.
Is this something that can be implemented? How hard it is?
No dependencies are listed on the main repo page however I've encountered the following error when trying to install. I do not currently have Xcode installed on my Mac:
bash-3.2$ npm install webworker-threads
npm WARN package.json [email protected] No repository field.
npm http GET https://registry.npmjs.org/webworker-threads
npm http 200 https://registry.npmjs.org/webworker-threads
npm http GET https://registry.npmjs.org/webworker-threads/-/webworker-threads-0.4.7.tgz
npm http 200 https://registry.npmjs.org/webworker-threads/-/webworker-threads-0.4.7.tgz
[email protected] install /Users//Documents/git/node-geoservices-adaptor/node_modules/webworker-threads
node-gyp rebuild
gyp http GET http://nodejs.org/dist/v0.10.15/node-v0.10.15.tar.gz
gyp http 200 http://nodejs.org/dist/v0.10.15/node-v0.10.15.tar.gz
xcode-select: Error: No Xcode is selected. Use xcode-select -switch , or see the xcode-select manpage (man xcode-select) for further information.
gyp: Error 2 running xcodebuild
gyp ERR! configure error
gyp ERR! stack Error: gyp
failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:424:16)
gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:789:12)
gyp ERR! System Darwin 12.4.0
gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users//Documents/git/node-geoservices-adaptor/node_modules/webworker-threads
gyp ERR! node -v v0.10.15
gyp ERR! node-gyp -v v0.10.6
gyp ERR! not ok
npm ERR! weird error 1
npm ERR! not ok code 0
bash-3.2$
Hi,
I try to use webworker-threads on my node webkit project.
After a npm-install, I got the following error when trying to run nw . :
Uncaught Error: dlopen(.../WebWorkerThreads.node, 1): no suitable image found. Did find:
/.../build/Release/WebWorkerThreads.node: mach-o, but wrong architecture", source: module.js (346)
I tried to rebuild with nw-gyp rebuild.
I also tried to modify the binding.gyp and add the following flags '-arch i386', '-arch x86_64', but still the same error.
But still the same error.
Do you have any idea ?
When installing with npm on Windows 8.1 I get the following error:
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V110\Microsoft.Cpp.Platform.targets(44,5): error MSB8020: The builds tools for v120 (Platform Toolset = 'v120') cannot be found. To build using the v120 build tools, either click the Project menu or right-click the solution, and then select "Update VC++ Projects...". Install v120 to build using the v120 build tools. [S:\dev\nodegyptest\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
I am not sure how to change the node-gyp build settings so a work around for anyone experiencing this issue follows:
Builds without issue for me using this method.
Setup:
Windows 8.1
Visual Studio 2013
Python 2.7.9
When I run a loop inside thread, thread still keeps running at 50% cpu even after loop finished.
Node v0.10.26, Windows 8.1
Is there explicit need to close the Worker/Thread? Thread should be idle, waiting for another onmessage and not exhausting processor.
Use case:
var Threads = require('webworker-threads');
var worker = new Threads.Worker(function() {
function loop() {
var a = 0;
for(var i = 0; i < 900000000; i++) {
a = Math.pow(2, 2) + i;
a /= 4;
}
return a;
}
function fibo (n) {
return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
}
this.onmessage = function (event) {
// works ok
//fibo(44);
//console.log('thread finished');
// does not work: if not closed, keeps exhausting CPU
loop();
console.log('thread finished');
//self.close(); // I dont want to close, just keep in the background to reuse
};
});
console.log('starting thread');
worker.postMessage();
It seems like it locks on cpu usage end keep "using" :) Once loop exhaust CPU to > 50%, it stays there even when loop has finished.
Executing fibo function works as expected, both functions exhaust CPU and last several seconds, but loop behaves wrong.
I'm investigating this library as a replacement for child processes for most situations, but I've run into a problem while testing the library -- an Ubuntu server with build-essential
and node-gyp
can't compile the code.
> [email protected] install /home/damocles/webworker-childproc-showdown/node_modules/webworker-threads
> node-gyp rebuild
make: Entering directory `/home/damocles/webworker-childproc-showdown/node_modules/webworker-threads/build'
CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
../src/WebWorkerThreads.cc: In function ‘char* readFile(v8::Handle<v8::String>)’:
../src/WebWorkerThreads.cc:649:25: warning: ignoring return value of ‘size_t fread(void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result [-Wunused-result]
SOLINK_MODULE(target) Release/obj.target/WebWorkerThreads.node
/usr/bin/ld: errno: TLS definition in /lib/x86_64-linux-gnu/libc.so.6 section .tbss mismatches non-TLS reference in Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
/lib/x86_64-linux-gnu/libc.so.6: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [Release/obj.target/WebWorkerThreads.node] Error 1
make: Leaving directory `/home/damocles/webworker-childproc-showdown/node_modules/webworker-threads/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/node-gyp/lib/build.js:232:23)
gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:91:17)
gyp ERR! stack at Process._handle.onexit (child_process.js:674:10)
gyp ERR! System Linux 3.2.0-23-virtual
gyp ERR! command "node" "/usr/local/bin/node-gyp" "rebuild"
gyp ERR! cwd /home/damocles/webworker-childproc-showdown/node_modules/webworker-threads
gyp ERR! node -v v0.8.8
gyp ERR! node-gyp -v v0.8.0
gyp ERR! not ok
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! `sh "-c" "node-gyp rebuild"` failed with 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the webworker-threads package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls webworker-threads
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 3.2.0-23-virtual
npm ERR! command "node" "/usr/bin/npm" "install"
npm ERR! cwd /home/damocles/webworker-childproc-showdown
npm ERR! node -v v0.8.8
npm ERR! npm -v 1.1.59
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /home/damocles/webworker-childproc-showdown/npm-debug.log
npm ERR! not ok code 0
According to this forum, this is the offending line of code, and the solution is to replace that line of code with #include <errno.h>
I will be forking and testing this. If this fixes it (and doesn't screw up OS X) I will make a PR.
Hi
Firstly, great work. I have a huge OCD for not using 3rd party libraries, but your 'node-webworker-threads' feels like it should be part of the core node.
As context, I've written a browser based wep-app that relies on WebWorkers, and now I'm able to use your library to simulate a headless client at the server side with node, with almost no coding changes to the App. A real breakthrough for me!
I've come up against a challenge though, as I also require WebSockets - initiated from within the WebWorker.
Inside my worker code I have :
connection = new WebSocket( url, protocol);
I've run some successfull tests from a node main thread using:
var WebSocket = require('websocket').client; [from https://github.com/Worlize/WebSocket-Node]
However I can't use require to pull in modules into a node-webworker.
Can you let me know your recommended approach to get around this?
Many thanks.
I'm attempting to use the library to run a few background threads in a node app, each processing an 8-32MB Uint8Array. Running with 4 workers and 4MB of data each, it currently takes more than 15 minutes to post the initial messages (before it can even start processing). The performance is rather bad, since typed arrays are supposed to be Transferable.
So far as I can tell, each array is being serialized repeatedly when being transferred between threads. This appears to be happening here (thanks to apsillers for finding that). Producing those strings peaks around 7GB of RES memory.
My test app consists of two files:
var Worker = require('webworker-threads').Worker;
var workerCount = 4, finishedCount = 0, bufferSize = (1024 * 1024 * 4);
var workers = {};
var buffers = {};
function messageReceiver(event) {
var data = event.data;
var id = data.id;
var buffer = data.buffer;
console.log('Received', buffer[0], 'from', id);
workers[id].terminate();
if (++finishedCount === workerCount) {
console.log('All workers finished.');
}
}
for (var i = 0; i < workerCount; ++i) {
console.log('Spawning worker', i);
var worker = new Worker('worker.js');
worker.onmessage = messageReceiver;
workers[i] = worker;
}
console.log('All workers spawned.');
for (var i = 0; i < workerCount; ++i) {
console.log('Allocating buffer', i);
buffers[i] = new Uint32Array(new Buffer(bufferSize));
}
console.log('All buffers allocated.');
for (var i = 0; i < workerCount; ++i) {
console.log('Launching worker', i);
var w = workers[i];
var a = buffers[i];
var data = {
id: i,
buffer: a.buffer
};
w.postMessage(data, [data.buffer]);
}
console.log('All workers launched.');
function finishWorker(id, buffer) {
console.log('Thread ' + id + ' finished filling buffer.');
var data = {
id: id,
buffer: buffer
};
postMessage(data, [data.buffer]);
}
function fillBuffer(buffer) {
for (var i = 0; i < buffer.byteLength; ++i) {
if (i % (1024 * 1024) === 0) {
console.log('Thread ' + id + ' filled ' + i + ' bytes');
}
buffer[i] = ((Math.random() * 255) | 0) % 255;
}
}
this.onmessage = function (event) {
var id = event.data.id, buffer = event.data.buffer;
console.log('Thread ' + id + ' received buffer of ' + buffer.byteLength);
fillBuffer(buffer);
finishWorker(id, buffer);
};
I am trying to pass an object to a worker
function update(data) {
var worker = new Worker(function() {
postMessage('start processing');
this.onmessage = function(event) {
// ...
postMessage('finished processing');
self.close();
};
});
worker.onmessage = function(event) {
console.log("Worker said : " + event.data);
};
worker.postMessage(data);
}
when running on small strings / objects it works fine, when trying to load my big object in
(couple of kb) I get
#
# Fatal error in ../deps/v8/src/api.h, line 297
# CHECK(allow_empty_handle || that != __null) failed
#
any idea ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.