anothersmith / node-duckdb Goto Github PK

View Code? Open in Web Editor NEW

48.0 48.0 13.0 1.36 MB

DuckDB NodeJS bindings

License: MIT License

CMake 1.67% C++ 26.72% JavaScript 0.77% TypeScript 70.06% Dockerfile 0.50% Shell 0.29%

node-duckdb's People

Contributors

Stargazers

Watchers

Forkers

willium jpkli bayestechnologies andrei-bitca-dc pewh bermi rostislavprovodenko antonycourtney chenecharry ozysky rprovodenko karthik-pave rgoupil

node-duckdb's Issues

Provide pre-built binaries for alpine/musl

It'd be great if node-duckdb would provide pre-built binaries for Alpine builds, which relies on musl rather than glibc.

Alpine is a very common choice of docker base image for CD/CI and production systems. As no pre-built binaries of node-duckdb can be found for musl, it currently forces all these users to rebuild the package from scratch, which in our case adds an extra 15 to 25 minutes to the GitHub pipeline.
In order to fight this, we are then forced to implement caching mechanism to those pipelines or to swap to a debian docker base image, which further increase the cost and complexity of integrating node-duckdb to a project.

Building binaries for musl would solve the issue for all users relying on alpine variants or other musl-based flavors without forcing them to invest more time and effort into adapting their existing system to fit node-duckdb.

I am willing to take care of the PR for that. From my understanding it involves adding an alpine build image in the docker-compose.yml file, adding a prebuild:alpine rule to the package.json, modifying the github pipeline before updating the dev doc to reflect the changes for manual build.
If that's correct, I would like some hint for the modification to the Github pipelines as I don't see where the different OS build are spawned.

Many thanks and I wish you the best for 2022!

Windows build?

Love the simplicity of the API and nice packaging (like including parquet support) of node-duckdb!

I'm developing a packaged application (Tad, a tabular data viewer). I've got it working nicely using node-duckdb on Mac and Linux, but I'd also like to include a Windows build as I did for past releases that were built on sqlite.

Has anyone attempted to build node-duckdb for Windows? Any known / expected compatibility issues? How dependent is the build process on being able to run Unix-like commands and shell scripts?

Support for the C++ Appender API

It'd be great to be able to bulk imports from JavaScript in-memory data. DuckDB exposes a C++ Appender API for this purpose, and it's much more performant than INSERT statements.

Failed to compile with webpack 4

Hi, receiving these errors when compiling with webpack 4

[./node_modules/node-duckdb/build/Release/node-duckdb-addon.node] 279 bytes {main} [built] [failed] [1 error]
    + 1544 hidden modules

ERROR in ./node_modules/node-duckdb/dist/addon/duckdb.js
Module not found: Error: Can't resolve 'fs' in '/home/xxxx/projectRoot/node_modules/node-duckdb/dist/addon'
 @ ./node_modules/node-duckdb/dist/addon/duckdb.js 4:13-26
 @ ./node_modules/node-duckdb/dist/addon/index.js
 @ ./node_modules/node-duckdb/dist/index.js
 @ ./components/Map/index.js
 @ ./components/main/Desktop/index.js
 @ ./Routes.jsx
 @ ./App.jsx
 @ ./index.js

ERROR in ./node_modules/node-duckdb/build/Release/node-duckdb-addon.node 1:0
Module parse failed: Unexpected character '' (1:0)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
(Source code omitted for this binary file)
 @ ./node_modules/node-duckdb/dist/addon-bindings/connection-binding.js 6:23-76
 @ ./node_modules/node-duckdb/dist/addon-bindings/index.js
 @ ./node_modules/node-duckdb/dist/addon/duckdb.js
 @ ./node_modules/node-duckdb/dist/addon/index.js
 @ ./node_modules/node-duckdb/dist/index.js
 @ ./components/Map/index.js
 @ ./components/main/Desktop/index.js
 @ ./Routes.jsx
 @ ./App.jsx
 @ ./index.js
Child html-webpack-plugin for "index.html":
     1 asset
    Entrypoint undefined = index.html
       4 modules
ℹ ｢wdm｣: Failed to compile.

Library not loaded libcrypto.1.1.dylib

Hello,
I have issue when running app with node-duckdb v0.0.78 installed (running on Mac OS)

Error: dlopen(/Users/user/project/node_modules/node-duckdb/build/Release/node-duckdb-addon.node, 1): Library not loaded: /opt/local/lib/libcrypto.1.1.dylib
  Referenced from: /Users/user/project/node_modules/node-duckdb/build/Release/node-duckdb-addon.node
  Reason: image not found
    at Object.Module._extensions..node (node:internal/modules/cjs/loader:1183:18)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:94:18)
    at Object.<anonymous> (/Users/user/project/node_modules/node-duckdb/src/addon-bindings/connection-binding.ts:12:24)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)

yarn add duckdb errors out using node 19

This work fine for me using node 18. Im on Apple mac Intel chip.
Here is the output i get when I yarn add duckdb using node 19

warning Error running install script for optional dependency: "/Users/chris/flipside/product-eng/node_modules/unix-dgram: Command failed.
Exit code: 7
Command: node-gyp rebuild
Arguments:
Directory: /Users/chris/flipside/product-eng/node_modules/unix-dgram
Output:
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | darwin | x64
gyp info find Python using Python version 3.9.13 found at \"/usr/local/opt/[email protected]/bin/python3.9\"
gyp ERR! UNCAUGHT EXCEPTION
gyp ERR! stack TypeError: Cannot assign to read only property 'cflags' of object '#<Object>'
gyp ERR! stack     at createConfigFile (/Users/chris/flipside/product-eng/node_modules/node-gyp/lib/configure.js:117:21)
gyp ERR! stack     at /Users/chris/flipside/product-eng/node_modules/node-gyp/lib/configure.js:84:9
gyp ERR! stack     at FSReqCallback.oncomplete (node:fs:189:23)
gyp ERR! System Darwin 20.6.0
gyp ERR! command \"/Users/chris/.nvm/versions/node/v19.0.0/bin/node\" \"/Users/chris/flipside/product-eng/node_modules/.bin/node-gyp\" \"rebuild\"
gyp ERR! cwd /Users/chris/flipside/product-eng/node_modules/unix-dgram
gyp ERR! node -v v19.0.0
[14/28] ⠠ @prisma/client
[21/28] ⠠ duckdb
[23/28] ⠠ @temporalio/core-bridge
[9/28] ⠠ sharp
warning Error running install script for optional dependency: "/Users/chris/flipside/product-eng/node_modules/deasync: Command failed.
Exit code: 7
Command: node ./build.js
Arguments:
Directory: /Users/chris/flipside/product-eng/node_modules/deasync
Output:
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | darwin | x64
gyp info find Python using Python version 3.9.13 found at \"/usr/local/opt/[email protected]/bin/python3.9\"
gyp ERR! UNCAUGHT EXCEPTION
gyp ERR! stack TypeError: Cannot assign to read only property 'cflags' of object '#<Object>'
gyp ERR! stack     at createConfigFile (/Users/chris/flipside/product-eng/node_modules/node-gyp/lib/configure.js:117:21)
gyp ERR! stack     at /Users/chris/flipside/product-eng/node_modules/node-gyp/lib/configure.js:84:9
gyp ERR! stack     at FSReqCallback.oncomplete (node:fs:189:23)
gyp ERR! System Darwin 20.6.0
gyp ERR! command \"/Users/chris/.nvm/versions/node/v19.0.0/bin/node\" \"/Users/chris/flipside/product-eng/node_modules/.bin/node-gyp\" \"rebuild\"
gyp ERR! cwd /Users/chris/flipside/product-eng/node_modules/deasync
gyp ERR! node -v v19.0.0
gyp ERR! node-gyp -v v7.1.2
[14/28] ⠄ @prisma/client
[21/28] ⠄ duckdb
[-/28] ⠄ waiting...
[9/28] ⠄ sharp
error /Users/chris/flipside/product-eng/node_modules/duckdb: Command failed.
Exit code: 1
Command: node-pre-gyp install --fallback-to-build
Arguments:
Directory: /Users/chris/flipside/product-eng/node_modules/duckdb
Output:
node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using [email protected]
node-pre-gyp info using [email protected] | darwin | x64
node-pre-gyp info check checked for "/Users/chris/flipside/product-eng/node_modules/duckdb/lib/binding/duckdb.node" (not found)
node-pre-gyp http GET https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.1-node-v111-darwin-x64.tar.gz
node-pre-gyp ERR! install response status 403 Forbidden on https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.1-node-v111-darwin-x64.tar.gz
node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v111 ABI, unknown) (falling back to source compile with node-gyp)
node-pre-gyp WARN Hit error response status 403 Forbidden on https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.1-node-v111-darwin-x64.tar.gz
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | darwin | x64
gyp info ok
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | darwin | x64
gyp info find Python using Python version 3.9.13 found at "/usr/local/opt/[email protected]/bin/python3.9"
gyp ERR! UNCAUGHT EXCEPTION
gyp ERR! stack TypeError: Cannot assign to read only property 'cflags' of object '#<Object>'
gyp ERR! stack     at createConfigFile (/Users/chris/flipside/product-eng/node_modules/node-gyp/lib/configure.js:117:21)
gyp ERR! stack     at /Users/chris/flipside/product-eng/node_modules/node-gyp/lib/configure.js:84:9
gyp ERR! stack     at FSReqCallback.oncomplete (node:fs:189:23)
gyp ERR! System Darwin 20.6.0
gyp ERR! command "/Users/chris/.nvm/versions/node/v19.0.0/bin/node" "/Users/chris/flipside/product-eng/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--module=/Users/chris/flipside/product-eng/node_modules/duckdb/lib/binding/duckdb.node" "--module_name=duckdb" "--module_path=/Users/chris/flipside/product-eng/node_modules/duckdb/lib/binding" "--napi_version=8" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v111"
gyp ERR! cwd /Users/chris/flipside/product-eng/node_modules/duckdb
gyp ERR! node -v v19.0.0
gyp ERR! node-gyp -v v7.1.2
gyp ERR! Node-gyp failed to build your package.
gyp ERR! Try to update npm and/or node-gyp and if it does not help file an issue with the package author.
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute '/Users/chris/.nvm/versions/node/v19.0.0/bin/node /Users/chris/flipside/product-eng/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/Users/chris/flipside/product-eng/node_modules/duckdb/lib/binding/duckdb.node --module_name=duckdb --module_path=/Users/chris/flipside/product-eng/node_modules/duckdb/lib/binding --napi_version=8 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v111' (7)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/Users/chris/flipside/product-eng/node_modules/@mapbox/node-pre-gyp/lib/util/compile.js:89:23)
node-pre-gyp ERR! stack     at ChildProcess.emit (node:events:513:28)
node-pre-gyp ERR! stack     at maybeClose (node:internal/child_process:1098:16)
node-pre-gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:304:5)
node-pre-gyp ERR! System Darwin 20.6.0
node-pre-gyp ERR! command "/Users/chris/.nvm/versions/node/v19.0.0/bin/node" "/Users/chris/flipside/product-eng/node_modules/duckdb/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /Users/chris/flipside/product-eng/node_modules/duckdb
node-pre-gyp ERR! node -v v19.0.0
node-pre-gyp ERR! node-pre-gyp -v v1.0.10
node-pre-gyp ERR! not ok




error Command failed.
Exit code: 1
Command: /Users/chris/.nvm/versions/node/v19.0.0/bin/node
Arguments: /usr/local/Cellar/yarn/1.22.19/libexec/lib/cli.js add duckdb
Directory: /Users/chris/flipside/product-eng/apps/query-retrieve
Output:

info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command.

node-duckdb install fails on ARM mac

Using yarn to install node-duckdb fails with the following error message:

CMake Error at /opt/homebrew/Cellar/cmake/3.23.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
  system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY
  OPENSSL_INCLUDE_DIR)
Call Stack (most recent call first):
  /opt/homebrew/Cellar/cmake/3.23.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /opt/homebrew/Cellar/cmake/3.23.0/share/cmake/Modules/FindOpenSSL.cmake:578 (find_package_handle_standard_args)
  CMakeLists.txt:468 (find_package)
  src/main/extension/CMakeLists.txt:9 (add_extension_definitions)


-- Configuring incomplete, errors occurred!
See also "/Users/jsteelz/Transit/duckdb-test/node_modules/node-duckdb/duckdb/build/release/CMakeFiles/CMakeOutput.log".
See also "/Users/jsteelz/Transit/duckdb-test/node_modules/node-duckdb/duckdb/build/release/CMakeFiles/CMakeError.log".
make: *** [release] Error 1
error Command failed with exit code 2.

To Reproduce

in an empty directory on an ARM mac (with yarn)
yarn init with whatever
yarn add node-duckdb - crash occurs

add a synchronous api

As the node.js driver better-sqlite3 shows, when using in-process databases that do little to no IO, performances are vastly improved when using the database synchronously instead of having everything go through the event loop.

Since duck db is pretty focused on analytics, I actually doubt that this driver will be used in many other settings than a single user process crunching numbers.

Maybe it would be a good idea to add a synchronous way of speaking with duckdb ?

Cannot install the package - 'rm' is not recognized as an internal or external command

When I run npm install node-duckdb, or run yarn add node-duckdb, I always get errors, and nothing gets installed. At the end it says rm is not recognized as an internal or external command (rm is in fact a command I can run in the command prompt)

Full Error

npm ERR! code 1
npm ERR! path C:\Users\name\Desktop\programm\experiment\stealer\grabberAPI\node_modules\node-duckdb
npm ERR! command failed
npm ERR! command C:\WINDOWS\system32\cmd.exe /d /s /c prebuild-install --verbose -d -r napi || (yarn download-duckdb && yarn build:duckdb && yarn prebuild:current-target)
npm ERR! yarn run v1.22.17
npm ERR! $ rm -rf duckdb && curl -L https://github.com/cwida/duckdb/archive/v0.2.8.tar.gz > duckdb.tar.gz && tar xf duckdb.tar.gz && mv duckdb-0.2.8 duckdb && rm duckdb.tar.gz
npm ERR! info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
npm ERR! prebuild-install info begin Prebuild-install version 6.1.4
npm ERR! prebuild-install info looking for local prebuild @ prebuilds\node-duckdb-v0.0.76-napi-v6-win32-x64.tar.gz
npm ERR! prebuild-install info looking for cached prebuild @ C:\Users\name\AppData\Local\npm-cache\_prebuilds\bc89a1-node-duckdb-v0.0.76-napi-v6-win32-x64.tar.gz
npm ERR! prebuild-install http request GET https://github.com/deepcrawl/node-duckdb/releases/download/v0.0.76/node-duckdb-v0.0.76-napi-v6-win32-x64.tar.gz
npm ERR! prebuild-install http 404 https://github.com/deepcrawl/node-duckdb/releases/download/v0.0.76/node-duckdb-v0.0.76-napi-v6-win32-x64.tar.gz
npm ERR! prebuild-install WARN install No prebuilt binaries found (target=6 runtime=napi arch=x64 libc= platform=win32)
npm ERR! 'rm' is not recognized as an internal or external command,
npm ERR! operable program or batch file.
npm ERR! error Command failed with exit code 1.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\name\AppData\Local\npm-cache\_logs\2021-11-26T22_44_49_370Z-debug.log

I found some older issues opened on other repos, however, they either just ended up not being solved, or not helpful, or I didn't search for long enough.

Does anyone have any idea what could be wrong?

Cannot load @rpath/libduckdb.dylib

Thank you for sharing this awesome work!

I tried to use node-duckdb, but encountered the following error when using with webpack 5.x and node-loader on Mac OS:

Error: Cannot open /..../build/node-duckdb-addon.node: Error: dlopen(/..../build/node-duckdb-addon.node, 1): Library not loaded: @rpath/libduckdb.dylib
  Referenced from: /.../build/node-duckdb-addon.node
  Reason: image not found
    at Object.../../node_modules/node-duckdb/build/Release/node-duckdb-addon.node (node-duckdb-addon.node:1)

I have to manually copy the "libduckdb.dylib" from the node_modules/node-duckdb/build/Release folder to my build folder to work around this error. It would be great if this can be fixed. Thanks again!

Node process should not exit if DuckDB throws exception on C++ side

Steps to reproduce, two node-duckb instances on NodeJS side. One opens a DuckDB database in AccessMode.ReadWrite and the other one AccessMode.ReadOnly concurrently. One of them throws.

  terminate called after throwing an instance of 'duckdb::IOException'
   what():  IO Error: Could not set lock on file "test.db": Resource temporarily unavailable

The same thing happens with the DuckDB client:

# shell 1
% duckdb test.duckdb
v0.2.4 d1b02c270
Enter ".help" for usage hints.
D CREATE TABLE t(t INTEGER);
D

# shell 2
% duckdb -readonly test.duckdb
Error: unable to open database "test.duckdb":

Is that possible to use this client with remote files

Hi, please help, I would like to use remote files like:

SELECT * FROM PARQUET_SCAN('https://github.com/KTH-Library/cordis-data/releases/download/0.0.1/projectPublications.parquet?raw=true') LIMIT 10;

Is that possible?
Thank you.

anothersmith / node-duckdb Goto Github PK

node-duckdb's People

Contributors

Stargazers

Watchers

Forkers

node-duckdb's Issues

To Reproduce

Recommend Projects

Recommend Topics

Recommend Org

Jobs