stieneee / simple-zstd Goto Github PK
View Code? Open in Web Editor NEWNode.js interface to system installed zstandard.
License: MIT License
Node.js interface to system installed zstandard.
License: MIT License
I am facing error while using function ZSTDCompress function for compression of stream data to compressed buffer.
Node Version: 10.16.0
Npm Version: 6.9.0
OS: Ubuntu 18.04
What can be the solution to resolve this error?
Following are the error logs :
`error: uncaughtException: Command failed: Incorrect parameters
Usage :
zstd [args] [FILE(s)] [-o file]
FILE : a filename
with no FILE, or when FILE is - , read standard input
Arguments :
-# : # compression level (1-19, default:3)
-d : decompression
-D file: use file
as Dictionary
-o file: result stored into file
(only if 1 input file)
-f : overwrite output without prompting and (de)compress links
--rm : remove source file(s) after successful de/compression
-k : preserve source file(s) (default)
-h/-H : display help/long help and exit
{ date: 'Thu Jul 18 2019 11:00:01 GMT+0530 (India Standard Time)',
process:
{ pid: 8622,
uid: 0,
gid: 0,
cwd: '/home/gdmtrunk',
execPath: '/usr/local/bin/node',
version: 'v10.16.0',
argv:
[ '/usr/local/bin/node',
'/home/gdmtrunk/server.js' ],
memoryUsage:
{ rss: 170455040,
heapTotal: 129097728,
heapUsed: 112122016,
external: 72180011 } },
os:
{ loadavg: [ 1.04345703125, 1.24267578125, 1.625 ],
uptime: 2035 },
trace:
[ { column: 12,
file:
'/home/gdmtrunk/node_modules/duplex-child-process/index.js',
function: 'ChildProcess.onExit',
line: 116,
method: 'onExit',
native: false },
{ column: 20,
file: 'events.js',
function: 'Object.onceWrapper',
line: 286,
method: 'onceWrapper',
native: false },
{ column: 13,
file: 'events.js',
function: 'ChildProcess.emit',
line: 198,
method: 'emit',
native: false },
{ column: 20,
file: 'domain.js',
function: 'ChildProcess.EventEmitter.emit',
line: 448,
method: 'emit',
native: false },
{ column: 16,
file: 'internal/child_process.js',
function: 'maybeClose',
line: 982,
method: null,
native: false },
{ column: 11,
file: 'internal/child_process.js',
function: 'Socket.stream.socket.on',
line: 389,
method: 'on',
native: false },
{ column: 13,
file: 'events.js',
function: 'Socket.emit',
line: 198,
method: 'emit',
native: false },
{ column: 20,
file: 'domain.js',
function: 'Socket.EventEmitter.emit',
line: 448,
method: 'emit',
native: false },
{ column: 12,
file: 'net.js',
function: 'Pipe._handle.close',
line: 606,
method: 'close',
native: false } ],
stack:
[ 'Error: Command failed: Incorrect parameters',
'Usage : ',
' zstd [args] [FILE(s)] [-o file] ',
'',
'FILE : a filename ',
' with no FILE, or when FILE is - , read standard input',
'Arguments : ',
' -# : # compression level (1-19, default:3) ',
' -d : decompression ',
' -D file: use file
as Dictionary ',
' -o file: result stored into file
(only if 1 input file) ',
' -f : overwrite output without prompting and (de)compress links ',
'--rm : remove source file(s) after successful de/compression ',
' -k : preserve source file(s) (default) ',
' -h/-H : display help/long help and exit ',
'',
' at ChildProcess.onExit (/home/gdmtrunk/node_modules/duplex-child-process/index.js:116:12)',
' at Object.onceWrapper (events.js:286:20)',
' at ChildProcess.emit (events.js:198:13)',
' at ChildProcess.EventEmitter.emit (domain.js:448:20)',
' at maybeClose (internal/child_process.js:982:16)',
' at Socket.stream.socket.on (internal/child_process.js:389:11)',
' at Socket.emit (events.js:198:13)',
' at Socket.EventEmitter.emit (domain.js:448:20)',
' at Pipe._handle.close (net.js:606:12)' ] }
18 Jul 11:00:01 - [red] Uncaught Exception:
18 Jul 11:00:01 - Error: Command failed: Incorrect parameters
Usage :
zstd [args] [FILE(s)] [-o file]
FILE : a filename
with no FILE, or when FILE is - , read standard input
Arguments :
-# : # compression level (1-19, default:3)
-d : decompression
-D file: use file
as Dictionary
-o file: result stored into file
(only if 1 input file)
-f : overwrite output without prompting and (de)compress links
--rm : remove source file(s) after successful de/compression
-k : preserve source file(s) (default)
-h/-H : display help/long help and exit
at ChildProcess.onExit (/home/gdmtrunk/node_modules/duplex-child-process/index.js:116:12)
at Object.onceWrapper (events.js:286:20)
at ChildProcess.emit (events.js:198:13)
at ChildProcess.EventEmitter.emit (domain.js:448:20)
at maybeClose (internal/child_process.js:982:16)
at Socket.stream.socket.on (internal/child_process.js:389:11)
at Socket.emit (events.js:198:13)
at Socket.EventEmitter.emit (domain.js:448:20)
at Pipe._handle.close (net.js:606:12)`
hello, is it possible and can you provide an example of decompressing/compressing a uint8array that would build and work in a browser environment (in my case webpack)
Hi
Great library, thanks for contributing.
This might be me being naive; apologies if so.
I want to uncompress a large text file on the fly, and I have different phases in the code (i.e. the file starts with metadata lines which I want to read all of, then data lines, which I want to expose as an async iterator. However, I'm at a simpler point so far.
When I run the following code in Mocha, it succeeds, but then hangs. If I take out the break
it works just fine.
import * as readline from "readline";
import * as fs from "fs";
import * as path from "path";
import { ZSTDDecompressMaybe } from "simple-zstd";
it("process actual lines", async () => {
console.log(__dirname);
let filePath = path.join(__dirname, "/data/NA24385.trio.subset.vcf.zst");
const stream = fs.createReadStream(filePath).pipe(ZSTDDecompressMaybe());
const lines = readline.createInterface({
input: stream,
crlfDelay: Infinity,
});
let n = 0;
for await (const line of lines) {
n += line.length;
if (n > 1024*1024) {
break;
}
}
console.log(n);
});
I presume I need to clean up somehow. Is there a better way to do this?
I assume the behaviour doesn't particularly depend on the specific compressed data. My file is 4.4Mb, which is a bit fat for a repro, but if you think it necessary, I can cut it down and see if the behaviour still exists.
in node_modules\simple-zstd\index.js
the regexp to remove zstd newline should not be just \n but \r?\n to support windows platform
Line 34 in 030fd0d
${lvl}
is used which could be a string. If a user of this library calls the function with unsanitized user-input on the compLevel argument, it is possible for the argument to be a string, which is then passed as the command line argument.
I'm not sure if there is a command line flag in zstd that would allow an RCE, especially because as far as I know, it is treated as only one process argument, so no other program can be started with it directly.
This can be fixed by checking the variable to be a number, for example by doing this:
if (!lvl || typeof lvl !== "number") lvl = 3;
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.