questdb / nodejs-questdb-client

QuestDB Node.js Client

 

English | 简体中文 | 繁體中文 | العربية | Italiano | Українська | Español | Português | 日本語 | Türkçe | हिंदी | Tiếng Việt

---

QuestDB is the fastest growing open-source time-series database offering blazingly fast, high throughput ingestion and dynamic, low-latency SQL queries. The entire high-performance codebase is built from the ground up in Java, C++ and Rust with no dependencies and zero garbage collection.

We achieve high performance via a column-oriented storage model, parallelized vector execution, SIMD instructions, and low-latency techniques. In addition, QuestDB is hardware efficient, with quick setup and operational efficiency.

QuestDB implements ANSI SQL with native time-series SQL extensions. These SQL extensions make it simple to analyze, filter and downsample data, or to correlate data from multiple sources using relational and time-series joins.

Ready to go? Jump to the Get started section.

 

QuestDB Web Console - click to launch demo

 

Benefits of QuestDB

QuestDB excels with:

• financial market data
• IoT sensors with high data cardinality
• real-time dashboards

Feature highlights include:

• SQL with powerful, SIMD-optimized time-series extensions
• High-speed ingestion via the InfluxDB Line Protocol
• Strong and efficient performance on limited hardware
• Columnar storage format (native or Apache Parquet), partitioned and ordered by time
• Responsive and intuitive Web Console for query and data management, with error handling
• Excellent performance with high data cardinality - see benchmarks

And why use a time-series database?

Beyond performance and efficiency, with a specialized time-series database, you don't need to worry about:

• out-of-order data
• duplicates
• exactly one semantics
• streaming data (low latency)
• high volumes of concurrent requests
• volatile and "bursty" data
• adding new columns - change schema "on the fly" while streaming data

Try QuestDB, demo and dashboards

The live, public demo is provisioned with the latest QuestDB release and sample datasets:

• Trips: 10 years of NYC taxi trips with 1.6 billion rows
• Trades: live crypto market data with 30M+ rows per month
• Pos: geolocations of 250k unique ships over time

Use example queries or write your own!

The public demo queries over 1.6BN rows and uses a r6a.12xlarge 48 vCPU and 348GB RAM instance.

Query	Execution time
SELECT sum(double) FROM trips	0.15 secs
SELECT sum(double), avg(double) FROM trips	0.5 secs
SELECT avg(double) FROM trips WHERE time in '2019'	0.02 secs
SELECT time, avg(double) FROM trips WHERE time in '2019-01-01' SAMPLE BY 1h	0.01 secs
SELECT * FROM trades LATEST ON timestamp PARTITION BY symbol	0.00025 secs

We also have some public, real-time demo dashboards using our Grafana-native plugin:

• Real-time crypto trades: executed trades on OKX from more than 20 assets in real time
• NYC taxi geolocation data: real-time replay of 146,393,317 taxi rides across New York City in 2016

QuestDB performance vs. other oss databases

QuestDB performs very well in performance benchmarks compared to alternatives.

For deep dives into internals and performance, see the following blog posts:

• QuestDB vs InfluxDB
• QuestDB vs TimescaleDB
• QuestDB vs MongoDB

As always, we encourage you to run your own benchmarks.

Get started

Use Docker to start quickly:

docker run -p 9000:9000 -p 9009:9009 -p 8812:8812 questdb/questdb

Or macOS users can use Homebrew:

brew install questdb
brew services start questdb

questdb start
questdb stop

Alternatively, to kickoff the full onboarding journey, start with our concise quick start guide.

First-party ingestion clients

QuestDB clients for ingesting data via the InfluxDB Line Protocol:

• Python
• .NET
• C/C++
• Go
• Java
• NodeJS
• Rust

Connect to QuestDB

Interact with QuestDB and your data via the following interfaces:

• Web Console for an interactive SQL editor and CSV import on port 9000
• InfluxDB Line Protocol for streaming ingestion on port 9000
• PostgreSQL Wire Protocol for programmatic queries on port 8812
• REST API for CSV import and cURL on port 9000

Popular third-party tools

Popular tools that integrate with QuestDB include:

• Apache Kafka
• Grafana
• Superset
• Telegraf
• Apache Flink
• qStudio
• MindsDB

End-to-end code scaffolds

From streaming ingestion to visualization with Grafana, start with code scaffolds in from our quickstart repository.

Configure QuestDB for production workloads

Find our capacity planning to fine-tune QuestDB for production workloads.

QuestDB Enterprise

For secure operation at greater scale or within larger organizations.

Additional features include:

• multi-primary ingestion
• read replica(s)
• cold storage integration
• role-based access control
• TLS encryption
• native querying of Parquet files via object storage
• support SLAs, enhanced monitoring and more

Visit the Enterprise page for further details and contact information.

Additional resources

📚 Read the docs

• QuestDB documentation: begin the journey
• Product roadmap: check out our plan for upcoming releases
• Tutorials: learn what's possible with QuestDB, step by step

❓ Get support

• Community Discourse forum: join technical discussions, ask questions, and meet other users!
• Public Slack: chat with the QuestDB team and community members
• GitHub issues: report bugs or issues with QuestDB
• Stack Overflow: look for common troubleshooting solutions

🚢 Deploy QuestDB

• AWS AMI
• Google Cloud Platform
• Official Docker image
• DigitalOcean droplets
• Kubernetes Helm charts

Contribute

Contributions welcome!

We appreciate:

• source code
• documentation (see our documentation repository)
• bug reports
• feature requests or feedback.

To get started with contributing:

• Have a look through GitHub issues labelled "Good first issue"
• For Hacktoberfest, see the relevant labelled issues
• Read the contribution guide
• For details on building QuestDB, see the build instructions
• Create a fork of QuestDB and submit a pull request with your proposed changes
• Stuck? Join our public Slack for assistance

✨ As a sign of our gratitude, we send QuestDB swag to our contributors!

A big thanks goes to the following wonderful people who have contributed to QuestDB emoji key:

clickingbuttons 💻 🤔 📓	ideoma 💻 📓 ⚠️	tonytamwk 💻 📓	sirinath 🤔	igor-suhorukov 💻 🤔	mick2004 💻 📦	rawkode 💻 🚇
solidnerd 💻 🚇	solanav 💻 📖	shantanoo-desai 📝 💡	alexprut 💻 🚧	lbowman 💻 ⚠️	chankeypathak 📝	upsidedownsmile 💻
Nagriar 💻	piotrrzysko 💻 ⚠️	mpsq 💻	siddheshlatkar 💻	Yitaek ✅ 💡	gabor-boros ✅ 💡	kovid-r ✅ 💡
TimBo93 🐛 📓	zikani03 💻	jaugsburger 💻 🚧	TheTanc 📆 🖋 🤔	davidgs 🐛 🖋	kaishin 💻 💡	bluestreak01 💻 🚧 ⚠️
patrickSpaceSurfer 💻 🚧 ⚠️	chenrui333 🚇	bsmth 📖 🖋	Ugbot 💬 📓 📢	lepolac 💻 🔧	tiagostutz 📓 🐛 📆	Lyncee59 🤔 💻
rrjanbiah 🐛	sarunas-stasaitis 🐛	RiccardoGiro 🐛	duggar 🐛	postol 🐛	petrjahoda 🐛	t00 🐛
snenkov 📓 🐛 🤔	marregui 💻 🤔 🎨	bratseth 💻 🤔 📓	welly87 🤔	fuzzthink 🤔 📓	nexthack 💻	g-metan 🐛
tim2skew 🐛 📓	ospqsp 🐛	SuperFluffy 🐛	nu11ptr 🐛	comunidadio 🐛	mugendi 🤔 🐛 📖	paulwoods222 🐛
mingodad 🤔 🐛 📖	houarizegai 📖	jjsaunier 🐛	zanek 🤔 📆	Geekaylee 📓 🤔	lg31415 🐛 📆	null-dev 🐛 📆
ultd 🤔 📆	ericsun2 🤔 🐛 📆	giovannibonetti 📓 🐛 📆	wavded 📓 🐛	puzpuzpuz 📖 💻 📓	rstreics 💻 🚇 📖	mariusgheorghies 💻 🚇 📖
pswu11 🖋 🤔 🎨	insmac 💻 🤔 🎨	eugenels 💻 🤔 🚧	bziobrowski 💻 📆	Zapfmeister 💻 📓	mkaruza 💻	DylanDKnight 📓 🐛
enolal826 💻	glasstiger 💻	argshook 💻 🤔 🎨 🐛	amunra 💻 📖 🐛	GothamsJoker 💻	kocko 💻	jerrinot 💻 🤔 🐛
rberrelleza 💻	Cobalt-27 💻	eschultz 💻	XinyiQiao 💻	terasum 📖	PlamenHristov 💻	tris0laris 📝 🤔
HeZean 💻 🐛	iridess 💻 📖	selmanfarukyilmaz 🐛	donet5 🤔 🐛	Zahlii 🐛	salsasepp 🐛	EmmettM 🐛 ⚠️
robd003 🤔	AllenEdison 🐛	CSharpDummy 🐛	shimondoodkin 🐛 🤔	huuhait 🐛 🤔	alexey-milovidov 🐛	suconghou 🐛
allegraharris 💻	oliver-daniel 💻	kerimsenturk5734 📖

This project adheres to the all-contributors specification. Contributions of any kind are welcome!

nodejs-questdb-client's Issues

Missing method to insert binary data

Hi,
in the QuestDB spec there's a column type "binary".
We want to use this column type to store base64 encoded data.
Unfortunately, there's no "binaryColumn" method in sender.js.
Is this planned or is there any other way to insert binary data using this library?
Thanks

Specify minimal supported Node.js version or support older ones like 14

Scenario:

• Create a QuestDB Cloud instance
• Run generated ILP ingestion script for Node.js with Node.js 14, e.g. v14.17.6
• Observe the following error:

$ node insert-client.js 
Successfully connected to <host>:<port>
Authenticating with <host>:<port>
(node:392696) UnhandledPromiseRejectionWarning: TypeError [ERR_INVALID_ARG_TYPE]: The "key.key" property must be of type string or an instance of Buffer, TypedArray, or DataView. Received an instance of Object
    at prepareAsymmetricKey (internal/crypto/keys.js:288:13)
    at Object.createPrivateKey (internal/crypto/keys.js:349:5)
    at authenticate (/home/puzpuzpuz/projects/node-experiments/node_modules/@questdb/nodejs-client/src/sender.js:387:40)
    at TLSSocket.<anonymous> (/home/puzpuzpuz/projects/node-experiments/node_modules/@questdb/nodejs-client/src/sender.js:133:43)
    at TLSSocket.emit (events.js:400:28)
    at addChunk (internal/streams/readable.js:290:12)
    at readableAddChunk (internal/streams/readable.js:265:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:204:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:392696) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:392696) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

The reason is that crypto.createPrivateKey(key) supports JWK object starting from v15.12.0: https://nodejs.org/api/crypto.html#cryptocreateprivatekeykey

Cant insert data no matter what I do

Hi,

Just installed questdb and trying to insert some data. I have created the table as such:

CREATE TABLE 'chartfeed' (
timestamp TIMESTAMP,
name SYMBOL capacity 256 CACHE,
quoteid INT,
marketid INT,
open INT,
high INT,
low INT,
close INT,
) timestamp (timestamp);

And below is the code Im using to insert:

` const { Sender } = require('@questdb/nodejs-client');

async function run() {
// create a sender using HTTP protocol
const sender = new Sender({ protocol: 'tcp', host: 'localhost', port: 9009, bufferSize: 4096 });
await sender.connect();

// add rows to the buffer of the sender
await sender.table('chartfeed')
    .symbol('name', 'TEST')
    .intColumn('quoteid', 1234)
    .intColumn('marketid', 5678)
    .intColumn('open', 33450)
    .intColumn('high', 33480)
    .intColumn('low', 33410)
    .intColumn('close', 33460)
    .timestampColumn('timestamp',1715809721)


// flush the buffer of the sender, sending the data to QuestDB
// the buffer is cleared after the data is sent, and the sender is ready to accept new data
await sender.flush();

// close the connection after all rows ingested
await sender.close();

}

run().then(function(){
console.log('good')
}).catch(function(err){
console.error('error',err)
}); `

Console log prints:

Successfully connected to localhost:9009
Connection to 127.0.0.1:9009 is closed
good

But however no data is inserted in the table when I inspect with the web console.

Cant see what Im doing wrong?

How to query?

I see there is sender implementation but where is reading?

Inserting data from more than one script/pm2 process causes Web console to refuse connection

Hi,

I have just managed to code a script that pulls data from an API and inserts into my questdb. I am using PM2 to manage the processes where each process runs at every second.

I have setup 4 processes to pull data from 4 different markets and the moment I have all 4 running, when I try to visit web console it gives me a refused connection. Dialling back PM2 to 1 process only and I can login just fine.

Is there a limitation on clients connected somewhere that I can change?

It seems everything grinds to a halt the moment I have all 4 script inserting data

Here is a gist:

https://gist.github.com/hmpmarketing/b18ba2b6db4fb0af370fbfea1f49c366

geohash support

would be great to have the geohash support, I am willing to contribute but I would need some guidence

Add connection pool

Many of our users write to QuestDB as a part of HTTP request processing. The simplest way to do that is to open a new connection on each HTTP request, but this has high overhead and also puts no limit on the number of active connections. We should implement a connection pool similar to the pg's one: https://github.com/brianc/node-postgres/blob/b03c071d2d15af259e1e008e9628191c865e58fa/packages/pg-pool/index.js

Reduce the number of necessary parameters when using ILP auth

For connecting via ILP auth, only the key_id and the private_key/d parameter should be needed. However, this client also requests the public key pair, which is annoying as those are not really needed, and the developer need to treat them as secrets.

Some official clients like Go, JAVA, or C# don't need the extra parameters.

Once changed, the documentation should be updated to reflect the simplified connection params.

Messages in console always visible

I want to use this module, but since it uses console.info to show some messages, it is a little bit slow due to the sync code with console.
The debug/info messages should be optional and defined as part of the options when the user create the sender

Client should skip columns if value is null

When adding columns to the sender, if we pass a null value, as in

sender.table('prices').symbol('instrument', 'EURUSD')
            .floatColumn('bid', null).at(Date.now(), 'ms');

It would be convenient that the column was ignored, rather than raising an error. It would make things easier for programmatic access

QuestDB NodeJS client library in Node-red

See full description here: questdb/questdb#2891

Flush - is there a way to check for buffer length or is it really required to manual flush?

I'm currently using HTTP (not TCP), and I wanted to optimize the buffer capacity. So instead of manually flushing after each entry;

• is there a way to check if the buffer can fit more data, then auto flush and auto close if the next message exceed the capacity?
• I saw autoFlush() and DEFAULT_HTTP_AUTO_FLUSH_ROWS so it seems manual flush is actually optional?

Example how to use Sender log level

As per title, how do I pass the log function setting 'debug'? Using version 3. Thanks!