Simple Yandex ClickHouse insert collector. It collect requests and send to ClickHouse servers.
Download binary for you platorm
or
or from sources (Go 1.13+):
git clone https://github.com/nikepan/clickhouse-bulk
cd clickhouse-bulk
go build
- Group n requests and send to any of ClickHouse server
- Sending collected data by interval
- Tested with VALUES, TabSeparated formats
- Supports many servers to send
- Supports query in query parameters and in body
- Supports other query parameters like username, password, database
- Supports basic authentication
For example:
INSERT INTO table3 (c1, c2, c3) VALUES ('v1', 'v2', 'v3')
INSERT INTO table3 (c1, c2, c3) VALUES ('v4', 'v5', 'v6')
sends as
INSERT INTO table3 (c1, c2, c3) VALUES ('v1', 'v2', 'v3')('v4', 'v5', 'v6')
- -config - config file (json); default config.json
{
"listen": ":8124",
"flush_count": 10000, // check by \n char
"flush_interval": 1000, // milliseconds
"clean_interval": 0, // how often cleanup internal tables - e.g. inserts to different temporary tables, or as workaround for query_id etc. milliseconds
"remove_query_id": true, // some drivers sends query_id which prevents inserts to be batched
"dump_check_interval": 300, // interval for try to send dumps (seconds); -1 to disable
"debug": false, // log incoming requests
"dump_dir": "dumps", // directory for dump unsended data (if clickhouse errors)
"clickhouse": {
"down_timeout": 60, // wait if server in down (seconds)
"connect_timeout": 10, // wait for server connect (seconds)
"tls_server_name": "", // override TLS serverName for certificate verification (e.g. in cases you share same "cluster" certificate across multiple nodes)
"insecure_tls_skip_verify": false, // INSECURE - skip certificate verification at all
"servers": [
"http://127.0.0.1:8123"
]
},
"metrics_prefix": "prefix"
}
CLICKHOUSE_BULK_DEBUG
- enable debug loggingCLICKHOUSE_SERVERS
- comma separated list of serversCLICKHOUSE_FLUSH_COUNT
- count of rows for insertCLICKHOUSE_FLUSH_INTERVAL
- insert intervalCLICKHOUSE_CLEAN_INTERVAL
- internal tables clean intervalDUMP_CHECK_INTERVAL
- interval of resend dumpsCLICKHOUSE_DOWN_TIMEOUT
- wait time if server is downCLICKHOUSE_CONNECT_TIMEOUT
- clickhouse server connect timeoutCLICKHOUSE_TLS_SERVER_NAME
- server name for TLS certificate verificationCLICKHOUSE_INSECURE_TLS_SKIP_VERIFY
- skip certificate verification at allMETRICS_PREFIX
- prefix for prometheus metrics
./clickhouse-bulk
and send queries to :8124
manual check main metrics
curl -s http://127.0.0.1:8124/metrics | grep "^ch_"
ch_bad_servers 0
- actual count of bad serversch_dump_count 0
- dumps saved from launchch_queued_dumps 0
- actual dump files id directorych_good_servers 1
- actual good servers countch_received_count 40
- received requests count from launchch_sent_count 1
- sent request count from launch
For better performance words FORMAT and VALUES must be uppercase.
clickhouse-bulk's People
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.