To design and develop a high performance, high compression ratio, highly available, distributed cloud native time series database, which meets the following objectives.
- Seperate storage and computation; theoretically uncapped support time series expansion; support horizontal/vertical scaling.
- Focus performance and cost balance; high performance io, Run-to-Completion scheduling model, support for hierarchical storage using object storage.
- Lossy compression with reduced precision at the user's option.
- Implemented query engine by using Apache Arrow and Datafusion.
- Support for vectorized execution of the query engine to execute complex query statements.
- Support for standard SQL, Flux, rich aggregate queries and arithmetic.
- Design for multi-tenant-oriented, providing more configuration parameters, able to provide more flexible configuration of resources.
- CDC, WAL can provide subscription and distribution to other nodes, more flexible deploy and support.
- Support Ecological compatibility with the K8s ecosystem, reducing shared memory.
- integration with other data ecosystems to support import/export of parquet files.
- Be compatible with major international and domestic public cloud ecosystems.
Important modules of Store Engine:
WAL: Write-Ahead Logging for recovering Memcache after downtime.
Memcache: memtable and immutmemtable in-memory cache data.
TSM: columnar storage format for temporal data.
Summary (TSM's MetaData): metadata file generated by TSM file version changes, used to recover data.
Versionset: global view of kv store, similar to storage manager.
Tsfamily: column cluster of series, a basic unit of LSM.
Compression of TSM: supports compression of multiple field types.
Lossy compression of TSM: support for data compression with reduced data precision.
Operatios of storage engine:
- write operation: grpc -> wal -> memcache.
- Read operation: support point query and range query, can read data from memtable and tsm.
- Token delete: delete files by compact, memtable data is cleared in real time.
- Flush: immutcache flush to L0 level smaller tsm files.
- Compact: TSM file merge after deletion.
- Other: configuration file, supports reading from environment variables and configuration.
- Implemented catalog provider
- schema storage
- Support assemble the data in TSM file into recordbatch in arrow.
- Support table scan, parse the data in the TSM file.
- Maintain system table & infomation schema for data statistics.
- Support orthogonal index
- Support inverted index
- Support custom indexing
- Support database management moudle
- Query optimization
- fs: direct IO of user-state cache.
- schedule: Run-to-Completion model.
- Native distributed system based on Rust language
- Distributed architecture, overall framework design, computational storage separation
- Data slicing rules, based on consistent hashing
- Scaling and shrinking data migration
- Disaster tolerance switching
- Copy synchronization
- Operation and maintenance tools
- Data backup and restore
- Achieve eventual consistency
- Hinted-Handoff
- Read Repair
- Tombstone mechanism
- anti-entropy
Cnosdb Isipho will supports and is compatible with many good ecologies.
-
CnosDB Isipho must support
- line protocol
- http interface
-
Ecological partner products that need to be supported
- Third-party data heterogeneity with CnosDB
- Telegraf
- Influxdb
- Prometheus
- Timescales
- Kafka/Pulsar
- MySQL/PgSQL
-
Data management tools involving engine layer tools
- Data file backup (you can specify DB, shard, time period, etc.) and restore
- Export to row protocol (can follow several different dimensions: DB, shard, time period) and bulk import
- Disk files analysis tools (analysis of TSM, WAL, index files, etc., index reconstruction, file legitimacy checks, etc.)
- Data deletion by multiple dimensions: DB, table, shard, time period
- Basic testing
- Continuous stress testing
- Cluster full-featured testing framework
- Chaos testing
kv:
- Read operation filter delete and add test cases
- Delete operation complete and add test cases
- compact merge iterator
- Support discrete point data production delta file
- Supplemental basic test cases build ci auto-run access control supplemental pressure test
- Configuration file, support reading from environment variables and configuration
- Optimize unnecessary locks (long term work)
query:
- impl catalog provider
- Schema storage docking arrow schema
- Assemble the data in TSM file into recordbatch in arrow