hstreamdb / hstream

HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.

HStreamDB

HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.

Main Features

• Push real-time data to your apps

  By subscribing to streams in HStreamDB, any update of the data stream will be pushed to your apps in real-time, and this promotes your apps to be more responsive.

  You can also replace message brokers with HStreamDB and everything you do with message brokers can be done better with HStreamDB.

• Stream processing with familiar SQL

  HStreamDB provides built-in support for event time-based stream processing. You can use your familiar SQL to perform basic filtering and transformation operations, statistics and aggregation based on multiple kinds of time windows and even joining between multiple streams.

• Easy integration with a variety of external systems

  With connectors provided, you can easily integrate HStreamDB with other external systems, such as MQTT Broker, MySQL, Redis and ElasticSearch. More connectors will be added.

• Real-time query based on live materialized views

  With maintaining materialized views incrementally, HStreamDB enables you to gain ahead-of-the-curve data insights that respond to your business quickly.

• Reliable persistent storage with low latency

  With an optimized storage design based on LogDevice, not only can HStreamDB provide reliable and persistent storage but also guarantee excellent performance despite large amounts of data written to it.

• Seamless scaling and high availability

  With the architecture that separates compute from storage, both compute and storage layers of HStreamDB can be independently scaled seamlessly. And with the consensus algorithm based on the optimized Paxos, data is securely replicated to multiple nodes which ensures the high availability of our system.

For more information, please visit HStreamDB homepage.

Quickstart

For detailed instructions, follow HStreamDB quickstart.

1. Install HStreamDB.
2. Start a local standalone HStream server.
3. Start HStreamDB's interactive CLI and create your first stream.
4. Run a continuous query.
5. Start another interactive CLI, then insert some data into the stream and get query results.

Documentation

Check out the documentation.

Community, Discussion, Construction and Support

You can reach the HStreamDB community and developers via the following channels:

• Slack
• Twitter
• Reddit

Please submit any bugs, issues, and feature requests to hstreamdb/hstream.

How to build (for developers only)

Pre-requirements

1. You have python3 and docker installed.
2. [Optional] You can run docker without sudo. For details, see this docs
3. [Optional] You can clone the GitHub repository by ssh key.

Get the source code

git clone --recursive [email protected]:hstreamdb/hstream.git
cd hstream/

Update images

script/dev-tools update-images

Start all required services

You must have all required services started before entering an interactive shell to do further development (especially for running tests).

script/dev-tools start-services

To see information about all started services, run

script/dev-tools info

All datas are stored under your-project-root/local-data

Enter in an interactive shell

script/dev-tools shell

Build as other Haskell projects

Inside the interactive shell, you have all extra dependencies installed.

I have no name!@649bc6bb75ed:~$ cabal update
I have no name!@649bc6bb75ed:~$ make
I have no name!@649bc6bb75ed:~$ cabal build all

License

HStreamDB is under the BSD 3-Clause license. See the LICENSE file for details.

Acknowledgments

• Thanks LogDevice for the powerful storage engine.

hstream's Issues

Details about how to build hstream through docker

• How to build Haskell projects by docker? ( with some extra shared library, e.g., logdevice, zookeeper
• How to make Haskell-Language-Server work?
• If I want to edit locally and build remotely, show me the steps.

FAQs:

• Q: Why my hls doesn't find hsc2hs but cabal build success?
• A: Short answer: run cabal install hsc2hs

merge hstream-server into hstream package

show debug message while receive segfault

hstore: more logdevice admin api

• getVersion: wrapper for getVersion() Thrift method
• getStatus: wrapper for getStatus() Thrift method
• aliveSince: wrapper for aliveSince() Thrift method
• getPid: wrapper for getPid() Thrift method
• get_nodes_state
• add_nodes
• update_nodes
• remove_nodes
• get_nodes_config
• add_nodes
• update_nodes
• remove_nodes
• mark_shards_as_provisioned
• bump_node_generation
• get_maintenances
• apply_maintenance
• remove_maintenances
• mark_all_shards_unrecoverable
• check_impact
• get_log_tree_info
• get_replication_info
• get_settings
• take_log_tree_snapshot
• take_maintenance_log_snapshot
• get_log_group_throughput
• get_log_group_custom_counters

hstore rpc: does topic exists?

Add a RPC has-topic: topic -> bool

need facebook::logdevice::DataRecordAttributes

client: data will not be shown properly when containing `{`, `}` and `"`

For example:

INSERT INTO foo (a,  b) VALUES (1, "xxx{}");

╭─────┬─────╮
│  a  │  b  │
╞═════╪═════╡
│ 1.0 │ xxx │
╰─────┴─────╯

need void facebook::logdevice::Reader::waitOnlyWhenNoData()

hstore: tests for checkpoint store

ignore grpc gen-src

CREATE STREAM with an existed stream doesn't show proper error message

get loggroup by logid

ref: logdevice::Client::getLogGroupById

ld-dev-cluster: add --admin-server-port param

Server will crash when executing `SELECT` query and then two `INSERT` queries

To reproduce the bug, run

1. cabal run hstream-server
2. cabal run hstream-client and then SELECT * FROM foo EMIT CHANGES;
3. cabal run hstream-client and then two INSERT INTO foo (a, b) VALUES (1, 2);

Details:

hstream-server: Data.HashMap.Internal.(!): key not found
CallStack (from HasCallStack):
  error, called at ./Data/HashMap/Internal.hs:785:16 in unordered-containers-0.2.14.0-4623fa3c6c98778c0d1a2c886970786fa56dd6d9087fb385e168ce0ef5ba869b:Data.HashMap.Internal
  !, called at src/HStream/Processing/Processor.hs:130:49 in hstream-processing-0.1.0.0-inplace:HStream.Processing.Processor

It means that the processor crashed. Logs say that the srcStream in the second SourceRecord does not match the real one:

2021-06-08 07:34:58.410204: [debug] polled 1 records
@(src/HStream/Processing/Processor.hs:122:11)
2021-06-08 07:34:58.410312: [debug] ---------------------------------
@(src/HStream/Processing/Processor.hs:126:17)
2021-06-08 07:34:58.410348: [debug] srcSource: fool﮿
@(src/HStream/Processing/Processor.hs:127:17)
2021-06-08 07:34:58.410377: [debug] expected: foo
@(src/HStream/Processing/Processor.hs:128:24)
2021-06-08 07:34:58.410406: [debug] ---------------------------------

Add bnfc, alex, happy to haskell docker image

hstream-server: error while loading shared libraries: libgrpc.so.14: cannot open shared object file: No such file or directory

I follow the Quickstart instructions, when I run this command
docker run -td --rm --name some-hstream-store -v dbdata:/data/store --network host hstreamdb/logdevice ld-dev-cluster --root /data/store --use-tcp ,
then shows
hstream-server: error while loading shared libraries: libgrpc.so.14: cannot open shared object file: No such file or directory
and Start HStreamDB Server command also

Test case fail some time

e.g. https://github.com/hstreamdb/hstream/runs/2770152235

async version of write checkpoints

checkpointed reader

c api: "logdevice/include/CheckpointedReaderBase.h"

• CheckpointedReaderBase.asyncWriteCheckpoints
• CheckpointedReaderBase.asyncWriteCheckpoints (writeLastCheckpointsSync)

checkpoint store

c api: "logdevice/include/CheckpointStore.h"

• CheckpointStore.getLSN
• CheckpointStore.updateLSN

Logdevice API: Client::findTime

Rewrite ldshell: command 'logs'

• logs show
• logs rename
• logs info

Move hstream-sql's autogen modules to a sepreate directory

Some SQL keywords are case insensitive, but others are not.

EMIT CHANGES must be capital now, or it will get a parsing error. we should make all SQL keywords case insensitive.

hstore: export stopReading

Stop reading by stream name

split all connectors to a new package

switch hstream CLI implementation to grpc

hstore-admin: logs show

logs show --path [...] --id [...] --max-depth ... --verbose

Tasks

• Add logDirLogsNames
• Add logDirChildFullName and logDirLogFullName, which will be used to get log groups and sub directories
• Add logDirectoryGetAttrs
• ldLogAttrsPtrToHsAttrs which reads attributes from a foreign pointer to a map in Haskell
• Update optparser

topic attributes

colourful hstore-admin cli out

hstore api: trim

ref: logdevice::Clinet::trim

hstore-admin: logs rename

hstore: batched writer

• exactly-once
• automatically

API: delete & rename topic

• rename topics. We need a HStream.Store.Stream.Topic.rename function to rename our topics. (Btw, a possible logdevice API is Client.rename.)
• remove topics. (Client.removeLogGroup may help)

move all submodules to external directory

zk based checkpoint store

Create connector throw grpc error when stream subscribing doesn't exist

Describe the bug

> CREATE SOURCE | SINK CONNECTOR mysqltest22 WITH (type = "mysql", host = "0.0.0.0", port = 3306, streamname = "aaa") ;
Client Error: ClientIOError (GRPCIOBadStatusCode StatusCancelled (StatusDetails {unStatusDetails = "Cancelled"}))

To Reproduce
see above

Expected behavior
display warning: Stream doesn't exist

Can you bind in to hstream directly from a Haskell app?

We have a Haskell application and I'm wondering since this is written in Haskell if it's possible to just bind directly into the query system without using SQL?

Log based CheckpointStore

HStore: Return more detailed information instead of just StreamName in findStreams

Since we need to provide an admin dashboard.
We will need provide some http apis for stream. But now we can only get StreamNames from HStore by findStreams.
Maybe we should return more detailed information (such as replicationFactor and beginTimestamp) instead of just StreamName .
Also provide another api for query stream info by streamname.

HStore: Create a batch of Streams

Connect to HStreamDB server programmatically

I couldn't find any documentation on how I can connect with HStreamDB server programmatically. I am looking for Java connector. If the connector does not exist then what is the best way to create one.

(ConnectionFailure Network.Socket.connect: <socket: 12>: does not exist (Connection refused))

创建流，ctl 报错

CREATE STREAM demo WITH (FORMAT = "JSON");

HttpExceptionRequest Request {
host = "localhost"
port = 6570
secure = False
requestHeaders = [("Content-Type","application/json; charset=utf-8")]
path = "/create/query"
queryString = ""
method = "POST"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
proxySecureMode = ProxySecureWithConnect
}
(ConnectionFailure Network.Socket.connect: <socket: 12>: does not exist (Connection refused))

HStream-Server 报错
hstream-server: LOGS_SECTION_MISSING {name:LOGS_SECTION_MISSING, description:LOGS_SECTION_MISSING: Configuration file misses logs section, callstack:CallStack (from HasCallStack):
throwStreamErrorIfNotOK', called at ./HStream/Store/Exception.hs:324:28 in hstream-store-0.1.0.0-00fd332cb31e900c02128ac35e4d993df94b7e2480f7781981c70e854378ff8e:HStream.Store.Exception}

ldshell command

Required but not implemented yet:

• logs update
• logs set-range
• logs trim
• nodes-config edit
• nodes-config shrink
• maintenance
• SQL: show tables;
• SQL: describe table_name;
• SQL: select ...

Other commands in ldshell:

• check-internal-logs

HStore-Admin API: Create logdevice client

command "SELECT * FROM demo WHERE humidity > 70 EMIT CHANGES" parse error

>SELECT * FROM demo WHERE humidity > 70 EMIT CHANGES;
ParseException at <unknown position>: syntax error at line 1, column 40 before `EMIT'
CallStack (from HasCallStack):
  ParseException, called at src/HStream/SQL/Parse.hs:23:36 in hstream-sql-0.1.0.0-5c9dc46b93814cbb293cf4f8ba64a8d7fd1e433bdd707f130f0d94c72e0b6ca1:HStream.SQL.Parse
  parse, called at src/HStream/SQL/Parse.hs:30:24 in hstream-sql-0.1.0.0-5c9dc46b93814cbb293cf4f8ba64a8d7fd1e433bdd707f130f0d94c72e0b6ca1:HStream.SQL.Parse
  parseAndRefine, called at client/Main.hs:99:33 in main:Main

Following the steps in the quick start.

hstore-admin: logs create

hstore api: async `getTailLSN`

Improve hstream-client prints (including error message and sql results)

• display empty info while running show streams; if there are no streams.
• some sql examples (:h)

ci: push image to docker hub failed

e.g. https://github.com/hstreamdb/hstream/runs/2408537502

hstream-server/processor: end task (safely) when call canceled from client

hstore-admin: logs remove