Comments (1)
At the time of that talk I had a queue implementation that would have been kafka like, but I decided to scrap it for something better.
The last few months I have worked on a general purpose c library that can be used to implement a kafka like server, a WAL for a database or anything else that needs to write to disk really fast in a fifo manner.
It should result in an order of magnitude better performance. Worker threads no longer touch any io directly and do not require mutexes. When an actor writes data to WAL, it is suspended using libctx. Once write is done it gets switched back for processing.
In the meantime worker thread is free to do processing for other actors and they never block for io. There is a dedicated write io thread that uses async io primitives of the local system. There is a pool of threads for blocking operations (like reading from lmdb).
All in all a design that is completely hardware scalable. The more cores or disk io you throw at it the better it will run. Threads no longer get in each others way. Allocations are predictable, thread syncronization is just a lock free queue.
The library works well and it already works with sqlite. What's missing is raft replication for it. For that I will almost certainly use https://github.com/willemt/raft
Once the replication aspect is finalized it will be open sourced.
After that some decisions must be made how the next iteration of actordb will look like. I am leaning towards Rust for the actual fast path. That is receiving requests (thrift/mysql protocol), parsing and actor execution+replication. The higher level stuff of shard balancing would be nice to keep as is. Nothing wrong with using a high level (slower) language for it.
As for the map-reduce bit. It seems to me that apache spark's rdd concept is the way of the future. A rdd implementation as a library (with a c api) seems to me like a better way to go.
from actordb.
Related Issues (20)
- Unable to open req pipe MacOS 10.13.6 HOT 2
- Error: {error,{error,<<"unknown_actor_type: type1">>}} HOT 3
- How to execute actorDB's load test? HOT 1
- SQLight drop in replacement build for C/C++ HOT 1
- LMDB still grows after delete then insert HOT 3
- actordb crash: exception exit: badmatch, sqlite_error, "SQL logic error or missing database" HOT 1
- Timeout error when executing init.example.sql on OSX HOT 2
- Connecting using MySQL CLI on OS X fails with error HOT 5
- PRAGMA LIST not updating after PRAGMA COPY HOT 1
- How to send multiple parameterised queries through the thrift interface? HOT 9
- Using mysql driver HOT 2
- Execute query timeout by using actordb_console
- Same name columns in query result are swallowed and column order can't be specified HOT 1
- Queuing of queries (basic question) HOT 3
- sqlcipher HOT 1
- Sqlite upgarde
- Running protractor test using docker-compose but getting error HOT 3
- Are clusters immutable?
- vm.args needs to have a -name parameter. -sname is not supported.
- Storage for FastAPI service
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from actordb.