niamtokik / stream

Erlang Stream Library

Founder of Erlang Punch project.

Work on small Erlang code on my spare time.

stream's Issues

Add QoS feature

Add QoS feature (like rate limiting, queueing limitation, statistics...)

add debug function

We can use default OTP debug feature, but, we can also extend this feature.

Add bookmarks feature

Add bookmarks feature, multiple process could have its own bookmarks and navigate with it

Convert to application

We should offer a way to switch easily to an Erlang application/release with a stream manager.

add pre/post callback

add pre/post callback or action with alerting. When we receive a stream, we should act before add it to our buffer, and we send it, we should also act to the rest of our buffer. This feature should be disabled by default and activated on demand or during startup.

try gen_statem handle_event_function

re-implement it with handle_event_function instead of state_function callback mode (more readability?)

Add different backend for buffer

Add different backend for buffer (will be a bit hard to use it, but could be cool). We could use something like redis buffer or external database.

Add reference management

Add reference management, when a cut is made, we keep copy reference up to date with removed data, based on client state.

add active mode

add active mode (this behaviour connecting to something). Currently, our state machine is passive and waits for some input. In active mode, our state machine request stream.

benchmark testing

benchmark testing from random bytes generator, local files and more. We should be able to optimize our code based on metric.

parsing feature

Add automatic parsing feature based on FSM lambda function or module. When executed, raw value is passed to another process or returned to caller.

Add filter function

Add filter function, to clean buffer and alter its return from caller.

Use reference instead of pid

Don't know if its a good idea to use references instead of pid when we want to communicate to our stream. A references could be a tuple containing more information about stream and should be more flexible (or not). Need to test it.

Make different protocol interfaces

Make different protocol interfaces (ftp, http, ssh...). This should be easy to branch any kind of protocol and stream it.

Copied mode

Copied mode updated dynamically from parent process

create a simple converter interface

Create a simple converter interface from ports/tcp/udp, to send buffer/stream easily.

Create standard/custom size

Create standard/custom size. Currently, I am using something based on state function name, its currently not the right way to do.

Size, Shifting and other altering data in API

All altering data should be pre-calculed on API side, when a caller want its information. We can kept that in gen_statem state or with another method. Why using that? Because calculation is costly.

handle other kind of structure

handle other kind of structure (list, map, custom one...), currently, we supports only bitstrings.

Shared stream

Shared stream (act as bucket without ordering, got data and just push it on buffer).

Add subscription feature

Add subscription feature, relaying data from buffer to another process or react to some events

choose when we want to alter buffer.

Add "cut" or "copy", we should choose when we want to alter buffer (or not). These functions are already implemented but should be reviewed.

Add search function

Add search function, to find a specific patterns and react on it. A particular process (or caller) will be alerted on this pattern.

Create common test

Currently, only unit testing is used. We need real world testing based on different type of data.

Create a transactional datastructure

Create a transactional datastructure when something is written on buffer. Currently, we don't have write log feature and we can't guaranty our data are written in order (or not).

Make stream as Erlang behavior

stream library should be used as behavior, currently, we are using it as simple gen_statem.

Add buffer limits

Add buffer limits, when specific size is reached, we do some action. This could be like rate limiting.

add more documentation and example usage

This is for the first release. We need lot of documentation and example usage. No documentation == no release.

add compression feature

add compression feature, compress the current stream on the fly.

try gen_server behavior

re-implement it with gen_server behavior (more performance?)

Add action function

Add action function, will react when some kind of pattern arrive, its a bit like search, but with custom action.

Poll events automatically

Polling event automatically based on some kind of inputs or external sources.

Add Copy mode with splitted buffer

Add Copy mode with splitted buffer (create a new process with buffer copied from original)

Need to add optional parameters to each functions

Each functions behavior should be overloaded (size, callback...) with extra-options.

Add forwarder size limit

Add forwarder size limit. When we reach a forward limit an action is executed.

dynamic size request

dynamic size (change based on data received). Currently, when caller do some request on our state machine, the pattern size is fixed. We should add a feature to dynamically change it based on caller or on stream.

handle error based on bitstring size

find a way to handle error correctly based on bitstring size. I have worked on it, and, currently, no error are generated, instead, our state is modified with insufficient resource

automatic loop on stream

create an API to generate loop over stream automatically.

add Erlang code version

add Erlang code version to manage state machine update.

split files

stream should be splitted in multiple little files containing all logical behavior. In our case, something like:

• stream.erl: main state machine, with full API
• gen_stream.erl: Erlang stream behavior
• stream_passive.erl: passive stream, receive only data
• stream_active.erl: active stream, responsible of its own stream (manage its own fd/ports)

use timeout instead of errors

Use timeout instead of error message. This feature must be set with a default value when stream is started.

add deduplication feature

add deduplication feature, like compression. Don't know if its a good idea.

Add callback manager

Add callback manager (logging event & more)

dedicated stream

dedicated stream (process sharing common reference and access input only from each one)