epi / flod Goto Github PK
View Code? Open in Web Editor NEWStream processing pipelines in D
License: Boost Software License 1.0
Stream processing pipelines in D
License: Boost Software License 1.0
just compose it backwards. can still look left-to-right if ufcs is not used, eg:
mmappedFile("foo").peekPush(somePushFilter(nextPushFilter(pushSink()))
E.g. test performance of byLine vs. File.byLine(); or write an example parser and test its performance vs. existing equivalent in C.
peek
and alloc
could accept a reference to the buffer, just like reallocate in allocator API.This means that element type must be defined explicitly in peek sink or alloc source, either literally or by using templates to agree with peek source's or alloc sink's expected types.
Question: This could also make writing an efficient FallbackBuffer easier, couldn't it?
Missing adapters that would use flod.buffer are:
Other missing adapters:
Some stage structs can't be copied, other shouldn't be moved. Movable structs should be marked @movable, UDA will be checked by startPipe() and appendPipe() and then they'll create non-movable, non-copyable stages using allocator API.
built-in arrays as peek sources (trivial)
input ranges (pull source, copy by element to sink's buffer)
alloc
always returns the buffer capable of holding at least the requested number of elements. TODO: decide if it should throw or return null/false if it can't provide such buffer.commit
returns a size_t
that should == the requested number of elements. Returning a smaller number means that everything is ok, but the sink is not interested in reading more data.InSituBuffer has a fixed-size (set at compile time) buffer inside it.
alloc() may move data on wrap
FallbackBuffer contains two buffers and uses the first one until it fails to alloc() the requested size. FallbackBuffer then tries to alloc() using the other buffer and moves the data to it, if successful.
Initial experiments with @satisfies
made it look promising.
Without verbose constraint checkers it is next to impossible to find what's wrong in stage's code.
Before doing anything else, current traits should be extended with constraint checkers.
Multiple-inverter unit tests in flod.pipeline that are commented out cause either compilation or run-time errors.
An active source with step can be wrapped like this:
Sink(AdapterSource(Source(AdapterSink)))
or Sink(AdapterSource(Source(ItsSource, AdapterSink)))
Call byChunk() without arguments if you don't care about the size of a chunk and you're able to process chunks of any size.
Call byChunk(size_t chunkSize) if you require chunks of specific size.
Call byChunk(T)(T[] buf) if you require chunks of specific size and you want to manage the buffer where they'll be stored.
byChunk should implement input range interface as well as opApply() iteration.
byChunk should be optimized according to the pipeline's last stage source method and required chunk size and buffer ownership.
Semantics of Buffer.alloc() and Stage.alloc() is not scritly
defined yet.
Buffer should use bool alloc(ref T[], size_t)
so that e.g. FallbackBuffer is possible.
Stage should probably just throw when it's unable to fulfill
the request. Or maybe it should return false and give the
largest buffer it can at the moment, and let the source decide.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.