epi / flod Goto Github PK

E.g. test performance of byLine vs. File.byLine(); or write an example parser and test its performance vs. existing equivalent in C.

• peek and alloc could accept a reference to the buffer, just like reallocate in allocator API.

This means that element type must be defined explicitly in peek sink or alloc source, either literally or by using templates to agree with peek source's or alloc sink's expected types.
Question: This could also make writing an efficient FallbackBuffer easier, couldn't it?

Missing adapters that would use flod.buffer are:

• push-peek
• push-pull
• alloc-pull
• alloc-peek
• alloc-push

Other missing adapters:

• pull-alloc
• peek-push
• peek-alloc
• push-alloc

Semantics of Buffer.alloc() and Stage.alloc() is not scritly
defined yet.
Buffer should use bool alloc(ref T[], size_t)
so that e.g. FallbackBuffer is possible.
Stage should probably just throw when it's unable to fulfill
the request. Or maybe it should return false and give the
largest buffer it can at the moment, and let the source decide.

just compose it backwards. can still look left-to-right if ufcs is not used, eg:
mmappedFile("foo").peekPush(somePushFilter(nextPushFilter(pushSink()))

An active source with step can be wrapped like this:
Sink(AdapterSource(Source(AdapterSink)))
or Sink(AdapterSource(Source(ItsSource, AdapterSink)))

1. Sink calls pull or peek on AdapterSource
2. AdapterSource calls step(n) on Source, where n is a hint on how many elements are to be written to AdapterSink. (TODO: should n be just a hint or an exact required number?)
3. Step calls alloc/commit or push on AdapterSink.
4. AdapterSource gets data from AdapterSink and passes them to Sink.

Multiple-inverter unit tests in flod.pipeline that are commented out cause either compilation or run-time errors.

InSituBuffer has a fixed-size (set at compile time) buffer inside it.
alloc() may move data on wrap
FallbackBuffer contains two buffers and uses the first one until it fails to alloc() the requested size. FallbackBuffer then tries to alloc() using the other buffer and moves the data to it, if successful.

• alloc always returns the buffer capable of holding at least the requested number of elements. TODO: decide if it should throw or return null/false if it can't provide such buffer.
• commit returns a size_t that should == the requested number of elements. Returning a smaller number means that everything is ok, but the sink is not interested in reading more data.
  Write errors (including broken pipe or disk full) should cause an exception to be thrown.

built-in arrays as peek sources (trivial)
input ranges (pull source, copy by element to sink's buffer)

Some stage structs can't be copied, other shouldn't be moved. Movable structs should be marked @movable, UDA will be checked by startPipe() and appendPipe() and then they'll create non-movable, non-copyable stages using allocator API.

Call byChunk() without arguments if you don't care about the size of a chunk and you're able to process chunks of any size.
Call byChunk(size_t chunkSize) if you require chunks of specific size.
Call byChunk(T)(T[] buf) if you require chunks of specific size and you want to manage the buffer where they'll be stored.

byChunk should implement input range interface as well as opApply() iteration.
byChunk should be optimized according to the pipeline's last stage source method and required chunk size and buffer ownership.

Initial experiments with @satisfies made it look promising.
Without verbose constraint checkers it is next to impossible to find what's wrong in stage's code.
Before doing anything else, current traits should be extended with constraint checkers.