jolin-io / continuables.jl

Extremely fast generator-like alternative to Julia Channels

Continuables

TLDR: Python / C# yield with performance matching plain Julia iterators (i.e. unbelievably fast)

Continuables are generator-like higher-order functions which take a continuation as an extra argument. The key macro provided by the package is @cont which will give access to the special function cont within its scope and wraps the computation in a special Type Continuables.Continuable. It is best to think of cont in the sense of yield from Python's Generators. It generates values and takes feedback from the outer process as return value.

If you come from Python, use Continuables wherever you would use generators. If you are Julia-native, Continuables can be used instead of Julia's Channels in many place with drastic performance-improvements (really drastic: in the little benchmark example below it is 20 million times faster!).

This package implements all standard functions like e.g. collect, reduce, any and others. As well as functionalities known from Base.Iterators and IterTools.jl like take, dropwhile, groupby, partition, nth and others.

For convenience, all methods also work for plain iterables.

Installation

Install like

using Pkg
pkg"add Continuables"

Use it like

using Continuables

For further information take a look at the documentation.

Example: flexible alternative to walkdir

Sometimes you recursively want to read files, skipping certain directories and doing other individual adaptations. Using Continuables you get full flexibility with very well readable code and good performance:

list_all_juliafiles(path=abspath(".")) = @cont begin
    if isfile(path)
        endswith(path, ".jl") && cont(path)
    elseif isdir(path)
        basename(path) in (".git",) && return
        for file in readdir(path)
            foreach(cont, list_all_juliafiles(joinpath(path, file)))
        end
    end
end

collect(list_all_juliafiles())

continuables.jl's Issues

Are you aware of ResumableFunctions.jl?

https://github.com/BenLauwens/ResumableFunctions.jl

Wonder if there is a pros and cons and compare and contrasts?

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Prevent allocations when using views

Hey!

First of all, cool packages!, I really did not want to go through the hassle of defining all iterator syntax... and this seems the prefect solution for that hehe

I'm working on a 0 alloc file reader and to make this work I use views of the lines, however when benchmarking I saw my code results in allocations when used with Continuables. I made this minimal example on array views (instead of char views):

using Continuables
using BenchmarkTools

view_test(n::Integer, arr::Vector{Int}) = @cont begin
  for i in 1:length(arr)
      v = view(arr, i:n)
      cont(v)
  end
end

function t1(arr::Vector{Int64})
    tot = 0
    for i in 1:length(arr)
        v = view(arr, i:length(arr))
        tot += length(v)
    end
    return tot
end

function t2(arr::Vector{Int64})
    tot = 0
    foreach(view_test(length(arr), arr)) do x
        tot += length(x)
    end
    return tot
end

test_arr = zeros(Int64, 1_000)
@btime t1($test_arr)
@btime t2($test_arr)

Which gives:

  1.780 μs (0 allocations: 0 bytes)
  26.453 μs (1490 allocations: 23.28 KiB)

P.s. I saw the doc about Ref(). While I might be able to use Ref(tot) here, in my file reader I basically "yield" lines based on numerous condition rather than simply adding up something (see my Julia discourse implementation of the reader).

Is there any way to prevent all these allocs? That would be awesome :)

docs stable link is broken

The link in the README to the stable docs is broken

Dependencies

I have noticed your package and I think it is faster in my case then ResumableFunctions.jl. I currently added your package to dependencies of OpenQuantumSystems.jl, but I would like to make sure, that no unnecessary dependencies are left.

Looking at your Project.toml

name = "Continuables"
uuid = "79afa230-ca09-11e8-120b-5decf7bf5e25"
authors = ["Sahm Stephan <[email protected]>"]
version = "1.0.0"

[deps]
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
Compat = "34da2185-b29b-5c13-b0c7-acf172513d20"
DataTypesBasic = "83eed652-29e8-11e9-12da-a7c29d64ffc9"
ExprParsers = "c5caad1f-83bd-4ce8-ac8e-4b29921e994e"
OrderedCollections = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
SimpleMatch = "a3ae8450-d22f-11e9-3fe0-77240e25996f"

[compat]
BenchmarkTools = "0.5"
Compat = "2.1, 3"
DataTypesBasic = "1.0"
ExprParsers = "1.0"
SimpleMatch = "1.0"
OrderedCollections = "1.3"
julia = "1.3"

[extras]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Test"]

It looks like BenchmarkTools.jl, OrderedCollections.jl, Compat.jl are just imported but not used in src folder. I would also suggest using Inequality specifiers for [compat] section if backward compatibility is really needed.