nozeio / noze.io Goto Github PK

Claas reported that swift build –config release fails to build, apparently with errors in base64. Need to check.

Now master branch is default, so for the first time we saw a tree not updated while a month, we were afraid: "is this project alive?"

But develop branch is living surely. I think it is better.

Noze.io streams work on batches of items. The items T can be any Swift type due to the use of generics. You can have streams that read and/or write bytes (UInt8) or Characters, or String-lines, or database records, or tweets, etc. In Noze the items are always processed in batches to be efficient. E.g. if a parser reads 5 lines, it won't issue 5 read calls, but just one containing all 5 lines.

The original idea was to make a 'bucket of items' a protocol, like:

protocol Bucket<T> { ... }

And then make existing bucket data structures conform to that, say:

Array<T>: Bucket<T>

and

DispatchData : Bucket<UInt8>

Any stream which would process Bucket<UInt8>'s would have been able to deal with [ UInt8 ] or DispatchData or a memory mapped file handle, etc.

Unfortunately that didn't/doesn't fly (yet?) in Swift 2/3 because you can't use a generic protocol as a type. (essentially this doesn't work: let someBucket : Bucket<UInt8>).

So as a temporary measure we skipped the issue and just used Swift arrays as 'buckets'. That is [T].
Now of course this is pretty slow in a lot of circumstances :-)

Summary: we need more efficient buckets.

One non-ideal solution is to use a base class for buckets and subclasses which wrap the various container classes. Like:

class Bucket<T> {}

class ArrayBucket<T> : Bucket<T> { let items: [ T ] }

class DispatchDataBucket : Bucket<UInt8> { let items : DispatchData }

Presumably this is much better than the [T] and may be the way to go, but obviously having an extra heap allocated wrapper isn't 1337.
Better suggestions are very welcome.

nice work
do you plan to support firebase queue?
https://github.com/firebase/firebase-queue

We'd like to use the upstream C http_parser instead of the C port (performance, stability, etc). This is easy to when using Swift Package Manager (which can compile C targets). But we also want an out-of-the-box Xcode experience.

Summary: Need to adjust the Xcode project so that the C http_parser can be built as part of it. Needs some module map magic (contributions are welcome :-)

As object _ArrayType doesn't exist in Swift 3

The Freddy embedded in Noze.io is a little outdated. Functionality should be OK, but the API of Freddy changed a little since the fork. We should update the code to be compatible with upstream.

I have a Freddy fork which builds on Linux over here: https://github.com/helje5/Freddy (untested)

Debug is working, it could be Xcode 8.0.1 having issue?

Compile Swift Module 'httpd_helloworld' (1 sources)
Linking ./.build/release/httpd-helloworld
duplicate symbol __TMRbBp in:
    /Users/test/Noze.io/Samples/httpd-helloworld/.build/release/fs.build/FileSource.swift.o
    /Users/test/Noze.io/Samples/httpd-helloworld/.build/release/fs.build/FileTarget.swift.o
duplicate symbol __TMRbBp in:
    /Users/test/Noze.io/Samples/httpd-helloworld/.build/release/fs.build/FileSource.swift.o
    /Users/test/Noze.io/Samples/httpd-helloworld/.build/release/fs.build/GCDChannelBase.swift.o
ld: 2 duplicate symbols for architecture x86_64
<unknown>:0: error: build had 1 command failures
error: exit(1): /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-build-tool -f /Users/test/Noze.io/Samples/httpd-helloworld/.build/release.yaml

... old Linux Swift used to require a lot of public declarations. I think this is not required anymore, at least not if proper clang/go.ld are being used.

Summary: go through the source code, drop #linux-public tags and try whether everything still works fine. (Note: I may not have tagged everything #linux-public :-)

E.g.: https://travis-ci.org/NozeIO/Noze.io/jobs/215874965

Cloning /home/travis/build/NozeIO/Noze.io
error: The dependency graph could not be satisfied. The package (/home/travis/build/NozeIO/Noze.io) with version tag in range (0.5.0..<0.5.9223372036854775807) is not found. Found tags ([0.3.1337])
make[1]: Leaving directory `/home/travis/build/NozeIO/Noze.io/Samples'
The command "make samples" exited with 0.

No idea what is going on here. It says make samples exited with 0 even though we got the error. Is that our fault or is SPM not setting a proper exit code?

There is Swift NIO
https://github.com/apple/swift-nio

Is it possible to change the underlying engine of Noze.io with this?

This library looks really great, but it doesn't appear to be officially licensed as open source. I was wondering if that was intentional, or if perhaps you're still deciding on a license? If you're still deciding, I think most developers in the community would love to see it MIT licensed like Node.js and the majority of open source Swift projects as well.

Thanks!

We currently invoke lstat instead of stat on Swift 4. Need to fix that.

Noze.io has the concept of streams (GReadableStream, GWritableStream) and generators/sinks (GReadableSourceType, GReadableTargetType). The latter just read and write items while the former provide a buffer and all the event and piping logic.

That separation of concerns seems actually quite good, but also makes the solution quite a bit more complex and adds an extra layer of callback closures.
So in retrospective I think we should rather remove them and just have streams with the relevant logic embedded in subclasses.

Need improvement to handle load, would it be idea to participate the Swift Server API and consider using their HTTP with Node.js HTTP Parser (C language) for better performance than Node.js itself if the team decide to choose that.

Error output during wrk testing:

wrk -d10 -c100 -t2 http://localhost:1337/
Running 10s test @ http://localhost:1337/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    68.09ms   22.21ms 132.56ms   69.66%
    Req/Sec   370.48     68.96   560.00     65.00%
  7379 requests in 10.01s, 1.50MB read
  Socket errors: connect 0, read 795, write 1, timeout 0
Requests/sec:    737.34
Transfer/sec:    152.98KB