openziti / dilithium Goto Github PK
View Code? Open in Web Editor NEWFramework for high-performance streaming over message-passing systems. High-performance WAN protocols over UDP datagrams. Implemented in golang.
License: Apache License 2.0
Framework for high-performance streaming over message-passing systems. High-performance WAN protocols over UDP datagrams. Implemented in golang.
License: Apache License 2.0
Iterate our basic v0.1
protocol to support "inline" acknowledgement. Inline acknowledgements are also cumulative, and indicate an acknowledgement "high water mark".
Slightly tricky in that we'll have to send acknowledgements even when there is no outbound data ready for shipment.
Capture westworld2
configuration details alongside the metrics output. Use these details to create configuration annotations that are available in the analyzer.
Retransmission events and duplicate ack events currently share the throttle_fraction
property. Should incorporate different fractions for both signal types.
Survey:
RBUDP is a data transport tool and protocol specifically designed to move very large files over wide area high-speed networks (on the order of gigabits per second). This is a simple replacement for FTP which tends to do poorly over long distance high-speed networks.
Reliable Blast has two goals. The first is to keep the network pipe as full as possible during bulk data transfer. The second goal is to avoid TCP’s per-packet interaction so that acknowledgments are not sent per window of transmitted data, but aggregated and delivered at the end of a transmission phase.
https://www.evl.uic.edu/cavern/RBUDP/Reliable%20Blast%20UDP.html
Use an inline flag to indicate session close.
Head to the APAC regions with the most onerous conditions, and square off with tcp
.
Memory handling trace. Finalization?
https://golang.org/pkg/runtime/#SetFinalizer
Can we augment our memory pool framework to include stack details, allowing us to trace sources of allocations and unref
s?
TCP Westwood-inspired v0.2
implementation.
Digest RFC 1323:
Incorporate stats
into the trace
instrument, creating the uber-instrument for development.
What can we learn from the Westwood[+]
implementation of TCP?
TCP Westwood significantly increases fairness wrt TCP (New) Reno in wired networks and throughput over wireless links.
Implement a pluggable soft-start and congestion control implementation for proving grounds testing.
Implement a v0.1
version of the paired TCP+UDP protocol.
Implement keepalives in westworld2
.
Capture and analyze perceived tunnel performance at the client and server, regardless of underlying protocol. This will allow us to produce metrics and graphs regardless of protocol.
Minimize the burn on runtime.futex
. Get state management working without sync.Mutex
and sync.Cond
.
Implement a clean close signal.
Experiment with using the size of the admitted data payload as a scaling factor for increasing the txPortal
size, instead of a static increase.
Get Google Netstack up and running in our testing harness.
https://github.com/google/netstack/
See what we can learn from their user-space golang network stack implementation.
Similar to using every-n scaling to decrease portal size, use every-n with an accumulator to increase the portal size.
We're producing a lot of metrics data. Need to summarize this into something easier to visualize and work with.
Thorough survey of QUIC:
Implement basic congestion control, similar to the stock Nagle implementation:
Congestion Control in IP/TCP Internetworks:
https://tools.ietf.org/html/rfc896
TCP Congestion Control:
A survey of more advanced congestion control topics and implementations.
The Addition of Explicit Congestion Notification (ECN) to IP:
https://tools.ietf.org/html/rfc3168
Robust Explicit Congestion Notification (ECN) Signaling with Nonces:
https://tools.ietf.org/html/rfc3540
Proportional Rate Reduction for TCP:
https://tools.ietf.org/html/rfc6937
Implement packetization according to a MSS.
Implement soft close in westworld2
.
Implement high and low limits for txPortal
sizing. There are likely corner cases where the current implementation can starve itself of portal size and die.
Implement an Instrument
, which can produce detailed trace output.
Develop a parallel implementation of westwood
supporting the lockless model.
An initial implementation of memory pooling for westworld.
Modularize the dilithium influx
code, such that it continues to work from the dilithium
cmd, but make it embeddable so that it can be inserted into fablab
to automate retrieval of remote metrics and insertion into the analyzer.
Implement buffer pooling.
Modify our Hello
protocol to include a final acknowledgement.
Next iteration of westworld/wb
and the pooled buffer system behind them.
Configure RX/TX socket buffer sizes independently, and separately from poolBufferSz
.
iperf
is nice and all, and people seem to respect that it's an arbiter of "truth" in terms of the performance of network pipes... but we also need a performance measurement tool, which verifies the integrity of the data traveling across the network.
Probably will look something like the loop2
tool in the ziti fabric.
Implement a proving grounds for comparing protocols across varyous WAN links, with multiple types of virtual machines, in fablab
.
Implement a "metrics" instrument, which is able to capture changing data over time, for visualization.
We're going to want to run various configurations of westworld
against one another, without having to recompile our dilithium
executable. Externalize the various tunables, such that we can run different variants of the configuration against one another for comparison.
Westworld really wants to be built using connected UDP.
Because dilithium.
We're currently using a hard-coded RETX timeout.
Compute expected ACK arrival time based on observed RTT. If ACK does not arrive in window, RETX.
I have a theory that the jumbo frame "cheat" is ultimately adjusting down the sensitivity of the portal scaling. Introducing an "every-n" sensitivity adjustment to portal scaling events might achieve the same effect.
Implement basic MSS packetization. Currently relying on os-level fragmentation.
Implement random starting sequence numbers. The hello portion of the protocol already supports starting from a non-zero sequence number.
Ensure that sequence number rollover is operating correctly (it most certainly is not, currently).
Implement retransmission.
https://github.com/google/flatbuffers
Potentially much higher performance than ProtoBufs. Will probably cooperate better with memory pooling.
Current RTT time average is accumulated over a fixed 8-RTT window. Make window length configurable.
RTT probes currently at 1 second intervals. Make configurable. Higher RTT probe resolution may allow more efficient retx behavior.
Survey the entirety of TCP to check for implementation coverage (or intentional omittance):
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.