A series of tubes.
Home Page: https://twisted.github.io/tubes/
License: Other
It seems likely to be useful to have some tubes that encode and decode strs.
(Mostly because of #67, otherwise it'd be a 1-liner to inline create such a thing)
Rabbit would be a great example, but generally, Tubes are a fantastic way to write a queue consumer; this would really show how to leverage Tubes to do interesting high-level things in the context of a real system.
It seems a bit tedious (and procedural) to need to write
def connectStuff(flow):
flow.fount.flowTo(stuff).flowTo(flow.fount.drain)
Specifically if I already have a series and just want to listen and flow through that series, rather than just passing that series in somewhere (to a Listener?) I have to manually rig it up as above. Is this not common enough to automate?
(If such a thing already exists maybe this is another doc request)
docs should be published to readthedocs
I've found the direct links to the source code in twisted to be an invaluable aid to understanding, well, just about everything. What the doc is talking about, what implications of the docs are actually true, to say nothing of a ready-made example of what I need to do if I need something somewhat different than what's available.
I suspect, for instance, that I only managed to figure out how to build my conch-based call-in support monster because of the quick linking. (If said monster is unpalatable, forget I said anything :p)
While the source is obviously sitting right near-by my code in my properly configured virtualenv, it's still a bit of a nontrivial half-second-to-three-second speed bump to find my place, and leaves open the possibility of looking at a similarly named but very different thing, with the associated confusion caused thereby. I don't know of any troublesome examples of that in tubes yet, but I hope you agree it's only a matter of time.
tubes.tube.series seems likely to be pretty common.
If you get pretty drunk, does it seem like maybe something like tube > tube > tube do series(tube, tube, tube)?
If you construct a fan.In that feeds a fan.Out which in turn feeds that same fan.In (even if this is indirected through a number of intermediary Tubes) it will create a flow graph that will be permanently paused, never passing any traffic. This is an inscrutable failure mode. It would be nice to see immediately that you have set up a cycle and get an exception.
It might be interesting to implement this in terms of a general-purpose interface for getting topology information from a Fount or Drain (maybe a Node interface with two interfaces: "downstreams" and "upstreams"). This sort of interface could also be used for visualizations (graphviz here we come)
In the pre-move commit twisted/twisted@3877aad, the return value for Router.newRoute changed so that it no longer returns a tuple of token and fount, but just the fount. The example in the module docstring should be adjusted to reflect this change.
we shouldn't be importing twisted anywhere outside of specific integration modules
In the intro docs, under "Getting Connected: an Echo Server", after the example, which looks like this:
def echoFlow(flow):
flow.fount.flowTo(flow.drain)the next paragraph makes the outrageous claim that "In the above example, echoFlow takes two things". IT TAKES ONE THING, or I'm insane (I'm going to assume the former).
I get stuck here wondering if I can't count, and then when I accept that I'm simply being misled, I'm left wondering the degree to which this evil plot to confuse me is trying to drive me nuts, which them makes me wonder if perhaps I'm already nuts, and I should have assumed the latter.
WTF is flow? Is it a type I should know about? Or some random thingo with these two attributes attached to it? OMG I DON'T KNOW! And then it says here that the function is a "flow". WHAT?!? Why is the argument called the thing that the function is?
Anyway, I feel like I'd be feeling a lot less stress if this function actually took two arguments:
def echoFlow(fount, drain):
fount.flowTo(drain)Then you can say it takes two thing, and even call the function a flow and I'd be cool with all of that.
Peace be unto you.
From the docstring on tube:
"""
L{tube} is a class decorator which declares a given class to be an
implementer of L{ITube} and fills out any methods or attributes which are
not present on the decorated type with null-implementation methods (those
which return None) and None attributes.
@param cls: A class with some or all of the attributes or methods described
by L{ITube}.
However, the readme shows:
@tube
def numbersToLines(value):
yield str(value).encode("ascii")
Indeed, when I try to decorate a function, I get:
zope.interface.exceptions.BrokenMethodImplementation: The implementation of started violates its because implementation requires too many arguments.
The intro doc does things like
@tube
def foo(item):
yield item
which presumably would create a simple tube using that function as the received, but that does not actually appear to be a thing tubes.tube.tube does (and doing so just complains that the resulting object does not have a started attribute when you hook it up to a flow).
The intro doc contains an example of an echo server which listens indefinitely, but it should likely also show how to process only a finite set of input and then exit.
we should be able to bind directly to the transport layer in asyncio
all Twisted imports should be moved into a sub-package and possibly moved into a different repository entirely.
there should be a tube which runs within asyncio and converts asyncio.Future objects into their results
log.err will hopefully shortly go away even in twisted, and it is unnecessary coupling to twisted itself. there should be some mechanism for setting the global exception handler.
(Carried over from IRC to an actual bug tracker :/)
The API doc links from e.g. https://tubes.readthedocs.org/en/latest/tube.html#an-introduction-to-tubes point at twistedmatrix.com when they seem to want to point at twisted.github.io now.
Also, the second example (with a Reverser) seems incomplete, or at least, reading through it, I cannot yet tell how to make it work.
If I put
from tubes.protocol import factoryFromFlow
from twisted.internet.endpoints import serverFromString
from twisted.internet.defer import Deferred
def reverseFlow(fount, drain):
from tubes.framing import bytesToLines, linesToBytes
lineReverser = series(bytesToLines(), Reverser(), linesToBytes())
fount.flowTo(lineReverser).flowTo(drain)
class Reverser(object):
def received(self, item):
yield b"".join(reversed(item))
def main(reactor, listenOn="stdio:"):
endpoint = serverFromString(reactor, listenOn)
endpoint.listen(factoryFromFlow(reverseFlow))
return Deferred()
if __name__ == '__main__':
from twisted.internet.task import react
from sys import argv
react(main, argv[1:])
in a file and run it, I get a thing that silently does nothing, seemingly.
There's possibly a mix of an upstream issue here, but after some quick debugging:
if I stop listening on stdio, now I get tracebacks on stdout when I make a connection:
/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/endpoints.py:29: DeprecationWarning: twisted.internet.interfaces.IStreamClientEndpointStringParser was deprecated in Twisted 14.0.0: This interface has been superseded by IStreamClientEndpointStringParserWithReactor.
from twisted.internet.interfaces import (
2015-11-19T14:51:22-0500 [-] Factory starting on 1234
2015-11-19T14:51:22-0500 [twisted.internet.protocol.Factory#info] Starting factory <twisted.internet.protocol.Factory instance at 0x0000000108c6c720>
asdf
Unhandled Error
Traceback (most recent call last):
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/python/log.py", line 84, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/selectreactor.py", line 149, in _doReadOrWrite
why = getattr(selectable, method)()
--- <exception caught here> ---
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/tcp.py", line 1074, in doRead
protocol.makeConnection(transport)
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/protocol.py", line 494, in makeConnection
self.connectionMade()
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/tubes/protocol.py", line 231, in connectionMade
self._flow(self._fount, self._drain)
File "send.py", line 23, in reverseFlow
lineReverser = series(bytesToLines(), Reverser(), linesToBytes())
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/tubes/tube.py", line 220, in series
drains = map(IDrain, tubes)
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/zope/interface/interface.py", line 142, in __call__
raise TypeError("Could not adapt", obj, self)
exceptions.TypeError: ('Could not adapt', <__main__.Reverser object at 0x0000000108c59fa0>, <InterfaceClass tubes.itube.IDrain>)
2015-11-19T14:51:27-0500 [twisted.internet.protocol.Factory] Unhandled Error
Traceback (most recent call last):
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/python/log.py", line 84, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/selectreactor.py", line 149, in _doReadOrWrite
why = getattr(selectable, method)()
--- <exception caught here> ---
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/tcp.py", line 1074, in doRead
protocol.makeConnection(transport)
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/protocol.py", line 494, in makeConnection
self.connectionMade()
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/tubes/protocol.py", line 231, in connectionMade
self._flow(self._fount, self._drain)
File "send.py", line 23, in reverseFlow
lineReverser = series(bytesToLines(), Reverser(), linesToBytes())
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/tubes/tube.py", line 220, in series
drains = map(IDrain, tubes)
File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/zope/interface/interface.py", line 142, in __call__
raise TypeError("Could not adapt", obj, self)
exceptions.TypeError: ('Could not adapt', <__main__.Reverser object at 0x0000000108c59fa0>, <InterfaceClass tubes.itube.IDrain>)
which seems to indicate that that isn't in fact enough to get a working IDrain?
It's also really off-putting I think to new users if tracebacks are completely silenced there. That was my first 5 minutes with Tubes unfortunately and I had literally no idea what was going wrong, and also there were 2 logging frameworks to disect to figure out where my tracebacks went (along the way I discovered that if I added both:
globalLogPublisher.addObserver(textFileLogObserver(sys.stderr))
and
log.startLogging(sys.stderr)
to try and get both frameworks to not swallow anything, I get instead an infinite stream of recursion on stderr which I think is the two of them fighting with each other:
# ... truncated
2015-11-19 14:58:12-0500 [-] File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/defer.py", line 121, in execute
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/defer.py", line 121, in execute
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/defer.py", line 121, in execute
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/defer.py", line 121, in execute
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/defer.py", line 121, in execute
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/defer.py", line 121, in execute
2015-11-19 14:58:12-0500 [-] UNFORMATTABLE OBJECT WRITTEN TO LOG with fmt '%(log_legacy)s', MESSAGE LOST
2015-11-19 14:58:12-0500 [-] result = callable(*args, **kw)
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] result = callable(*args, **kw)
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] result = callable(*args, **kw)
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] result = callable(*args, **kw)
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] result = callable(*args, **kw)
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] result = callable(*args, **kw)
2015-11-19 14:58:12-0500 [-] UNFORMATTABLE OBJECT WRITTEN TO LOG with fmt '%(log_legacy)s', MESSAGE LOST
2015-11-19 14:58:12-0500 [-] File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/posixbase.py", line 478, in listenTCP
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/posixbase.py", line 478, in listenTCP
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/posixbase.py", line 478, in listenTCP
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/posixbase.py", line 478, in listenTCP
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/posixbase.py", line 478, in listenTCP
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] File "/Users/Julian/.local/share/virtualenvs/kafka_faf/site-packages/twisted/internet/posixbase.py", line 478, in listenTCP
2015-11-19 14:58:12-0500 [-] UNFORMATTABLE OBJECT WRITTEN TO LOG with fmt '%(log_legacy)s', MESSAGE LOST
2015-11-19 14:58:12-0500 [-] p.startListening()
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] p.startListening()
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] p.startListening()
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] p.startListening()
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] p.startListening()
2015-11-19 14:58:12-0500 [-] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] 2015-11-19T14:58:12-0500 [stderr#error] p.startListening()
# ... truncated
-- it might be that not having tracebacks shown is really an upstream issue which is "maybe tracebacks should be going to stderr when listening on a stdio endpoint? or something?"
Sorry to file such a tangled issue report, as I learn a bit more about tubes over the next hour I'll hopefully be able to untangle, but that was a brain dump of the first 10 minutes or so.
(I remember somewhere this idea being brought up but don't remember where that was)
It seems like it might be nice to -- just like endpoints do so for listening+connecting -- have injectable ways of describing what framing is being used on those endpoints, whether it belongs in tubes or otherwise.
E.g., if I decide to run a server using line-delimited messages for testing, I'd like to write something like line:(stdio:):line to get a series that does bytesToLines -> something -> linesToBytes and then maybe later when I go to prod decide I want int32netstring:(stdio:):line because now there's a server on the other end.
Thoughts?
The intro doc should show how to take an API that returns a deferred and flow the result of the deferred into further tubes in a series.
I would expect flowStopped to be called on each participant in the flow. It seems that stopped is called on the tube with the buggy received method but the exception doesn't appear to be propagated beyond that.
I actually have no idea where zope.interface's docs are or if pydoctor can generate a compatible link structure. Maybe intersphinx support would help if we had a link to its sphinx output?
Sometimes, when encountering backpressure – assuming that the output type of the given fount is a discrete message and not part of a continuous stream like IFragment – the thing the fount actually wants to do instead of delivering things is to drop them. For example, in the case of a fan.Out keeping a bunch of observers up to date about the current state of something, you may want to allow some of those observers to just not see messages while their respective buffers are full.
See #35
Presently on trunk, tubes are hooked up to the outside world (i.e.: Twisted) by converting a "flow function", a function taking a fount and drain, to a Factory, which is then hooked up to Twisted.
However, this is limiting in one major regard: the backpressure of not accepting future connections is impossible to model properly.
A better way to conceptualize a listening socket is a fount whose outputType is an object with both Fount and Drain attributes. This way, we can flow the stream of incoming connections to a "listening" drain, which can exert backpressure (most obviously by simply limiting the number of concurrently active connections).
As it happens, this is also a feature missing from Twisted which makes it hard to implement fairness in queueing. By making tubes treat listening sockets and connected sockets consistently, we can open up a whole new area of correct-by-default behavior under high levels of load.
https://github.com/twisted/tubes/blob/master/sketches/amptube.py#L115 typoes twisted.iterne
There are only two mentions of it, and they're both in the source code of the first example
Add an IOFount class that makes a fount from a readable FLO.
The code below works, but it's doing a blocking read on the data, and should be streaming instead.
@implementer(IFount)
@attrs(frozen=False)
class IOFount(object):
"""
Fount that reads from a file-like-object.
"""
outputType = ISegment
_source = attrib() # type: BinaryIO
drain = attrib(
validator=optional(provides(IDrain)), default=None, init=False
) # type: IDrain
_paused = attrib(validator=instance_of(bool), default=False, init=False)
def __attrs_post_init__(self):
# type: () -> None
self._pauser = Pauser(self._pause, self._resume)
def _flowToDrain(self):
# type: () -> None
if self.drain is not None and not self._paused:
data = self._source.read()
if data:
self.drain.receive(data)
self.drain.flowStopped(Failure(StopIteration()))
# FIXME: this should stream.
def flowTo(self, drain):
# type: (IDrain) -> IFount
result = beginFlowingTo(self, drain)
self._flowToDrain()
return result
def pauseFlow(self):
# type: () -> None
return self._pauser.pause()
def stopFlow(self):
# type: () -> None
return self._pauser.resume()
def _pause(self):
# type: () -> None
self._paused = True
def _resume(self):
# type: () -> None
self._paused = False
self._flowToDrain()
the code is mostly twistedchecker-clean, we should ensure it stays that way
I have no explicit pause / unpause logic in my code but I reliably get this after the first value dribbles through the tube:
2017-05-02T13:49:41-0400 [twisted.internet.defer#critical]
Traceback (most recent call last):
File "/home/exarkun/Environments/kubetop/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 651, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/home/exarkun/Environments/kubetop/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1030, in _cbDeferred
self.callback(self.resultList)
File "/home/exarkun/Environments/kubetop/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 457, in callback
self._startRunCallbacks(result)
File "/home/exarkun/Environments/kubetop/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 565, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/home/exarkun/Environments/kubetop/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 651, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/home/exarkun/Environments/kubetop/local/lib/python2.7/site-packages/tubes/undefer.py", line 28, in done
pause.unpause()
File "/home/exarkun/Environments/kubetop/local/lib/python2.7/site-packages/tubes/kit.py", line 39, in unpause
self._friendPauser._actuallyResume()
File "/home/exarkun/Environments/kubetop/local/lib/python2.7/site-packages/tubes/_siphon.py", line 184, in _actuallyResume
fp.unpause()
File "/home/exarkun/Environments/kubetop/local/lib/python2.7/site-packages/tubes/kit.py", line 39, in unpause
self._friendPauser._actuallyResume()
File "/home/exarkun/Environments/kubetop/local/lib/python2.7/site-packages/tubes/_siphon.py", line 184, in _actuallyResume
fp.unpause()
exceptions.AttributeError: 'NoneType' object has no attribute 'unpause'
The intro doc shows how to set up a basic echo server, and separately introduces series() when it discusses "real" logic, but does not show how to take the series example into a working server.
Besides the broken docstring for the Router as reported in #21 the Router also generally appears not to work; or am I missing something simple here?
see my attempt at creating unit tests for Router:
https://github.com/david415/tubes/tree/add_routing_tests.0
https://github.com/david415/tubes/blob/add_routing_tests.0/tubes/test/test_routing.py
When the Router's newRoute() method is called this results in:
$ trial tubes.test.test_routing
tubes.test.test_routing
TestBasicRouter
test_basic_router ... [ERROR]
===============================================================================
[ERROR]
Traceback (most recent call last):
File "/home/user/tubes/tubes/test/test_routing.py", line 64, in test_basic_router
evenOddTube.addRoutes()
File "/home/user/tubes/tubes/test/test_routing.py", line 41, in addRoutes
self.evenRoute = self.newRoute()
File "/home/user/tubes/tubes/routing.py", line 179, in newRoute
fount = self._out.newFount().flowTo(received)
File "/home/user/tubes/tubes/fan.py", line 253, in flowTo
return beginFlowingTo(self, drain)
File "/home/user/tubes/tubes/kit.py", line 104, in beginFlowingTo
return drain.flowingFrom(fount)
exceptions.AttributeError: '_Tubule' object has no attribute 'flowingFrom'
tubes.test.test_routing.TestBasicRouter.test_basic_router
-------------------------------------------------------------------------------
Ran 1 tests in 0.018s
FAILED (errors=1)
Sometimes, when a fount experiences backpressure, the right thing to do is to just give up immediately. For example, in a pub/sub service using fan.Out, if one of the drains is falling behind and tells its fount to pauseFlow, we may instead want to simply stopFlow.
The same goes for ITube.received, the tutorial starts defining it (in its first Reversed example) as if it's obvious that it's part of some interface and is how you'd connect into tubes, but only mentions that fact later down.
I'm fairly certain this line has got to be wrong:
https://github.com/twisted/tubes/blob/master/tubes/protocol.py#L407
The string _pauseForNoDrain only appears once in the source tree:
https://github.com/twisted/tubes/search?utf8=%E2%9C%93&q=_pauseForNoDrain
If you do the above, which I assume everyone will do at one point or another, you get the horrid:
Traceback (most recent call last):
File "/home/julian/.local/share/virtualenvs/pkg/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/julian/Development/pkg/pkg/_cli.py", line 33, in crawl
react(_crawl, [endpoint])
File "/home/julian/.local/share/virtualenvs/pkg/site-packages/twisted/internet/task.py", line 908, in react
finished = main(_reactor, *argv)
File "/home/julian/.local/share/virtualenvs/pkg/site-packages/twisted/internet/defer.py", line 1532, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
File "/home/julian/.local/share/virtualenvs/pkg/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
result = g.send(result)
File "/home/julian/Development/pkg/pkg/_cli.py", line 40, in _crawl
fount.flowTo(Listener(crawlerFlow))
File "/home/julian/.local/share/virtualenvs/pkg/site-packages/tubes/protocol.py", line 353, in flowTo
self._aFlowFunction(f, d)
File "/home/julian/.local/share/virtualenvs/pkg/site-packages/tubes/protocol.py", line 411, in aFlowFunction
listening.impl.drain.receive(Flow(fount, drain))
File "/home/julian/.local/share/virtualenvs/pkg/site-packages/tubes/listening.py", line 93, in receive
item.drain))
File "/home/julian/Development/pkg/pkg/_cli.py", line 53, in crawlerFlow
flow.fount.flowTo(crawler).flowTo(flow.drain)
File "/home/julian/.local/share/virtualenvs/pkg/site-packages/tubes/_siphon.py", line 215, in flowTo
result = beginFlowingTo(self, drain)
File "/home/julian/.local/share/virtualenvs/pkg/site-packages/tubes/kit.py", line 104, in beginFlowingTo
return drain.flowingFrom(fount)
File "/home/julian/.local/share/virtualenvs/pkg/site-packages/tubes/protocol.py", line 107, in flowingFrom
beginFlowingFrom(self, fount)
File "/home/julian/.local/share/virtualenvs/pkg/site-packages/tubes/kit.py", line 128, in beginFlowingFrom
fount=fount, drain=drain))
exceptions.TypeError: the output of <Fount for <tubes.framing._CarriageReturnRemover object at 0x0000000001a24e90>>, <InterfaceClass tubes.itube.IFrame>, is not compatible with the required input type of <tubes.protocol._TransportDrain object at 0x0000000001a24ec8>, <InterfaceClass tubes.itube.ISegment>
which is pretty hard to decypher.
Add fountToBytes, bytesToFount functions:
def fountToBytes(fount):
# type: (IFount) -> Deferred[bytes]
def collect(chunks):
# type: (Iterable[bytes]) -> bytes
return b"".join(chunks)
d = fountToDeferred(fount)
d.addCallback(collect)
return d
def bytesToFount(data):
# type: (bytes) -> IFount
…
These are at least useful in testing, so that one can create a fount with some data that can can be fed into a tube, or get out the data pouring out of a tube.
https://warehouse.python.org/project/Tubes/ doesn't link to homepage in either prose or metadata.
The comment in tubes.tube.tube goes into a bit of detail on why it's better than subclassing. That seems great!
But why's it a decorator, instead of just being a function:
tube(
started=lambda self: None,
stopped=lambda self: None,
received=lambda self: None,
inputType=None,
outputType=None,
) -> class implementing ITube
(This would also satisfy #67)
builds should be automated
Greetings,
tldr;
exceptions.AttributeError: _ProtocolPlumbing instance has no attribute 'pauseProducing'
We seem to be exercising a bug in flowFountFromEndpoint with our tubes coroutine pipeline and unit test here:
https://github.com/david415/bananaphone/tree/add-tubes.0
https://github.com/david415/bananaphone/blob/56c871e697c9050e85cf4d9fa8df059deb6f4a0a/bananaphone.py#L901-L934
https://github.com/david415/bananaphone/blob/56c871e697c9050e85cf4d9fa8df059deb6f4a0a/test/test_utils.py#L17-L33
(virtenv-tubes)user@python-dev1:~/bananaphone$ trial test/test_utils.py
test_utils
CoCoTubeDrainTests
test_drain_changeWordSize ... coroutine drain flowing from
coroutine fount flow to
coroutine drain receive
coroutine drain receive
[OK]
TubeEndpointProxyTests
test_tube_proxy ... args
['markov', '/usr/share/dict/words', 'words', 'sha1', 2]
[ERROR]
===============================================================================
[ERROR]
Traceback (most recent call last):
File "/home/user/virtenv-tubes/local/lib/python2.7/site-packages/tubes/protocol.py", line 400, in listening
listening.impl = _FountImpl(portObject, aFlowFunction, preListen)
File "/home/user/virtenv-tubes/local/lib/python2.7/site-packages/tubes/protocol.py", line 331, in __init__
self._pauser = Pauser(portObject.pauseProducing,
exceptions.AttributeError: _ProtocolPlumbing instance has no attribute 'pauseProducing'
test_utils.TubeEndpointProxyTests.test_tube_proxy
-------------------------------------------------------------------------------
Ran 2 tests in 0.035s
FAILED (errors=1, successes=1)
Actually running sketches/fanchat.py produces confusingly broken behavior. I haven't conclusively traced everything to a single source, but I will enumerate some problems related to its dysfunction here:
stopFlowchannels defaultdict can't construct Channel objects because the constructor takes an argument_TransportDrain.flowingFrom breaks when self.fount is not Nonepython fanchat.py and then ^C results in a traceback since _OutFount.stopFlow is not idempotentand probably others
Sometimes when a drain starts experiencing backpressure, it wants its fount to roll up all the messages that can't be delivered into a single message, compressing them. For example, an IRC consumer might want to batch together N messages into a message that says "missed 10 messages due to slow connection".
Failure is a big, messy object and it would be good to use it only if we absolutely need its functionality. We don't. flowStopped needs to know how the flow was ended. If the flow was ended due to an application exception, said exception should be unconditionally logged. Flows may end for a variety of reasons which are not exceptions though. It may be worthwhile to have a specific Reason type, or we may just want to go with an Exception instance.
If you were to make a script of JSON commands for fanchat and use the stdio endpoint to hook them together into a little test, then do something like
$ cat fcscript.txt | python fanchat.pyyou would get very weird non-deterministic behavior where half of the output would vanish.
This is because, in the absence of a protocol providing IHalfCloseableProtocol as the protocol, "EOF" means "shut down both read and write halves of the connection".
Understandably, further writes to the StandardIO would be lost.
To remedy this, IHalfCloseableProtocol should be implemented - which, by the way, would also allow us to separate the fount and drain termination on transports which support such a distinction.
presently tubes depends on Twisted and should declare that
we should run pyflakes over the codebase in an automatic fashion
It should be able to consume N Deferreds at a time and add callbacks to all of them. Perhaps even allow them to come back out of order?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.