aht / stream.py Goto Github PK

If you are processing a stream that has a chance of failure, each workers in your ThreadPool/ProcessPool will eventually hit a failure case and die.

Once all of the workers have died, processing stops.

For example, your "Retrieving web pages concurrently" only works because your ThreadPool has more workers than failures (4 workers vs 2 failures). If you reduce the number of workers to 1, it only processes the first two URLs before dying.

The documentation states:

PCollectors can collect from ForkedFeeder‘s or ProcessPool‘s (via system pipes)
...
class PCollector([waittime=0.1]): Collect items from many ForkedFeeder‘s or ProcessPool‘s.

However, when you try to pipe the output of a ProcessPool into a PCollector, it fails:

AttributeError: 'ProcessPool' object has no attribute 'outpipe'

As the error suggests, ForkedFeeder has an attribute 'outpipe', but ProcessPool does not.

I'm not sure how I could be calling it incorrectly, but I'd be glad to hear I was.