twisted / ampoule

Process pool for Twisted, using AMP

Twisted

_ _ _ _

For information on changes in this release, see the NEWS file.

What is this?

Twisted is an event-based framework for internet applications, supporting Python 3.6+. It includes modules for many different purposes, including the following:

• twisted.web: HTTP clients and servers, HTML templating, and a WSGI server
• twisted.conch: SSHv2 and Telnet clients and servers and terminal emulators
• twisted.words: Clients and servers for IRC, XMPP, and other IM protocols
• twisted.mail: IMAPv4, POP3, SMTP clients and servers
• twisted.positioning: Tools for communicating with NMEA-compatible GPS receivers
• twisted.names: DNS client and tools for making your own DNS servers
• twisted.trial: A unit testing framework that integrates well with Twisted-based code.

Twisted supports all major system event loops -- select (all platforms), poll (most POSIX platforms), epoll (Linux), kqueue (FreeBSD, macOS), IOCP (Windows), and various GUI event loops (GTK+2/3, Qt, wxWidgets). Third-party reactors can plug into Twisted, and provide support for additional event loops.

Installing

To install the latest version of Twisted using pip:

$ pip install twisted

Additional instructions for installing this software are in the installation instructions.

Documentation and Support

Twisted's documentation is available from the Twisted Matrix website. This documentation contains how-tos, code examples, and an API reference.

Help is also available on the Twisted mailing list.

There is also an IRC channel, #twisted, on the Libera.Chat network. A web client is available at web.libera.chat.

Unit Tests

Twisted has a comprehensive test suite, which can be run by tox:

$ tox -l                       # to view all test environments
$ tox -e nocov                 # to run all the tests without coverage
$ tox -e withcov               # to run all the tests with coverage
$ tox -e alldeps-withcov-posix # install all dependencies, run tests with coverage on POSIX platform

You can test running the test suite under the different reactors with the TWISTED_REACTOR environment variable:

$ env TWISTED_REACTOR=epoll tox -e alldeps-withcov-posix

Some of these tests may fail if you:

• don't have the dependencies required for a particular subsystem installed,
• have a firewall blocking some ports (or things like Multicast, which Linux NAT has shown itself to do), or
• run them as root.

Static Code Checkers

You can ensure that code complies to Twisted coding standards:

$ tox -e lint   # run pre-commit to check coding stanards
$ tox -e mypy   # run MyPy static type checker to check for type errors

Or, for speed, use pre-commit directly:

$ pipx run pre-commit run

Copyright

All of the code in this distribution is Copyright (c) 2001-2024 Twisted Matrix Laboratories.

Twisted is made available under the MIT license. The included LICENSE file describes this in detail.

Warranty

THIS SOFTWARE IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE USE OF THIS SOFTWARE IS WITH YOU.

IN NO EVENT WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE LIBRARY, BE LIABLE TO YOU FOR ANY DAMAGES, EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Again, see the included LICENSE file for specific legal details.

ampoule's Issues

Ampoule logging is exceedingly verbose

Ampoule copies each child's output into the parent's log output. Since the child is also using Twisted logging, this results in log messages with lots of duplicate information. For example,

2010-04-03 15:11:04-0400 [-] FROM 0: 2010-04-03 15:11:04-0400 [HTMLOnlyPageGetter,client] Found links [...]

This would be better if the redundancy were elided. For example, formatting the above event like this instead would a bit of an improvement:

2010-04-03 15:11:04-0400 [Ampoule 0,HTMLOnlyPageGetter,client] Found links [...]

This preserves all of the information, but presents it better.

One approach which could work to implement this would be to have the child processes use a different logger. Rather than writing to stdout, it could make AMP calls back onto the parent with the log information.

Meanwhile, the parent would implement a receiver for these logging commands which re-published them to the log observer in the parent. The system keyword argument to log.msg can be used to make sure the information about which child the event is from is preserved.

ampoule test suite is unreliable, has intermittent failures

These tests fail for me when I run the ampoule test suite, sometimes:

test_BootstrapContext
test_BootstrapContextInstance
test_deferToAMPProcess
test_recyclingWithQueueOverload

ProcessPool spawns more and more processes

It seems that under certain circumstances, if a worker process exits abnormally, ProcessPool will spawn two new processes to replace it. This ignores the maximum worker limit the ProcessPool is configured with, so eventually enough processes will be spawned so as to consume all available system resources.

I don't completely understand why this happens, but I know that the "second" replacement process is spawned by the startAWorker call attached to the stopAWorker Deferred in _cb_doWork. This may be because the Shutdown call issued by stopAWorker fails with ProcessTerminated. This results in the Deferred succeeding instead of failing, allowing the startAWorker callback to run. Additionally, when the process exits, it gets restarted by the logic in _addProcess. Just a guess, though.

(In general I think all this code should (and could be) be a lot simpler.)

don't attempt to "correct" PYTHONPATH configuration for subprocesses unless asked

When Ampoule was first written, it was common practice for applications to do weird, potentially non-replicable startup nonsense to get sys.path to be correct and include all of one's sources. Since virtualenv, venv, and pip have taken over the Python developer-setup space, it's much more likely that your Python's path and dependency configuration is correct if you just don't mess with it.

Setting PYTHONPATH to point at the standard dist dirs can also have weird and unintended side-effects. For example, any shadowing of stdlib modules (enum34, for example) might create inscrutable breakages when .pth files try to import things (like coverage).

#24 is another symptom of this monkeying around with sys.path.

Let's leave the environment untouched, and provide a hook to tweak it in the now-unusual case where something has broken.

PYTHONPATH is cleared by default rather than preserved

The code in question: https://github.com/twisted/ampoule/blob/master/ampoule/main.py#L268

This makes some modules impossible to find

Ampoule serializes job execution

Each process in an Ampoule process pool will only be given one job at a time to work on.

All of the pieces are in place to support concurrent async job execution within each child, except that the ProcessPool class intentionally serializes job distribution. There should at least be an option to allow more than one job to be sent to a worker process at once.

ProcessPool.callRemote Deferred sometimes fires with None instead of result

This is a weird behavior I ran into when issuing a new job from an errback on an earlier job. The attached illustrates better than I can do in English. The expected result is to see a result of {}, though, since that is what the "Fast" command in the worker returns. Instead, the result is None.

ampoule trunk won't work with twisted trunk on win32

Ampoule trunk won't work with Twisted trunk on win32, because twisted.internet.stdio.StandardIO (being _win32stdio really) won't accept multiple parameters for the init function.

This should be fixed in Twisted IMO, but in the meantime -- the patch is included.

---

Imported from Launchpad using lp2gh.

• date created: 2009-07-19T18:55:42Z
• owner: dotz
• assignee: dialtone
• the launchpad url was https://bugs.launchpad.net/bugs/401460

Too hard to associate Ampoule worker IDs with real processes

Ampoule assigns its own IDs to worker processes. This isn't necessarily bad, but the lack of PID information in any of the logging related to workers means that it's very challenging to associate an actual process with an Ampoule worker. This kind of association is very useful for debugging. For example, a process might begin to use lots of CPU or RAM. Being able to tie this to a particular Ampoule worker process can be helpful in determining which job that process is executing.

sets module deprecated

On recent versions of Python, whenever a worker process starts, it spits out this warning:

.../ampoule/main.py:4: exceptions.DeprecationWarning: the sets module is deprecated

Bootstrap script should be a separate file for compatability with other executables

https://github.com/twisted/ampoule/blob/master/ampoule/main.py#L107

The bootsrap script is a string that gets passed as a command to sys.executable with the -c option which presumes that this is being run by python.

https://github.com/twisted/ampoule/blob/master/ampoule/main.py#L282

As a script file is more generic and likely to be accepted as the main argument to whatever executable is running, that should be used instead.

ampoule twistd plugin uses child class for parent

bzr diff twisted/plugins/ampoule_plugin.py
=== modified file 'twisted/plugins/ampoule_plugin.py'
--- twisted/plugins/ampoule_plugin.py	2008-11-15 23:27:23 +0000
+++ twisted/plugins/ampoule_plugin.py	2010-03-23 21:56:20 +0000
@@ -41,7 +41,7 @@
             """
             self['child'] = reflect.namedAny(self['child'])
             if self['parent'] is not None:
-                self['parent'] = reflect.namedAny(self['child'])
+                self['parent'] = reflect.namedAny(self['parent'])
             if self['name']:
                 self['name'] = self['name'].decode('utf-8')

Add non-basic usage example.

The current basic examples are fine... but it uses a single python mode for both the main process and the AMP workers.

When I started to see how to use ampule I was a bit confused about what is executed in the main reactor and what is executed in the child process.

I have created this example in which there are 2 python modules.
It helped me as an example of what is executed in the child.
It uses the setproctitle module so that at least in Linux, when you run ps you don't see the whole bootstrap code.

https://gist.github.com/adiroiban/d2bb96a52634d7aa07493ac3df2ecfb0

Let me know if you think that this example makes sense and it it worth being included in the repo.

---

For my work, I ended up using Python stdlib multiprocess.Pool

ampoule doesn't handle childConnectionLost

/opt/001-python/lib/python3.8/site-packages/ampoule/pool.py:397: in stop
    l = [self.stopAWorker(process) for process in self.processes]
/opt/001-python/lib/python3.8/site-packages/ampoule/pool.py:397: in <listcomp>
    l = [self.stopAWorker(process) for process in self.processes]
/opt/001-python/lib/python3.8/site-packages/ampoule/pool.py:351: in stopAWorker
    child.callRemote(commands.Shutdown
/opt/001-python/lib/python3.8/site-packages/twisted/protocols/amp.py:969: in callRemote
    return co._doCommand(self)
/opt/001-python/lib/python3.8/site-packages/twisted/protocols/amp.py:1998: in _doCommand
    d = proto._sendBoxCommand(self.commandName,
/opt/001-python/lib/python3.8/site-packages/twisted/protocols/amp.py:900: in _sendBoxCommand
    box._sendTo(self.boxSender)
/opt/001-python/lib/python3.8/site-packages/twisted/protocols/amp.py:723: in _sendTo
    proto.sendBox(self)
/opt/001-python/lib/python3.8/site-packages/twisted/protocols/amp.py:2384: in sendBox
    self.transport.write(box.serialize())
/opt/001-python/lib/python3.8/site-packages/ampoule/main.py:82: in write
    self.transport.writeToChild(TO_CHILD, data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <Process pid=1030 status=-1>, childFD = 3
data = b'\x00\x04_ask\x00\x012\x00\x08_command\x00\x08Shutdown\x00\x00'
    def writeToChild(self, childFD, data):
>       self.pipes[childFD].write(data)
E       KeyError: 3

Ampoule doesn't work < python2.5

After a lot of mucking around making setuptools work (because Ampoule wanted it) on an old FC4 box I finally was able to proceed with the install. Much to my surprise this included downloading and installing Twisted, the same version already installed on the system - though I imagine this duplication is the fault of setuptools itself.

During this install I saw [1]. Anyhow the install proceeded and exited cleanly, so I tried ampoule and was greeted by [2]. I guess yield has changed or wasn't supported before python2.5 - now that I think about it this is probably a bigger problem if it affects Twisted, but it nevertheless seems to break Ampoule

[1]
Downloading http://tmrc.mit.edu/mirror/twisted/Twisted/8.2/Twisted-8.2.0.tar.bz2#md5=c85f151999df3ecf04c49a781b4438d2
Processing Twisted-8.2.0.tar.bz2
Running Twisted-8.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install--UxITk/Twisted-8.2.0/egg-dist-tmp-A9O7-p
twisted/protocols/_c_urlarg.c: In function ‘unquote’:
twisted/protocols/_c_urlarg.c:65: warning: pointer targets in passing argument 2 of ‘PycStringIO->cwrite’ differ in signedness
twisted/protocols/_c_urlarg.c:75: warning: pointer targets in passing argument 2 of ‘PycStringIO->cwrite’ differ in signedness
twisted/protocols/_c_urlarg.c:83: warning: pointer targets in passing argument 2 of ‘PycStringIO->cwrite’ differ in signedness
twisted/protocols/_c_urlarg.c:85: warning: pointer targets in passing argument 2 of ‘PycStringIO->cwrite’ differ in signedness
twisted/protocols/_c_urlarg.c:93: warning: pointer targets in passing argument 2 of ‘PycStringIO->cwrite’ differ in signedness
twisted/protocols/_c_urlarg.c:96: warning: pointer targets in passing argument 2 of ‘PycStringIO->cwrite’ differ in signedness
twisted/protocols/_c_urlarg.c:97: warning: pointer targets in passing argument 2 of ‘PycStringIO->cwrite’ differ in signedness
File "build/bdist.linux-i686/egg/twisted/test/generator_failure_tests.py", line 66
yield
^
SyntaxError: invalid syntax
File "/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-linux-i686.egg/twisted/test/generator_failure_tests.py", line 66
yield
^
SyntaxError: invalid syntax
File "/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-linux-i686.egg/twisted/test/generator_failure_tests.py", line 66
yield
^
SyntaxError: invalid syntax
Adding Twisted 8.2.0 to easy-install.pth file
Installing pyhtmlizer script to /usr/bin
Installing cftp script to /usr/bin
Installing ckeygen script to /usr/bin
Installing manhole script to /usr/bin
Installing tap2deb script to /usr/bin

[2]
[blair@nimrod1 ampoule-0.1]$ python examples/pid.py
File "examples/pid.py", line 26
result = yield pp.doWork(Pid)
^
SyntaxError: invalid syntax

ProcessPool should take a reactor parameter

It's kind of a pain to test when it uses the global reactor and doesn't clean up after itself.

Add a tox to run unit tests on reasonable versions of Python (2.7, 3.5, 3.6)

switch from travis to github actions

Travis hasn't worked for a long time.

test_changeChildReactor fails without installed ampoule

test_changeChildReactor does not pass packages=("twisted", "ampoule") to its ProcessStarter, which means the test fails if there is no system-wide ampoule for the child to pick up. This seems like a simple oversight: all other ProcessStarter calls do pass it, and doing the same thing here makes the test pass on my system.

setup.py imports ampoule (breaks if Twisted is missing at setup time)

In the setup.py file for Ampoule, line 19 imports ampoule. The only place it is used is for version information. This import breaks the install if Twisted is not available on the system path at the time. Would it be possible to explicitly enter the version in setup.py or read it from a file instead of importing the module?

KeyError on worker shutdown sometimes

This exception pops up somewhat often:

2010-05-12 11:31:23-0400 [-] Unhandled error in Deferred:
2010-05-12 11:31:23-0400 [-] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ses-181-py2.6.egg/ses/email_client.py", line 342, in scrapeURL
d = self._pool.callRemote(ScrapeURL, url=url)
File "/usr/lib/python2.6/site-packages/ampoule-0.2.0-py2.6.egg/ampoule/pool.py", line 286, in callRemote
return self.doWork(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/ampoule-0.2.0-py2.6.egg/ampoule/pool.py", line 299, in doWork
self._cb_doWork(command, _d=d, **kwargs)
File "/usr/lib/python2.6/site-packages/ampoule-0.2.0-py2.6.egg/ampoule/pool.py", line 278, in _cb_doWork
return defer.maybeDeferred(child.callRemote, command, **kwargs
--- ---
File "/usr/lib/python2.6/site-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py", line 117, in maybeDeferred
result = f(*args, **kw)
File "/usr/lib/python2.6/site-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/protocols/amp.py", line 802, in callRemote
return co._doCommand(self)
File "/usr/lib/python2.6/site-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/protocols/amp.py", line 1686, in _doCommand
self.requiresAnswer)
File "/usr/lib/python2.6/site-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/protocols/amp.py", line 733, in _sendBoxCommand
box._sendTo(self.boxSender)
File "/usr/lib/python2.6/site-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/protocols/amp.py", line 558, in _sendTo
proto.sendBox(self)
File "/usr/lib/python2.6/site-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/protocols/amp.py", line 2007, in sendBox
self.transport.write(box.serialize())
File "/usr/lib/python2.6/site-packages/ampoule-0.2.0-py2.6.egg/ampoule/main.py", line 76, in write
self.transport.writeToChild(TO_CHILD, data)
File "/usr/lib/python2.6/site-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/internet/process.py", line 684, in writeToChild
self.pipes[childFD].write(data)
exceptions.KeyError: 3

It may only happen when the worker process exits abnormally, ie after being SIGKILL'd or encountering a platform enforced resource limit.

Add timeout for subprocess calls

Sometimes calls take too much time and they get stuck, there should be a timeout mechanism to avoid reaching a stuck condition in the pool.

ampoule.pool.ProcessPool misdocuments its "starter" attribute

starter is documented as "A process starter that implements L{iampoule.IStarter}.". However, what's actually expected is an object which "provides" IStarter, not "implements". In other words, the docs are asking for a class like ampoule.main.ProcessStarter, but they mean to be asking for an instance of such a class.

ampoule should provide a facility to work with frozen or py2exe'd files

One of the typical problems with process pools (as opposed to thread pools) is that they're sensitive to changes in the environment which make it impossible to start up a process.

One way that this can be mitigated is to provide an explicit integration mechanism for weird deployment environments, where tools like py2exe might be in use.

This may require some explicit cooperation from the deployment mechanism (a command-line flag saying "this should be a worker process", for example) but it should be explicitly accounted for.

Twisted warning after ampoule installation

Since i installed ampoule i receive a warning when importing twisted.

This is always reproducible by issuing:
$ python

import twisted
import pkg_resources
/usr/bin/ipython:1: UserWarning: Module twisted was already imported from /usr/lib64/python2.7/site-packages/twisted/init.pyc, but /usr/lib/python2.7/site-packages/ampoule-0.2.1-py2.7.egg is being added to sys.path
#!/usr/bin/python

if i erase ampoule the warning disappears.

I think this is due to ampoule uncorrectly stating that he owns the twisted package:
$ cat /usr/lib/python2.7/site-packages/ampoule-0.2.1-py2.7.egg/EGG-INFO/top_level.txt
ampoule
twisted

It also may be the same as this (see msg230):
http://bugs.python.org/setuptools/issue36

I'm using the latest version from trunk, on fedora 14 with python-setuptools 0.6.14 (the Distribute fork) an python 2.7.
Ampoule is installed using "python setup.py install".

A solution (which if i'm not wrong is what they're doing with nevow) could be to declare the twisted ampoule plugin as package data in setup.py.

modernize logging

Ampoule presently makes heavy use of (old, one-day-hopefully deprecated) log.msg, and worse yet, frequently does string interpolation on log messages (see here for example).

Let's clean this up to use twisted.logger.

Ampoule 0.1 does not work on Windows XP

Hello,

I am trying to run Ampoule on windows. On my windows XP (SP3) machine I got the following error when trying to run the examples:

    Traceback (most recent call last):
      File "c:\python25\lib\site-packages\twisted\internet\defer.py", line 7

37, in _inlineCallbacks
result = g.send(result)
File "C:\Documents and Settings\dia\Desktop\ampoule-0.1\examples\pid.p
y", line 25, in _run
yield pp.start()
File "c:\python25\lib\site-packages\ampoule-0.1-py2.5.egg\ampoule\pool
.py", line 97, in start
return self.adjustPoolSize()
File "c:\python25\lib\site-packages\twisted\internet\defer.py", line 8
22, in unwindGenerator
return _inlineCallbacks(None, f(*args, **kwargs), Deferred())
--- ---
File "c:\python25\lib\site-packages\twisted\internet\defer.py", line 7
37, in _inlineCallbacks
result = g.send(result)
File "c:\python25\lib\site-packages\ampoule-0.1-py2.5.egg\ampoule\pool.py", line 358, in adjustPoolSize
self.startAWorker()
File "c:\python25\lib\site-packages\ampoule-0.1-py2.5.egg\ampoule\pool.py", line 192, in startAWorker
ampParent=self.ampParent)
File "c:\python25\lib\site-packages\ampoule-0.1-py2.5.egg\ampoule\main.py", line 212, in startAMPProcess
return self.startPythonProcess(prot, self.childReactor, fullPath)
File "c:\python25\lib\site-packages\ampoule-0.1-py2.5.egg\ampoule\main.py", line 228, in startPythonProcess
usePTY=self.usePTY, packages=self.packages)
File "c:\python25\lib\site-packages\ampoule-0.1-py2.5.egg\ampoule\main.py", line 260, in spawnProcess
childFDs={0:"w", 1:"r", 2:"r", 3:"w", 4:"r"})
File "c:\python25\lib\site-packages\twisted\internet\posixbase.py", line 231, in spawnProcess
raise ValueError("Customizing childFDs is not supported on Windows.")
exceptions.ValueError: Customizing childFDs is not supported on Windows.

Any help is appreciated.