mjcaley / aiospamc

Python asyncio-based library that implements the SPAMC/SPAMD client protocol used by SpamAssassin.

aiospamc

aiospamc is a client for SpamAssassin that you can use as a library or command line tool.

The implementation is based on asyncio; so you can use it in your applications for asynchronous calls.

The command line interface provides user-friendly access to SpamAssassin server commands and provides both JSON and user-consumable outputs.

Documentation

Detailed documentation can be found at: https://aiospamc.readthedocs.io/

Requirements

• Python 3.9 or higher
• certifi for updated certificate authorities
• loguru for structured logging
• typer for the command line interface

Examples

Command-Line Tool

aiospamc is your interface to SpamAssassin through CLI. To submit a message for a score, use:

# Take the output of gtube.msg and have SpamAssasin return a score
$ cat ./gtube.msg | aiospamc check
1000.0/5.0

# Ping the server
$ aiospamc ping
PONG

Library

import asyncio
import aiospamc


GTUBE = """Subject: Test spam mail (GTUBE)
Message-ID: <[email protected]>
Date: Wed, 23 Jul 2003 23:30:00 +0200
From: Sender <[email protected]>
To: Recipient <[email protected]>
Precedence: junk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

This is the GTUBE, the
    Generic
    Test for
    Unsolicited
    Bulk
    Email

If your spam filter supports it, the GTUBE provides a test by which you
can verify that the filter is installed correctly and is detecting incoming
spam. You can send yourself a test mail containing the following string of
characters (in upper case and with no white spaces and line breaks):

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

You should send this test mail from an account outside of your network.
""".encode("ascii")


# Ping the SpamAssassin server
async def is_alive():
    pong = await aiospamc.ping()
    return True if pong.status_code == 0 else False

asyncio.run(is_alive())
# True


# Get the spam score of a message
async def get_score(message):
    response = await aiospamc.check(message)
    return response.headers.spam.score, response.headers.spam.threshold

asyncio.run(get_score(GTUBE))
# (1000.0, 5.0)


# List the modified headers
async def list_headers(message):
    response = await aiospamc.headers(message)
    for line in response.body.splitlines():
        print(line.decode())

asyncio.run(list_headers(GTUBE))
# Received: from localhost by DESKTOP.
#         with SpamAssassin (version 4.0.0);
#         Wed, 30 Aug 2023 20:11:34 -0400
# From: Sender <[email protected]>
# To: Recipient <[email protected]>
# Subject: Test spam mail (GTUBE)
# Date: Wed, 23 Jul 2003 23:30:00 +0200
# Message-Id: <[email protected]>
# X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on DESKTOP.
# X-Spam-Flag: YES
# X-Spam-Level: **************************************************
# X-Spam-Status: Yes, score=1000.0 required=5.0 tests=GTUBE,NO_RECEIVED,
#         NO_RELAYS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no
#         version=4.0.0
# MIME-Version: 1.0
# Content-Type: multipart/mixed; boundary="----------=_64EFDAB6.3640FAEF"

aiospamc's Issues

Suppress exceptions

Consider adding a parameter for the frontend functions to suppress throwing exceptions.

Travis CI Python 3.7 build fails due to setuptools_scm installing as 0.0.0

pytest-runner is unable to install since it identifies 0.0.0 as being below the minimum required version.

Issue can be found at: pytest-dev/pytest-runner#23

Update and add examples

Update the examples to use the new frontend API.
Add other examples for using the Client class.

Remove beta category

Mark project as stable.

Simplify certificate generation

Right now there's code in the integration tests to generate a certificate. See if the https://github.com/python-trio/trustme library can be used to replace this code.

Remove loop parameter

The loop parameter for asyncio.open_connection and asyncio.open_unix_connection is optional. It can be removed as a parameter from the entire library.

Use docstring coverage tool

Verify docstring coverage within the CI pipeline in order to ensure the API documentation quality doesn't drop.

A tool to consider is https://github.com/econchick/interrogate

Add CLI interface

Mimic the spamc client to provide a CLI.

Evaluate new logging library

• Evaluate logging libraries that are available with their features
• Is the extra dependency worth it

Currently considering both structlog and loguru.

Improve parser

I'd like to explore improving the parser so it's not creating the end objects and instead returning dictionaries only.

The idea is headers will be able to use parts of the parser without worrying about circular imports. Then it can end up that the user can supply headers as strings instead of importing and instantiating objects. It's kind of mimicking requests headers parameter.

As an example:
aiospamc.request(verb="CHECK", headers={"Spam": "True; 4.0/2.0")

One thing to keep in mind is that the parser only works on bytes right now. That may need to be taken into account.

Git tracks "changes" directory

The "changes" directory needs to be tracked by git when it's empty.

Add Python 3.10 to pipelines

Add Python 3.10 to CI pipelines.
Make Python 3.10 the default for release pipeline.

Response.__eq__: lenient type annotation

I had some sloppy code which basically tried

if aiospamc.response() != '':

This no longer works, and throws a fugly traceback:

 File "/var/task/aiospamc/responses.py", line 104, in __eq__
   self.version == other.version

I notice that aiospamc 0.7.0 introduced an __eq__ method for the Response class, but this method has a very lenient type annotation:

    def __eq__(self, other: Any) -> bool:

Shouldn't we require other to also be a Response, since it requires the object to have the same attributes? (The code below the statement from the traceback proceeds to check that other has attributes other.headers, other.status_code, other.message, and other.body which are also equal to the same attributes of self.)

Consider replacing towncrier with reno

Display message in response __repr__

Display the message in the repr method to show the response type.

Release stage moved into azure-pipelines.yml

Right now the release to PyPI is in Azure DevOps release section. I'd like to move that into the YAML file so it lives with the code.

Unicode characters may cause an exception

If the data to be checked contains unciode characters, it will result in Content-Length mismatch.
I will give examples in Chinese.

Here is the test code：

import asyncio
from aiospamc import *


GTUBE = '''
From: wevsty <[email protected]>
Subject: =?UTF-8?B?5Lit5paH5rWL6K+V?=
Message-ID: <[email protected]>
Date: Tue, 15 Jan 2019 15:26:51 +0800
X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0;
 attachmentreminder=0; deliveryformat=4
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.3.3
MIME-Version: 1.0
Content-Type: text/html; charset=gbk
Content-Transfer-Encoding: 8bit

这是Unicode文字.
This is Unicode characters.
'''

loop = asyncio.get_event_loop()
client = Client(host='127.0.0.1', port=783)
responses = loop.run_until_complete(client.ping())
responses = loop.run_until_complete(client.process(GTUBE))
print(responses)

Run the this code, SPAMD will return the following results

SPAMD/1.0 76 Bad header line: (Content-Length mismatch: Expected 484 bytes, got 492 bytes)

aiospamc will throw an exception

Exception for request (1422449975648)when composing response: <aiospamc.responses.Response object at 0x0000014B30A02FD0>
Traceback (most recent call last):
  File "D:\python_project\test\venv\lib\site-packages\aiospamc\client.py", line 247, in send
    self._raise_response_exception(response)
  File "D:\python_project\test\venv\lib\site-packages\aiospamc\client.py", line 169, in _raise_response_exception
    raise ProtocolException(response)
aiospamc.exceptions.ProtocolException: <aiospamc.responses.Response object at 0x0000014B30A02FD0>
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm_Community\helpers\pydev\pydevd.py", line 1741, in <module>
    main()
  File "C:\Program Files\JetBrains\PyCharm_Community\helpers\pydev\pydevd.py", line 1735, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files\JetBrains\PyCharm_Community\helpers\pydev\pydevd.py", line 1135, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm_Community\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "D:/python_project/test/test_aiospamc.py", line 26, in <module>
    responses = loop.run_until_complete(client.process(GTUBE))
  File "C:\Python\Python3\lib\asyncio\base_events.py", line 568, in run_until_complete
    return future.result()
  File "D:\python_project\test\venv\lib\site-packages\aiospamc\client.py", line 513, in process
    response = await self.send(request)
  File "D:\python_project\test\venv\lib\site-packages\aiospamc\client.py", line 30, in wrapper
    return await func(cls, request)
  File "D:\python_project\test\venv\lib\site-packages\aiospamc\client.py", line 47, in wrapper
    return await func(cls, request)
  File "D:\python_project\test\venv\lib\site-packages\aiospamc\client.py", line 247, in send
    self._raise_response_exception(response)
  File "D:\python_project\test\venv\lib\site-packages\aiospamc\client.py", line 169, in _raise_response_exception
    raise ProtocolException(response)
aiospamc.exceptions.ProtocolException: <aiospamc.responses.Response object at 0x0000014B30A02FD0>

GitHub release doesn't track enhancement

Issues and PRs tagged with "enhacement" aren't being added to the GitHub release notes in their own section.

SSL no longer working

The SSL flag is set in the client, but is never passed to a connection.

Exception when parsing timeout message

Spamd sends a timeout message that the parser fails to interpret.

The response is: b'SPAMD/1.0 79 Timeout: (30 second timeout while trying to CHECK)\r\n'

Exception raised is:

Error parsing response
Traceback (most recent call last):
  File "/home/mjcaley/aiospamc/aiospamc/incremental_parser.py", line 285, in parse_response_status
    protocol_version, status_code, message = list(
ValueError: too many values to unpack (expected 3)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mjcaley/aiospamc/aiospamc/frontend.py", line 83, in request
    parsed_response = parser.parse(response)
  File "/home/mjcaley/aiospamc/aiospamc/incremental_parser.py", line 95, in parse
    self.status()
  File "/home/mjcaley/aiospamc/aiospamc/incremental_parser.py", line 116, in status
    self.result = {**self.result, **self.status_parser(status_line)}
  File "/home/mjcaley/aiospamc/aiospamc/incremental_parser.py", line 290, in parse_response_status
    raise ParseError(
aiospamc.exceptions.ParseError: Could not parse response status line, not in recognizable format
Traceback (most recent call last):
  File "/home/mjcaley/aiospamc/aiospamc/incremental_parser.py", line 285, in parse_response_status
    protocol_version, status_code, message = list(
ValueError: too many values to unpack (expected 3)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mjcaley/aiospamc/aiospamc/frontend.py", line 83, in request
    parsed_response = parser.parse(response)
  File "/home/mjcaley/aiospamc/aiospamc/incremental_parser.py", line 95, in parse
    self.status()
  File "/home/mjcaley/aiospamc/aiospamc/incremental_parser.py", line 116, in status
    self.result = {**self.result, **self.status_parser(status_line)}
  File "/home/mjcaley/aiospamc/aiospamc/incremental_parser.py", line 290, in parse_response_status
    raise ParseError(
aiospamc.exceptions.ParseError: Could not parse response status line, not in recognizable format

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mjcaley/aiospamc/ssl_debug.py", line 5, in <module>
    print(repr(asyncio.run(aiospamc.check(b"abc", port=11783, verify=False))))
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
    return future.result()
  File "/home/mjcaley/aiospamc/aiospamc/frontend.py", line 210, in check
    return await request(req, connection, parser)
  File "/home/mjcaley/aiospamc/aiospamc/frontend.py", line 90, in request
    raise BadResponse(response) from error
aiospamc.exceptions.BadResponse: b'SPAMD/1.0 79 Timeout: (30 second timeout while trying to CHECK)\r\n'

Update Poetry build system reference

Currently we reference Poetry directly, but this has been updated to use poetey-core.

[build-system]
requires = ["poetry_core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Fix case of "localhost" literal in frontend functions

Host parameters were written with an uppercase character (e.g. Localhost). Change it so that it's lowercase.

Build and Coverage shields pointing to wrong URL

The shields for build and coverage of the README weren't updated since updating the pipelines.

Add header value to support failures in decoding

Header values that can't be decoded into UTF8 will throw an exception.
Add a header value that will give the user a bytes object so at least there's no loss of data through an exception.

Add type annotations

Type annotations would be a good addition to the library.

Connection hangs with SSL and compression

Connecting to a spamd instance that's SSL and sending a body that's compressed will hang the connection.

Spam value header can contain "Yes" or "No"

Instead of just "True" or "False" the Spam header value can contain "Yes" or "No" values. The parser will need to handle these cases.

It might be good to use case insensitive matching too.

Integration tests

See if SpamAssassin can be installed in the build pipeline in order to have pytest run integration tests against a live server.

The tests wouldn't have to be required to be run on a developers machine (just unit tests), but it can be a gate for merging into the development branch.

Add warning for known bug

Add a warning when combining SSL and compression options due to known bug: https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7183

Update async tests

pytest-asyncio 0.17 removes the requirement to mark tests with pytest.mark.asyncio.

Add methods to convert objects to dictionaries

Add support for user objects (request, response, headers) to convert to dictionaries.

Will help support converting to JSON later on. Use basic types as much as possible. Leave the body as a bytes object, we'll let the user decide how to convert it to a string.

Add timeout options

Add timeout options for the client. Connection, response, and total timeout values should be allowed (like aiohttp).

Example fails with ValueError: loop argument must agree with Future

I attempted to exercise the example code but I got an error:

Clean slate (MacOS High Sierra, Homebrew Python 3.5.1):

bash$ python3 -m venv aiotest
... stuff ...

bash$ . ./aiotest/bin/activate

(aiotest) bash$ pip install aiospamc
Collecting aiospamc
  Using cached https://files.pythonhosted.org/packages/38/4f/2f320f2579e42c0bc178b747a8bc2565ead7c3bf8a782ba02dd9cb4a098c/aiospamc-0.4.1-py3-none-any.whl
Installing collected packages: aiospamc
Successfully installed aiospamc-0.4.1

(aiotest) bash$ python
Python 3.5.1 (default, Dec 26 2015, 18:08:53)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> from aiospamc import *
>>> GTUBE = '''Subject: Test spam mail (GTUBE)
... Message-ID: <[email protected]>
........ (etc etc) .......
... XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
...
... You should send this test mail from an account outside of your network.
... '''
>>> loop = asyncio.new_event_loop()
>>> client = Client(host='localhost')
>>> responses = loop.run_until_complete(asyncio.gather(
...     client.ping(),
...     client.check(GTUBE),
...     client.headers(GTUBE)
... ))
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/base_events.py", line 317, in run_until_complete
    future = tasks.ensure_future(future, loop=self)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/tasks.py", line 534, in ensure_future
    raise ValueError('loop argument must agree with Future')
ValueError: loop argument must agree with Future

Based on https://stackoverflow.com/questions/46806174/python-3-6-and-valueerror-loop-argument-must-agree-with-future I was able to fix this by adding

asyncio.set_event_loop(loop)

before attempting to use the client.

Add support for Python 3.9

Python 3.9 is scheduled to be released in October 2020.

Make sure that the package's classifiers lists Python 3.9 and any CI tests against it.

We might need to wait for Microsoft's agent to support it. (https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml)

dependabot troubleshooting

@dependabot ignore

Add Python 3.11 support

Add support for Python 3.11.

Documentation can link to GitHub issues

Allow GitHub issues be linked from documentation

Deprecate support for Python 3.6

Deprecate support for Python 3.6. It is end-of-life in December 2021.

Update project definition to support 3.10

README updated to reflect supported Python versions
Poetry configuration updated to support Python 3.10 and remove support for 3.6

Update dependencies

Dependencies updated to latest versions that support Python 3.7 and later.

Remove warning for warn_spamd_bug_7938

Not a SpamAssassin bug. The body should be formatted as a standard email message which contains a newline after every line of the body.

Use dataclasses for headers and options

Add nicer __str__ methods for objects

Add str methods added to all classes to give a nicer summary of the contents of the object. Especially SpamcHeaders, Request and Response.

Support other async libraries

Add support for other asynchronous libraries like Trio and Curio.

The library sniffio can help with detection.
The library anyio abstracts Trio and Asyncio.

Update library to use Python 3.6 features

Since dropping support for Python 3.5 we can use newer features made available in 3.6.

• f-strings
• Enum's auto members
• Variable type annotations

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

requirements.txt is UTF-16

This threw me off, but actually seems to work. But is it on purpose? Other than that, the file is pure ASCII; and where other files in the tree are not ASCII, they use UTF-8 (namely, docs/protocol.rst).

Change name of 'master' branch to 'main'

Pipelines will need to be updated to accommodate the change.

Exception when importing ActionOption on Python 3.11

Python 3.11 made a change to data class fields that requires defaults be hashable, causing an exception.

ValueError: mutable default <class 'aiospamc.options.ActionOption'> for field action is not allowed: use default_factory

Messages not ending with a newline might cause the connection to hang

If a message doesn't end with a newline the connection could hang waiting for a response from spamd. Change the connection implementation to send an EOF to close the connection.