progval / irctest Goto Github PK

It would be nice to have a way to write tests that work for both peers.
The problem is tests would have to be both loose (what they accept) and exact (what they send) for each event.

This is needed to test SASL with servers.

[Just dumping some thoughts here]

In the last week I added CI to the repo + made it so GH Actions exports the pytest results as junit XML.

So far this isn't very useful, because the only UI we have is the shitty bot comments on PRs, and a list of successful/failed builds, without per-test details.

So I'd like to make a dashboard (probably just a cron generating XHTML files) that reads these junit XML files, and produces large support tables, highlighting what tests pass for each ircd, and what tests are automatically skipped (with the reason) based on these XML files, and what tests are manually excluded from the collection (also with the reason) somehow (parsing comments in the makefile?)

Ideally this should be readable to people unfamiliar with irctest (eg. by including each test's docstring) so they can use it to see exactly what is implemented by each IRCd.

ircv3/ircv3-specifications#439 specifies the bot mode; Oragono's master (0db7f886374ce616 and later) implements the spec as written, including the new draft/bot tag. It would be neat to get some tests for this (they can be fully ircd-agnostic, since the value of the bot mode is published in 005).

I accidentally switched the batch type (chathistory) and batch parameter (canonical name of the message target) in Ergo. This is the fix for reference: ergochat/ergo#2070

This is a MUST in the spec, so we should have a test for it:

If the batch capability was negotiated, the server MUST reply to a successful CHATHISTORY command using a batch. For subcommands that return message history (i.e. all subcommands other than TARGETS), the batch MUST have type chathistory and take a single additional parameter, the canonical name of the target being queried.

I broke one of my ZNC tests in development and:

        self.sendLine("viewer", "PRIVMSG *playback :play * %d" % (early_time,))
        messages = extract_playback_privmsgs(self.getMessages("viewer"))
>       self.assertEqual(set(messages), set([dm] + echo_messages))
E       AssertionError: None

irctest/server_tests/znc_playback.py:95: AssertionError

It looks like there's no sane default for msg here?

Spec: ircv3/ircv3-specifications#501

WIP tests in this branch: https://github.com/progval/irctest/compare/metadata2?expand=1 (no PR because it would be noisy)

• SUB before connect
• UNSUB
• SYNC
• rate-limiting
• quote a spec paragraph in every docstring
• test Unreal module on Github CI (pull https://github.com/progval/unrealircd-contrib/blob/metadata2/files/metadata2.c )

https://github.com/ProgVal/irctest/blob/4ded96fbbacc9b9085a8dbe58e6469dacbf49cbd/irctest/irc_utils/junkdrawer.py#L11-L15

This does not work with Python 3.6, and does work with 3.8 (I'm not sure about 3.7).

There may be other, similar issues.

Since IRC ECDSA SASL isn't a standard, I don't think it should be seen as a failure for clients to not support it in all cases.

I've got a (dodgy) attempt at a fix here, but maybe it would be nice to allow controllers to set a variable like:

self.supported_sasl_mechs = ['PLAIN']

And if that list exists and the given mechanism isn't in that list, just skip the testing of it?