alternator suite: when run by test.py, used scylla from the pool about scylladb

[nyh]

The first step is to switch alternator suite type from Run to Python.

I think as far as this issue is concerned, it should be the only step, and it should be trivial to do - exactly like test/cql-pytest no longer uses test/cql-pytest/run (which run all of Alternator's tests) and instead starts Scylla using test.py's builtin Scylla-starter and runs pytest separately on each test file.

If I understand correctly (see your commit 1955f91) all it takes to complete this issue is a patch like this:

--- a/test/alternator/suite.yaml
+++ b/test/alternator/suite.yaml
@@ -1 +1,2 @@
-type: Run
+type: Python
+pool_size: 4

Basically right after such a change, test.py will start running test/alternator test files correctly with its correct conftest.py and everything, and everything should work. Let me know if I'm missing anything.

[xtrey]

These changes will be not enough. pytest_adoption() should be changed and at least alternator should be moved to conftest, since currently, it started from the run that will be skipped with the aforementioned changes. I expected there would be more things to fix, but hope not too much.

[xtrey]

Draft PR added #18442 however two tests fail and run can not be used. WIP

[nyh]

Please note that not only must test/alternator/run remain usable, also 'cd alternator; pytest' must continue working (that's also how one runs tests against AWS). I think you shouldn't need to modify test/alternator/conftest.py at all.

[xtrey]

it's not possible to migrate to the Python suite without modifying conftest. Python suite starting the scylla and we need to provide this information to the test.
what is the point of having run when you can just use pytest to start tests? we need to move logic from run to fixtures and just use pure pytest

[nyh]

You are oversimplifying the situation and missing much of the power of the Alternator test suite by focusing on the Jenkins-like use case. The Alternator test suite is designed to run against any server which supports the DynamoDB API. This case be any version of Scylla with Alternator enabled running locally, can be Scylla Cloud running remotely and can be Amazon Dynamodb. This is why cd test/alternator; pytest doesn't run Scylla - it connects to an existing server! It is an external process, be it test/alternator/run or test.py, which needs to start Scylla before running pytest.

Exactly the same thing already happens in the cql-pytest test suite. It too is designed to work against any server supporting CQL (including Cassandra), so it too does not start Scylla - rather test/cql-pytest/run or test.py starts Scylla before running pytest.

What is true, however, is that the cql-pytest and alternator suites would like test.py to start Scylla with slightly different options, eg the Alternator test suite needs Alternator to be enabled. This has two possible solutions in test.py:

1. Test.py can run a single-node Scylla using options which will work for both cql-pytest and alternator use cases.
2. Test.py can use suite.yaml to decide on suite-specific options to use for Scylla.

I think that option 2 will be better in the long run.
But note that neither option needs any modifications to test/alternator/conftest.py.

[temichus]

PR #18442 in master