Cannot start scylla-manager-agent due missing DataDirectory info about scylladb

[yaronkaikov]

@xemul Can you please have a look?

[xemul]

What is "artifact tests" and where can I find their source code? @soyacz

Presumably the failures are because scylla is started with --workdir and the test gets current DataDirectory from API or system.config table, and at least in the system.config this option would be empty. But need to check the test source code to be sure

[soyacz]

What is "artifact tests" and where can I find their source code? @soyacz

Presumably the failures are because scylla is started with --workdir and the test gets current DataDirectory from API or system.config table, and at least in the system.config this option would be empty. But need to check the test source code to be sure

https://github.com/scylladb/scylla-cluster-tests/blob/master/artifacts_test.py
But this is not interesting, because it fails during manager setup - which also is not relevant if we wrongly start scylla.
I don't see we used --workdir: 2024-01-30T23:53:28.492 artifacts-ubuntu2204-jenkins-db-node-a5687b31-0-1 !INFO | scylla[11936]: command used: "/usr/bin/scylla --blocked-reactor-notify-ms 25 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 0-1 --lock-memory=1"
more test details in argus: https://argus.scylladb.com/test/6dd5afca-3db4-46c4-a898-050660995a22/runs?additionalRuns[]=a5687b31-f47c-4ad4-837c-043a62a1c220

We didn't adapt any SCT code related to Scylla startup - if something changed in that area, was there docs update about it?

[xemul]

The weird thing is that neither CLI nor the scylla.yaml has any directories configured -- data, commitlog, anything else -- all are missing

[xemul]

OK, it's agent that tries to access this endpoint:

        Method:             "GET",
        PathPattern:        "/config/data_file_directories",

[xemul]

Hm...

$ curl -X GET 'http://127.0.0.1:10000/storage_service/release_version'
"3.0.8"
$ curl -X GET 'http://127.0.0.1:10000/config/data_file_directories'
{"message": "Not found", "code": 404}

[xemul]

$ curl -X GET 'http://127.0.0.1:10000/v2/config/data_file_directories'
["/home/xemul/scylla-roots/data"]

[xemul]

How can I get a node, that ... "is the same as the one used in this test"? @soyacz

[mykaul]

@xemul - please revert the commit first of all, then we'll get this fixed.

[xemul]

Already did 7c5c89b

[soyacz]

How can I get a node, that ... "is the same as the one used in this test"? @soyacz

Rebuild job with keep db nodes alive, ping me on slack if you need more guidance

[xemul]

Yes, with the mentioned update it's now empty

$ curl -X GET 'http://127.0.0.1:10000/v2/config/data_file_directories'
[]

[pwrobelse]

Does the problem still occur? The change that introduced the regression was reverted. The revert is already present on master branch.

After discussion with @xemul, the original change will not be re-introduced. More details in the following comment:
#5626 (comment)

[mykaul]

@soyacz ?

[soyacz]

Yes, it's fixed. Closing.