Arangobench unreliable behavior about arangodb

[jsteemann]

@maininformer : the --collection option is ignored when using --custom-query or --custom-query-file.
When using a custom query, all that matters to arangobench is the actual custom query. So not declaring all collections used in the query via the --collection option is not an error.

When using a non-existing collection in the custom query, the query will fail. The same is true for any other hard errors. arangobench will even print warnings about this, such as:

2024-03-14T08:26:23Z [6568] WARNING [fb835] {bench} single request for URL '/_api/cursor: got invalid response from server: HTTP 404: 'Not Found'
2024-03-14T08:26:23Z [6568] WARNING [6daf1] {bench} ...more warnings...
...
2024-03-14T08:26:23Z [6568] WARNING [a826b] {bench} 1000 arangobench request(s) failed!

In case of hard errors, the generated results JSON file will also contain these in the "failures" attribute.

Regarding the typo in your collection names inside the traversal (i.e. FOR vertex IN 1..9999 OUTBOUND 'foos/878' GRAPH loadtest_graph instead of FOR vertex IN 1..9999 OUTBOUND 'foo/878' GRAPH loadtest_graph):
This is not considered a hard error by ArangoDB, as during query parsing 'foos/878' is just a string literal.
You can even run this query in ArangoDB just fine (at least on single server), as there may be edges starting at foos/878 in any of the edge collections used in the graph.

[maininformer]

Thank you this is helpful.