marqo-ai / marqo-api-tests Goto Github PK

doc:
https://docs.google.com/document/d/1YTFnMkoMnJQ1PIXmyBxK6QrrBcskW2jm1_IevMJJunU/

The issue:
The marqo github CI workflow to test S2Search doesn't actually connect to any S2Search URL, it just acts like DIND.
Proof: Observe this run that passed: https://github.com/marqo-ai/marqo/actions/runs/5107294728/jobs/9180084294#step:8:4868
It runs its own OpenSearch container.

There are several causes:

1. The current S2Search URL (github secret) is outdated and has already been torn down. There is a new URL we should use (supplied by @VitusAcabado )
2. The tox does not properly pass env variables (from running conf) to the setup script. I think this is because the env vars set from running conf in clone_marqo_repo.sh in [testenv] commands_pre do not get carried over to the startup scripts in the child envs. Because of this, the s2search startup script reads $S2SEARCH_URL as empty, and starts its own marqo-os.

I don't know if this way of running conf used to work before and just doesn't now, or if it was always like this. Worth having a chat about!

Proposed solution:

1. Remove any setting of env vars and running conf in clone_marqo_repo.sh
2. Transfer LOCAL_OPENSEARCH_URL export declaration to conf file, so all exports are in 1 place (1 source of truth)
3. In all startup scripts, call . "$(pwd)/conf" at the start, thus env vars are properly set before running marqo/marqo-os
4. In the marqo repo, add the working S2Search URL as secret
5. Add skip flag for some tests in the S2Search environment. Tests where we change the marqo configuration (like max replicas) will fail because of the internal S2Search limit. For example, I get this error:

marqo.errors.MarqoWebError: MarqoWebError: MarqoWebError Error message: {'message': "{'error': {'root_cause': [{'reason': 'provided number of number_of_replicas 4 exceeds your Marqo limit. Current limit for number_of_replicas is 1', 'type': 'marqo_error'}]}, 'status': 400}\nPlease create an issue on Marqo's GitHub repo (https://github.com/marqo-ai/marqo/issues) if this problem persists.", 'code': 'unhandled_backend_error', 'type': 'backend_error', 'link': ''}
E       status_code: 500, type: backend_error, code: unhandled_backend_error, link:

Problem:
We often forget to increment the Marqo's in-code version number. This results in the root endpoint returning a version not representative of the actual version.

Proposed solution:

1. Check if the given version number is in the list of GitHub release tags:

curl https://api.github.com/repos/marqo-ai/marqo/tags

3. Echo warnings that can be parsed by Github:
   https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions

We need to add new functionality to Marqo API end-to-end tests to support the following:

• Testing marqo with different Env Vars within the same test script

In particular, for throttling tests, we want to run tests on instances of marqo with different MARQO_MAX_CONCURRENT_INDEX and MARQO_MAX_CONCURRENT_SEARCH.

Proposed solution: https://docs.google.com/document/d/1YTFnMkoMnJQ1PIXmyBxK6QrrBcskW2jm1_IevMJJunU/edit?usp=sharing