juhoinkinen
In PR #879 some functions of Schemathesis testing were disabled: https://github.com/NatLibFi/Annif/blob/ef5d8aeb1758e662ad3c15b25e5b07668bebf3a3/tests/test_openapi.py#L12-L13 If these are enabled, two tests fail: ``` ________________________________________________ test_openapi_fuzzy[POST /projects/{project_id}/suggest] ________________________________________________ + Exception Group Traceback (most recent call last): | File "/home/jmminkin/git/Annif/.venv/lib/python3.11/site-packages/hypothesis/core.py", line 2287, in wrapped_test | raise the_error_hypothesis_found | schemathesis.core.failures.FailureGroup: Schemathesis found 1 distinct failure | | - API rejected schema-compliant request | | Valid data should have been accepted | Expected: 2xx, 401, 403, 404, 409, 5xx | Hint: The request body contains 1 additional property not defined in the schema (``). The server likely rejects unexpected fields. Add `additionalProperties: false` to your schema to prevent this. | | [400] Bad Request: | | `{"type": "about:blank", "title": "Bad Request", "detail": "language \"d\ud9d9\uddbc\" not supported by vocabulary", "status": 400}` | | Reproduce with: | | curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' -d 'text=d%C2%9F%C2%AD%C3%A6%1B&=%C3%8F%0A&limit=3200206&threshold=1.2279522352668104e-107&language=d%F2%86%96%BC' http://localhost/v1/projects/dummy-en/suggest | | (1 sub-exception) +-+---------------- 1 ---------------- | schemathesis.openapi.checks.RejectedPositiveData: API rejected schema-compliant request | | Valid data should have been accepted | Expected: 2xx, 401, 403, 404, 409, 5xx | Hint: The request body contains 1 additional property not defined in the schema (``). The server likely rejects unexpected fields. Add `additionalProperties: false` to your schema to prevent this. +------------------------------------ ``` and ``` _____________________________________________ test_openapi_fuzzy[POST /projects/{project_id}/suggest-batch] _____________________________________________ + Exception Group Traceback (most recent call last): | File "/home/jmminkin/git/Annif/tests/test_openapi.py", line 77, in test_openapi_fuzzy | case.call_and_validate() | File "/home/jmminkin/git/Annif/.venv/lib/python3.11/site-packages/schemathesis/generation/case.py", line 601, in call_and_validate | self.validate_response( | File "/home/jmminkin/git/Annif/.venv/lib/python3.11/site-packages/schemathesis/generation/case.py", line 560, in validate_response | raise FailureGroup(_failures, message) from None | schemathesis.core.failures.FailureGroup: Schemathesis found 1 distinct failure | | - API accepted schema-violating request | | Invalid data should have been rejected | Expected: 400, 401, 403, 404, 405, 406, 409, 422, 428, 5xx | Invalid component: in query - object with unexpected properties | | [200] OK: | | `[]` | | Reproduce with: | | curl -X POST -H 'Content-Type: application/json' -d '{"documents": []}' 'http://localhost/v1/projects/dummy-fi/suggest-batch?threshold=0&x-schemathesis-unknown-property=42' | | (1 sub-exception) +-+---------------- 1 ---------------- | schemathesis.core.failures.AcceptedNegativeData: API accepted schema-violating request | | Invalid data should have been rejected | Expected: 400, 401, 403, 404, 405, 406, 409, 422, 428, 5xx | Invalid component: in query - object with unexpected properties +------------------------------------ ``` I think it should be possible to restrict the generation of languages to resolve the first failure (the error response "not supported by vocabulary" is correct here, it just is not possible to be expressed in OpenAPI spec?). For the second failure, I speculated on an resolution: > In addition to upgrading to Schemathesis 4, this PR currently enables [`strict_validation` in Connexion](https://connexion.readthedocs.io/en/stable/validation.html#parameter-validation) "to disallow any extra parameters that are not defined in your specification". I think this would be a good feature, because then typos in API requests parameters are noticed at once, not after debugging (with strict validation e.g. using `theshold` instead of `threshold` gives 400 Bad request error, without it the extra, nonfunctional parameter is just silently ignored). > > However, Annif 1.4 introduced `metadata_*` parameters to suggest/, suggest_batch/ and learn/ API methods (#867), and to return a 400 error for requests with parameters _not_ starting with `metadata_` I made this: [Add Custom request body validator](https://github.com/NatLibFi/Annif/pull/879/commits/c2aa7fa652debb4a9986f37beb45e45201367cad). _Originally posted by @juhoinkinen in https://github.com/NatLibFi/Annif/issues/879#issuecomment-3631177005_ Also, on some runs a third test has failed (can't remember of find again which or reproduce); to not make this happen surprisingly, Schemathesis could be made generate the test cases determistically: https://github.com/schemathesis/schemathesis/discussions/3531#discussioncomment-15972537