"orderOfOperations" doesn't honor "*" wildcard about portman

[thim81]

@Fetchinator7 It is an interesting request.

The goal of the ordering operations was to provide a way to change the order of specific requests, to facilitate simple request scenarios or reordering then what has been set in the OpenApi doc.

The wildcard option was indeed not foreseen. The order of operations option, would only rearrange the specific request and maintain the order of the untouched items, within the folder. Mostly to overcome strangely sorted OAS documents.

Some questions, that will help us to better understand your use-case:
If you would be able use wildcards? What would be the order for the items that match for example the POST::/*? How would the POST requests be ordered? And would you expect them to respect the folders?

[Fetchinator7]

I guess there was a misunderstanding on my part. I thought ordering requests by type was supposed to work for all routes so this would have been a bug.

BTW, congrats on being being featured in OpenAPI.Tools. I'm really excited to see where this project goes!

To answer your questions:

• What would be the order for the items that match for example the POST::/*?
  • It doesn't really matter to me since most of my resources aren't directly connected to each other (it'd probably in order of appearance in the openapi) but for sure all the POST requests should be first. I may still specify an order of routes for interconnected resources though such as running "POST::/users" before "POST::/pets" since in this example a pet requires a user owner. In that case it should probably evaluate from top to bottom as ["POST::/users", "POST::/pets", "POST::/*"]
• Would you expect them (ordered requests) to respect the folders?
  • With the current Portman project setup I would indeed expect them to respect the folders. However, I'd be nice if I could run all "happy" and "unhappy" POST requests consecutively before the corresponding DELETE requests (see below for more details). It doesn't matter to me if there are no folders or the folders had additional tests in them (maybe "TAG/ENDPOINT/happy and unhappy").

As far as my use-case goes:
I start with a fresh db every time before all of the tests run and my goal is to automate as much of the testing as possible with as little per-test configuration as possible.

I've been using Dredd for openapi testing but I would rather be using this project. However, Dredd allows you to sort requests "in a sensible way" which helps with my minimal per-test configuration.

Per Dredd's documentation: "Sorts requests in a sensible way so that objects are not modified before they are created. POST, GET, HEAD, PUT, PATCH, LINK, UNLINK, DELETE, TRACE."

Sorting requests by type is helpful because since I start with an empty db every time by running the POST request first the resource is there for a GET request vs the default order where a GET going first would result in a 404 since no resources exist yet.

Expanding on the new feature: consecutive "happy" and "unhappy" requests

This is similar to sorting in general but helps with what may be an edge case of a 409 unique value error:

Currently the "unhappy" assertions happen after the "happy" ones because of the folder structure but it'd be nice if they were consecutive for automatic 409 unique value errors. (Like if "[email protected]" is already in use then if a new user tried to use that same email address they'd get a 409 since the email is unique.)
Say I have a /pet and /pet/{id} endpoint where pets have a unique pet value. It'd be nice if I could run the tests based on existing data instead of having to add it before each failure case tests. So thee order would look like:

(I omitted some status codes for brevity).

• /pet POST 201 Adds the pet when previously 0 pets existed
  • Pre request script: Add another valid pet with an id of 2
• /pet/2 PATCH 409 Attempts to change the new pet to the default pet but since pets have a unique field and that field is already in use for pet 1 the PATCH fails.
• /pet/1 DELETE 200 Deletes the pet so it doesn't exist anymore

However since Portman sorts requests into "happy" and "unhappy" folders it would go:

1st folder of "happy" requests:

• /pet POST 201 Adds the pet when previously 0 pets existed
• /pet/1 DELETE 200 Deletes the pet so it doesn't exist anymore

2nd folder of "unhappy" requests:

• Pre request script: Add another valid pet with an id of 2
• /pet/2 PATCH 409 FAILS, gets 201 since the first pet doesn't exist anymore.

Yes, I could just add a second pet with the default values beforehand via a second pre-request script and the 409 would work as expected but just running the request before the original pet was deleted is easier/simpler.

Another feature I would like to see if more support for generating "unhappy" requests via a "one size fits most" wildcard config that can match against the expected status code

Currently we can match requests against the request type and route (GET::/crm/leads) but having a more specific selector that matches against the expected response code (GET:409::/crm/leads or something) would help with config reusability.
Dredd automatically runs a separate request for each possible response code on individual routes. However, (from my understanding) Postman/Portman doesn't have a way to filter the requests by response type so I can't (for example) add a 409 test that applies to all routes via a * since most routes can't even return a 409. I only want to add the additional tests for the routes that are supposed to have a 409 response.

[thim81]

Thanks for the clear explanation and use-case description.

We will need some time to investigate the ordering functionality.

Just double-checking the 1ste case:

• you would be happy with the option to define a re-ordering by method (get, post,...) that would apply on all matching wildcard operations, but the result would apply per folder.
  Your main objective would be to facilitate having all POST ops to be executed 1st.

[Fetchinator7]

Hi. Sorry for the late reply, and thanks again for this great project!

Yes, my main objective would be to facilitate having all POST opt be executed 1st. Based on current functionality, I would indeed expect matching against wildcards to be per folder. However, I'm personally fine with removing folders entirely because I can imagine some situations where people may want POST requests to go in a certain order that doesn't necessarily go in the same order a folder/tag would. Take this folder structure for example:

Folder/tag 1:

• POST Dog,
• POST Leash

Folder/tag 2

• POST Collar

Say that a dog has to exist, a collar needs a dog, and the leash needs a collar. I can't run group two first because the dog doesn't exist yet. However, if I run the first group by folder it would try and post a leash second which would fail because the collars don't exist yet. So if there were no groups I could just order it Dog, Leash, Collar and not run into any issues.
Granted, I doubt this scenario would come up very often if ever, but if the issue of one folder relying on the output of another folder did come up it could be fixed by not having any folders in the first place for that greater customization.

In hindsight the last part of my previous message really should have been its own dedicated feature request, but since I already put it there would you still like me to open an issue for "unhappy" requests via a "one size fits most" wildcard config that can match against the expected status code?
That's the other big thing I'm wondering about because of cases like 409 response codes that don't apply to all routes so I can't just apply a 409 "unhappy" test to every request. Please correct me if I'm wrong about not having the ability to selectively apply "unhappy" requests or there's something else I'm missing. Thanks.

[thim81]

@Fetchinator7 We implemented your suggested fix for supporting the wildcard in the orderOfOperations. It seems to cover the cases which could think off. We will include this in the next release of Portman.

Currently we have kept support for orderOfOperations using the wildcard, where Portman is respecting the grouping per folder. So if you have POST::/*, GET::/* defined as orderOfOperations, they will be sorted in that order per folder. If this is not desired, we suggest to skip the grouping in folders using the Postman configuration folderStrategy .

Update: The wildcard support for orderOfOperations has landed in Portman version 1.13.

🙌 Thank you for contributing to Portman, since your example helped us to bring to fix.

[thim81]

Another feature I would like to see if more support for generating "unhappy" requests via a "one size fits most" wildcard config that can match against the expected status code

Currently we can match requests against the request type and route (GET::/crm/leads) but having a more specific selector that matches against the expected response code (GET:409::/crm/leads or something) would help with config reusability. Dredd automatically runs a separate request for each possible response code on individual routes. However, (from my understanding) Postman/Portman doesn't have a way to filter the requests by response type so I can't (for example) add a 409 test that applies to all routes via a * since most routes can't even return a 409. I only want to add the additional tests for the routes that are supposed to have a 409 response.

We have introduced this in release 1.10, with the option to define a "openApiResponse" for a variation.

openApiResponse (String | optional) : References to the OpenAPI response object code/name for which a variation will be created. (example: "404"). If not defined, the 1st response object from OpenAPI will be taken as expected response. If the configured openApiResponse code is not defined in the OpenAPI document, Portman will not generate a variation for the targeted operations.

You can review the example for some more details

[thim81]

Expanding on the new feature: consecutive "happy" and "unhappy" requests

This is similar to sorting in general but helps with what may be an edge case of a 409 unique value error:

Currently the "unhappy" assertions happen after the "happy" ones because of the folder structure but it'd be nice if they were consecutive for automatic 409 unique value errors. (Like if "[email protected]" is already in use then if a new user tried to use that same email address they'd get a 409 since the email is unique.) Say I have a /pet and /pet/{id} endpoint where pets have a unique pet value. It'd be nice if I could run the tests based on existing data instead of having to add it before each failure case tests. So thee order would look like:

(I omitted some status codes for brevity).

• /pet POST 201 Adds the pet when previously 0 pets existed

  • Pre request script: Add another valid pet with an id of 2

• /pet/2 PATCH 409 Attempts to change the new pet to the default pet but since pets have a unique field and that field is already in use for pet 1 the PATCH fails.

• /pet/1 DELETE 200 Deletes the pet so it doesn't exist anymore

However since Portman sorts requests into "happy" and "unhappy" folders it would go:

1st folder of "happy" requests:

• /pet POST 201 Adds the pet when previously 0 pets existed
• /pet/1 DELETE 200 Deletes the pet so it doesn't exist anymore

2nd folder of "unhappy" requests:

• Pre request script: Add another valid pet with an id of 2
• /pet/2 PATCH 409 FAILS, gets 201 since the first pet doesn't exist anymore.

Yes, I could just add a second pet with the default values beforehand via a second pre-request script and the 409 would work as expected but just running the request before the original pet was deleted is easier/simpler.

I struggle to fully understand the desired outcome.
Could you create a seperate issue for this and provide some more details, that will help me understand the use-case better?