Empty schema in requestBody for application/x-www-form-urlencoded POST endpoints when using requestParameters as Spring REST Docs suggests about restdocs-api-spec

[afronski]

If application/x-www-form-urlencoded content type is used with POST/PUT request, check the request body example for form fields that match parameter names. Then render those appropriately as form parameters, instead of query/path w/e parameters.

Yes, @ozscheyge - exactly my thoughts about the implementation. If we are okay with this "workaround", then I can prepare something along those lines.

[CalamarBicefalo]

I was thinking @ozscheyge maybe we could make hasRequestBody true if it is application/x-www-form-urlencoded and improve the example generation so it can also parse params in this case?

[ozscheyge]

Hey @afronski , thanks for reaching out!

I can only reproduce this as a minor issue in the openapi3 generator gradle-plugin.

Test setup

        resultActions = mockMvc.perform(
                        post("/api/payloadtest")
                        .contentType(MediaType.APPLICATION_FORM_URLENCODED_VALUE)
                        .content("name=Bob&street=AbcStreet")
                ).andDo(print());

        resultActions
                .andExpect(status().isOk())
                .andDo(document("callbacks.parameterTest", ..., requestParameters(
                        parameterWithName("name").description("name desc"),
                        parameterWithName("street").description("street desc")
                )));

api-spec snippets, openapi (2) generator, postman generator are working as intended by Spring Restdocs

restdocs-api-spec generates intermediate snippets being a structured data dump of the documented request. Those snippets are then converted to the target format using the gradle-plugins.

{
  "operationId" : "callbacks.parameterTest",
  "summary" : "blub",
  "description" : "blubbo",
  "privateResource" : false,
  "deprecated" : false,
  "request" : {
    "path" : "/api/payloadtest",
    "method" : "POST",
    "contentType" : "application/x-www-form-urlencoded",
    "headers" : [ ],
    "pathParameters" : [ ],
    "requestParameters" : [ {
      "name" : "name",
      "attributes" : { },
      "description" : "name desc",
      "ignored" : false,
      "type" : "STRING",
      "optional" : false
    }, {
      "name" : "street",
      "attributes" : { },
      "description" : "street desc",
      "ignored" : false,
      "type" : "STRING",
      "optional" : false
    } ],
    "requestFields" : [ ],
    "example" : "name=Bob&street=AbcStreet",
    "securityRequirements" : null
  },
  "response" : {
    "status" : 200,
    "contentType" : null,
    "headers" : [ ],
    "responseFields" : [ ],
    "example" : null
  },
  "tags" : [ "api" ]
}

Notice how both parameters from the payload body are described in the parameters section as intended by Spring Restdocs. If you use it with asciidoctor, it generates the following snippet too:

|===
|Parameter|Description

|`+name+`
|name desc

|`+street+`
|street desc

|===

The generated postman entry looks like:

{
    "id" : "callbacks.parameterTest",
    "name" : "/api/payloadtest",
    "description" : "blubbo",
    "variable" : [ ],
    "event" : [ ],
    "request" : {
      "url" : {
        "protocol" : "https",
        "host" : "localhost",
        "path" : "/api/payloadtest",
        "port" : "8080",
        "query" : [ {
          "key" : "name",
          "disabled" : false,
          "description" : "name desc"
        }, {
          "key" : "street",
          "disabled" : false,
          "description" : "street desc"
        } ]
      },
      "method" : "POST",
      "header" : [ {
        "key" : "Content-Type",
        "value" : "application/x-www-form-urlencoded",
        "disabled" : false
      } ],
      "body" : {
        "mode" : "raw",
        "raw" : "name=Bob&street=AbcStreet",
        "urlencoded" : [ ]
      }
    },
    "response" : [ {
      "id" : "callbacks.parameterTest",
      "name" : "200-null",
      "originalRequest" : {
        "url" : {
          "protocol" : "https",
          "host" : "localhost",
          "path" : "/api/payloadtest",
          "port" : "8080",
          "query" : [ {
            "key" : "name",
            "disabled" : false,
            "description" : "name desc"
          }, {
            "key" : "street",
            "disabled" : false,
            "description" : "street desc"
          } ]
        },
        "method" : "POST",
        "header" : [ {
          "key" : "Content-Type",
          "value" : "application/x-www-form-urlencoded",
          "disabled" : false
        } ],
        "body" : {
          "mode" : "raw",
          "raw" : "name=Bob&street=AbcStreet",
          "urlencoded" : [ ]
        }
      },
      "cookie" : [ ],
      "code" : 200
    } ]
  }

Again the parameters are documented, an example payload body is listed.

Similar for the openapi (2) generator:

{
    "/api/payloadtest" : {
      "post" : {
        "tags" : [ "api" ],
        "summary" : "blub",
        "description" : "blubbo",
        "operationId" : "callbacks.parameterTest",
        "consumes" : [ "application/x-www-form-urlencoded" ],
        "parameters" : [ {
          "name" : "name",
          "in" : "query",
          "description" : "name desc",
          "required" : true,
          "type" : "string"
        }, {
          "name" : "street",
          "in" : "query",
          "description" : "street desc",
          "required" : true,
          "type" : "string"
        }, {
          "in" : "body",
          "name" : "",
          "required" : false,
          "schema" : {
            "$ref" : "#/definitions/api_payloadtest-1942036431"
          },
          "x-examples" : {
            "application/x-www-form-urlencoded" : "name=Bob&street=AbcStreet"
          }
        } ],
        "responses" : {
          "200" : {
            "description" : "",
            "examples" : { }
          }
        }
      }
    }

further in the schema section:

"api_payloadtest-1942036431" : {
      "example" : "name=Bob&street=AbcStreet"
},

So for the body there is no schema derived, just the example listed, which is ok since the parameters are already documented.

openapi3 generator makes a minor error

I can confirm what you reported with the openapi3 generator:

 "/api/payloadtest" : {
      "post" : {
        "tags" : [ "api" ],
        "summary" : "blub",
        "description" : "blubbo",
        "operationId" : "callbacks.parameterTest",
        "parameters" : [ {
          "name" : "name",
          "in" : "query",
          "description" : "name desc",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        }, {
          "name" : "street",
          "in" : "query",
          "description" : "street desc",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "requestBody" : {
          "content" : {
            "application/x-www-form-urlencoded" : {
              "schema" : {
                "$ref" : "#/components/schemas/api-payloadtest1116014589"
              },
              "examples" : {
                "callbacks.parameterTest" : {
                  "value" : "name=Bob&street=AbcStreet"
                }
              }
            }
          }
        },
        "responses" : {
          "200" : {
            "description" : "200"
          }
        }
      }
    }

Further the down the schema is incomplete:

"api-payloadtest1116014589" : {
        "type" : "object"
      },

Summary

• restdocs-api-spec, the openapi (2) and postman generators seem to work fine
• The schema generated by the openapi3 gradle-plugin could be better or just include the example like with openapi (2)
• Having the request body fields documented as parameters for application/x-www-form-urlencoded content-type is in accordance with https://docs.spring.io/spring-restdocs/docs/current/reference/html5/#documenting-your-api-request-parameters , Spring Restdocs + asciidoctor would also not document request body fields

So it looks like a minor issue of the openapi3 generator. Would you mind providing a PR?

[afronski]

Thanks, @ozscheyge for a comprehensive analysis!

Let me start that I'd love to provide PR. However, I do have further questions, so let's align on those elements before implementation:

1. We are agreeing about the openapi3 issue, which generates an empty schema. That's one thing to fix for sure.
2. Spring REST Docs generates proper examples, no doubt in that as well.

However, there is a 3rd point, where I either do not understand something or generated specs are not exactly right:

As we are sending POST with payload as application/x-www-form-urlencoded I would expect to see the corresponding payload elements (in your example name and street) not present in the section parameters with type query as those are describing query parameters in the URL.

Example: https://swagger.io/docs/specification/describing-parameters/#query-parameters

IMHO they should land only in requestBody as schema and example. Am I missing something?

I think the culprit lies here: https://docs.spring.io/spring-restdocs/docs/2.0.3.RELEASE/reference/html5/#documenting-your-api-request-parameters

It is confusing that they are reusing the same method in Spring REST Docs for query parameters and POST payload with application/x-www-form-urlencoded MIME type, but for their use case, it does not matter what they are documenting - the output will be correct either way.

BTW. being able to describe and list individual elements of application/x-www-form-urlencoded would be nice, but it's totally separate and for now limited by the Spring REST Docs capabilities:

spring-projects/spring-restdocs#510

[ozscheyge]

On a second look, you are correct @afronski :

• Openapi 3 wants to have the parameters in the request body schema and to use the object type: https://swagger.io/docs/specification/describing-request-body/ (see section "Form Data")
• Openapi 2 wants to have it in the parameters block, but with in: formData: https://swagger.io/docs/specification/2-0/describing-parameters/#form-parameters

I think, this is pretty tough, since the restdocs parameter snippet does not contain this info, so we don't have it in the resource.json dump (see my 2nd code snippet in #113 (comment) ).

When reading this information, the plugins would need to apply the following logic:

If application/x-www-form-urlencoded content type is used with POST/PUT request, check the request body example for form fields that match parameter names. Then render those appropriately as form parameters, instead of query/path w/e parameters.

[afronski]

@ozscheyge better late than never. 😉

PR #137 should fix the aforementioned issue - feel free to review and provide any remarks. I've just fixed the parameter types generation for the OpenAPI 2.0 and 3.0, skipping problem of empty schema generation for the OpenAPI 3.0

[CalamarBicefalo]

This does not really work for rest easy, I'm afraid.

Given the current test - the legit way of doing this according to spring docs + a content type to make this PR happy

@Test
    fun `post-token-client-credentials`() {
        RestAssured.given(this.spec).urlEncodingEnabled(true)
                .formParam("grant_type", "client_credentials")
                .formParam("client_id", clientId)
                .formParam("client_secret", clientSecret)
                .contentType(ContentType.URLENC)
                .filter(
                        document(
                                identifier = "{method-name}",
                                description = TOKEN_DESCRIPTION,
                                summary = TOKEN_SUMMARY,
                                requestPreprocessor = preprocessRequest(
                                        Preprocessors.modifyParameters().set("client_id", "***").set("client_secret", "***")
                                ),
                                responsePreprocessor = hideResponseTokensPreprocessor,
                                snippets = *arrayOf(tokenResponseFields,
                                        requestParameters(
                                                grantTypeParam,
                                                clientIdParam,
                                                clientSecretParam
                                        ))
                        )
                )
                .`when`().post("/v1/token").apply { prettyPrint() }
                .then().assertThat().statusCode(CoreMatchers.`is`(200))
    }

The whole thing falls apart when building the data model because of this line:

com/epages/restdocs/apispec/ResourceSnippet.kt:68

if (hasRequestBody) getContentTypeOrDefault(operation.request.headers) else null

Unfortunately hasRequestBody says no. If you try to set the actual body as body (instead of formParams) spring fails.

Any ideas?

[ozscheyge]

Sounds reasonable, let's continue in #145