Handling multiple request body Content-Type on one endpoint

[tillig]

We have a specification that has a single endpoint for PATCH but allows either standard JSON patch (application/json-patch+json) or JSON merge patch (application/merge-patch+json) as input in the request body.

I find that only the first one ends up generating contract tests.

I did see this issue about different content types (one being non-JSON) having some challenges getting handled, and this issue about handling JSON patch. They seem to slightly overlap with what I'm talking about, so I don't know if it's all connected or what.

Here's an example schema to show what I'm talking about:

schema.json (click to expand)

{
  "components": {
    "responses": {
      "400badrequest": {
        "description": "The request could not be processed by the server."
      },
      "401unauthorized": {
        "description": "The client has not authenticated to perform the operation."
      },
      "403forbidden": {
        "description": "The authenticated client is not authorized to perform the operation."
      },
      "404notfound": {
        "description": "The server has not found a resource matching the request URI."
      }
    },
    "schemas": {
      "jsonPatchDocument": {
        "description": "RFC 6902 JSON Patch document - http://json.schemastore.org/json-patch",
        "items": {
          "oneOf": [
            {
              "additionalProperties": false,
              "properties": {
                "op": {
                  "description": "The operation to perform.",
                  "enum": [
                    "add",
                    "replace",
                    "test"
                  ],
                  "type": "string"
                },
                "path": {
                  "$ref": "#/components/schemas/jsonPatchPath"
                },
                "value": {
                  "description": "The value to add, replace or test."
                }
              },
              "required": [
                "value",
                "op",
                "path"
              ]
            },
            {
              "additionalProperties": false,
              "properties": {
                "op": {
                  "description": "The operation to perform.",
                  "enum": [
                    "remove"
                  ],
                  "type": "string"
                },
                "path": {
                  "$ref": "#/components/schemas/jsonPatchPath"
                }
              },
              "required": [
                "op",
                "path"
              ]
            },
            {
              "additionalProperties": false,
              "properties": {
                "from": {
                  "$ref": "#/components/schemas/jsonPatchPath"
                },
                "op": {
                  "description": "The operation to perform.",
                  "enum": [
                    "move",
                    "copy"
                  ],
                  "type": "string"
                },
                "path": {
                  "$ref": "#/components/schemas/jsonPatchPath"
                }
              },
              "required": [
                "from",
                "op",
                "path"
              ]
            }
          ]
        },
        "type": "array"
      },
      "jsonPatchPath": {
        "description": "A JSON Pointer path pointing to the location of the patch operation.",
        "type": "string"
      },
      "user": {
        "allOf": [
          {
            "$ref": "#/components/schemas/userPatch"
          },
          {
            "properties": {
              "id": {
                "$ref": "#/components/schemas/userId"
              }
            }
          }
        ],
        "description": "The identity for an individual person.",
        "required": [
          "id",
          "familyName",
          "givenName"
        ]
      },
      "userId": {
        "description": "A unique and immutable identifier used to identify a user resource.",
        "example": "214d9ca3-3aaf-4312-902d-03818fd0f4e6",
        "maxLength": 40,
        "minLength": 1,
        "type": "string"
      },
      "userPatch": {
        "description": "Base properties for a user.",
        "properties": {
          "familyName": {
            "description": "The family/last name for the user.",
            "example": "Smith",
            "type": "string"
          },
          "givenName": {
            "description": "The given/first name for the user.",
            "example": "Alice",
            "type": "string"
          }
        },
        "type": "object"
      }
    }
  },
  "info": {
    "contact": {
      "name": "Demo"
    },
    "description": "Patch demo API.",
    "title": "Patch Demo",
    "version": "0.0.1"
  },
  "openapi": "3.0.3",
  "paths": {
    "/users/{id}": {
      "patch": {
        "description": "Makes changes to the profile of a user.",
        "operationId": "patch-user",
        "parameters": [
          {
            "description": "The identifier of the user to update.",
            "in": "path",
            "name": "id",
            "required": true,
            "schema": {
              "$ref": "#/components/schemas/userId"
            }
          }
        ],
        "requestBody": {
          "content": {
            "application/json-patch+json": {
              "example": [
                {
                  "op": "replace",
                  "path": "/givenName",
                  "value": "Alison"
                }
              ],
              "schema": {
                "$ref": "#/components/schemas/jsonPatchDocument"
              }
            },
            "application/merge-patch+json": {
              "example": {
                "givenName": "Alison"
              },
              "schema": {
                "$ref": "#/components/schemas/userPatch"
              }
            }
          },
          "description": "A JSON Patch document with the set of profile element updates.",
          "required": true
        },
        "responses": {
          "200": {
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/user"
                }
              }
            },
            "description": "All operations in the patch document succeeded. The response will contain the updated user profile."
          },
          "400": {
            "$ref": "#/components/responses/400badrequest"
          },
          "401": {
            "$ref": "#/components/responses/401unauthorized"
          },
          "403": {
            "$ref": "#/components/responses/403forbidden"
          },
          "404": {
            "$ref": "#/components/responses/404notfound"
          }
        },
        "summary": "Update a user profile",
        "tags": [
          "resource-users"
        ]
      }
    }
  },
  "servers": [
    {
      "description": "Demo API",
      "url": "https://api.demo.com"
    }
  ],
  "tags": [
    {
      "description": "User/identity-related services.",
      "name": "resource-users"
    }
  ]
}

As the schema is now, it will generate tests for application/json-patch+json because that's the first one listed in the requestBody.content listing. If you reorder them so application/merge-patch+json is first, those are the tests you'll get.

How do I get both?

[thim81]

hi @tillig

Portman typically looks for the 1st JSON content type it comes across and uses that to generate the schema validation.
In your case, you would want to have tests generated for multiple content-types within a response?

[tillig]

Yeah, because each JSON type has a different schema associated - merge patch looks similar to the document getting patched, while JSON patch has a specific schema that specifies operations using JSONPath. Both are JSON, both have a JSON schema, but different schema per content type.

[thim81]

hi @tillig

Not sure if this is still relevant (if not feel free to close this issue).

But we made some improvement in the latest version of Portman with regards the request body & the schema validation.

You can specify as option, via the postman config --postmanConfigFile, with "preferredRequestBodyType", which request body content-type to take from OpenAPI. (typically: raw OR first-listed)

Example

{
  "folderStrategy": "Tags",
  "requestParametersResolution": "Example",
  "exampleParametersResolution": "Example",
  "keepImplicitHeaders": true,
  "enableOptionalParameters": false,
  "preferredRequestBodyType": "raw"
}

[tillig]

Is there a way to generate tests for all of the content types? That's really what I'm looking for with this issue - not just defaulting to one type, but actually generating tests based on all the types [assuming they're JSON compatible]. In this case, JSON patch vs. JSON merge patch - both are JSON, both are accepted, both have a schema associated, so it seems like both could get tests generated.

[thim81]

Using Variations and some new config that could be an option, that we could explore.

With variations (and fuzzing) we already are using the original requests and making variations from them.

How would you expect to configure this? How would that variation config look like?

[tillig]

I mean, I guess I wouldn't really expect to configure it a lot, but I haven't really thought it through deeply. I guess I'd just figure it's a foreach over the set of content types and that's about it. Sort of the simplest thing that could possibly work. Perhaps I'm missing something or oversimplifying?

[thim81]

We have had the request to generate multiple variations for different responses but you are looking for the case to generate variations per request content-type?

something like this:

{
  "variationTests": [
    {
      "openApiOperationId": "leadsAdd",
      "openApiResponse": "200",
      "requestContentType": [
        "application/json",
        "application/x-www-form-urlencoded"
      ],
      "responseContentType": [
        "application/json",
        "application/xml"
      ],
      "variations": []
    }
  ]
}

which would lead to requests:

• POST::/leads - Request application/json - Response - application/json
• POST::/leads - Request application/json - Response - application/xml
• POST::/leads - Request application/x-www-form-urlencoded - Response - application/json
• POST::/leads - Request application/x-www-form-urlencoded - Response - application/xml

[thim81]

@nicklloyd What would be your input on this idea?