Default Response Object in variation tests

[Giackgamba]

I'm trying to create variation tests for an OpenAPI where we define default responses for all error (not 2xx) codes.

Are default responses ssupported in any way? I guess statusCode contractTests would become quite problematic, but the other tests should be valid.

If a variation with "openApiResponse": "default" is set, examples end up in the postman collection, but no tests are created for it.

Any strategy/workaround without enumerating all the possible error codes in the openAPI spec and in the portman-config?

Example of a openapi spec:

openapi: 3.1.0

info:
  title: test API
  description: |
    The following document contains the documentation of test API
  version: 1.0.0
  
servers:
  - url: https://localhost
    description: local test environment

paths:
  /persons:
    
    get:
      tags:
        - persons
      summary: Get all persons.
      description: |
        Return all the data of the persons registered by the partner to the Platform.
      operationId: getPersons
      responses:
        '200':
          description: Persons retrieved.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PersonGetAllResponse'
        default:
          description: Error occurred.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PersonErrorResponse'
                
components:
  schemas:
    PersonGetAllResponse:
      title: Persons
      type: object
      properties:
        persons:
          $ref: '#/components/schemas/PersonCreationResponseArray'
    PersonCreationResponseArray:
      title: Person Response Array
      type: array
      items:
        $ref: '#/components/schemas/PersonCreationResponse'
    PersonCreationResponse:
      title: Person
      allOf:
        - type: object
          required:
            - personId
          properties:
            personId:
              type: string
              format: uuid
              description: Unique identifier of a person.
              example: '123e4567-e89b-12d3-a456-426614174000'
        - $ref: '#/components/schemas/PersonCreationRequest'
    PersonCreationRequest:
      title: New person
      type: object
      required:
        - personId
        - firstName
        - lastName
      properties:
        personId:
          type: string
          description: Unique identifier of a person.
          example: '123e4567-e89b-12d3-a456-426614174000'
        firstName:
          type: string
          description: First name.
          example: 'John'
        lastName:
          type: string
          description: Last name.
          example: 'Doe'
    PersonErrorResponse:
      required:
        - errors
      type: object
      properties:
        errors:
          type: array
          items:
            type: object
            properties:
              code:
                type: string
                description: Error code.
                enum:
                  - PERSON_NOT_FOUND
                  - INVALID_FIELD_FORMAT
                  - MISSING_HEADER
                  - INVALID_HEADER_CONTENT
                  - PERSON_ALREADY_EXISTS
                example: PERSON_NOT_FOUND
              message:
                type: string
                description: Human readable representation of the error.
                example: Could not find person '1234'

Partial portman-config:

 "variationTests": [
      {
        "openApiOperation": "*::/*",
        "openApiResponse": "default",
        "variations": [
          {
            "name": "Default errors",
            "fuzzing": [
              {
                "requestBody": [
                  {
                    "maxLengthFields": {
                      "enabled": true
                    },
                    "maximumNumberFields": {
                      "enabled": true
                    },
                    "minLengthFields": {
                      "enabled": false
                    },
                    "minimumNumberFields": {
                      "enabled": true
                    },
                    "requiredFields": {
                      "enabled": true
                    }
                  }
                ],
                "requestQueryParams": [
                  {
                    "maxLengthFields": {
                      "enabled": true
                    },
                    "maximumNumberFields": {
                      "enabled": true
                    },
                    "minLengthFields": {
                      "enabled": false
                    },
                    "minimumNumberFields": {
                      "enabled": true
                    },
                    "requiredFields": {
                      "enabled": true
                    }
                  }
                ]
              }
            ],
            "tests": {
              "contractTests": [
                {
                  "statusCode": {
                    "enabled": true
                  },
                  "contentType": {
                    "enabled": true
                  },
                  "jsonBody": {
                    "enabled": true
                  },
                  "schemaValidation": {
                    "enabled": true
                  },
                  "headersPresent": {
                    "enabled": true
                  }
                }
              ]
            }
          }
        ]

Partial resulting postman collection

        {
          "id": "ea15058f-acc5-4fab-a686-b5f96411021e",
          "name": "persons Variations",
          "item": [
            {
              "id": "4f70e408-bd48-4466-9ced-419085f558ac",
              "name": "Update a person.[Default errors][required personId]",
              "request": {
                "name": "Update a person.[Default errors][required personId]",
                "description": {
                  "content": "Update the data of an existing person.\n",
                  "type": "text/plain"
                },
                "url": {
                  "path": [
                    "persons",
                    ":personId"
                  ],
                  "host": [
                    "{{baseUrl}}"
                  ],
                  "query": [],
                  "variable": [
                    {
                      "disabled": false,
                      "description": {
                        "content": "(Required) Unique identifier of a person.",
                        "type": "text/plain"
                      },
                      "type": "string",
                      "value": "{{personId}}",
                      "key": "personId"
                    }
                  ]
                },
                "header": [
                  {
                    "key": "Content-Type",
                    "value": "application/json"
                  },
                  {
                    "key": "Accept",
                    "value": "application/json"
                  }
                ],
                "method": "PUT",
                "body": {
                  "mode": "raw",
                  "raw": "{\n    \"firstName\": \"John\",\n    \"lastName\": \"Doe\",\n    \"emailAddress\": \"[email protected]\",\n    \"dateOfBirth\": \"2000-07-21\",\n    \"placeOfBirth\": {\n        \"city\": \"Rome\",\n        \"country\": \"IT\"\n    },\n    \"citizenship\": \"IT\",\n    \"fiscalCode\": \"RSSPPP00L21H501U\",\n    \"gender\": \"MALE\",\n    \"homeAddress\": {\n        \"line1\": \"3972 Eagles Nest Drive\",\n        \"postalCode\": \"00152\",\n        \"city\": \"Placerville\",\n        \"country\": \"IT\",\n        \"line2\": \"Apt. 123\",\n        \"state\": \"Italy\"\n    },\n    \"domicileAddress\": {\n        \"line1\": \"3972 Eagles Nest Drive\",\n        \"postalCode\": \"00152\",\n        \"city\": \"Placerville\",\n        \"country\": \"IT\",\n        \"line2\": \"Apt. 123\",\n        \"state\": \"Italy\"\n    },\n    \"phoneNumber\": \"+399991234567\"\n}",
                  "options": {
                    "raw": {
                      "language": "json"
                    }
                  }
                }
              },
              "response": [
                {
                  "id": "9fa1c80d-c015-42e9-ace0-c37798181df0",
                  "name": "Person's data updated.",
                  "originalRequest": {
                    "url": {
                      "path": [
                        "persons",
                        ":personId"
                      ],
                      "host": [
                        "{{baseUrl}}"
                      ],
                      "query": [],
                      "variable": [
                        {
                          "disabled": false,
                          "description": {
                            "content": "(Required) Unique identifier of a person.",
                            "type": "text/plain"
                          },
                          "type": "any",
                          "value": "123e4567-e89b-12d3-a456-426614174000",
                          "key": "personId"
                        }
                      ]
                    },
                    "header": [
                      {
                        "description": {
                          "content": "Added as a part of security scheme: bearer",
                          "type": "text/plain"
                        },
                        "key": "Authorization",
                        "value": "Bearer <token>"
                      }
                    ],
                    "method": "PUT",
                    "body": {
                      "mode": "raw",
                      "raw": "{\n  \"personId\": \"123e4567-e89b-12d3-a456-426614174000\",\n  \"firstName\": \"John\",\n  \"lastName\": \"Doe\",\n  \"emailAddress\": \"[email protected]\",\n  \"dateOfBirth\": \"2000-07-21\",\n  \"placeOfBirth\": {\n    \"city\": \"Rome\",\n    \"country\": \"IT\"\n  },\n  \"citizenship\": \"IT\",\n  \"fiscalCode\": \"RSSPPP00L21H501U\",\n  \"gender\": \"MALE\",\n  \"homeAddress\": {\n    \"line1\": \"3972 Eagles Nest Drive\",\n    \"postalCode\": \"00152\",\n    \"city\": \"Placerville\",\n    \"country\": \"IT\",\n    \"line2\": \"Apt. 123\",\n    \"state\": \"Italy\"\n  },\n  \"domicileAddress\": {\n    \"line1\": \"3972 Eagles Nest Drive\",\n    \"postalCode\": \"00152\",\n    \"city\": \"Placerville\",\n    \"country\": \"IT\",\n    \"line2\": \"Apt. 123\",\n    \"state\": \"Italy\"\n  },\n  \"phoneNumber\": \"+399991234567\"\n}",
                      "options": {
                        "raw": {
                          "language": "json"
                        }
                      }
                    }
                  },
                  "status": "OK",
                  "code": 200,
                  "header": [
                    {
                      "key": "Content-Type",
                      "value": "application/json"
                    }
                  ],
                  "body": "{\n  \"citizenship\": \"IT\",\n  \"dateOfBirth\": \"2000-07-21\",\n  \"emailAddress\": \"[email protected]\",\n  \"firstName\": \"John\",\n  \"fiscalCode\": \"RSSPPP00L21H501U\",\n  \"lastName\": \"Doe\",\n  \"personId\": \"123e4567-e89b-12d3-a456-426614174000\",\n  \"placeOfBirth\": {\n    \"city\": \"Rome\",\n    \"country\": \"IT\"\n  },\n  \"gender\": \"MALE\",\n  \"homeAddress\": {\n    \"line1\": \"3972 Eagles Nest Drive\",\n    \"postalCode\": \"00152\",\n    \"city\": \"Placerville\",\n    \"country\": \"IT\",\n    \"line2\": \"Apt. 123\",\n    \"state\": \"Italy\"\n  },\n  \"domicileAddress\": {\n    \"line1\": \"3972 Eagles Nest Drive\",\n    \"postalCode\": \"00152\",\n    \"city\": \"Placerville\",\n    \"country\": \"IT\",\n    \"line2\": \"Apt. 123\",\n    \"state\": \"Italy\"\n  },\n  \"phoneNumber\": \"+399991234567\"\n}",
                  "cookie": []
                }
            ...
          ],
          "event": []

[thim81]

hi @Giackgamba

At first looks, it seems that "default" is not supported.

We could investigate to bring support for it, with the assumption that you would have to define the expected "statusCode" since we cannot know what is expected statusCode that the contract should be tested against.

Let me look into it some more and report back and most likely ask some more questions about your expected behavior.
The fact that you shared already an example is really helpful.

[thim81]

hi @Giackgamba

While looking into bringing support for the "default" response, I took your example but it does not seem to generate any response in the "Variation test" folder?!

Do you experience the same?

[Giackgamba]

You are right, indeed the provided MVE does not create variations tests.
I'll try to conjure up an example that does, because I just retried and my complete configuration does that (but it's too complex and protected to be shared).

[thim81]

@Giackgamba I want to be able to reproduce it, so if you have an easy way with 1 default response to reproduce it that would be sufficient. It would reduce the time needed to get the setup going and make it easier to develop a fix.