Check multiple response codes

[JavierSierraBlazquez]

This question could be related to #13

Having an OpenAPI document and the API implementation, we would like to check whether the API is in compliance with the contract.

In an OpenAPI document you can specify more than one response code with different body in each case. It would be nice we could test each one by making multiple requests for the same endpoint, one for each status code.

To obtain different response types from the API we could add an extra parameter that could be specified using OpenAPI extension x-something https://swagger.io/docs/specification/openapi-extensions/ that would result in a header that API will interpret to return the desired response.

What do you think about?

[javier-sierra-sngular]

Hello @mishushakov, what do you think about this?

[mishushakov]

It's a bit more involved and I don't have time to contribute a fix atm. Would it be possible for your company to sponsor the development of an improved OpenAPI generator? Otherwise you could try use GPT-4 to generate the workflow from OpenAPI spec

[javier-sierra-sngular]

We are interested in collaborating on the project. If you agree, we can submit a pull request and work on it. If you don't have any specific ideas in mind, we can propose a solution.

[mishushakov]

Yeah, that works too, just submit a PR and I will review

[javier-sierra-sngular]

Hello @mishushakov. We send you the proposed solution in case you want to add any comments

Proposal

Objective

We want to verify if the implementation of the API complies with the contract established with the OpenAPI document.

In an OpenAPI document you can specify more than one response code with a different body in each case. It would be nice if we could test each one by making multiple requests for the same endpoint, one for each status code.

The problem that arises from this idea is how to trigger a different response code for the same endpoint.

We have thought about defining an additional file, with the intention of decoupling it from the original document. In this file, the final user specifies how to trigger a response code through a request. The format of this document will also follow the OpenAPI specification, using the “examples” field. This way, it will be easy to process the information in the workflow generation and we can use the Swagger validator to check if the document is well-formed.

Use example

The examples file could be specified by using a flag or as a second parameter of stepci generate:

stepci generate path/contract.yml --request-examples path/contract.examples.yml
stepci generate path/contract.yml path/contract.examples.yml

OpenAPI contract (starter point)

The following OpenAPI document is the specification of the API that will be referred to in the subsequent examples.

openapi: 3.0.0
info:
  title: Example API
  version: 1.0.0
servers:
  - url: http://localhost:1234
paths:
  /example-headers:
    get:
      responses:
        '200':
          description: This is the response for the HTTP status code 200.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/SuccessResponse'
        '500':
          description: This is the response for the HTTP status code 500.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'

  /example-query:
    get:
      parameters:
        - name: foo
          in: query
          schema:
            type: string
      responses:
        '200':
          description: This is the response for the HTTP status code 200.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/SuccessResponse'
        '500':
          description: This is the response for the HTTP status code 500.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'

  /example-request-body:
    post:
      requestBody:
        content:
          'application/json':
            schema:
              type: object
              properties:
                property1:
                  type: string
                property2:
                  type: string
      responses:
        '200':
          description: This is the response for the HTTP status code 200.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/SuccessResponse'
        '500':
          description: This is the response for the HTTP status code 500.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'

components:
  schemas:
    SuccessResponse:
      type: object
      properties:
        data:
          type: string
    ErrorResponse:
      type: object
      properties:
        error:
          type: integer

Examples document structure

Several options are provided for the user to specify how to trigger a specific HTTP response code through an HTTP request. All of them share the use of examples field.

Object Examples

The object key represents an HTTP response code, and the value is an object with the following properties:

• summary (optional): Explanatory text.

• value (required): Payload value to be sent. It must follow the same format as the associated schema, according to the OpenAPI specification:

  *“In all cases, the example value is expected to be compatible with the type schema of its associated value. Tooling implementations MAY choose to validate compatibility automatically, and reject the example value(s) if incompatible.”*

  https://spec.openapis.org/oas/v3.1.0#example-object

Placement

1. parameters (https://spec.openapis.org/oas/v3.1.0#parameter-object)

   1. header
      The user specifies the header to be added to the request, with the goal of having the API interpret it and return the expected response code. The header name, its value, and interpretation are on the user's side.

      In this example, Prefer: statusCode=200 and Prefer: statusCode=500 headers are used.

      get:
        parameters:
          - name: Prefer          
            in: header
            schema:
              type: string
            examples:
              '200':
                summary: Expects a 200 status code from the API
                value: 'statusCode=200'
              '500':
                summary: Expects a 500 status code from the API
                value: 'statusCode=500'

   2. query

      get:
        parameters:
          - name: foo          
            in: query
            schema:
              type: string
            examples:
              '200':
                summary: This is a correct parameter example
                value: bar
              '500':
                summary: This is a wrong parameter example
                value: xxx

2. requestBody (https://spec.openapis.org/oas/v3.1.0#request-body-object)

   post:
     requestBody:
       content:
         'application/json':
           examples:
             '200':
               summary: This is a correct parameter example
               value:
                 property1: 'foo'
                 property2: 'bar'
             '500':
               summary: This is a wrong parameter example
               value: 'xxx'

Complete example

openapi: 3.0.0
info:
  title: Example API
  version: 1.0.0
paths:
  /example-headers:
    get:
      parameters:
        - name: Prefer
          in: header
          schema:
            type: string
          examples:
            '200':
              summary: Expects a 200 status code from the API
              value: 'statusCode=200'
            '500':
              summary: Expects a 500 status code from the API
              value: 'statusCode=500'
      responses:
        '200':
          description: This is the response for the HTTP status code 200.
        '500':
          description: This is the response for the HTTP status code 500.

  /example-query:
    get:
      parameters:
        - name: foo
          in: query
          schema:
            type: string
          examples:
            '200':
              summary: This is a correct parameter example
              value: bar
            '500':
              summary: This is a wrong parameter example
              value: xxx
      responses:
        '200':
          description: This is the response for the HTTP status code 200.
        '500':
          description: This is the response for the HTTP status code 500.

  /example-request-body:
    post:
      requestBody:
        content:
          'application/json':
            examples:
              '200':
                summary: This is a correct parameter example
                value:
                  property1: 'foo'
                  property2: 'bar'
              '500':
                summary: This is a wrong parameter example
                value: 'xxx'
      responses:
        '200':
          description: This is the response for the HTTP status code 200.
        '500':
          description: This is the response for the HTTP status code 500.