-
Notifications
You must be signed in to change notification settings - Fork 16
/
dredd.javascript.txt
302 lines (243 loc) · 16.9 KB
/
dredd.javascript.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
DREDD
ALTERNATIVES ==> # - dredd (prefered depending on case):
# - OpenAPI 2.0/3.0, API blueprint
# - high-profile, nice CLI
# - one test per OPERATION + 2** RESP
# - parameters values: PARAM.x-example|default|enum or dummy
# - validates: status code, RESP.headers keys, RESP.schema or RESP.examples.MIME
# - URL: manually passed
# - swagger-test-templates (prefered depending on case):
# - OpenAPI 2.0
# - generates tests from templates
# - one test per OPERATION + RESP + content-type + parameter value
# - parameters values: manually passed
# - can pass SECURITY_DEF values manually in .env file
# - validates: status code, RESP.schema
# - URL: uses SPEC
# - brunty cigar (prefered when needs fast and simple):
# - no specification: simple OBJ_ARR with only url|status|content|content-type
# - simplistic
VERSION ==> #11.2.19
#Automated integration testing (assertions on HTTP request|responses on live server) using
#an API specification file.
#Supports API Blueprint, OpenAPI 2.0/3.0
INSTALLATION ==> #Unless npm install --no-optional is used, drafter (C++ library) will be used, i.e. will compile
#The library is used for parsing API Blueprint, i.e. not needed if only used with OpenAPI.
/=+===============================+=\
/ : : \
)==: REQUESTS :==(
\ :_______________________________: /
\=+===============================+=/
REQUESTS ==> #Fires the following requests:
# - CONF.endpoint (not SERVER)
# - 1 request per pair of OPERATION + 2** RESP
# - all PARAM with PARAM.x-example (priority) or PARAM.default or first PARAM.enum (uses its value)
# - if PARAM.required, one of them must be defined
# - for "in" "body|formData", a dummy value can be generated based on the SCHEMA
# - Content-Type [C] according to SPEC|OPERATION.consumes (even charset must match) (only for JSON)
#Since it is performed against live database, state might persist, i.e. tests won't be
#isolated unless hooks do cleanup.
ASSERTS ==> #Against the response:
# - STATUS_CODE
# - RESP.headers:
# - Content-Type [S] according to SPEC|OPERATION.produces (even charset must match) (only for JSON)
# or RESP.examples.MIME
# - specified headers names are all present
# - RESP.schema SCHEMA (priority) or RESP.examples.MIME VAL (if OBJ, same keys but not values) (for body)
NOT CHECKED|USED (IMPORTANT) ==> # - different responses according to different requests
# - sequences of requests|responses:
# - can fix it by using specification order, and using several SPEC_FILE
# - error responses:
# - only 2** responses are checked
# - other status codes always error, even if matching the specification
# - endpoints are skipped if either:
# - there is at least one non-2** RESP, unless there is another RESP that is 2**
# - there is a "default" STATUS_CODE
# - can fix by setting TRANS.skip = false in before* hooks
# - RESP.headers values
# - can fix with after* hooks
# - SECURITY_DEFS|USES
# - can fix with CONF.header|user or before* hooks
NOT CHECKED|USED (LESS IMPORTANT) # - SCHEMA.collectionFormat "ssv|tsv|pipes" (but "csv|multi" are used)
==> # - example PARAM value is only partially checked against [SMALL_]SCHEMA during parsing (type|enum)
# - RESP.schema.multipleOf|minProperties|maxProperties
/=+===============================+=\
/ : : \
)==: CONF :==(
\ :_______________________________: /
\=+===============================+=/
CONF #Can be:
# - ./dredd.yml
# - dredd --config CONF_FILE
# - new Dredd(OPTS), using OPTS.options CONF
# - CLI flags
#They are all dasherized
#When not in CLI:
# - ARR are used when "can be used several times"
# - OBJ is used for --custom
dredd init #Creates a CONF file
/=+===============================+=\
/ : : \
)==: RUN :==(
\ :_______________________________: /
\=+===============================+=/
dredd [SPEC_FILE] [SERVER_URL] #SPEC_FILE is CONF.blueprint, e.g. openapi.yml
-p SPEC_FILE|URL|GLOB #Additional specification files. Can be used several times.
--path SPEC_FILE|URL|GLOB #Unless CLI, required (even empty array)
--endpoint SERVER_URL #
-g STR
--server STR #Run a shell command (to start server), then kills process after testing.
--server-wait NUM #Wait NUMsec (def: 3) before testing.
-j 'cwd:DIR'
--custom 'cwd:DIR' #To resolve spec files and hook files (def: process.cwd())
-r REPORTER #Among: dot, nyan, markdown, html, xunit, apiary (SaaS)
--reporter REPORTER #Can be used several times.
-j 'VAR:VAL,...'
--custom 'VAR:VAL,...' #Custom options, mostly for reporters
-o FILE #Required by REPORTER markdown|html|xunit
--output FILE #Can be used several times.
-l #Among 'silent|error|warn|skip|request|hook|expected|actual|complete|fail|pass|test', 'info' (def), 'verbose|debug|silly'
--loglevel STR #Note that test failure is just 'info'
-d
--details #Show detailed information for passed tests, not just failed ones
-e
--inline-errors #Display errors as they happen instead of all at the end
--no-color #
-s #Sort tests by HTTP method: CONNECT, OPTIONS, POST, GET, HEAD, PUT, PATCH, DELETE, TRACE
--sorted #Otherwise, sort tests by specification order
-n
--names #Only list all the tests TEST_NAME
-y
--dry-run #
-h 'VAR:VAL'
--header 'VAR:VAL' #Sets HTTP request headers. Can be used several times.
-u 'USER:PASSWORD'
--user 'USER:PASSWORD' #HTTP Basic authentication
-x 'TEST_NAME'
--only 'TEST_NAME' #Only run those tests. Can be used several times.
-m 'METHOD'
--method 'METHOD' #Only run tests with this HTTP method. Can be used several times.
/=+===============================+=\
/ : : \
)==: PROGRAMMATIC :==(
\ :_______________________________: /
\=+===============================+=/
new Dredd(OPTS) #DREDD
#OPTS:
# - server STR: instead of CONF.endpoint
# - options CONF
# - emitter EVENTEMITTER with following events:
# - start(CONF)
# - end()
# - 'test start'(TEST)
# - 'test pass|fail|error|skip'(TEST)
# - data.PATH STR: like CONF.files but as a string
DREDD.run(FUNC(ERROR, STATS)) #ERROR is not test failure, but parsing or conf failure
STATS.result.tests #NUM
STATS.result.
failures|errors|passes|skipped #NUM
STATS.result.start|end #DATE
STATS.result.duration #NUM
/=+===============================+=\
/ : : \
)==: HOOKS :==(
\ :_______________________________: /
\=+===============================+=/
dredd
-f HOOKS_FILES_GLOB
--hookfiles HOOKS_FILES_GLOB #
-a
--language STR #Language of the hooks files, among 'nodejs' (def), 'ruby|python|php|perl|go|rust'
--hooks-worker-handler-host|port #Hooks server
#Def: '127.0.0.1' and 61321
--hooks-worker-connect- #Connection retries to hooks server
retry|timeout NUM #Def: 500|1500
--hooks-worker-after-connect-wait #Wait NUMms after connecting to hooks server, before starting testing
NUM #Def: 100
--hooks-worker-timeout NUM #Timeout to start hooks server
#Def: 5000
--hooks-worker-term- #Disconnection retries to hooks server
retry|timeout NUM #Def: 500|5000
/=+===============================+=\
/ : : \
)==: HOOK FILES :==(
\ :_______________________________: /
\=+===============================+=/
SHARED STATE ==> #Simply re-use a variable in parent scope
ERROR ==> #Either throw an error (including through an assertion library) or call FUNC2(ERROR)
HOOKS.*(..., FUNC2([ERROR])) #All hooks can be async by adding a FUNC2 parameter
HOOKS.beforeAll|afterAll
(FUNC(TRANS_ARR[, FUNC2])) #Fired before|after all tests
HOOKS.beforeEach(FUNC(TRANS)) #Fired for each test, before HTTP request|response
HOOKS.beforeEachValidation
(FUNC(TRANS)) #Fired for each test, after HTTP request|response, before assertions
HOOKS.afterEach(FUNC(TRANS)) #Fired for each test, after assertions
HOOKS.before|beforeValidation|
after('TEST_NAME', FUNC(TRANS)) #Same with a specific test
HOOKS.log(VAL) #To use instead of console.log(VAL)
HOOKS.configuration #CONF
HOOKS.transactions.TEST_NAME #TRANS
/=+===============================+=\
/ : : \
)==: TRANSACTION :==(
\ :_______________________________: /
\=+===============================+=/
TRANS #HTTP request|response
#Changing actually modifies the HTTP request|response, the assertion and the results
TRANS.id #'METHOD (STATUS_CODE) /PATH'
TRANS.name #'TEST_NAME', i.e. 'TAG > /PATH > OPERATION.summary > RESP.STATUS_CODE [> RESP.produces]'
TRANS.origin.filename #'SPEC_FILE'
TRANS.origin.apiName #STR, e.g. SPEC.title
TRANS.origin.resourceGroupName #STR, e.g. TAG
TRANS.origin.resourceName #STR, e.g. /PATH
TRANS.origin.actionName #STR, e.g. OPERATION.summary
TRANS.origin.exampleName #STR, e.g. STATUS_CODE
TRANS.protocol #'http[s]:'
TRANS.host #STR
TRANS.port #NUM
TRANS.fullPath #STR. After URI templating.
TRANS.request #HTTP request
TRANS.request.method #STR
TRANS.request.url #'/PATH'
TRANS.request.headers #OBJ
TRANS.request.body #STR
TRANS.request.bodyEncoding #'ENCODING'
TRANS.expected #Assertions on HTTP response
TRANS.expected.statusCode #STR
TRANS.expected.headers #OBJ
TRANS.expected.body #STR
TRANS.expected.bodySchema #OBJ (JSON schema)
TRANS.real #HTTP response
TRANS.real.statusCode #STR
TRANS.real.headers #OBJ
TRANS.real.body #STR
TRANS.skip #BOOL. Set to true to skip test
TRANS.fail #BOOL|STR. Set to 'ERROR' to make test fail.
TRANS.test #TEST. Testing session.
TEST.start|end #DATE
TEST.startedAt #NUM (Unix timestamp)
TEST.duration #NUM (in ms)
TEST.status #'pass|fail|skip'
TEST.valid #BOOL
TEST.message #All output concatenated as single STR
TEST.title|request|actual|expected
|origin #Same as TRANS.id|request|real|expected|origin
TRANS|TEST.results #RESULTS
RESULTS.general.results #GERROR_ARR
RESULTS.statusCode|headers|body #GOUTPUT
GOUTPUT.results #GERROR_ARR
GOUTPUT.expectedType|realType #'MIME'
GOUTPUT.validator #STR
GOUTPUT.rawData #GJSONSCHEMA_ARR or GTEXTDIFF
GJSONSCHEMA.length #NUM. Number of errors
GJSONSCHEMA['NUM'].property #JSON_POINTER_ARR to the errored property
GJSONSCHEMA['NUM'].propertyValue #VAL
GJSONSCHEMA['NUM'].attributeName #JSON schema proeprty name STR
GJSONSCHEMA['NUM'].attributeValue #VAL
GJSONSCHEMA['NUM'].message #'ERROR'
GTEXTDIFF #STR looking like a GNU diff.
GERROR.pointer #JSON_POINTER
GERROR.severity #STR
GERROR.message #'ERROR'