Skip to content

Commit 797e6f4

Browse files
arcuri82arcuri82
committed
documentation for Overlay
1 parent 5cf86a8 commit 797e6f4

2 files changed

Lines changed: 190 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,8 @@ Note, since version 4.0.0, now _EvoMaster_ by default also creates an interactiv
138138

139139
* _Schema_: REST APIs must provide a schema in [OpenAPI format](https://www.openapis.org/). We support versions _2.0_, _3.0_ and _3.1_. Unfortunately, support for version _3.2_ is currently on hold due to [swagger-parser](https://github.com/swagger-api/swagger-parser/issues/2248).
140140

141+
* _OAI Overlay_: for REST APIs, we natively support [Overlay](https://github.com/OAI/Overlay-Specification) transformations. This is needed for testers that want to add test data via "examples" entries without modifying the OpenAPI scheme directly, [see documentation](./docs/overlay.md).
142+
141143
* _Output_: the tool generates _JUnit_ (version 4 or 5) tests, written in either _Java_ or _Kotlin_, as well as test suites in _Python_ and _JavaScript_. For a complete list, see the documentation for the CLI parameter [--outputFormat](docs/options.md).
142144
Some examples are: PYTHON_UNITTEST, KOTLIN_JUNIT_5, JAVA_JUNIT_4 and JS_JEST.
143145
Note that the generated tests rely on third-party libraries (e.g., to make HTTP calls).

‎docs/overlay.md‎

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
# Adding "Examples" with OAI Overlay
2+
3+
## Intro
4+
5+
EvoMaster has native support for [OAI Overlay](https://github.com/OAI/Overlay-Specification) files.
6+
This is done with [option parameters](./options.md) such as `--overlay`, `--overlayFileSuffixes` and `--overlayLenient`.
7+
For example, EvoMaster can be run with:
8+
9+
`evomaster --schema $SCHEMA --overlay $OVERLAY`
10+
11+
with `$OVERLAY` specifying where the Overlay file(s) is/are located.
12+
13+
14+
> Goal: The main reason to have native Overlay support in a fuzzer is to be able to provide `examples` entries to help the fuzzing.
15+
16+
Most fuzzers are able to use data in the `example` and `examples` fields of an OpenAPI schema.
17+
However, having testers manually adding those entries in the schema for their testing needs (e.g., specific combinations of values, or ids of data in the databases of their testing environment) is not viable.
18+
What if a new version of the schema come?
19+
That would require manually re-adding all the example entries to the new version.
20+
Putting such data in an Overlay file solves the problem.
21+
22+
## Problem Statement
23+
24+
In black-box testing of REST APIs, where the only information available is an OpenAPI schema and the dynamic feedback/responses from interacting with the tested API, a fuzzer can struggle to generate good inputs in a reasonable time.
25+
Hence, users can provide "hints" to the fuzzer, e.g., ids existing in the database, to help the fuzzing process.
26+
27+
Currently, most fuzzers already can support this, as they can use `example` and `examples` entries in the OpenAPI schema when generating input data.
28+
The problem, though, is that this would require to modify the schema with test data.
29+
This is not viable in the long run.
30+
As soon as there is a new update in the schema, all the examples would need to be manually re-added to the new schema.
31+
The alternative to use a separated, custom format to provide input data hints to the fuzzer is not a particularly good option.
32+
It would create a "vendor-lockin" for each specific fuzzer.
33+
Also, IDE support for such format (e.g., autocomplete and validation) would be necessarily limited, besides the time needed to invest in learning a new custom format.
34+
35+
A possible solution is to use [OAI Overlay](https://spec.openapis.org/overlay/latest.html).
36+
It provides a way to do define transformation rules that can be applied to an existing OpenAPI schema.
37+
Such transformations can be used to add `examples` entries in the OpenAPI schema of the tested API.
38+
If it works, it would solve the addressed problem, as Overlay is a standard, used also outside of software testing needs.
39+
The transformation could be applied on an OpenAPI schema before giving it as input to the fuzzer (e.g., using any existing Overlay merge tool).
40+
Alternatively, if a fuzzer has "native" support for Overlay (and EvoMaster does!), then it is just a matter of giving the Overlay file as input (and EvoMaster will automatically apply the transformation internally).
41+
42+
43+
44+
## Writing Overlay Files
45+
46+
Overlay is a relative new standard.
47+
Documentation, examples, and IDE-support is still at early stages.
48+
But we can expect things to get better in the next coming years.
49+
This means that, today, we cannot provide you precise instructions or suggestions on how those files should be written (e.g., any good IDE-plugin that support Overlay files).
50+
If there is any error when applying an Overlay file to an existing OpenAPI schema, then EvoMaster will crash and give you an error message.
51+
This is a start, but not as good as having an IDE that validates the files while writing them.
52+
53+
Also note that, as it is an official standard, it is quite likely that your LLM of choice might have been trained on it, and being able to write some draft-template of Overlay transformations for you.
54+
55+
Another things to consider is that you should avoid adding `example` (which is deprecated in OpenAPI) but rather used _named_ `examples`.
56+
Names of examples should be unique.
57+
Names re-used in different parameters (query and path) or body payloads will be treated as the same example combination by EvoMaster.
58+
For example, let's say you have a `POST /api/foo?x=1&y=2` with body `B`.
59+
If you want to test a specific combination of `x`, `y` and `B`, you can create `examples` entries with the same name.
60+
61+
You can search online on tips and instructions on how to write Overlay files.
62+
Here, we provide an example to use as reference.
63+
Consider the following simple API with 1 POST endpoint, having 2 query parameters `x` and `y`, and a body payload:
64+
65+
```
66+
openapi: 3.1.0
67+
info:
68+
title: Simple API
69+
version: 1.0.0
70+
servers:
71+
- url: https://api.example.com/v1
72+
paths:
73+
/api:
74+
post:
75+
parameters:
76+
- name: x
77+
in: query
78+
schema:
79+
type: string
80+
- name: y
81+
in: query
82+
schema:
83+
type: string
84+
requestBody:
85+
required: true
86+
content:
87+
application/json:
88+
schema:
89+
type: object
90+
properties:
91+
id:
92+
type: string
93+
name:
94+
type: string
95+
responses:
96+
'200':
97+
description: Successful response
98+
```
99+
100+
Now, consider adding 2 named examples `foo` and `bar`.
101+
`foo` adds an entry just for the parameter `y`, whereas `bar` adds to the combination `x`, `y` and body.
102+
This can be represented with the following Overlay definition:
103+
104+
```
105+
overlay: 1.1.0
106+
info:
107+
title: Add Examples to API
108+
version: 1.0.0
109+
actions:
110+
- target: '$.paths["/api"].post.parameters[?(@.name=="y")]'
111+
description: Add foo example to query parameter y
112+
update:
113+
examples:
114+
foo:
115+
value: Foo
116+
117+
- target: '$.paths["/api"].post.parameters[?(@.name=="x")]'
118+
description: Add bar example to query parameter x
119+
update:
120+
examples:
121+
bar:
122+
value: Bar
123+
124+
- target: '$.paths["/api"].post.parameters[?(@.name=="y")]'
125+
description: Add bar example to query parameter y
126+
update:
127+
examples:
128+
bar:
129+
value: Bar
130+
131+
- target: '$.paths["/api"].post.requestBody.content["application/json"]'
132+
update:
133+
examples:
134+
bar:
135+
value:
136+
id: "bar-id-123"
137+
name: "Bar Name Example"
138+
```
139+
140+
When these 4 transformation actions are applied, the resulting schema will be:
141+
142+
```
143+
openapi: 3.1.0
144+
info:
145+
title: Simple API
146+
version: 1.0.0
147+
servers:
148+
- url: https://api.example.com/v1
149+
paths:
150+
/api:
151+
post:
152+
parameters:
153+
- name: x
154+
in: query
155+
schema:
156+
type: string
157+
examples:
158+
bar:
159+
value: Bar
160+
- name: y
161+
in: query
162+
schema:
163+
type: string
164+
examples:
165+
bar:
166+
value: Bar
167+
foo:
168+
value: Foo
169+
requestBody:
170+
required: true
171+
content:
172+
application/json:
173+
schema:
174+
type: object
175+
properties:
176+
id:
177+
type: string
178+
name:
179+
type: string
180+
examples:
181+
bar:
182+
value:
183+
id: "bar-id-123"
184+
name: "Bar Name Example"
185+
responses:
186+
'200':
187+
description: Successful response
188+
```

0 commit comments

Comments
 (0)