howdoesschemathesiswork.org

Schemathesis internals overview

High-level overview of the execution path of Schemathesis, which is not easy to piece together and not well-documented.

Test case generation

1. Runner class runs _run_tests() with maker=schema.get_all_tests and template=network_test
2. _run_tests runs maker (which is schema.get_all_tests) with func=network_test
3. Schema get_all_tests takes every API operation and calls create_test with operation and test=func=network_test
4. create_test:
   1. turns operation into a Hypothesis data generation strategy by calling operation.as_strategy
   2. Wraps the test=network_test function in a hypothesis.given annotation with case=strategy as kwarg
5. _run_tests receives the Hypothesis test from create_test and calls run_test on the hypothesis test
6. run_test runs the hypothesis test (which wraps network_test) and extracts some data from it
7. network_test is called by Hypothesis with the case argument populated by Hypothesis given (???) using the strategy defined in create_test
8. network_test calls _network_test which
   1. calls case.call() which sends the HTTP request
   2. checks the response against the defined checks

Strategy generation

1. operation.as_strategy calls schema.get_case_strategy(operation) and then applies hooks to it.
2. This forwards to the actual method, which generates the actual case.
   1. Calls generate_parameter for path params, headers, cookies, and query params through generate_parameter
   2. Fills in the body by selecting a schema for the body and passing it to _get_body_strategy then applying hooks to the strategy
3. _get_body_strategy prepares the jsonschema and calls the strategy_factory passed to it, which is usually make_positive_strategy. It also adds the case of not-set if the body is optional.
4. make_positive_strategy defers to hypothesis_jsonschema.from_schema

Schemathesis stateful

Seemingly based purely on OpenAPI links, and mostly undocumented. Not useful for Onweer, as links are not often specified.

get_case_strategy

Method which turns an APIOperation into a strategy producing Case that can be executed.

Any parameters which are filled in in the APIOperation object will be kept and supplemented with randomly generated inputs where necessary.

Except for the body, which if it is filled in will be used exactly as it is. It is not supplemented.

• Major limitation, would be nice to resolve but not urgent (request bodies are not too essential, hopefully)

APIOperation

APIOperation is generic in the type of the parameters so it can be reused between openapi and graphql.

Current working theory: the concrete type of the P parameter is either OpenAPIParameter or its child class OpenAPI30Parameter (or the same with openapi 2.0 but we’re not using that). The differences between these two don’t seem to matter too much, so we can just work on OpenAPIParameter if this is the correct class.

This parameter class seems to have a relatively comprehensive and usable interface, should be useful for matching data dependencies. Need to look at how RESTler and WuppieFuzz do this matching.