You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+73-22Lines changed: 73 additions & 22 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,6 +10,7 @@ This project is a tool designed to test a microservice system. The most novel pa
10
10
-[Dependencies](#dependencies)
11
11
-[Installation](#installation)
12
12
-[Preparation](#preparation)
13
+
-[Integration with other tools](#Integration-with-other-tools)
13
14
-[Usage](#usage)
14
15
-[Configuration](#configuration)
15
16
-[License](#license)
@@ -53,10 +54,51 @@ To install the tool, follow these steps:
53
54
## Preparation
54
55
55
56
1. Prepare the OpenAPI specification file for the system under test. The tool supports OpenAPI 3 by default.
56
-
2. Prepare a protobuf file for all RPC which internal services use.
57
+
2. Prepare a protobuf file for all RPC which internal services use, or the OpenAPI specification file for all REST APIs which internal services use.
57
58
- We use [protoc-gen-openapi](https://github.com/google/gnostic/tree/main/cmd/protoc-gen-openapi) to convert protobuf to openapi.
58
59
- You should annotate the proto file, and you can refer to this [issue](https://github.com/google/gnostic/issues/412).
59
-
3. Ensure that your service includes a unique trace ID in the headers of each response. You can configure the header key in the settings.
60
+
- If you use OpenAPI spec, you should ensure that the OpenAPI spec is in the same format as a file generated by the tool (including `operationId`). Note that you must add tag `APIType_HTTP` to all HTTP APIs, as we treat internal service APIs as RPC APIs by default. You can refer to the example at the end of this section.
61
+
3. Optionally, you can provide API dependencies to enhance fuzzing, including system API dependencies and internal service API dependencies (a map from service name to single service dependencies). We support parse dependency file generated by RESTler now, you can refer to its [Github Repo](https://github.com/microsoft/restler-fuzzer) for more information.
62
+
4. Ensure that your service includes a unique trace ID in the headers of each response. You can configure the header key in the settings.
63
+
64
+
Internal service API Spec example:
65
+
```yaml
66
+
/v1/cart/add:
67
+
post:
68
+
tags:
69
+
- CartService
70
+
- APIType_gRPC
71
+
operationId: CartService_AddItem
72
+
requestBody:
73
+
content:
74
+
application/json:
75
+
schema:
76
+
$ref: '#/components/schemas/AddItemRequest'
77
+
required: true
78
+
responses:
79
+
"200":
80
+
description: OK
81
+
content:
82
+
application/json:
83
+
schema:
84
+
$ref: '#/components/schemas/Empty'
85
+
default:
86
+
description: Default error response
87
+
content:
88
+
application/json:
89
+
schema:
90
+
$ref: '#/components/schemas/Status'
91
+
```
92
+
93
+
## Integration with Other Tools
94
+
95
+
### Restler
96
+
97
+
Restler is a tool designed to generate REST API test cases from OpenAPI specifications. During the compilation process, it produces a dependency file for the system's APIs. Our tool can parse this dependency file to enhance the fuzzing process. For more details, refer to the `--dependency-file` and `--dependency-file-type` options.
98
+
99
+
### CoSREST
100
+
101
+
CoSREST is a tool that leverages Large Language Models (LLMs) to generate REST API test cases from OpenAPI specifications. It enhances test case generation by using LLMs to make decisions on input space partitioning (e.g., parameter generation), which improves both efficiency and effectiveness. Our tool supports parsing the space partition file generated by CoSREST and converts it into a fuzz value dictionary. This helps the fuzzer create more diverse and valid test cases. For more details, refer to our [script](scripts/CoSREST/convert_sisp_to_dict.py)
60
102
61
103
## Usage
62
104
@@ -88,32 +130,41 @@ In addition, we provide vscode tasks to use the tool. You can build, run and deb
88
130
89
131
The tool can be configured using command-line arguments. The following options are available:
90
132
91
-
- `--openapi-spec`: Path to the OpenAPI specification file.
92
-
- `--server-base-url`: Base URL of the server to test.
93
-
- `--internal-service-openapi-spec`: Path to the internal service OpenAPI specification file.
94
-
- `--trace-backend-url`: URL of the trace backend.
95
-
- `--trace-backend-type`: Type of the trace backend. Currently only supports 'Jaeger'.
96
-
- `--fuzzer-type`: Type of fuzzer to use (e.g., Basic).
97
-
- `--fuzzer-budget`: Time budget for fuzzing (e.g., 30s).
- `--output-dir`: Directory to save the output reports.
100
133
- `--config-file`: Path to the config file. If an argument is provided in both the config file and command line, the config file argument will be used.
101
134
- `--dependency-file`: Path to the dependency file generated by other tools or manually.
102
-
- `--dependency-file-type`: Type of the dependency file. Currently only supports 'Restler'.
103
-
- `--extra-headers`: Extra headers to be added to the request, in the format of stringified JSON, e.g., '{"header1": "value1", "header2": "value2"}'. If you want to add complex logic or modify the parameters and body, you can use the `--http-middleware-script` argument.
104
-
- `--fuzz-value-dict-file`: Path to the file containing the dictionary of fuzz values, in the format of a JSON list. Each element in the list is a dictionary with two key-value pairs, one is `name` (value is of type string) and the other is `value` (value can be any JSON).
105
-
- `--log-to-file`: Should log to file.
106
-
- `--trace-id-header-key`: The response header key to be used for trace ID.
107
-
- `--http-middleware-script`: Path to the script file that contains the HTTP middleware functions, see [HTTP Middleware Script](#about-http-middleware-script).
108
-
109
-
You can also use a configuration file with `--config-file` option to set the options. The configuration file should be in JSON format, we provide an example configuration file [here](configs/config.json).
110
-
111
-
In addition, fortoken, private key, or other sensitive information, you can use environment variables. You can set an environment variable with the same name as thatin configuration file (but in uppercase) to override the value in the configuration file. For example, you can set`SERVER_BASE_URL`in`.env` file:
135
+
- `--dependency-file-type`: Type of the dependency file. Currently only supports 'Restler'. Required if `--dependency-file` is provided.
136
+
- `--enable-energy-operation`: Enable energy (priority) of test operations. If true, energy affects the test operation selection when extending the test scenario.
137
+
- `--enable-energy-scenario`: Enable energy (priority) of test scenarios. If true, energy affects the test scenario selection when starting a new test loop.
138
+
- `--extra-headers`: Extra headers to be added to the request, in the format of stringified JSON, e.g., `{"header1": "value1", "header2": "value2"}`.
139
+
- `--fuzz-value-dict-file`: Path to the file containing the dictionary of fuzz values, in JSON format. Each element is a dictionary with `name` (string) and `value` (any JSON).
140
+
- `--fuzzer-budget`: The maximum time the fuzzer can run, in seconds (default: 5).
141
+
- `--fuzzer-type`: Type of the fuzzer. Currently only supports 'Basic' (default: Basic).
142
+
- `--http-client-dial-timeout`: Timeout for the HTTP client dial, in seconds (default: 30).
143
+
- `--http-middleware-script`: Path to the script file that contains the HTTP middleware functions.
144
+
- `--internal-service-api-dependency-file`: Path to the internal service API dependency file generated by other tools or manually. It should be a map of service name to a list of API dependencies.
145
+
- `--internal-service-openapi-spec`: Path to the internal service OpenAPI specification file (required).
- `--log-to-file`: Whether to log to a file (default: false).
148
+
- `--max-ops-per-scenario`: Maximum number of operations to execute in each scenario (default: 1).
149
+
- `--max-allowed-operation-case-executed-count`: Maximum number of times a test operation case can be executed (default: 14).
150
+
- `--max-allowed-operation-cases`: Maximum number of test operation cases in the queue of an API method (default: 7).
151
+
- `--max-allowed-scenario-executed-count`: Maximum number of times a test scenario can be executed (default: 6).
152
+
- `--max-allowed-scenarios`: Maximum number of test scenarios in the queue (default: 114).
153
+
- `--openapi-spec`: Path to the OpenAPI specification file (required).
154
+
- `--output-dir`: Directory to save the output reports (default: ./output).
155
+
- `--server-base-url`: Base URL of the server to test (default: https://www.example.com).
156
+
- `--trace-backend-type`: Type of the trace backend. Currently supports 'Jaeger' and 'Tempo' (default: Jaeger).
157
+
- `--trace-backend-url`: URL of the trace backend (required).
158
+
- `--trace-id-header-key`: The response header key to be used for trace ID (default: X-Trace-Id).
159
+
- `--use-internal-service-api-dependency`: Indicates whether to use the internal service API dependency. If true, the internal service API dependency will be used to enhance the external service API dependency.
160
+
161
+
You can also use a configuration file with the `--config-file` option to set the options. The configuration file should be in JSON format. We provide an example configuration file [here](configs/config.json).
162
+
163
+
For sensitive information like tokens or private keys, you can use environment variables. Set an environment variable with the same name as the configuration file key (in uppercase) to override the value in the configuration file. For example, you can set `SERVER_BASE_URL` in a `.env` file:
112
164
```sh
113
165
SERVER_BASE_URL=http://localhost:6789
114
166
```
115
167
116
-
117
168
## About HTTP Middleware Script
118
169
119
170
The HTTP Middleware Script allows you to intercept and modify HTTP requests and responses using a Starlark script. The script can modify headers, path parameters, query parameters, and the body of the request.
log.Err(err).Msgf("[main] Failed to parse internal service API dependency map file, path: %s", config.GlobalConfig.InternalServiceAPIDependencyFilePath)
0 commit comments