Skip to content

Commit b6c55b0

Browse files
sabbansabban
committed
Merge branch 'improve/traefik' of github.com:crowdsecurity/crowdsec-docs into improve/traefik
2 parents 5a2889a + 374de5a commit b6c55b0

79 files changed

Lines changed: 4463 additions & 1469 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎crowdsec-docs/docs/appsec/api_validation.md‎

Lines changed: 233 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,233 @@
1+
---
2+
id: api_validation
3+
title: OpenAPI Schema Validation
4+
sidebar_position: 5
5+
---
6+
7+
The Application Security Component can validate incoming HTTP requests against an [OpenAPI 3](https://swagger.io/specification/) schema you provide. Requests that do not conform to the schema (unknown route, unexpected method, missing or malformed parameters, invalid request body, missing/invalid authentication credentials, …) can be rejected before they ever reach the protected application.
8+
9+
This is a positive-security model layered on top of the negative-security model implemented by the WAF rules: instead of describing what an attacker looks like, you describe what a valid client looks like and reject everything else.
10+
11+
## How it works
12+
13+
Schema validation is exposed through the [hooks](hooks.md) system:
14+
15+
- An `on_load` hook loads one or more OpenAPI schemas at startup, each under a short string `ref`.
16+
- A `pre_eval` hook calls `ValidateRequestWithSchema(ref)` to validate the current request. The function returns `true` when the request is valid, `false` otherwise.
17+
- When validation fails, structured details about the failure are published to `hook_vars` so the same hook (or a later one) can build a meaningful drop reason, enrich an event, etc.
18+
19+
## Storing schemas
20+
21+
Schemas are loaded from the `schemas/` subdirectory of the CrowdSec [`data_dir`](/configuration/crowdsec_configuration.md#data_dir) (typically `/var/lib/crowdsec/data/schemas/`).
22+
23+
Filenames passed to the loader **must be relative** to that directory.
24+
25+
```
26+
/var/lib/crowdsec/data/schemas/
27+
├── users-api.yaml
28+
└── billing-api.yaml
29+
```
30+
31+
OpenAPI 3.0 and Swagger schemas in YAML or JSON are both accepted.
32+
33+
## Loading schemas (`on_load`)
34+
35+
Loading is done from an `on_load` hook using one of two helpers:
36+
37+
| Helper | Description |
38+
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
39+
| `LoadAPISchemaWithName(ref str, filename str)` | Load `<data_dir>/schemas/<filename>` and register it under `ref`, with default policies. |
40+
| `LoadAPISchemaWithOptions(ref str, filename str, opts map)` | Same as above, but lets you override per-schema policies (see below). |
41+
| `RegisterAPISchemaBodyDecoder(content_type str, decoder str)` | Enable a non-default body decoder for a given Content-Type (see below). |
42+
43+
`ref` is an arbitrary string you choose; you will use it later in `pre_eval` to refer to this schema. A schema name cannot be loaded twice.
44+
45+
```yaml
46+
name: custom/my-appsec-config
47+
inband_rules:
48+
- crowdsecurity/base-config
49+
on_load:
50+
- apply:
51+
- LoadAPISchemaWithName("users_api", "users-api.yaml")
52+
- LoadAPISchemaWithName("billing_api", "billing-api.yaml")
53+
```
54+
55+
If the schema file is missing, malformed, or not a valid OpenAPI 3 document, the datasource will fail to start and log the underlying error.
56+
57+
### Schema options
58+
59+
`LoadAPISchemaWithOptions` accepts the following keys, all strings:
60+
61+
| Key | Values | Default | Effect |
62+
| -------------------------------- | ----------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
63+
| `on_route_not_found` | `drop` / `ignore` | `drop` | What to do when no path in the schema matches the request URL. |
64+
| `on_method_not_allowed` | `drop` / `ignore` | `drop` | What to do when a path matches but the method does not (e.g. schema only declares `GET`, request is `POST`). |
65+
| `on_unsupported_security_scheme` | `drop` / `ignore` | `drop` | What to do when an unsupported security schema is encountered (`openid`, `oauth2`). If `ignore`, the security schema will not be validated when checking a request |
66+
67+
`drop` (the default) treats the unmatched route as a validation failure — `ValidateRequestWithSchema` returns `false` and the validation error is surfaced via `hook_vars`. `ignore` lets the request through the validator without inspection (the function returns `true`), which is useful when your schema only covers a subset of your API.
68+
69+
```yaml
70+
on_load:
71+
- apply:
72+
- >
73+
LoadAPISchemaWithOptions("public_api", "public-api.yaml", {
74+
"on_route_not_found": "ignore",
75+
"on_method_not_allowed": "drop",
76+
})
77+
```
78+
79+
### Body decoders
80+
81+
The validator uses the request `Content-Type` to pick a decoder for the body. By default, only the following Content-Types are decoded:
82+
83+
- `application/json` and the JSON variants `application/json-patch+json`, `application/merge-patch+json`, `application/ld+json`, `application/hal+json`, `application/vnd.api+json`, `application/problem+json`
84+
- `application/x-www-form-urlencoded`
85+
- `multipart/form-data`
86+
87+
A request whose Content-Type is not in this list will fail validation if the matching operation in the schema declares a request body.
88+
89+
To enable validation of additional Content-Types, register a decoder from `on_load`:
90+
91+
```yaml
92+
on_load:
93+
- apply:
94+
- RegisterAPISchemaBodyDecoder("application/yaml", "yaml")
95+
- RegisterAPISchemaBodyDecoder("text/csv", "csv")
96+
```
97+
98+
Available decoder names:
99+
100+
| Decoder | Use for |
101+
| ------------ | ----------------------------------------------------- |
102+
| `json` | JSON payloads |
103+
| `urlencoded` | `application/x-www-form-urlencoded` |
104+
| `multipart` | `multipart/form-data` |
105+
| `yaml` | YAML payloads |
106+
| `csv` | CSV payloads |
107+
| `plain` | `text/plain` |
108+
| `file` | Raw binary uploads (`application/octet-stream`, etc.) |
109+
110+
:::warning
111+
Body decoders are registered process-wide. If you run several AppSec datasources in the same CrowdSec process, they share the same set of registered decoders.
112+
:::
113+
114+
## Validating requests (`pre_eval`)
115+
116+
In a `pre_eval` hook, call `ValidateRequestWithSchema(ref)` with the `ref` you used at load time. It returns `true` if the request matches the schema, `false` otherwise.
117+
118+
| Helper | Type | Description |
119+
| --------------------------- | -------------------- | -------------------------------------------------------------------------------------------------- |
120+
| `ValidateRequestWithSchema` | `func(ref str) bool` | Validate the current request against the schema registered under `ref`. Returns `true` on success. |
121+
122+
A typical pattern is to fail closed — on validation failure, drop the request and use the failure details to build a human-readable reason:
123+
124+
```yaml
125+
name: custom/my-appsec-config
126+
on_load:
127+
- apply:
128+
- LoadAPISchemaWithName("users_api", "users-api.yaml")
129+
inband:
130+
pre_eval:
131+
- filter: req.URL.Path startsWith "/users" && !ValidateRequestWithSchema("users_api")
132+
apply:
133+
- |
134+
DropRequest("schema validation failed: " + hook_vars.validation_error_message)
135+
```
136+
137+
You can also use the result to pick a softer remediation, send a custom event, etc.
138+
139+
### Validation result variables
140+
141+
When `ValidateRequestWithSchema` returns `false`, the following keys are set on `hook_vars`. They are available to the `apply` block of the same hook, to later hooks in the same request, and to `on_match` / `post_eval` hooks. The same keys are also propagated to the resulting CrowdSec event.
142+
143+
| `hook_vars` key | Description |
144+
| --------------------------- | ---------------------------------------------------------------------------------------------------------------- |
145+
| `validation_error` | Full human-readable error string (combination of reason, field and message). |
146+
| `validation_error_reason` | Failure category — `parameter`, `request_body`, `security`, `route_not_found`, `method_not_allowed`, `internal`. |
147+
| `validation_error_field` | Name of the offending field (e.g. query parameter, header, body property) when applicable. |
148+
| `validation_error_message` | The underlying error message from the validator. |
149+
| `validation_error_value` | The offending value, truncated to 100 characters. |
150+
| `validation_error_expected` | Short description of what the schema expected (e.g. `type: integer, min: 18`). |
151+
152+
On success these keys are absent.
153+
154+
## Authentication
155+
156+
If your OpenAPI schema declares a `security` requirement on an operation, the validator enforces it as part of validation. Failure to satisfy the security requirement is reported as a `security` reason in `hook_vars`.
157+
158+
| Security scheme | Supported | Notes |
159+
| ------------------------- | --------- | ---------------------------------------------------------------------------------------------- |
160+
| `http` `basic` | Yes | Checks that an `Authorization: Basic …` header is present and non-empty. |
161+
| `http` `bearer` | Yes | Checks that an `Authorization: Bearer …` header is present and non-empty. |
162+
| `apiKey` (`header`) | Yes | Checks that the named header is present and non-empty. |
163+
| `apiKey` (`query`) | Yes | Checks that the named query parameter is present and non-empty. |
164+
| `apiKey` (`cookie`) | Yes | Checks that the named cookie is present and non-empty. |
165+
| `oauth2`, `openIdConnect` | No | A warning is logged at schema load. Any request guarded by such a scheme will fail validation. |
166+
167+
The validator only verifies that the credential **is present and well-formed** — it does not verify the credential against any backing store.
168+
169+
## End-to-end example
170+
171+
`/var/lib/crowdsec/data/schemas/users-api.yaml`:
172+
173+
```yaml
174+
openapi: 3.0.0
175+
info:
176+
title: Users API
177+
version: "1.0.0"
178+
paths:
179+
/users:
180+
post:
181+
requestBody:
182+
required: true
183+
content:
184+
application/json:
185+
schema:
186+
type: object
187+
required: [username, email]
188+
additionalProperties: false
189+
properties:
190+
username:
191+
type: string
192+
minLength: 3
193+
maxLength: 20
194+
email:
195+
type: string
196+
format: email
197+
responses:
198+
"201":
199+
description: created
200+
```
201+
202+
AppSec configuration:
203+
204+
```yaml
205+
name: custom/my-appsec-config
206+
on_load:
207+
- apply:
208+
- LoadAPISchemaWithName("users_api", "users-api.yaml")
209+
inband:
210+
pre_eval:
211+
- filter: req.URL.Path startsWith "/users" && !ValidateRequestWithSchema("users_api")
212+
apply:
213+
- |
214+
DropRequest("API schema violation on '" + hook_vars.validation_error_field + "': " + hook_vars.validation_error_message)
215+
```
216+
217+
With this configuration:
218+
219+
- `POST /users` with `{"username": "ab", "email": "x"}` is dropped (`username` too short, `email` malformed).
220+
- `POST /users` with a valid body passes validation and is then evaluated by the WAF rules as usual.
221+
- `GET /users` is dropped with reason `method_not_allowed` (default policy).
222+
- `POST /admin` is dropped with reason `route_not_found` (default policy).
223+
224+
## Metrics
225+
226+
Two Prometheus counters are exposed:
227+
228+
| Metric | Labels | Description |
229+
| ----------------------------------- | ------------------------------------------------- | -------------------------------------------------------------- |
230+
| `cs_appsec_validation_ok_total` | `source`, `appsec_engine`, `schema_ref` | Requests that passed schema validation. |
231+
| `cs_appsec_validation_failed_total` | `source`, `appsec_engine`, `schema_ref`, `reason` | Requests that failed schema validation, broken down by reason. |
232+
233+
`reason` values match `validation_error_reason`: `parameter`, `request_body`, `security`, `route_not_found`, `method_not_allowed`, `internal`.

‎crowdsec-docs/docs/appsec/hooks.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,9 @@ This hook is intended to be used to disable rules at loading (eg, to temporarily
5454
| `SetRemediationByTag` | `func(tag str, remediation string)` | Change the remediation of the in-band rule identified by the tag (multiple rules can have the same tag) |
5555
| `SetRemediationByID` | `func(id int, remediation string)` | Change the remediation of the in-band rule identified by the ID |
5656
| `SetRemediationByName` | `func(name str, remediation string)` | Change the remediation of the in-band rule identified by the name |
57+
| `LoadAPISchemaWithName` | `func(ref str, filename str)` | Load an OpenAPI schema from `<data_dir>/schemas/<filename>` and register it under `ref`. See [OpenAPI Schema Validation](api_validation.md). |
58+
| `LoadAPISchemaWithOptions` | `func(ref str, filename str, opts map)` | Same as `LoadAPISchemaWithName` but accepts per-schema policy overrides (`on_route_not_found`, `on_method_not_allowed`). |
59+
| `RegisterAPISchemaBodyDecoder` | `func(content_type str, decoder str)` | Enable a non-default body decoder for a Content-Type. See [available decoders](api_validation.md#body-decoders). |
5760

5861
##### Example
5962

@@ -90,6 +93,8 @@ This hook is intended to be used to disable rules only for this particular reque
9093
| `SetRemediationByName` | `func(name str, remediation string)` | Change the remediation of the in-band rule identified by the name |
9194
| `req` | `http.Request` | Original HTTP request received by the remediation component |
9295
| `DropRequest` | `func(reason str)` | Stop processing the request immediately and instruct the remediation component to block the request |
96+
| `ValidateRequestWithSchema` | `func(ref str) bool` | Validate the current request against an OpenAPI schema previously loaded under `ref` (returns `true` on success). On failure, structured details are published to `hook_vars` (see [OpenAPI Schema Validation](api_validation.md#validation-result-variables)). |
97+
| `hook_vars` | `map[string]string` | Per-request scratch space shared with later hooks and propagated to the resulting event. Helpers such as `ValidateRequestWithSchema` publish their results here. |
9398

9499
#### Example
95100

‎crowdsec-docs/docs/appsec/quickstart/haproxy_spoa.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ Make sure the following are already done on the machine running HAProxy (each is
1313

1414
1. **CrowdSec Security Engine** installed and running — see the [Linux quickstart](/u/getting_started/installation/linux).
1515
2. **HAProxy** already running and proxying your application(s).
16-
3. **HAProxy SPOA bouncer** (`crowdsec-haproxy-spoa-bouncer`) installed and registered against the CrowdSec LAPI — see the [SPOA bouncer guide](/u/bouncers/haproxy_spoa).
16+
3. **HAProxy SPOA bouncer** (`crowdsec-haproxy-spoa-bouncer`) installed and registered against the CrowdSec LAPI. See the [SPOA bouncer guide](/u/bouncers/haproxy_spoa).
1717

1818
## 1. Install the AppSec rule collections
1919

‎crowdsec-docs/docs/appsec/rules_deploy.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ labels:
4040
4141
Once the rule behaves as expected, the remaining steps package it for CrowdSec, wire it into the acquisition pipeline, and test it end to end.
4242
43-
## Step 1 — Stage the Rule File
43+
## Step 1 - Stage the Rule File
4444
4545
CrowdSec loads AppSec rules from `/etc/crowdsec/appsec-rules/`. Copy your YAML rule into that directory (create a `custom/` subfolder to keep things tidy if you manage several rules):
4646

@@ -56,7 +56,7 @@ Make sure the `name` inside the rule file matches the file name convention you p
5656
If you run CrowdSec in a container, copy the file into the volume that is mounted at `/etc/crowdsec/appsec-rules/` inside the container.
5757
:::
5858

59-
## Step 2 — Create an AppSec Configuration
59+
## Step 2 - Create an AppSec Configuration
6060

6161
An AppSec configuration lists which rules to load and how to handle matches. Create a new file under `/etc/crowdsec/appsec-configs/` that targets your custom rule:
6262

@@ -73,7 +73,7 @@ Key points:
7373
- `inband_rules` (and/or `outofband_rules`) accept glob patterns, so you can load multiple rules with a single entry such as `custom/block-*`.
7474
- During the reload step CrowdSec validates the syntax; if anything is off, the reload fails and the service logs the parsing error.
7575

76-
## Step 3 — Reference the Configuration in the Acquisition File
76+
## Step 3 - Reference the Configuration in the Acquisition File
7777

7878
The AppSec acquisition file (`/etc/crowdsec/acquis.d/appsec.yaml`) controls which configurations are active for the WAF component. Add your configuration to the `appsec_configs` list. Order matters: later entries override conflicting defaults such as `default_remediation`.
7979

@@ -89,7 +89,7 @@ source: appsec
8989

9090
If you only want to run your custom configuration, remove other entries and keep the list with a single item.
9191

92-
## Step 4 — Reload CrowdSec and Validate the Load
92+
## Step 4 - Reload CrowdSec and Validate the Load
9393

9494
Apply the changes by reloading the CrowdSec service:
9595

@@ -106,7 +106,7 @@ sudo cscli appsec-configs list | grep block-nonnumeric-user-id
106106

107107
The rule should appear as `enabled`, and the configuration should show up in the list. CrowdSec logs confirm the configuration was loaded without errors.
108108

109-
## Step 5 — Functional Test with `curl`
109+
## Step 5 - Functional Test with `curl`
110110

111111
Trigger the behaviour your rule is meant to catch to ensure it blocks as expected. For the example rule, send a request with a non-numeric `user_id` value:
112112

‎crowdsec-docs/docs/getting_started/sdk_intro.mdx‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,9 @@ CrowdSec offers lightweight SDKs for Python and PHP to help developers seamlessl
99
By using these SDKs, you can report signals such as suspicious IP activity or confirmed attacks directly to the Central API (CAPI). In return, your users gain access to the CrowdSec Community Blocklist, a curated and constantly updated list of IPs involved in malicious behavior observed across the global CrowdSec network.
1010

1111
Why Integrate the SDK:
12-
- **Simple Integration** — Add signal sharing with just a few lines of code
13-
- **Community-Powered Protection** — Contributions help power our global threat intelligence network
14-
- **Mutual Benefit** — Your platform shares valuable intelligence and gains stronger real-time protection in return
12+
- **Simple Integration**: Add signal sharing with just a few lines of code
13+
- **Community-Powered Protection**: Contributions help power our global threat intelligence network
14+
- **Mutual Benefit**: Your platform shares valuable intelligence and gains stronger real-time protection in return
1515

1616
## Supported SDKs
1717

0 commit comments

Comments
 (0)