Skip to content

Commit 52a4b86

Browse files
feat: clean up bl-mirror docs and add aggregate option (#1001)
* feat: clean up bl-mirror docs and add aggregate option See crowdsecurity/cs-blocklist-mirror#122 * enhance: add other 0.0.6 option
1 parent e424a2a commit 52a4b86

1 file changed

Lines changed: 114 additions & 54 deletions

File tree

crowdsec-docs/unversioned/bouncers/blocklist-mirror.mdx

Lines changed: 114 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -9,22 +9,20 @@ import TabItem from '@theme/TabItem';
99
import useBaseUrl from '@docusaurus/useBaseUrl';
1010
import RemediationSupportBadges from '@site/src/components/remediation-support-badge';
1111

12-
13-
<p align="center">
12+
<div align="center">
1413
<img src={useBaseUrl('/img/crowdsec_http.svg')} width="400" height="300" />
15-
</p>
16-
17-
<p align="center">
18-
<p align="center">
19-
<img src="https://img.shields.io/badge/build-pass-green"></img>
20-
<img src="https://img.shields.io/badge/tests-pass-green"></img>
21-
</p>
22-
<p align="center">
23-
&#x1F4DA; <a href="#installation/">Documentation</a>
24-
&#x1F4A0; <a href="https://hub.crowdsec.net">Hub</a>
25-
&#128172; <a href="https://discourse.crowdsec.net">Discourse </a>
26-
</p>
27-
</p>
14+
</div>
15+
16+
<div align="center">
17+
<img src="https://img.shields.io/badge/build-pass-green" />
18+
<img src="https://img.shields.io/badge/tests-pass-green" />
19+
</div>
20+
21+
<div align="center">
22+
&#x1F4DA; <a href="#installation-from-repositories">Documentation</a>
23+
&#x1F4A0; <a href="https://hub.crowdsec.net">Hub</a>
24+
&#128172; <a href="https://discourse.crowdsec.net">Discourse</a>
25+
</div>
2826

2927
<RemediationSupportBadges
3028
MTLS
@@ -48,15 +46,15 @@ This Remediation Component exposes CrowdSec's active decisions via provided HTTP
4846
}>
4947
<TabItem value="debian">
5048

51-
```console
52-
$ sudo apt install crowdsec-blocklist-mirror
49+
```bash
50+
sudo apt install crowdsec-blocklist-mirror
5351
```
5452

5553
</TabItem>
5654
<TabItem value="rhel">
5755

58-
```console
59-
$ sudo yum install crowdsec-blocklist-mirror
56+
```bash
57+
sudo yum install crowdsec-blocklist-mirror
6058
```
6159

6260
</TabItem>
@@ -73,21 +71,21 @@ Refer to [docker hub](https://hub.docker.com/r/crowdsecurity/blocklist-mirror)
7371

7472
First, download the latest [`crowdsec-blocklist-mirror` release](https://github.com/crowdsecurity/cs-blocklist-mirror/releases).
7573

76-
```console
77-
$ tar xzvf crowdsec-blocklist-mirror.tgz
78-
$ sudo ./install.sh
74+
```bash
75+
tar xzvf crowdsec-blocklist-mirror.tgz
76+
sudo ./install.sh
7977
```
8078

8179
### From source
8280

8381
Run the following commands:
8482

85-
```console
86-
$ git clone https://github.com/crowdsecurity/cs-blocklist-mirror.git
87-
$ cd cs-blocklist-mirror/
88-
$ make release
89-
$ cd crowdsec-blocklist-mirror-v*/
90-
$ sudo ./install.sh
83+
```bash
84+
git clone https://github.com/crowdsecurity/cs-blocklist-mirror.git
85+
cd cs-blocklist-mirror/
86+
make release
87+
cd crowdsec-blocklist-mirror-v*/
88+
sudo ./install.sh
9189
```
9290

9391
### Configuration
@@ -96,14 +94,33 @@ For manual installations before starting the `crowdsec-blocklist-mirror` service
9694

9795
The default configuration file is located under : `/etc/crowdsec/bouncers/` as file `crowdsec-blocklist-mirror.yaml`.
9896

99-
If you need to download and restore the configuration file you can find an example on the [Respository](https://github.com/crowdsecurity/cs-blocklist-mirror/blob/main/config/crowdsec-blocklist-mirror.yaml)
97+
If you need to download and restore the configuration file you can find an example on the [Repository](https://github.com/crowdsecurity/cs-blocklist-mirror/blob/main/config/crowdsec-blocklist-mirror.yaml)
10098

10199
## Configuration Reference
102100

103101
### `crowdsec_config`
104102

105103
Used to nest the configuration related to crowdsec.
106104

105+
```yaml title="Example"
106+
crowdsec_config:
107+
lapi_url: http://127.0.0.1:8080
108+
lapi_key: ${LAPI_KEY}
109+
update_frequency: 10s
110+
supported_decisions_types:
111+
- ban
112+
include_scenarios_containing:
113+
- ssh
114+
- http
115+
exclude_scenarios_containing:
116+
- dns
117+
only_include_decisions_from:
118+
- crowdsec
119+
- cscli
120+
insecure_skip_verify: false
121+
listen_uri: 127.0.0.1:41412
122+
```
123+
107124
#### `lapi_url`
108125
> string
109126

@@ -138,13 +155,23 @@ Path to the CA file used to trust the LAPI certificate.
138155

139156
The component will poll the CrowdSec LAPI every `update_frequency` interval.
140157

158+
#### `supported_decisions_types`
159+
> [ ]string
160+
161+
Limit mirrored blocklists to specific decision types (for example `ban`, `captcha`).
162+
163+
- Empty/missing list: no type filtering (all types).
164+
- Case-insensitive matching.
165+
141166
#### `include_scenarios_containing`
142167
> [ ]string
143168

144169
Ignore IPs banned for triggering scenarios not containing the provided words.
145170

146171
```yaml title="Example"
147-
include_scenarios_containing: ["ssh", "http"]
172+
include_scenarios_containing:
173+
- ssh
174+
- http
148175
```
149176

150177
#### `exclude_scenarios_containing`
@@ -153,16 +180,20 @@ include_scenarios_containing: ["ssh", "http"]
153180
Ignore IPs banned for triggering scenarios containing the provided words.
154181

155182
```yaml title="Example"
156-
exclude_scenarios_containing: ["ssh", "http"]
183+
exclude_scenarios_containing:
184+
- ssh
185+
- http
157186
```
158187

159188
#### `only_include_decisions_from`
160189
> [ ]string
161190

162-
Only include IPs banned due to decisions orginating from provided sources.
191+
Only include IPs banned due to decisions originating from provided sources.
163192

164193
```yaml title="Example"
165-
only_include_decisions_from: ["cscli", "crowdsec"]
194+
only_include_decisions_from:
195+
- cscli
196+
- crowdsec
166197
```
167198

168199
#### `insecure_skip_verify`
@@ -183,6 +214,12 @@ listen_uri: "127.0.0.1:41412"
183214

184215
Prometheus metrics
185216

217+
```yaml title="Example"
218+
metrics:
219+
enabled: true
220+
endpoint: /metrics
221+
```
222+
186223
#### `enabled`
187224
> boolean
188225

@@ -198,15 +235,26 @@ The URI endpoint to serve the metrics on.
198235

199236
List of blocklists to serve. Each blocklist has the following configuration.
200237

238+
```yaml title="Example"
239+
blocklists:
240+
- format: plain_text
241+
aggregate: true
242+
endpoint: /security.txt
243+
authentication:
244+
type: none
245+
```
246+
201247
#### `format`
202248
> string
203249

204-
Format of the blocklist, the following are supported:
250+
Format of the blocklist. See the supported values and examples in the [Formats](#formats) section.
205251

206-
- `plain_text` : One IP per line
207-
- `mikrotik` : Generates a mikrotik script
208-
- `F5` : Lines for f5 appliances
209-
- `juniper`: One entry per line using CIDR notation
252+
#### `aggregate`
253+
> boolean
254+
255+
When enabled, aggregate decisions into CIDR ranges to reduce the size of the served blocklist.
256+
257+
This changes the output to CIDR ranges (including `/32` and `/128` for single IPs) and makes per-decision metadata unavailable, so runtime filters that rely on such metadata (for example `?origin=`) are not compatible.
210258

211259
#### `endpoint`
212260
> string
@@ -220,7 +268,7 @@ Configuration used to enforce or bypass authentication on the blocklist.
220268
##### `type`:
221269
> `none` | `basic` | `ip_based`
222270

223-
The type of authetentication to impose:
271+
The type of authentication to impose:
224272

225273
- `none` : No authentication required.
226274
- `basic` : Basic authentication required.
@@ -245,6 +293,12 @@ List of valid IPv4 and IPv6 IPs and ranges which have access to blocklist. It's
245293

246294
TLS Configuration is utilized to activate HTTPS on the mirror server.
247295

296+
```yaml title="Example"
297+
tls:
298+
cert_file: /etc/ssl/certs/blocklist-mirror.pem
299+
key_file: /etc/ssl/private/blocklist-mirror-key.pem
300+
```
301+
248302
#### `cert_file`:
249303
> string
250304

@@ -255,7 +309,7 @@ Path to certificate to use if TLS is to be enabled on the mirror server.
255309

256310
Path to certificate key file.
257311

258-
## Global RunTime Query Parameters
312+
## Global runtime query parameters
259313

260314
`?ipv4only` - Only return IPv4 addresses
261315

@@ -270,7 +324,7 @@ Example usage
270324
```
271325
http://localhost:41412/security/blocklist?ipv6only
272326
```
273-
`?nosort` - Do not sort IP's
327+
`?nosort` - Do not sort IPs
274328
275329
> Only use if you do not care about the sorting of the list, can result in better performance.
276330
@@ -279,13 +333,26 @@ Example usage
279333
http://localhost:41412/security/blocklist?nosort
280334
```
281335
282-
`?origin=` - Only return IP's by origin
336+
`?origin=` - Only return IPs by origin
337+
338+
Not compatible with `aggregate: true` (aggregation can’t preserve the original decision origin).
283339
284340
Example usage
285341
```
286342
http://localhost:41412/security/blocklist?origin=cscli
287343
```
288344
345+
`?supported_decisions_types=` - Override decision type filtering for this request
346+
347+
Accepts a comma-separated list or repeated query parameters:
348+
349+
```
350+
/security/blocklist?supported_decisions_types=ban,captcha
351+
/security/blocklist?supported_decisions_types=ban&supported_decisions_types=captcha
352+
```
353+
354+
If omitted, the YAML value applies. To include all types, omit the parameter and leave the YAML list empty.
355+
289356
You can then start the service via:
290357
291358
```bash
@@ -372,22 +439,15 @@ Example:
372439
373440
#### SRX Dynamic Address configuration sample
374441
375-
Using the blocklist on a Juniper SRX requires that the published url ends in .txt. This can be acieved by altering the endpoint config in `cfg.yaml` as follows:
442+
Using the blocklist on a Juniper SRX requires that the published URL ends in `.txt`. This can be achieved by altering the endpoint config in `cfg.yaml` as follows:
376443
377-
Sample `cfg.yaml`
444+
Sample `cfg.yaml`:
378445
```yaml
379-
####
380446
blocklists:
381-
- format: juniper # Supported formats are either of "plain_text", "mikrotik", "juniper"
382-
endpoint: /security/blocklist.txt #Must have .txt for juniper formatter.
447+
- format: juniper
448+
endpoint: /security/blocklist.txt # Must end with `.txt` for the Juniper formatter.
383449
authentication:
384-
type: none # Supported types are either of "none", "ip_based", "basic"
385-
user:
386-
password:
387-
trusted_ips: # IP ranges, or IPs which don't require auth to access this blocklist
388-
- 127.0.0.1
389-
- ::1
390-
####
450+
type: none
391451
```
392452
393453
This can then be configured on the SRX firewall as follows:

0 commit comments

Comments
 (0)