Skip to content

Commit bfde9de

Browse files
TheCrowdsecMattTheCrowdsecMatt
authored
Merge branch 'main' into premium-sapi
2 parents 9f60fbe + 04105cd commit bfde9de

14 files changed

Lines changed: 249 additions & 29 deletions

File tree

‎crowdsec-docs/docs/appsec/alerts_and_scenarios.md‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
---
2+
id: alerts_and_scenarios
3+
title: AppSec Alerts & Scenarios
4+
sidebar_position: 5
5+
---
6+
7+
## Generated Events Layout
8+
9+
HTTP requests that trigger _In-Band_ or _Out-Of-Band_ AppSec/WAF rules generate events. These events can trigger scenarios that react by banning or alerting when rules are matched.
10+
11+
The [`crowdsecurity/appsec-logs` parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-logs) is designed as a general-purpose tool to convert events into a format that is easier to process with scenarios.
12+
13+
14+
The generated event looks like this:
15+
16+
- `evt.Meta.service` is set to `appsec`
17+
- `evt.Meta.log_type`:
18+
- `appsec-block` for blocked requests (_In-Band_ rule matched, for example)
19+
- `appsec-info` for requests that triggered _Out-Of-Band_ rule (not blocked)
20+
- `evt.Meta.source_ip` is set to the source (client) IP
21+
- `evt.Meta.target_host` is set to the FQDN if present (`Host` header in the HTTP request)
22+
- `evt.Meta.target_uri` is set to the complete URI of the HTTP request
23+
- `evt.Meta.rule_name` is set to the name of the triggered rule
24+
- `evt.Meta.remediation_cmpt_ip` is set to the IP of the Remediation Component (Bouncer) that sent the HTTP request.
25+
26+
:::info
27+
The [`crowdsecurity/appsec-logs` parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-logs) is already part of the generic AppSec/WAF collections and doesn't have to be manually installed.
28+
:::
29+
30+
31+
## Creating Scenario Based on AppSec/WAF Events
32+
33+
### Triggering on _In-Band_ Rules
34+
35+
A simple yexample is the [`crowdsecurity/appsec-vpatch` scenario](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-vpatch) that will ban IPs triggering two distinct _In-Band_ rules:
36+
37+
```yaml title="/etc/crowdsec/scenarios/appsec-vpatch.yaml"
38+
type: leaky
39+
name: crowdsecurity/appsec-vpatch
40+
filter: "evt.Meta.log_type == 'appsec-block'"
41+
distinct: evt.Meta.rule_name
42+
leakspeed: "60s"
43+
capacity: 1
44+
groupby: evt.Meta.source_ip
45+
...
46+
```
47+
48+
:::info
49+
The [`crowdsecurity/appsec-vpatch` scenario](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-vpatch) is already part of the generic AppSec/WAF collections, and doesn't have to be manually installed.
50+
:::
51+
52+
### Triggering on Out-Of-Band Rules
53+
54+
Let's try to solve an imaginary scenario:
55+
56+
> We aim to prevent users from enumerating certain URLs (specifically, those that begin with `/foobar/*`) when a particular HTTP header is present (`something: *test*`). However, we want to impose this restriction only on users attempting to access two or more distinct `/foobar/*` URLs while this header is set.
57+
58+
:::info
59+
Keep in mind that _Out-Of-Band_ rules will generate an event instead of blocking the HTTP Request.
60+
:::
61+
62+
#### The AppSec/WAF Rule
63+
64+
This is our AppSec/WAF rule:
65+
66+
```yaml title="/etc/crowdsec/appsec-rules/foobar-access.yaml"
67+
name: crowdsecurity/foobar-access
68+
description: "Detect access to foobar files with the something header set"
69+
rules:
70+
- zones:
71+
- URI
72+
transform:
73+
- lowercase
74+
match:
75+
type: startsWith
76+
value: /foobar/
77+
- zones:
78+
- HEADERS
79+
variables:
80+
- something
81+
transform:
82+
- lowercase
83+
match:
84+
type: contains
85+
value: test
86+
```
87+
88+
Let ensure it's loaded as an _Out-Of-Band_ rule, first by creating a new appsec-config:
89+
90+
```yaml title="/etc/crowdsec/appsec-configs/appsec-oob.yaml"
91+
name: crowdsecurity/appsec-oob
92+
default_remediation: ban
93+
#Let's add our rule as an out-of-band rule
94+
outofband_rules:
95+
- crowdsecurity/foobar-access
96+
```
97+
98+
And then make sure this appsec-config is loaded:
99+
100+
```yaml title="/etc/crowdsec/acquis.d/appsec.yaml"
101+
appsec_configs:
102+
- crowdsecurity/appsec-default
103+
- crowdsecurity/appsec-oob
104+
labels:
105+
type: appsec
106+
listen_addr: 127.0.0.1:7422
107+
source: appsec
108+
```
109+
110+
#### The Scenario
111+
112+
We can now create a scenario that will trigger when a single IPs triggers this rule on distinct URLs:
113+
114+
```yaml title="/etc/crowdsec/scenarios/foobar-enum.yaml"
115+
type: leaky
116+
format: 3.0
117+
name: crowdsecurity/foobar-enum
118+
description: "Ban IPs repeateadly triggering out of band rules"
119+
filter: "evt.Meta.log_type == 'appsec-info' && evt.Meta.rule_name == 'crowdsecurity/foobar-access'"
120+
distinct: evt.Meta.target_uri
121+
leakspeed: "60s"
122+
capacity: 1
123+
groupby: evt.Meta.source_ip
124+
blackhole: 1m
125+
labels:
126+
remediation: true
127+
```
128+
129+
:::info
130+
The `filter` ensures only _Out-Of-Band_ events generated by our scenario are picked up, while the `capacity: 1` and `distinct: evt.Meta.target_uri` will ensure that the IP has to trigger the rule on at least 2 distinct URLs to trigger the scenario.
131+
:::
132+
133+
#### Testing
134+
135+
Let's now test our setup:
136+
137+
```bash
138+
$ curl -I localhost/foobar/1 -H 'something: test'
139+
HTTP/1.1 404 Not Found
140+
141+
$ curl -I localhost/foobar/2 -H 'something: test'
142+
HTTP/1.1 404 Not Found
143+
144+
$ curl -I localhost/foobar/3 -H 'something: test'
145+
HTTP/1.1 403 Forbidden
146+
```
147+
148+
And CrowdSec logs will show:
149+
150+
```
151+
INFO[2024-12-02T15:28:16+01:00] Ip ::1 performed 'crowdsecurity/foobar-enum' (2 events over 4.780233613s) at 2024-12-02 14:28:16.858419797 +0000 UTC
152+
INFO[2024-12-02T15:28:17+01:00] (test/crowdsec) crowdsecurity/foobar-enum by ip ::1 (/0) : 4h ban on Ip ::1
153+
```
154+
155+
As expected, the first two requests were processed without being blocked. The second one triggered the scenario, resulting in the third request being blocked by the bouncer.

‎crowdsec-docs/docs/appsec/benchmark.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,8 +33,8 @@ The benchmarks cover the following tests:
3333
Each test has been run with multiple cases:
3434

3535
- Application Security Component enabled but without any rules
36-
- Application Security Component enabled with 100 vpatch rules (in inband)
37-
- Application Security Component enabled with all the CRS (in inband)
36+
- Application Security Component enabled with 100 vpatch rules (in in-band)
37+
- Application Security Component enabled with all the CRS (in in-band)
3838

3939
On the system, we deployed:
4040

‎crowdsec-docs/docs/appsec/configuration.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Configuring the AppSec Component usually requires the use of multiple files:
1010

1111
- [AppSec rules](/appsec/rules_syntax.md) allow you to write a signature to detect and/or block malevolent requests. [You can find more information about the syntax here](/appsec/rules_syntax.md)
1212
- [acquisition configuration](/data_sources/appsec.md) indicates which port is the AppSec Component listening on, and which AppSec configuration it will use.
13-
- AppSec configuration tells which rules are loaded in inband (blocking) and out-of-band (non-blocking)
13+
- AppSec configuration tells which rules are loaded in in-band (blocking) and out-of-band (non-blocking)
1414
phases. [it as well allows you to tweak the behavior of the component via the powerful expr bindings](/appsec/rules_syntax.md)
1515

1616

@@ -40,11 +40,11 @@ A supplementary list of rules can be loaded during the out-of-band phase. These
4040

4141
### `inband_rules`
4242

43-
An optional list of rules to be loaded in inband phase. In band rules are blocking and evaluated before answering the remediation component. Useful for virtual patching, rules with no/low false positives.
43+
An optional list of rules to be loaded in in-band phase. In band rules are blocking and evaluated before answering the remediation component. Useful for virtual patching, rules with no/low false positives.
4444

4545
### `default_remediation`
4646

47-
An optional remediation for inband rules, defaults to `ban`. If set to `allow`, remediation component won't block the request (even if it matched rules). Any other value (including `captcha`) is passed as-is back to the remediation component.
47+
An optional remediation for in-band rules, defaults to `ban`. If set to `allow`, remediation component won't block the request (even if it matched rules). Any other value (including `captcha`) is passed as-is back to the remediation component.
4848

4949
### `default_pass_action`
5050

@@ -84,6 +84,6 @@ See the [dedicated doc](/docs/appsec/hooks.md#on_match)
8484

8585
### `inband_options` and `outofband_options`
8686

87-
Subset of options that can be applied to the inband/outofband rules:
87+
Subset of options that can be applied to the in-band/out-of-band rules:
8888
- `disable_body_inspection` : boolean, allows to disable HTTP body inspection
8989
- `request_body_in_memory_limit` : a number of byes indicating the maximum body size to be loaded in memory

‎crowdsec-docs/docs/appsec/installation.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ To have a functional AppSec Component, you need:
2424
Before diving into the practical steps, it's crucial to familiarize yourself with the core configuration aspects of the AppSec Component:
2525

2626
- **acquisition configuration**: Specifies how to acquire the AppSec Component stream of data
27-
- **AppSec Component configuration**: Tells which rules are loaded in inband (blocking) and out-of-band (non-blocking)
27+
- **AppSec Component configuration**: Tells which rules are loaded in in-band (blocking) and out-of-band (non-blocking)
2828
phases, [and allows tweaking the behavior of the component via the powerful expr bindings](/appsec/hooks.md). <!--@sbl we need anchor for the on_whatever and expr helpers -->
2929
- **rules** allow writing a [signature to detect and/or block malevolent requests](/appsec/rules_syntax.md).
3030

‎crowdsec-docs/docs/log_processor/data_sources/introduction.md‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
id: intro
3-
title: Introduction
3+
title: Acquisition Datasources Introduction
44
sidebar_position: 1
55
---
66

@@ -12,19 +12,19 @@ DataSources are configured via the [acquisition](/configuration/crowdsec_configu
1212

1313
Name | Type | Stream | One-shot
1414
-----|------|--------|----------
15-
[Appsec](/data_sources/appsec.md) | expose HTTP service for the Appsec component | yes | no
16-
[AWS cloudwatch](/data_sources/cloudwatch.md) | single stream or log group | yes | yes
17-
[AWS kinesis](/data_sources/kinesis.md)| read logs from a kinesis strean | yes | no
18-
[AWS S3](/data_sources/s3.md)| read logs from a S3 bucket | yes | yes
19-
[docker](/data_sources/docker.md) | read logs from docker containers | yes | yes
20-
[file](/data_sources/file.md) | single files, glob expressions and .gz files | yes | yes
21-
[HTTP](/data_sources/http.md) | read logs from an HTTP endpoint | yes | no
22-
[journald](/data_sources/journald.md) | journald via filter | yes | yes
23-
[Kafka](/data_sources/kafka.md)| read logs from kafka topic | yes | no
24-
[Kubernetes Audit](/data_sources/kubernetes_audit.md) | expose a webhook to receive audit logs from a Kubernetes cluster | yes | no
25-
[Loki](/data_sources/loki.md) | read logs from loki | yes | yes
26-
[syslog service](/data_sources/syslog_service.md) | read logs received via syslog protocol | yes | no
27-
[Windows Event](/data_sources/windows_event_log.md)| read logs from windows event log | yes | yes
15+
[Appsec](/log_processor/data_sources/appsec.md) | expose HTTP service for the Appsec component | yes | no
16+
[AWS cloudwatch](/log_processor/data_sources/cloudwatch.md) | single stream or log group | yes | yes
17+
[AWS kinesis](/log_processor/data_sources/kinesis.md)| read logs from a kinesis strean | yes | no
18+
[AWS S3](/log_processor/data_sources/s3.md)| read logs from a S3 bucket | yes | yes
19+
[docker](/log_processor/data_sources/docker.md) | read logs from docker containers | yes | yes
20+
[file](/log_processor/data_sources/file.md) | single files, glob expressions and .gz files | yes | yes
21+
[HTTP](/log_processor/data_sources/http.md) | read logs from an HTTP endpoint | yes | no
22+
[journald](/log_processor/data_sources/journald.md) | journald via filter | yes | yes
23+
[Kafka](/log_processor/data_sources/kafka.md)| read logs from kafka topic | yes | no
24+
[Kubernetes Audit](/log_processor/data_sources/kubernetes_audit.md) | expose a webhook to receive audit logs from a Kubernetes cluster | yes | no
25+
[Loki](/log_processor/data_sources/loki.md) | read logs from loki | yes | yes
26+
[syslog service](/log_processor/data_sources/syslog_service.md) | read logs received via syslog protocol | yes | no
27+
[Windows Event](/log_processor/data_sources/windows_event_log.md)| read logs from windows event log | yes | yes
2828

2929
## Common configuration parameters
3030

‎crowdsec-docs/sidebars.js‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
12
module.exports = {
23
// By default, Docusaurus generates a sidebar from the docs folder structure
34
//tutorialSidebar: [{type: 'autogenerated', dirName: '.'}],
@@ -760,6 +761,7 @@ module.exports = {
760761
{ type: "doc", id: "appsec/create_rules" },
761762
],
762763
},
764+
{ type: "doc", id: "appsec/alerts_and_scenarios", label: "Alerts & Scenarios" },
763765
{ type: "doc", id: "appsec/installation" },
764766
{ type: "doc", id: "appsec/protocol", label: "Communication Protocol" },
765767
{ type: "doc", id: "appsec/benchmark", label: "Benchmark" },

‎crowdsec-docs/sidebarsUnversioned.js‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,11 @@ module.exports = {
184184
label: "Getting started",
185185
id: "console/cti/getting_started",
186186
},
187+
{
188+
type: "doc",
189+
label: "CTI API Keys",
190+
id: "console/cti/cti_api_keys",
191+
},
187192
{
188193
type: "doc",
189194
label: "IP report",

‎crowdsec-docs/static/img/console/cti/cti_api_create_key.png‎

111 KB
Loading

‎crowdsec-docs/unversioned/console/blocklists/integrations/firewall.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ Every product product has its way to handle external blocklists. We provide a si
3636
| [F5](https://techdocs.f5.com/kb/en-us/products/big-ip-afm/manuals/product/big-ip-network-firewall-policies-and-implementations-14-0-0/07.html) | Custom | `192.168.38.187,32,BL,crowdsec-myf5Integration`<br /> `192.168.38.188,32,BL,crowdsec-myf5Integration` |
3737
| [Fortinet](https://docs.fortinet.com/document/fortigate/6.4.5/administration-guide/891236/external-blocklist-policy) | Plain text | `192.168.38.187`<br />`192.168.38.186` |
3838
| [Palo Alto](https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-admin/policy/use-an-external-dynamic-list-in-policy/external-dynamic-list#idf36cb80a-77f1-4d17-9c4b-7efe9fe426af) | Plain text | `192.168.38.187`<br />`192.168.38.186` |
39-
| [Sophos](https://docs.sophos.com/nsg/sophos-firewall/21.0/help/en-us/webhelp/onlinehelp/AdministratorHelp/ActiveThreatResponse/ThirdPartyThreatFeeds/index.html) | Plain text | `192.168.38.187`<br />`192.168.38.186` |
39+
| [Sophos](https://docs.sophos.com/nsg/sophos-firewall/latest/Help/en-us/webhelp/onlinehelp/AdministratorHelp/ActiveThreatResponse/ConfigureFeeds/ThirdPartyThreatFeeds/index.html) | Plain text | `192.168.38.187`<br />`192.168.38.186` |
4040
| Generic vendor | Plain text | `192.168.38.187`<br />`192.168.38.186` |
4141

4242
## How to bypass provider limit?

‎crowdsec-docs/unversioned/console/cti/cti_api_keys.mdx‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
title: CTI API Keys
3+
description: What can you access in the CTI API
4+
---
5+
6+
import AcademyPromo from '@site/src/components/AcademyPromo';
7+
import ThemedImage from "@theme/ThemedImage";
8+
import useBaseUrl from "@docusaurus/useBaseUrl";
9+
10+
This section will describe how to access to our CTI API and the [different plans available](#ip-lookup-plans).
11+
If you want to know more about the CTI API itself, you can check the [CTI API documentation](/u/cti_api/getting_started).
12+
13+
# Creating a CTI API Key
14+
15+
You can create a CTI API key via the **+** short cut on the top right corner that will lead you to the CrowdSec Console settings, CTI API Key section.
16+
17+
![CTI API Key Creation](/img/console/cti/cti_api_create_key.png)
18+
19+
<ThemedImage
20+
alt="CrowdSec Create API Key Page"
21+
sources={{
22+
light: useBaseUrl("/img/console_create_api_key_page_light.png"),
23+
dark: useBaseUrl("/img/console_create_api_key_page_dark.png"),
24+
}}
25+
/>
26+
27+
# CTI Plans
28+
29+
The CTI API contains multiple endpoints that are not all accessible with any plan.
30+
You can check the Swagger [There ↗️](https://crowdsecurity.github.io/cti-api/).
31+
32+
The different plans are the following:
33+
34+
## Ip Lookup Plans
35+
Those CTI API plans give you access to the search by IP and search query endpoints.
36+
They return CrowdSec CTI enriched information about IPS.
37+
IP returned will consume one token from your quota.
38+
39+
The CTI endpoint accessible with those keys are:
40+
`/smoke/{ip}` and `/smoke/search`
41+
42+
The plans quotas are the following:
43+
* Community CTI API Key (**1 free key** with a console account): **50 token/day**
44+
* Extended CTI API Key: [Contact us to activate an extended key for additional quota ↗️](https://www.crowdsec.net/contact-threat-intelligence-subscription)
45+
46+
## Advanced plans
47+
Those CTI API plans extend access to all CTI API endpoints:
48+
* `/smoke` : Allowing batch IP lookup
49+
* `/fire` : To stream all enriched IPs from our CrowdSec Intelligence Blocklist
50+
51+
[Contact us to activate an extended key for additional quota ↗️](https://www.crowdsec.net/contact-threat-intelligence-subscription)

0 commit comments

Comments
 (0)