Skip to content

Commit 2d8fab6

Browse files
committed
add doc about custom hub items
1 parent 5a75178 commit 2d8fab6

1 file changed

Lines changed: 187 additions & 0 deletions

File tree

crowdsec-docs/unversioned/getting_started/installation/kubernetes.mdx

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,193 @@ crowdsec-agent-kf9fr 1/1 Running 0 34s
148148
crowdsec-lapi-777c469947-jbk9q 1/1 Running 0 34s
149149
```
150150

151+
### Adding custom parsers and scenarios
152+
153+
You can ship your own parser and scenario files through Helm by using:
154+
155+
- `config.parsers`
156+
- `config.scenarios`
157+
158+
Those values are mounted into the agent pods as files and passed to CrowdSec as-is.
159+
160+
`config.parsers` is split by parser stage:
161+
162+
- `config.parsers.s00-raw`
163+
- `config.parsers.s01-parse`
164+
- `config.parsers.s02-enrich`
165+
166+
`config.scenarios` is a flat map of scenario files.
167+
168+
What is allowed:
169+
170+
- `config.parsers.<stage>.<fileName>: |` with a string value
171+
- `config.scenarios.<fileName>: |` with a string value
172+
- file names with or without a `.yaml` / `.yml` suffix
173+
- keys such as `my-parser`, `my-parser.yaml`, `test_scenario`, `custom.rule.yaml`
174+
175+
What is not allowed:
176+
177+
- nested objects below the file name
178+
- using a non-string value for a file entry
179+
- adding custom stages under `config.parsers`
180+
- forgetting to escape dots when using `--set` or `--set-file`
181+
182+
This is valid in a values file:
183+
184+
```yaml title="crowdsec-custom-items.yaml"
185+
config:
186+
parsers:
187+
s01-parse:
188+
test-parser: |
189+
name: mycorp/test-parser
190+
description: "Parse my custom application logs"
191+
filter: "evt.Parsed.program == 'myapp'"
192+
onsuccess: next_stage
193+
nodes:
194+
- grok:
195+
pattern: '^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}$'
196+
apply_on: message
197+
198+
another-parser.yaml: |
199+
name: mycorp/another-parser
200+
description: "A parser stored with an explicit .yaml suffix"
201+
filter: "evt.Parsed.program == 'another-app'"
202+
onsuccess: next_stage
203+
204+
scenarios:
205+
test-scenario: |
206+
type: leaky
207+
name: mycorp/test-scenario
208+
description: "Detect repeated errors from my application"
209+
filter: "evt.Meta.service == 'myapp' && evt.Parsed.level == 'ERROR'"
210+
groupby: evt.Meta.source_ip
211+
capacity: 5
212+
leakspeed: 1m
213+
blackhole: 5m
214+
215+
another-scenario.yaml: |
216+
type: trigger
217+
name: mycorp/another-scenario
218+
description: "Raise an alert on a specific event"
219+
filter: "evt.Meta.log_type == 'special_event'"
220+
blackhole: 10m
221+
```
222+
223+
Minimal valid parser file example:
224+
225+
```yaml title="parser.yaml"
226+
name: mycorp/minimal-parser
227+
description: "Parse a simple custom log format"
228+
filter: "evt.Parsed.program == 'myapp'"
229+
onsuccess: next_stage
230+
nodes:
231+
- grok:
232+
pattern: '^%{WORD:action} %{IP:source_ip}$'
233+
apply_on: message
234+
statics:
235+
- meta: log_type
236+
value: myapp_event
237+
```
238+
239+
The same parser embedded as a `config.parsers` item:
240+
241+
```yaml
242+
config:
243+
parsers:
244+
s01-parse:
245+
minimal-parser: |
246+
name: mycorp/minimal-parser
247+
description: "Parse a simple custom log format"
248+
filter: "evt.Parsed.program == 'myapp'"
249+
onsuccess: next_stage
250+
nodes:
251+
- grok:
252+
pattern: '^%{WORD:action} %{IP:source_ip}$'
253+
apply_on: message
254+
statics:
255+
- meta: log_type
256+
value: myapp_event
257+
```
258+
259+
This is not valid:
260+
261+
```yaml
262+
config:
263+
parsers:
264+
s01-parse:
265+
test-parser:
266+
yaml: |
267+
name: mycorp/test-parser
268+
```
269+
270+
The example above is invalid because `test-parser` becomes an object with a `yaml` field. Each file entry must be a single string.
271+
272+
#### Using a dedicated values file
273+
274+
If you have several custom items, the simplest approach is to keep them in a dedicated values file and pass it with `-f`:
275+
276+
```bash
277+
helm upgrade --install crowdsec crowdsec/crowdsec \
278+
-n crowdsec \
279+
-f crowdsec-values.yaml \
280+
-f crowdsec-custom-items.yaml
281+
```
282+
283+
This works well when:
284+
285+
- you manage several parsers and scenarios together
286+
- you want everything in Git
287+
- you do not want shell escaping issues in the command line
288+
289+
#### Using `--set-file`
290+
291+
You can also load file contents directly from local files:
292+
293+
```bash
294+
helm upgrade --install crowdsec crowdsec/crowdsec \
295+
-n crowdsec \
296+
-f crowdsec-values.yaml \
297+
--set-file "config.parsers.s01-parse.test-parser=/tmp/crowdsec-custom-items/parser.yaml" \
298+
--set-file "config.scenarios.test-scenario=/tmp/crowdsec-custom-items/scenario.yaml"
299+
```
300+
301+
If you want the key itself to contain `.yaml`, you must escape the dot in the Helm path:
302+
303+
```bash
304+
helm upgrade --install crowdsec crowdsec/crowdsec \
305+
-n crowdsec \
306+
-f crowdsec-values.yaml \
307+
--set-file "config.parsers.s01-parse.test-parser\.yaml=/tmp/crowdsec-custom-items/parser.yaml" \
308+
--set-file "config.scenarios.test-scenario\.yaml=/tmp/crowdsec-custom-items/scenario.yaml"
309+
```
310+
311+
This is important because Helm uses dots as path separators. Without escaping, this:
312+
313+
```bash
314+
--set-file "config.parsers.s01-parse.test-parser.yaml=/tmp/crowdsec-custom-items/parser.yaml"
315+
```
316+
317+
is interpreted as:
318+
319+
```yaml
320+
config:
321+
parsers:
322+
s01-parse:
323+
test-parser:
324+
yaml: <file-content>
325+
```
326+
327+
which is not valid for CrowdSec chart values.
328+
329+
Practical rules:
330+
331+
- `config.parsers.s01-parse.test-parser` is valid
332+
- `config.parsers.s01-parse.test-parser\.yaml` is valid
333+
- `config.scenarios.test-scenario` is valid
334+
- `config.scenarios.test-scenario\.yaml` is valid
335+
- `config.parsers.s03-custom.test-parser` is not valid
336+
- `config.parsers.s01-parse.test-parser.yaml` without escaping is not valid
337+
151338
### A word about source IPs
152339

153340
For CrowdSec to work in Kubernetes, it must see the real client IP. Otherwise

0 commit comments

Comments
 (0)