@@ -148,6 +148,193 @@ crowdsec-agent-kf9fr 1/1 Running 0 34s
148148crowdsec-lapi-777c469947-jbk9q 1/1 Running 0 34s
149149` ` `
150150
151+ # ## Adding custom parsers and scenarios
152+
153+ You can ship your own parser and scenario files through Helm by using :
154+
155+ - ` config.parsers`
156+ - ` config.scenarios`
157+
158+ Those values are mounted into the agent pods as files and passed to CrowdSec as-is.
159+
160+ `config.parsers` is split by parser stage :
161+
162+ - ` config.parsers.s00-raw`
163+ - ` config.parsers.s01-parse`
164+ - ` config.parsers.s02-enrich`
165+
166+ ` config.scenarios` is a flat map of scenario files.
167+
168+ What is allowed :
169+
170+ - `config.parsers.<stage>.<fileName> : |` with a string value
171+ - `config.scenarios.<fileName> : |` with a string value
172+ - file names with or without a `.yaml` / `.yml` suffix
173+ - keys such as `my-parser`, `my-parser.yaml`, `test_scenario`, `custom.rule.yaml`
174+
175+ What is not allowed :
176+
177+ - nested objects below the file name
178+ - using a non-string value for a file entry
179+ - adding custom stages under `config.parsers`
180+ - forgetting to escape dots when using `--set` or `--set-file`
181+
182+ This is valid in a values file :
183+
184+ ` ` ` yaml title="crowdsec-custom-items.yaml"
185+ config:
186+ parsers:
187+ s01-parse:
188+ test-parser: |
189+ name: mycorp/test-parser
190+ description: "Parse my custom application logs"
191+ filter: "evt.Parsed.program == 'myapp'"
192+ onsuccess: next_stage
193+ nodes:
194+ - grok:
195+ pattern: '^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}$'
196+ apply_on: message
197+
198+ another-parser.yaml: |
199+ name: mycorp/another-parser
200+ description: "A parser stored with an explicit .yaml suffix"
201+ filter: "evt.Parsed.program == 'another-app'"
202+ onsuccess: next_stage
203+
204+ scenarios:
205+ test-scenario: |
206+ type: leaky
207+ name: mycorp/test-scenario
208+ description: "Detect repeated errors from my application"
209+ filter: "evt.Meta.service == 'myapp' && evt.Parsed.level == 'ERROR'"
210+ groupby: evt.Meta.source_ip
211+ capacity: 5
212+ leakspeed: 1m
213+ blackhole: 5m
214+
215+ another-scenario.yaml: |
216+ type: trigger
217+ name: mycorp/another-scenario
218+ description: "Raise an alert on a specific event"
219+ filter: "evt.Meta.log_type == 'special_event'"
220+ blackhole: 10m
221+ ` ` `
222+
223+ Minimal valid parser file example :
224+
225+ ` ` ` yaml title="parser.yaml"
226+ name: mycorp/minimal-parser
227+ description: "Parse a simple custom log format"
228+ filter: "evt.Parsed.program == 'myapp'"
229+ onsuccess: next_stage
230+ nodes:
231+ - grok:
232+ pattern: '^%{WORD:action} %{IP:source_ip}$'
233+ apply_on: message
234+ statics:
235+ - meta: log_type
236+ value: myapp_event
237+ ` ` `
238+
239+ The same parser embedded as a `config.parsers` item :
240+
241+ ` ` ` yaml
242+ config:
243+ parsers:
244+ s01-parse:
245+ minimal-parser: |
246+ name: mycorp/minimal-parser
247+ description: "Parse a simple custom log format"
248+ filter: "evt.Parsed.program == 'myapp'"
249+ onsuccess: next_stage
250+ nodes:
251+ - grok:
252+ pattern: '^%{WORD:action} %{IP:source_ip}$'
253+ apply_on: message
254+ statics:
255+ - meta: log_type
256+ value: myapp_event
257+ ` ` `
258+
259+ This is not valid :
260+
261+ ` ` ` yaml
262+ config:
263+ parsers:
264+ s01-parse:
265+ test-parser:
266+ yaml: |
267+ name: mycorp/test-parser
268+ ` ` `
269+
270+ The example above is invalid because `test-parser` becomes an object with a `yaml` field. Each file entry must be a single string.
271+
272+ # ### Using a dedicated values file
273+
274+ If you have several custom items, the simplest approach is to keep them in a dedicated values file and pass it with `-f` :
275+
276+ ` ` ` bash
277+ helm upgrade --install crowdsec crowdsec/crowdsec \
278+ -n crowdsec \
279+ -f crowdsec-values.yaml \
280+ -f crowdsec-custom-items.yaml
281+ ` ` `
282+
283+ This works well when :
284+
285+ - you manage several parsers and scenarios together
286+ - you want everything in Git
287+ - you do not want shell escaping issues in the command line
288+
289+ # ### Using `--set-file`
290+
291+ You can also load file contents directly from local files :
292+
293+ ` ` ` bash
294+ helm upgrade --install crowdsec crowdsec/crowdsec \
295+ -n crowdsec \
296+ -f crowdsec-values.yaml \
297+ --set-file "config.parsers.s01-parse.test-parser=/tmp/crowdsec-custom-items/parser.yaml" \
298+ --set-file "config.scenarios.test-scenario=/tmp/crowdsec-custom-items/scenario.yaml"
299+ ` ` `
300+
301+ If you want the key itself to contain `.yaml`, you must escape the dot in the Helm path :
302+
303+ ` ` ` bash
304+ helm upgrade --install crowdsec crowdsec/crowdsec \
305+ -n crowdsec \
306+ -f crowdsec-values.yaml \
307+ --set-file "config.parsers.s01-parse.test-parser\. yaml=/tmp/crowdsec-custom-items/parser.yaml" \
308+ --set-file "config.scenarios.test-scenario\. yaml=/tmp/crowdsec-custom-items/scenario.yaml"
309+ ` ` `
310+
311+ This is important because Helm uses dots as path separators. Without escaping, this :
312+
313+ ` ` ` bash
314+ --set-file "config.parsers.s01-parse.test-parser.yaml=/tmp/crowdsec-custom-items/parser.yaml"
315+ ` ` `
316+
317+ is interpreted as :
318+
319+ ` ` ` yaml
320+ config:
321+ parsers:
322+ s01-parse:
323+ test-parser:
324+ yaml: <file-content>
325+ ` ` `
326+
327+ which is not valid for CrowdSec chart values.
328+
329+ Practical rules :
330+
331+ - ` config.parsers.s01-parse.test-parser` is valid
332+ - ` config.parsers.s01-parse.test-parser\. yaml` is valid
333+ - ` config.scenarios.test-scenario` is valid
334+ - ` config.scenarios.test-scenario\. yaml` is valid
335+ - ` config.parsers.s03-custom.test-parser` is not valid
336+ - ` config.parsers.s01-parse.test-parser.yaml` without escaping is not valid
337+
151338# ## A word about source IPs
152339
153340For CrowdSec to work in Kubernetes, it must see the real client IP. Otherwise
0 commit comments