diff --git a/src/pages/en/display_filters.md b/src/pages/en/display_filters.md index 3c59601..1e7f9d8 100644 --- a/src/pages/en/display_filters.md +++ b/src/pages/en/display_filters.md @@ -101,7 +101,7 @@ http && request.headers["content-type"] == "application/json" ``` timestamp >= now() # Live traffic only timestamp > timestamp("2024-02-01T14:00:00Z") # After specific time -elapsed_time <= 300000000 # Last 5 minutes +timestamp > now() - duration("5m") # Last 5 minutes ``` --- diff --git a/src/pages/en/filtering.md b/src/pages/en/filtering.md index 5bac863..e17ff69 100644 --- a/src/pages/en/filtering.md +++ b/src/pages/en/filtering.md @@ -1,206 +1,64 @@ --- -title: Display Filters (KFL - Kubeshark Filtering Language) -description: The filter input enables filtering results using Kubeshark Filter Language (KFL). +title: Display Filters (KFL) +description: Filter traffic using Kubeshark Filter Language (KFL), powered by CEL expressions. layout: ../../layouts/MainLayout.astro --- -> KFL is a rich filtering language (not a query language) +**Kubeshark Filter Language (KFL)** is a display filter language powered by [Common Expression Language (CEL)](https://github.com/google/cel-go). It filters network traffic according to specific rules — instead of searching for the needle in the haystack, it filters out the haystack to reveal the needle. -Inspired by Wireshark's display filters. Its primary purpose is to filter network traffic according to specific rules. Instead of searching for the needle in the haystack, it filters out the haystack to reveal the needle. +For example, to only see HTTP responses with client error status codes (400–499): -**Kubeshark Filter Language (KFL)** is a language that enables you filter traffic that matches a boolean expression. For example; to only see the items with HTTP response status codes (`400` – `499`), enter: - -```python -http and response.status == r"4.*" +```cel +http && status_code >= 400 && status_code < 500 ``` -and click the **Apply** button. Your traffic stream will look like this: +Click the **Apply** button to filter the traffic stream: ![Filter example](/filter-applied.png) -## KFL vs. Pod Targeting (Display vs. Capture Filters) +## KFL vs. Capture Filters + +KFL should not be confused with [Capture Filters](/en/pod_targeting) as they serve different purposes: -KFL should not be confused with [Pod Targeting](/en/pod_targeting) as they serve different purposes. KFL statements only affect the data presented in the Dashboard, hence they are referred as Display FIlters. Pod Targeting rules constitute CApture Filters. These rules determine which pods are targeted and, consequently, which traffic is tapped. +| Aspect | Display Filters (KFL) | Capture Filters | +|--------|------------------------|-----------------| +| Purpose | Filter what is displayed | Filter what is captured | +| Impact | Dashboard view only | Resource consumption | +| Scope | Single browser tab | Cluster-wide | +| Syntax | CEL expressions | Helm values / Dashboard | -For those familiar with Wireshark, KFL can be likened to Wireshark's Display Filters, and Pod Targeting to Wireshark's BPF (Berkeley Packet Filter) filters. +For those familiar with Wireshark, KFL is analogous to Wireshark's Display Filters, and Capture Filters to Wireshark's BPF (Berkeley Packet Filter) filters. ## Queryable UI Elements -When you hover over UI elements and they display a green plus sign, it means this element can be added to your query. Selecting an element with a green plus sign will add this element to the query. For example, selecting this queryable element: +When you hover over UI elements and they display a green **+** sign, the element can be added to your query. Clicking appends the corresponding filter expression. ![Queryable UI Elements Example](/filter-ui-example.png) -adds `response.status == 201` to your query and only displays `HTTP 201` responses in the live traffic streaming. - -## KFL Syntax Reference - -**Kubeshark Filter Language (KFL)** is the language implemented inside [`kubeshark/worker`](https://github.com/kubeshark/worker) that enables the user to filter the traffic efficiently and precisely. - -```python -http and request.method == "GET" and request.path != "/example" and (request.query.a > 42 or request.headers["x"] == "y") -``` - -The language as a whole evaluates into a boolean outcome, always. Such that the record which makes the boolean `true` is a record that matches the filter. - -There are certain helper methods that can do more than reduce into a boolean value. - -### Literals - -The language supports the following literals: - -- Nil `nil` -- Boolean `true` or `false` -- Number `42`, `3.14` etc. -- String `"hello world"` -- Regex `r"prefix.*"` - -### Operators - -Operations can be grouped (precedence) using parentheses `(...)` - -> *Note: Operators are evaluated from left to right, excluding parentheses `(...)`* - -The language supports the following operators: - -#### Logical - -`and`, `or` - -> *Note: `false and`, `true or` are evaluated fast* - -#### Equality - -`==`, `!=` - -#### Comparison - -`>=`, `>`, `<=`, `<` - -#### Unary - -`!`, `-` - -### Helpers - -Helpers in KFL are method invocations that enable filtering capability which cannot be provided through the syntax. These are the available helpers in KFL: - -#### `startsWith(string)` - -Returns `true` if the given selector's value starts with the `string`. e.g. `brand.name.startsWith("Chev")` - -#### `endsWith(string)` - -Returns `true` if the given selector's value ends with the `string`. e.g. `brand.name.endsWith("let")` - -#### `contains(string)` - -Returns `true` if the given selector's value contains the `string`. e.g. `brand.name.contains("ro")` - -#### `datetime(string) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time that's provided by the `string`. If the given date-time string is not in a recognized format then it evaluates to `false`. The format must be same as `"1/2/2006, 3:04:05 PM"` (`en-US`) e.g. `timestamp > datetime("10/19/2021, 6:29:02 PM")` - -It's equal to the `time.Now().UnixNano() / int64(time.Millisecond)` in Go. e.g. `1635190131000` - -#### `limit(integer)` +This makes it easy to build complex filters without typing the full expression. -Limits the number of records that are streamed back as a result of a query. Always evaluates to `true`. +## Quick Examples -#### `json()` +```cel +# HTTP server errors +http && status_code >= 500 -A decoding helper that decodes the given JSON field if it's possible. It's used through chain calls and the JSONPath that's wanted to be accessed can be chained to the end like; `response.body.json().brand.name == "Chevrolet"` +# Traffic to a specific namespace +dst.pod.namespace == "production" -#### `xml()` +# DNS queries for a domain +dns && "example.com" in dns_questions -A decoding helper that decodes the given XML field if it's possible. It's used through chain calls and the dot-notation path that's wanted to be accessed can be chained to the end like; `response.body.xml().brand.name == "Chevrolet"` +# Filter by pod labels (safe access) +map_get(local_labels, "app", "") == "payments" -#### `redact(string...)` +# Slow requests (over 5 seconds) +http && elapsed_time > 5000000 -A record altering helper that takes N number of `string` typed arguments. The arguments are dot-notation paths and it replaces the value on each matching path with `[REDACTED]` string. - -The `json()` and `xml()` helpers are supported inside the arguments of the `redact` helper. - -#### `now() integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time now. - -#### `seconds(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` seconds before or after from now according to the sign of the argument. - -#### `minutes(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` minutes before or after from now according to the sign of the argument. - -#### `hours(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` hours before or after from now according to the sign of the argument. - -#### `days(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` days before or after from now according to the sign of the argument. - -#### `weeks(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` weeks before or after from now according to the sign of the argument. - -#### `months(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` months before or after from now according to the sign of the argument. - -#### `years(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` years before or after from now according to the sign of the argument. - -> *Note: Calling an undefined helper makes the whole expression collapse and evaluate to `false`.* - -### Selectors - -Selectors in KFL are JSONPath(s) that refer to the path in a JSON document. In KFL, every record is a JSON document. Any selector that does not match a path is evaluated to `false`. - -Following are some selector examples: - -- `brand.name` basic selector -- `request.path[1]` index selector -- `request.headers["a"] == "b"` key selector - -Selectors can be combined with a subset of helpers such that they evaluate into a boolean without any operator. e.g. `brand.name.startsWith("Chev")` These are the list of helpers that can be used with selectors: - -- `startsWith(string)` -- `endsWith(string)` -- `contains(string)` - -Such that instead of writing a boolean expression like `brand.name == "Chevrolet"`, one of the following can be written: - -```python -brand.name.startsWith("Chev") -brand.name.endsWith("let") -brand.name.contains("ro") +# Recent traffic +timestamp > now() - duration("5m") ``` -A selector can be compared to another selector as well. Such that for these given JSON document, filters like those can be written: - -JSON: `{"model":"Camaro","brand":{"name":"Chevrolet"},"year":2021,"salesYear":2021}` Filter: `year == salesYear` - -JSON: `{"model":"Camaro","brand":{"name":"Chevrolet"},"year":2021,"salesYear":2020}` Filter: `year != salesYear` - -> *Note: A selector (JSONPath) that couldn't be found in the JSON document makes the whole expression collapse and evaluate to `false`.* - -#### Wildcard Selector - -Wildcard(`*`) selectors are evaluated into arrays and the elements in the array are checked using the respective operator against whether any or all elements are matching the given criteria or not. So, for example; - -- for a JSON like `{"request":{"path":["api","v1","example"]}}` the query `request.path.* == "v1"` returns `true` -- for a JSON like `{"request":{"path":["api","v1","example"]}}` the query `request.path.* == "v2"` returns `false` -- for a JSON like `{"request":{"path":["api","v1","example"]}}` the query `request.path.* != "v2"` returns `true` -- for a JSON like `{"request":{"path":[1, 2, 3]}}` the query `request.path.* > 2` returns `true` -- for a JSON like `{"request":{"path":[1, 2, 3]}}` the query `request.path.* > 4` returns `false` - -The wildcard selector can be used to compare two JSON arrays. So, for a JSON like `{"request":{"path":[1, 2, 3]},"response":{"header":[1, 2, 3]}}` the query `request.path.* == response.header.*` checks for full value equality and returns `true`. - -Empty arrays are evaluated to `false`, the rest is `true`: - -- for a JSON like `{"request":{"path":[{"x":1}, {"x":2}, {"x":3}]}}` the query `request.path.*.x and true` returns `true`. The part of the query `request.path.*.x` evaluates to `[1, 2, 3]`. -- for a JSON like `{"request":{"path":[]}}` the query `request.path.* and true` returns `true` +## Full Reference -Please check out [the unit tests](https://github.com/up9inc/basenine/blob/main/server/lib/eval_test.go) of the language for more examples. +For the complete KFL syntax, all supported variables, and advanced examples, see the [KFL Reference](/en/v2/kfl2). diff --git a/src/pages/en/v2/kfl2.md b/src/pages/en/v2/kfl2.md index d8fe3bf..796320f 100644 --- a/src/pages/en/v2/kfl2.md +++ b/src/pages/en/v2/kfl2.md @@ -34,8 +34,52 @@ dns && "google.com" in dns_questions # Large HTTP responses http && response_body_size > 10000 + +# Slow requests (over 5 seconds) +http && elapsed_time > 5000000 + +# Recent traffic (last 5 minutes) +timestamp > now() - duration("5m") ``` +## Operators + +| Category | Operators | +|----------|-----------| +| Comparison | `==`, `!=`, `<`, `<=`, `>`, `>=` | +| Logical | `&&`, `\|\|`, `!` | +| Arithmetic | `+`, `-`, `*`, `/`, `%` | +| Membership | `in` | +| Ternary | `condition ? true_val : false_val` | + +## String Functions + +| Function | Description | +|----------|-------------| +| `str.contains(substring)` | Substring search | +| `str.startsWith(prefix)` | Prefix match | +| `str.endsWith(suffix)` | Suffix match | +| `str.matches(regex)` | Regex match (RE2 syntax) | +| `size(str)` | String length | + +## Collection Functions + +| Function | Description | +|----------|-------------| +| `size(collection)` | List/map/string length | +| `key in map` | Key existence check | +| `map[key]` | Value access (errors if key missing) | +| `map_get(map, key, default)` | Safe access with default value | +| `value in list` | List membership | + +## Time Functions + +| Function | Description | +|----------|-------------| +| `timestamp("2026-03-14T22:00:00Z")` | Parse ISO timestamp | +| `duration("5m")` | Parse duration string | +| `now()` | Current time (snapshot at filter creation) | + ## Supported Variables ### Network-Level Variables @@ -50,29 +94,25 @@ http && response_body_size > 10000 ### Kubernetes Variables -| Variable | Type | Description | Example | -|----------|------|-------------|---------| -| `src.pod.name` | string | Source pod name | `"web-server-123"` | -| `dst.pod.name` | string | Destination pod name | `"database-456"` | -| `src.pod.namespace` | string | Source pod namespace | `"production"` | -| `dst.pod.namespace` | string | Destination pod namespace | `"default"` | -| `src.service.name` | string | Source service name | `"web-service"` | -| `dst.service.name` | string | Destination service name | `"db-service"` | -| `src.service.namespace` | string | Source service namespace | `"production"` | -| `dst.service.namespace` | string | Destination service namespace | `"default"` | -| `namespaces` | list | All namespaces involved | `["production", "default"]` | -| `pods` | list | All pod names involved | `["web-server-123", "db-456"]` | -| `services` | list | All service names involved | `["web-service", "db-service"]` | -| `node_name` | string | Node name | `"ks-node-001"` | -| `node_ip` | string | Node IP address | `"10.0.0.12"` | - -### DNS Resolution Variables - -| Variable | Type | Description | Example | -|----------|------|-------------|---------| -| `src.dns` | string | DNS resolution of the source peer's IP | `"web.example.com"` | -| `dst.dns` | string | DNS resolution of the destination peer's IP | `"db.example.com"` | -| `dns_resolutions` | list | All DNS resolutions from both peers (deduplicated) | `["web.example.com", "db.example.com"]` | +| Variable | Type | Description | +|----------|------|-------------| +| `src.pod.name` | string | Source pod name | +| `dst.pod.name` | string | Destination pod name | +| `src.pod.namespace` | string | Source pod namespace | +| `dst.pod.namespace` | string | Destination pod namespace | +| `src.service.name` | string | Source service name | +| `dst.service.name` | string | Destination service name | +| `src.service.namespace` | string | Source service namespace | +| `dst.service.namespace` | string | Destination service namespace | +| `namespaces` | []string | All namespaces involved (src + dst) | +| `pods` | []string | All pod names involved (src + dst) | +| `services` | []string | All service names involved (src + dst) | +| `node_name` | string | Node name | +| `node_ip` | string | Node IP address | +| `local_node_name` | string | Node name of local peer | +| `remote_node_name` | string | Node name of remote peer | + +Pod fields automatically fall back to service data when pod info is unavailable — `dst.pod.namespace` works even when only service-level resolution exists. #### Labels and Annotations @@ -85,53 +125,107 @@ http && response_body_size > 10000 | `local_process_name` | string | Process name on the local peer | | `remote_process_name` | string | Process name on the remote peer | +Always use `map_get()` for labels and annotations — direct access like `local_labels["app"]` errors if the key doesn't exist: + +```cel +map_get(local_labels, "app", "") == "checkout" +map_get(remote_labels, "version", "") == "canary" +"tier" in local_labels +``` + +#### DNS Resolution + +| Variable | Type | Description | +|----------|------|-------------| +| `src.dns` | string | DNS resolution of the source peer's IP | +| `dst.dns` | string | DNS resolution of the destination peer's IP | +| `dns_resolutions` | []string | All DNS resolutions from both peers (deduplicated) | + +#### Resolution Status + +| Variable | Type | Values | +|----------|------|--------| +| `local_resolution_status` | string | `""` (resolved), `"no_node_mapping"`, `"rpc_error"`, `"rpc_empty"`, `"cache_miss"`, `"queue_full"` | +| `remote_resolution_status` | string | Same as above | + ### Protocol Detection -Use these boolean variables to filter by protocol: +Boolean variables that indicate which protocol was detected. Use these as the first filter term for best performance. -| Variable | Description | -|----------|-------------| -| `http` | HTTP traffic | -| `dns` | DNS traffic | -| `tls` | TLS traffic | -| `tcp` | TCP traffic | -| `udp` | UDP traffic | -| `ws` | WebSocket traffic | -| `redis` | Redis traffic | -| `kafka` | Kafka traffic | -| `ldap` | LDAP traffic | -| `amqp` | AMQP traffic | -| `radius` | RADIUS traffic | -| `diameter` | Diameter traffic | -| `sctp` | SCTP traffic | -| `icmp` | ICMP traffic | +| Variable | Protocol | Variable | Protocol | +|----------|----------|----------|----------| +| `http` | HTTP/1.1, HTTP/2 | `redis` | Redis | +| `dns` | DNS | `kafka` | Kafka | +| `tls` | TLS/SSL | `amqp` | AMQP | +| `tcp` | TCP | `ldap` | LDAP | +| `udp` | UDP | `ws` | WebSocket | +| `sctp` | SCTP | `gql` | GraphQL (v1 + v2) | +| `icmp` | ICMP | `gqlv1` / `gqlv2` | GraphQL version-specific | +| `radius` | RADIUS | `conn` / `flow` | L4 connection/flow tracking | +| `diameter` | Diameter | `tcp_conn` / `udp_conn` | Transport-specific connections | +| | | `tcp_flow` / `udp_flow` | Transport-specific flows | + +### Identity and Metadata Variables + +| Variable | Type | Description | +|----------|------|-------------| +| `id` | int | BaseEntry unique identifier | +| `node_id` | string | Node identifier (assigned by hub) | +| `index` | int | Entry index for stream uniqueness | +| `stream` | string | Stream identifier (hex string) | +| `timestamp` | timestamp | Event time (UTC), use with `timestamp()` function | +| `elapsed_time` | int | Age since timestamp in microseconds | +| `worker` | string | Worker identifier | + +### Cross-Reference Variables + +| Variable | Type | Description | +|----------|------|-------------| +| `conn_id` | int | L7 to L4 connection cross-reference ID | +| `flow_id` | int | L7 to L4 flow cross-reference ID | +| `has_pcap` | bool | Whether PCAP data is available for this entry | + +### Capture Source Variables + +| Variable | Type | Description | Values | +|----------|------|-------------|--------| +| `capture_source` | string | Canonical capture source | `"unspecified"`, `"af_packet"`, `"ebpf"`, `"ebpf_tls"` | +| `capture_backend` | string | Backend family | `"af_packet"`, `"ebpf"` | +| `capture_source_code` | int | Numeric enum | 0=unspecified, 1=af_packet, 2=ebpf, 3=ebpf_tls | ### HTTP Variables | Variable | Type | Description | |----------|------|-------------| -| `url` | string | Complete URL path | -| `method` | string | HTTP method (GET, POST, etc.) | -| `status_code` | int | Response status code | +| `method` | string | HTTP method (GET, POST, PUT, DELETE, PATCH) | +| `url` | string | Full URL path and query string | +| `path` | string | URL path component (no query) | +| `status_code` | int | HTTP response status code | | `http_version` | string | HTTP version | -| `path` | string | URL path component | | `query_string` | map | URL query parameters | | `request.headers` | map | Request headers | | `response.headers` | map | Response headers | | `request.cookies` | map | Request cookies | | `response.cookies` | map | Response cookies | +| `request_headers_size` | int | Request headers size in bytes | | `request_body_size` | int | Request body size in bytes | +| `response_headers_size` | int | Response headers size in bytes | | `response_body_size` | int | Response body size in bytes | +GraphQL requests have `gql` (or `gqlv1`/`gqlv2`) set to true and all HTTP variables available. + ### DNS Variables | Variable | Type | Description | |----------|------|-------------| -| `dns_questions` | list | DNS question names | -| `dns_answers` | list | DNS answer names | +| `dns_questions` | []string | DNS question domain names | +| `dns_answers` | []string | DNS answer domain names | +| `dns_question_types` | []string | Record types (A, AAAA, CNAME, MX, TXT, SRV, PTR) | | `dns_request` | bool | Is DNS request | | `dns_response` | bool | Is DNS response | -| `dns_question_types` | list | DNS record types (A, AAAA, etc.) | +| `dns_request_length` | int | DNS request size in bytes | +| `dns_response_length` | int | DNS response size in bytes | +| `dns_total_size` | int | Sum of request + response sizes | ### TLS Variables @@ -141,33 +235,171 @@ Use these boolean variables to filter by protocol: | `tls_info` | string | TLS connection details | | `tls_request_size` | int | TLS request size in bytes | | `tls_response_size` | int | TLS response size in bytes | +| `tls_total_size` | int | Sum of request + response sizes | + +### TCP Variables + +| Variable | Type | Description | +|----------|------|-------------| +| `tcp_method` | string | TCP method information | +| `tcp_payload` | bytes | Raw TCP payload data | +| `tcp_error_type` | string | TCP error type (empty if none) | +| `tcp_error_message` | string | TCP error message (empty if none) | + +### UDP Variables + +| Variable | Type | Description | +|----------|------|-------------| +| `udp_length` | int | UDP packet length | +| `udp_checksum` | int | UDP checksum value | +| `udp_payload` | bytes | Raw UDP payload data | + +### SCTP Variables + +| Variable | Type | Description | +|----------|------|-------------| +| `sctp_checksum` | int | SCTP checksum value | +| `sctp_chunk_type` | string | SCTP chunk type | +| `sctp_length` | int | SCTP chunk length | + +### ICMP Variables + +| Variable | Type | Description | +|----------|------|-------------| +| `icmp_type` | string | ICMP type code | +| `icmp_version` | int | ICMP version (4 or 6) | +| `icmp_length` | int | ICMP message length | + +### WebSocket Variables + +| Variable | Type | Description | +|----------|------|-------------| +| `ws_opcode` | string | Operation code: `"text"`, `"binary"`, `"close"`, `"ping"`, `"pong"` | +| `ws_request` | bool | Is WebSocket request | +| `ws_response` | bool | Is WebSocket response | +| `ws_request_payload_data` | string | Request payload (safely truncated) | +| `ws_request_payload_length` | int | Request payload length in bytes | +| `ws_response_payload_length` | int | Response payload length in bytes | ### Redis Variables | Variable | Type | Description | |----------|------|-------------| -| `redis_type` | string | Redis command verb (GET, SET, etc.) | +| `redis_type` | string | Redis command verb (GET, SET, DEL, HGET) | | `redis_command` | string | Full Redis command line | -| `redis_key` | string | The Redis key | +| `redis_key` | string | The Redis key (truncated to 64 bytes) | | `redis_request_size` | int | Request size in bytes | | `redis_response_size` | int | Response size in bytes | +| `redis_total_size` | int | Sum of request + response sizes | ### Kafka Variables | Variable | Type | Description | |----------|------|-------------| | `kafka_api_key` | int | Kafka API key number | +| `kafka_api_key_name` | string | Human-readable API operation (PRODUCE, FETCH) | | `kafka_client_id` | string | Kafka client identifier | | `kafka_size` | int | Message size | | `kafka_request` | bool | Is Kafka request | | `kafka_response` | bool | Is Kafka response | +| `kafka_request_summary` | string | Request summary/topic | +| `kafka_request_size` | int | Request size in bytes | +| `kafka_response_size` | int | Response size in bytes | -### Timestamp Variables +### AMQP Variables | Variable | Type | Description | |----------|------|-------------| -| `timestamp` | timestamp | Event time (UTC) | -| `elapsed_time` | int | Age since timestamp in microseconds | +| `amqp_method` | string | AMQP method name (`"basic.publish"`, `"channel.open"`) | +| `amqp_summary` | string | Operation summary | +| `amqp_request` | bool | Is AMQP request | +| `amqp_response` | bool | Is AMQP response | +| `amqp_request_length` | int | Request length in bytes | +| `amqp_response_length` | int | Response length in bytes | +| `amqp_total_size` | int | Sum of request + response sizes | + +### LDAP Variables + +| Variable | Type | Description | +|----------|------|-------------| +| `ldap_type` | string | LDAP operation type | +| `ldap_summary` | string | Operation summary | +| `ldap_request` | bool | Is LDAP request | +| `ldap_response` | bool | Is LDAP response | +| `ldap_request_length` | int | Request length in bytes | +| `ldap_response_length` | int | Response length in bytes | +| `ldap_total_size` | int | Sum of request + response sizes | + +### RADIUS Variables + +| Variable | Type | Description | +|----------|------|-------------| +| `radius_code` | int | RADIUS code | +| `radius_code_name` | string | Code name (`"Access-Request"`) | +| `radius_request` | bool | Is RADIUS request | +| `radius_response` | bool | Is RADIUS response | +| `radius_request_authenticator` | string | Request authenticator (hex) | +| `radius_request_length` | int | Request size in bytes | +| `radius_response_length` | int | Response size in bytes | +| `radius_total_size` | int | Sum of request + response sizes | + +### Diameter Variables + +| Variable | Type | Description | +|----------|------|-------------| +| `diameter_method` | string | Method name | +| `diameter_summary` | string | Operation summary | +| `diameter_request` | bool | Is Diameter request | +| `diameter_response` | bool | Is Diameter response | +| `diameter_request_length` | int | Request size in bytes | +| `diameter_response_length` | int | Response size in bytes | +| `diameter_total_size` | int | Sum of request + response sizes | + +### L4 Connection Tracking Variables + +| Variable | Type | Description | +|----------|------|-------------| +| `conn` | bool | Connection tracking entry | +| `conn_state` | string | Connection state: `"open"`, `"in_progress"`, `"closed"` | +| `conn_local_pkts` | int | Packets from local peer | +| `conn_local_bytes` | int | Bytes from local peer | +| `conn_remote_pkts` | int | Packets from remote peer | +| `conn_remote_bytes` | int | Bytes from remote peer | +| `conn_l7_detected` | []string | L7 protocols detected on connection | +| `conn_group_id` | int | Connection group identifier | + +Use `tcp_conn` or `udp_conn` to filter by transport protocol. + +```cel +conn && conn_state == "open" && conn_local_bytes > 1000000 +tcp_conn && "HTTP" in conn_l7_detected +``` + +### L4 Flow Tracking Variables + +Flows extend connections with rate metrics (packets/bytes per second). + +| Variable | Type | Description | +|----------|------|-------------| +| `flow` | bool | Flow tracking entry | +| `flow_state` | string | Flow state: `"open"`, `"in_progress"`, `"closed"` | +| `flow_local_pkts` | int | Packets from local peer | +| `flow_local_bytes` | int | Bytes from local peer | +| `flow_remote_pkts` | int | Packets from remote peer | +| `flow_remote_bytes` | int | Bytes from remote peer | +| `flow_local_pps` | int | Local packets per second | +| `flow_local_bps` | int | Local bytes per second | +| `flow_remote_pps` | int | Remote packets per second | +| `flow_remote_bps` | int | Remote bytes per second | +| `flow_l7_detected` | []string | L7 protocols detected on flow | +| `flow_group_id` | int | Flow group identifier | + +Use `tcp_flow` or `udp_flow` to filter by transport protocol. + +```cel +flow && flow_local_pps > 1000 +tcp_flow && flow_local_bps > 5000000 +``` ## Filter Examples @@ -202,8 +434,8 @@ src.service.name == "api-gateway" && dst.service.name == "user-service" # Traffic involving production namespace "production" in namespaces -# Filter by pod labels -local_labels.app == "payments" || remote_labels.app == "payments" +# Filter by pod labels (safe access) +map_get(local_labels, "app", "") == "payments" # Filter by process name local_process_name == "nginx" @@ -235,6 +467,12 @@ http && url.matches(".*/api/v[0-9]+/.*") # Large responses http && response_body_size > 1000000 + +# Slow requests (over 5 seconds) +http && elapsed_time > 5000000 + +# GraphQL errors +gql && status_code >= 400 ``` ### DNS Filtering @@ -246,6 +484,12 @@ dns && dns_request # Specific domain queries dns && "google.com" in dns_questions +# Failed DNS lookups +dns && dns_response && status_code != 0 + +# A record queries +dns && "A" in dns_question_types + # DNS responses with answers dns && dns_response && size(dns_answers) > 0 ``` @@ -268,7 +512,7 @@ src.dns.contains("example.com") || dst.dns.contains("example.com") dst.dns != "" && !dst.dns.endsWith(".internal") ``` -### Database Filtering +### Database and Messaging Filtering ```cel # Redis GET commands @@ -277,11 +521,71 @@ redis && redis_type == "GET" # Redis key pattern redis && redis_key.startsWith("session:") +# Kafka produce operations +kafka && kafka_api_key_name == "PRODUCE" + +# Kafka topic filtering +kafka && kafka_request_summary.contains("orders") + # Large Kafka messages kafka && kafka_size > 10000 -# Large Redis responses -redis && redis_response_size > 8192 +# AMQP publish +amqp && amqp_method == "basic.publish" + +# LDAP bind requests +ldap && ldap_type == "bind" + +# RADIUS auth +radius && radius_code_name == "Access-Request" +``` + +### Transport and Connection Filtering + +```cel +# TCP errors +tcp && tcp_error_type != "" + +# Open connections with high volume +conn && conn_state == "open" && conn_local_bytes > 1000000 + +# High packet-rate flows +flow && flow_local_pps > 1000 + +# High-bandwidth TCP flows +tcp_flow && flow_local_bps > 5000000 + +# Connections with detected L7 protocols +conn && "HTTP" in conn_l7_detected +``` + +### Negation + +```cel +# Everything that is NOT HTTP +!http + +# HTTP responses that aren't 200 +http && status_code != 200 + +# Exclude health checks +http && !path.contains("/health") + +# Exclude system namespace +!(src.pod.namespace == "kube-system") +``` + +### Time-Based Filtering + +```cel +# After a specific time +timestamp > timestamp("2026-03-14T22:00:00Z") + +# Time range +timestamp >= timestamp("2026-03-14T22:00:00Z") && timestamp <= timestamp("2026-03-14T23:00:00Z") + +# Last 5 minutes +timestamp > now() - duration("5m") ``` ### Complex Filters @@ -299,68 +603,41 @@ src.pod.namespace == "production" && http && status_code >= 400 # Cross-namespace communication src.service.namespace != dst.service.namespace -# Time-based filtering -timestamp > timestamp("2024-11-14T22:00:00Z") - -# Recent traffic (last 5 minutes) -elapsed_time <= 300000000 +# Filter by capture source +capture_source == "ebpf_tls" ``` -## CEL Language Reference - -KFL uses [Common Expression Language (CEL)](https://github.com/google/cel-go) syntax. - -### Operators - -| Type | Operators | -|------|-----------| -| Comparison | `==`, `!=`, `<`, `<=`, `>`, `>=` | -| Logical | `&&`, `||`, `!` | -| Arithmetic | `+`, `-`, `*`, `/`, `%` | -| Membership | `in` | - -### String Functions +## Type Safety -| Function | Description | -|----------|-------------| -| `contains(substring)` | Check if string contains substring | -| `startsWith(prefix)` | Check if string starts with prefix | -| `endsWith(suffix)` | Check if string ends with suffix | -| `matches(regex)` | Match against regular expression | -| `size()` | Get string length | +KFL is statically typed. Common gotchas: -### Collection Functions +- `status_code` is `int`, not string — use `status_code == 200`, not `"200"` +- `elapsed_time` is in **microseconds** — 5 seconds = `5000000` +- `timestamp` requires the `timestamp()` function — not a raw string +- Map access on missing keys errors — use `key in map` or `map_get()` first +- List membership uses `value in list` — not `list.contains(value)` -| Function | Description | -|----------|-------------| -| `size(collection)` | Get collection size | -| `in` | Check membership (`"value" in list`) | -| `[index]` | Access element by index | -| `[key]` | Access map value by key | +## Default Values -### Examples +When a variable is not present in an entry, KFL uses these defaults: -```cel -# String operations -url.contains("/api") -src.ip.startsWith("10.") -url.matches(".*\\.(jpg|png|gif)$") - -# Collection operations -"production" in namespaces -size(dns_questions) > 1 -request.headers["content-type"] - -# Map key checking -"authorization" in request.headers -``` +| Type | Default | +|------|---------| +| string | `""` | +| int | `0` | +| bool | `false` | +| list | `[]` | +| map | `{}` | +| bytes | `[]` | ## Performance Tips -1. **Use protocol detection first** - More efficient to check `http &&` before checking HTTP-specific fields -2. **Port filtering is fast** - `dst.port == 80` is very efficient -3. **Prefer `startsWith`/`endsWith`** over `contains` for prefix/suffix matching -4. **Empty filters match all** - An empty filter string matches all traffic +1. **Protocol flags first** — `http && ...` is faster than `... && http` +2. **`startsWith`/`endsWith` over `contains`** — prefix/suffix checks are faster +3. **Specific ports before string ops** — `dst.port == 80` is cheaper than `url.contains(...)` +4. **Use `map_get` for labels** — avoids errors on missing keys +5. **Keep filters simple** — CEL short-circuits on `&&`, so put cheap checks first +6. **Empty filters match all** — an empty filter string matches all traffic ## KFL vs Capture Filters