|
| 1 | +# LLM Tag Filter Plugin |
| 2 | + |
| 3 | +The `llm_tag` filter plugin uses a Large Language Model (LLM) to **classify log records** and **route them by rewriting their tag** based on natural-language conditions. Instead of writing complex regex rules or static matchers, you describe what you're looking for in plain English, and the LLM decides which records match. |
| 4 | + |
| 5 | +It extracts the log payload (looking for `log` or `message` keys) and securely queries an OpenAI-compatible API using natural language prompts. If the LLM determines that a record matches a specific prompt classification, the plugin emits a copy of the record with a newly assigned tag. |
| 6 | + |
| 7 | +This is highly useful for semantic log routing, such as identifying security anomalies, analyzing sentiment, or catching obscure errors without writing complex regular expressions. |
| 8 | + |
| 9 | +This plugin is OpenAI API-compatible and works with OpenAI, Azure OpenAI, vLLM, Ollama (with the OpenAI-compatible endpoint), LM Studio, and any other server that exposes an OpenAI-compatible `/chat/completions` endpoint. |
| 10 | + |
| 11 | +## How It Works |
| 12 | + |
| 13 | +1. For each incoming record, the filter extracts the `log` or `message` field. |
| 14 | +2. It builds a **batch prompt** that includes every configured rule's prompt, and sends a single request to the configured LLM. |
| 15 | +3. The LLM responds with one `yes`/`no` decision per rule (e.g. `1: yes\n2: no\n3: yes`). |
| 16 | +4. For each rule that matched, the record is re-emitted with the rule's target tag through a shared internal emitter input. |
| 17 | +5. Depending on `keep_record` and `tags_match_mode`, the original record is either dropped, kept, or evaluation stops at the first match. |
| 18 | + |
| 19 | +> **Note:** Because the plugin re-emits records through an internal `emitter` input, downstream `[OUTPUT]` sections can match the new tags directly with the `Match` directive. |
| 20 | +
|
| 21 | +<details> |
| 22 | +<summary><strong>Architecture details (click to expand)</strong></summary> |
| 23 | + |
| 24 | +- A **single shared emitter** is created at init time (named `emitter_for_<filter_name>` by default), used for all re-emissions. This avoids creating one emitter per rule. |
| 25 | +- Records are batched into **one LLM call per log record** (not one per rule), which dramatically reduces latency and cost when you have many rules. |
| 26 | +- The plugin protects against **infinite loops** by skipping records that come from its own emitter, and it pauses cleanly when the emitter buffer is full or during shutdown (records are passed through untouched). |
| 27 | +- On any LLM failure (timeout, network error, parse failure), the original record is **always preserved** to prevent data loss. |
| 28 | + |
| 29 | +</details> |
| 30 | + |
| 31 | +## Configuration Parameters |
| 32 | + |
| 33 | +| Key | Description | Default | |
| 34 | +|---|---|---| |
| 35 | +| `model_endpoint` | **Required.** The LLM HTTP endpoint URL, e.g. `https://api.openai.com/v1/chat/completions`. | _(none)_ | |
| 36 | +| `model_id` | **Required.** The LLM model identifier sent in the request body (e.g. `gpt-4o-mini`, `llama3.1:8b`). | _(none)_ | |
| 37 | +| `model_api_key` | API key for authentication. If relying on a local/unauthenticated model, this can be omitted. | _(none)_ | |
| 38 | +| `model_timeout` | HTTP request timeout in milliseconds for the LLM API call. | `1000` | |
| 39 | +| `tags` | **Required.** The classification rules array. Each item must be an object containing `tag` (the new tag to apply) and `prompt` (the natural language condition). | _(none)_ | |
| 40 | +| `tags_match_mode` | `first` to stop at the first matching rule, or `all` to evaluate every rule and emit one record per match. | `first` | |
| 41 | +| `keep_record` | When `true`, also keep the original record (with the original tag) after re-emission. When `false`, the original is dropped if any rule matched. | `false` | |
| 42 | + |
| 43 | +### Rule structure |
| 44 | + |
| 45 | +Each entry in the `tags` array is an object with two string fields: |
| 46 | + |
| 47 | +| Field | Description | |
| 48 | +|---|---| |
| 49 | +| `tag` | The new tag to assign when this rule matches. | |
| 50 | +| `prompt` | A natural-language condition. The LLM will answer **yes** or **no** for this rule for every log line. | |
| 51 | + |
| 52 | +## Examples |
| 53 | + |
| 54 | +### Example 1: Route errors and security events with OpenAI |
| 55 | + |
| 56 | +This pipeline reads from a JSON file, asks the LLM to flag error and security-related lines, and routes them to different outputs. |
| 57 | + |
| 58 | +```yaml |
| 59 | +service: |
| 60 | + log_level: info |
| 61 | + |
| 62 | +pipeline: |
| 63 | + inputs: |
| 64 | + - name: tail |
| 65 | + path: /var/log/app.log |
| 66 | + tag: app.raw |
| 67 | + parser: json |
| 68 | + |
| 69 | + filters: |
| 70 | + - name: llm_tag |
| 71 | + match: app.raw |
| 72 | + model_endpoint: https://api.openai.com/v1/chat/completions |
| 73 | + model_id: gpt-4o-mini |
| 74 | + model_api_key: ${OPENAI_API_KEY} |
| 75 | + model_timeout: 5000 |
| 76 | + tags_match_mode: all |
| 77 | + keep_record: false |
| 78 | + tags: |
| 79 | + - tag: app.errors |
| 80 | + prompt: "Does this log indicate an application error, exception, or stack trace?" |
| 81 | + - tag: app.security |
| 82 | + prompt: "Does this log describe a security event such as failed login, unauthorized access, or suspicious activity?" |
| 83 | + - tag: app.slow |
| 84 | + prompt: "Does this log indicate slow performance, high latency, or a timeout?" |
| 85 | + |
| 86 | + outputs: |
| 87 | + - name: stdout |
| 88 | + match: app.errors |
| 89 | + - name: http |
| 90 | + match: app.security |
| 91 | + host: siem.internal |
| 92 | + port: 8080 |
| 93 | + - name: stdout |
| 94 | + match: app.slow |
| 95 | +``` |
| 96 | +
|
| 97 | +### Example 2: Local Ollama |
| 98 | +
|
| 99 | +```yaml |
| 100 | +filters: |
| 101 | + - name: llm_tag |
| 102 | + match: logs |
| 103 | + tags_match_mode: all |
| 104 | + model_endpoint: http://127.0.0.1:11434 |
| 105 | + model_id: phi3:mini |
| 106 | + model_timeout: 10000 |
| 107 | + model_api_key: "" |
| 108 | + keep_record: true |
| 109 | + tags: |
| 110 | + - tag: security |
| 111 | + prompt: "This log indicates a security incident or authentication failure" |
| 112 | + - tag: phishing |
| 113 | + prompt: "This log contains phishing attempt or credential request" |
| 114 | +``` |
| 115 | +
|
| 116 | +In this example `keep_record true` means the original record under `sys.raw` is also preserved, so you'll see each matched record twice: once with its new tag and once with the original. |
| 117 | + |
| 118 | +### Example 3: Match modes compared |
| 119 | + |
| 120 | +Given two rules and a log that matches both: |
| 121 | + |
| 122 | +| `tags_match_mode` | `keep_record` | Result | |
| 123 | +|---|---|---| |
| 124 | +| `first` | `false` | 1 record emitted with the **first** matching tag; original dropped. | |
| 125 | +| `first` | `true` | 1 record emitted with the first matching tag; original kept. | |
| 126 | +| `all` | `false` | 1 record emitted **per matching rule**; original dropped. | |
| 127 | +| `all` | `true` | 1 record emitted **per matching rule**; original kept. | |
| 128 | + |
| 129 | +If **no** rule matches, the original record is always passed through unchanged regardless of these settings. |
| 130 | + |
| 131 | +## Observability |
| 132 | + |
| 133 | +The plugin tracks four internal counters, logged at shutdown: |
| 134 | + |
| 135 | +- `requests_total` — total LLM requests issued |
| 136 | +- `requests_failed` — failed LLM requests (timeouts, errors, etc.) |
| 137 | +- `records_emitted` — records re-emitted with a new tag |
| 138 | +- `records_dropped` — original records dropped after a successful match |
| 139 | + |
| 140 | +Each LLM call also logs its latency at `info` level, e.g.: |
| 141 | + |
| 142 | +``` |
| 143 | +[ info] [filter:llm_tag:llm_tag.0] LLM API request completed in 412.55 ms |
| 144 | +``` |
| 145 | + |
| 146 | +Enable `log_level: debug` on the service to see the raw and parsed LLM responses, which is useful when tuning prompts. |
| 147 | + |
| 148 | +## Tips and Caveats |
| 149 | + |
| 150 | +- **Latency.** Every record incurs an LLM round-trip. For high-volume streams, place `llm_tag` after a sampling or rate-limiting filter, or use a small, fast local model. |
| 151 | +- **Cost.** With a hosted API, every record is a billable request. Batch mode keeps it to **1 request per record**, regardless of how many rules you define. |
| 152 | +- **Prompt design.** Phrase prompts as **yes/no questions**. Ambiguous prompts produce inconsistent classifications. Include short, unambiguous criteria. |
| 153 | +- **Field names.** The plugin looks for the log content in either a `log` or `message` field. Records without either field are passed through unmodified. |
| 154 | +- **Failure mode.** If the LLM call fails or returns an unparseable response, the original record is preserved — you will not silently lose data. |
| 155 | +- **Output matching.** Ensure to add outputs that match the new tags you define under `tags[].tag`, otherwise re-emitted records have nowhere to go. |
| 156 | +- **Recursion.** The plugin skips records from its own emitter to avoid infinite loops. |
| 157 | + |
| 158 | +## Security considerations |
| 159 | + |
| 160 | +Logs are sent to the configured LLM endpoint. Do not send sensitive data to third-party services unless permitted by your security policy. |
| 161 | + |
| 162 | +* Use environment variables or secrets management for `model_api_key`. |
| 163 | +* Consider redacting secrets before this filter if logs may contain credentials, tokens, personal data, or regulated information. |
| 164 | +* For sensitive environments, use a local OpenAI-compatible model endpoint. |
0 commit comments