Skip to content

Commit b02dd34

Browse files
inikeppatrickbirch
inikep
authored and
patrickbirch
committed
Improve and fix misc issues
1 parent a8c5ec3 commit b02dd34

28 files changed

Lines changed: 965 additions & 789 deletions

‎docs/_static/audit-data-flow.png‎

7.45 MB
Loading

‎docs/_static/audit-filter-hierarchy.png‎

5.37 MB
Loading

‎docs/_static/audit-log-filter-arch.png‎

6.39 MB
Loading

‎docs/_static/audit-log-filter-modification-lifecycle.png‎

6.32 MB
Loading

‎docs/_static/audit-logging-strategy.png‎

5.79 MB
Loading

‎docs/audit-log-filter-compression-encryption.md‎

Lines changed: 27 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -2,42 +2,40 @@
22

33
## Compression
44

5-
You can enable compression for any [format](audit-log-filter-formats.md) by setting the `audit_log_filter.compression` system variable when the server starts.
5+
Enable compression for any [format](audit-log-filter-formats.md) with `audit_log_filter.compression` at server startup.
66

7-
The `audit_log_filter.compression` variable can be either of the following:
7+
Allowed values:
88

9-
* NONE (no compression) - the default value
10-
* GZIP - uses the GNU Zip compression
9+
* `NONE` — default, no compression
10+
* `GZIP` — GNU zip compression
1111

12-
If compression and encryption are enabled, the component applies compression before encryption. If you must manually recover a file with both settings, first decrypt the file and then uncompress the file.
12+
With both compression and encryption enabled, the component compresses first, then encrypts. To recover a file manually, decrypt, then decompress.
1313

1414
## Encryption
1515

16-
You can encrypt any audit log filter file in any [format](audit-log-filter-formats.md). The audit log filter component generates the initial password, but you can use user-defined passwords after that. The component stores the passwords in the keyring, so that feature must be enabled.
16+
Encrypt any audit log format. The component generates the first password; you may rotate to custom passwords afterward. Passwords live in the keyring—enable a keyring first.
1717

18-
Set the `audit_log_filter.encryption` system variable with the server starts. The allowed values are the following:
18+
Set `audit_log_filter.encryption` at startup. Values:
1919

20-
* NONE - no encryption, the default value
21-
* AES - AES-256-CBC (Cipher Block Chaining) encryption
20+
* `NONE` — default, no encryption
21+
* `AES` — AES-256-CBC
2222

23-
The AES uses the 256-bit key size.
23+
AES uses a 256-bit key.
2424

25-
The following audit log filter functions are used with encryption:
25+
Encryption-related functions:
2626

2727
| Function name | Description |
2828
| ----------------- | -------------------- |
29-
| audit_log_encryption_password_set() | Stores the password in the keyring. If encryption is enabled, the function also rotates the log file by renaming the current log file and creating a log file encrypted with the password. |
30-
| audit_log_encryption_password_get() | Invoking this function without an argument returns the current encryption password. An argument that specifies the keyring ID of an archived password or current password returns that password by ID. |
29+
| audit_log_encryption_password_set() | Stores a password in the keyring. With encryption on, also rotates the log: renames the current file and starts a new file encrypted with the new password. |
30+
| audit_log_encryption_password_get() | With no argument, returns the active password. With a keyring ID, returns that archived or current password by ID. |
3131

32-
The `audit_log_filter.password_history_keep_days` variable is used with encryption. If the variable is not zero (0), invoking `audit_log_encryption_password_set()` causes the expiration of archived audit log passwords.
32+
`audit_log_filter.password_history_keep_days` controls how long archived passwords stay available. When non-zero, calling `audit_log_encryption_password_set()` can expire older keyring entries.
3333

34-
When the component starts with encryption enabled, the component checks if the keyring has an audit log filter encryption password. If no password is found, the component generates a random password and stores this password in the keyring. Use `audit_log_encryption_password_get()` to review this password.
35-
36-
If compression and encryption are enabled, the component applies compression before encryption. If you must manually recover a file with both settings, first decrypt the file and then uncompress the file.
34+
On startup with encryption enabled, if no audit password exists the component generates one and stores it. Call `audit_log_encryption_password_get()` to inspect it.
3735

3836
## Manually uncompressing and decrypting audit log filter files
3937

40-
To decrypt an encrypted log file, use the openssl command. For example:
38+
Decrypt with OpenSSL, for example:
4139

4240
```bash
4341
openssl enc -d -aes-256-cbc -pass pass:password
@@ -46,20 +44,25 @@ openssl enc -d -aes-256-cbc -pass pass:password
4644
-out audit.timestamp.log
4745
```
4846

49-
To execute that command, you must obtain a password and iterations. To do this, use `audit_log_encryption_password_get()`.
50-
51-
This function gets the encryption password, and the iterations count and returns this data as a JSON-encoded string. For example, if the audit log file name is `audit.20190415T151322.log.20190414T223342-2.enc`, the password ID is `{randomly-generated-alphanumeric-string}` and the keyring ID is `audit-log-20190414T223342-2`.
47+
You need the password and iteration count from `audit_log_encryption_password_get()`.
5248

53-
Get the keyring password:
49+
That function returns JSON, for example for file `audit.20190415T151322.log.20190414T223342-2.enc` with keyring ID `audit-log-20190414T223342-2`:
5450

5551
```sql
5652
SELECT audit_log_encryption_password_get('audit-log-20190414T223342-2');
5753
```
5854

59-
The return value of this function may look like the following:
60-
6155
??? example "Expected output"
6256

6357
```{.text .no-copy}
6458
{"password":"{randomly-generated-alphanumeric-string}","iterations":568977}
6559
```
60+
61+
## Additional reading
62+
63+
* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) — encryption and compression options
64+
* [Audit Log Filter security](audit-log-filter-security.md)
65+
* [Audit Log Filter file format overview](audit-log-filter-formats.md)
66+
* [Manage the Audit Log Filter files](manage-audit-log-filter.md)
67+
* [Keyring components and plugins overview](keyring-components-plugins-overview.md)
68+
* [Quickstart component keyring](quickstart-component-keyring.md)

‎docs/audit-log-filter-definition-fields.md‎

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
# Audit Log Filter definition fields
2+
3+
Canonical class, event, and field names for [`audit_log_filter_set_filter()`](audit-log-filter-variables.md#audit_log_filter_set_filterfilter_name-definition) validation—aligned with Percona Server source [filter_definition_fields.md](https://github.com/percona/percona-server/blob/8.4/components/audit_log_filter/filter_definition_fields.md).
4+
5+
[`audit_log_filter.event_mode`](audit-log-filter-variables.md#audit_log_filterevent_mode) decides which sections apply; notes below contrast `REDUCED` and `FULL`.
6+
7+
## Notes
8+
9+
* Names here are filter-definition names—they may differ from JSON log output keys.
10+
* Field type reflects the current server validator (`get_event_field_value_type()`).
11+
* Some numeric-looking fields validate as `string` when the server leaves them untyped.
12+
* Only documented class names pass validation.
13+
* When [`audit_log_filter.event_mode`](audit-log-filter-variables.md#audit_log_filterevent_mode) is `REDUCED` (the default from 8.4.9-9 onward), only these events are tracked and accepted by filter-definition validation:
14+
* `general`: `status`
15+
* `connection`: `connect`, `disconnect`, `change_user`
16+
* `table_access`: `read`, `insert`, `update`, `delete`
17+
* `message`: `internal`, `user`
18+
19+
In `REDUCED` mode, class names that only exist for extended auditing (`global_variable`, `command`, `query`, `stored_program`, `authentication`, `parse`) are rejected entirely. Subclass names that are not in the list above (for example `general/log`, `connection/pre_authenticate`) are also rejected during filter validation. At runtime, events outside the `REDUCED` set are silently skipped. With `FULL`, those six classes and their subclasses are valid filter targets.
20+
* Lifecycle-related records with class names `audit`, `server_startup`, and `server_shutdown` are not valid filter-definition targets. Startup and shutdown lifecycle events are ignored by the audit log filter if they are received.
21+
* For `connection.connection_type`, the validator accepts numeric values `0`–`5` and the pseudo-constants `::undefined`, `::tcp/ip`, `::socket`, `::named_pipe`, `::ssl`, and `::shared_memory`.
22+
23+
## `general`
24+
25+
Supported events: `log`, `error`, `result`, `status`
26+
REDUCED mode: only `status`
27+
28+
| Field name | Field type | Description |
29+
| --- | --- | --- |
30+
| `general_error_code` | integer | Event error code. |
31+
| `general_thread_id` | unsigned integer | Event thread ID. Currently an alias of `general_connection_id`. |
32+
| `general_connection_id` | unsigned integer | Event connection ID. |
33+
| `general_user.str` | string | User name recorded for the general event. |
34+
| `general_user.length` | unsigned integer | User name length. |
35+
| `general_command.str` | string | General command text, for example `Query`. |
36+
| `general_command.length` | unsigned integer | General command text length. |
37+
| `general_query.str` | string | SQL statement text associated with the event. |
38+
| `general_query.length` | unsigned integer | SQL statement text length. |
39+
| `general_host.str` | string | Client host name. |
40+
| `general_host.length` | unsigned integer | Client host name length. |
41+
| `general_sql_command.str` | string | SQL command name associated with the statement, for example `select`. |
42+
| `general_sql_command.length` | unsigned integer | SQL command name length. |
43+
| `general_external_user.str` | string | External user or OS login associated with the event. |
44+
| `general_external_user.length` | unsigned integer | External user or OS login length. |
45+
| `general_ip.str` | string | Client IP address. |
46+
| `general_ip.length` | unsigned integer | Client IP address length. |
47+
48+
## `connection`
49+
50+
Supported events: `connect`, `disconnect`, `change_user`, `pre_authenticate`
51+
REDUCED mode: `connect`, `disconnect`, `change_user`
52+
53+
| Field name | Field type | Description |
54+
| --- | --- | --- |
55+
| `status` | integer | Current connection event status. |
56+
| `connection_id` | unsigned integer | Connection ID. |
57+
| `user.str` | string | User name of this connection. |
58+
| `user.length` | unsigned integer | User name length. |
59+
| `priv_user.str` | string | Privileged user name. |
60+
| `priv_user.length` | unsigned integer | Privileged user name length. |
61+
| `external_user.str` | string | External user name or OS login. |
62+
| `external_user.length` | unsigned integer | External user name length. |
63+
| `proxy_user.str` | string | Proxy user used for the connection. |
64+
| `proxy_user.length` | unsigned integer | Proxy user name length. |
65+
| `host.str` | string | Connection host name. |
66+
| `host.length` | unsigned integer | Connection host name length. |
67+
| `ip.str` | string | Connection IP address. |
68+
| `ip.length` | unsigned integer | Connection IP address length. |
69+
| `database.str` | string | Default database specified at connection time. |
70+
| `database.length` | unsigned integer | Default database name length. |
71+
| `connection_type` | integer | Connection type code (numeric `0`–`5` or pseudo-constant; see below). |
72+
73+
`connection_type` values:
74+
75+
| Value | Meaning |
76+
| --- | --- |
77+
| `0` or `::undefined` | Undefined |
78+
| `1` or `::tcp/ip` | TCP/IP |
79+
| `2` or `::socket` | Socket |
80+
| `3` or `::named_pipe` | Named pipe |
81+
| `4` or `::ssl` | TCP/IP with encryption |
82+
| `5` or `::shared_memory` | Shared memory |
83+
84+
## `table_access`
85+
86+
Supported events: `read`, `insert`, `update`, `delete`
87+
REDUCED mode: all events
88+
89+
| Field name | Field type | Description |
90+
| --- | --- | --- |
91+
| `connection_id` | unsigned integer | Event connection ID. |
92+
| `sql_command_id` | integer | SQL command ID. |
93+
| `query.str` | string | SQL statement text. |
94+
| `query.length` | unsigned integer | SQL statement text length. |
95+
| `table_database.str` | string | Database name associated with event. |
96+
| `table_database.length` | unsigned integer | Database name length. |
97+
| `table_name.str` | string | Table name associated with event. |
98+
| `table_name.length` | unsigned integer | Table name length. |
99+
100+
## `global_variable` *(FULL mode only)*
101+
102+
Supported events: `get`, `set`
103+
104+
| Field name | Field type | Description |
105+
| --- | --- | --- |
106+
| `connection_id` | string | Event connection ID. |
107+
| `variable_name.str` | string | Variable name. |
108+
| `variable_name.length` | string | Variable name length. |
109+
| `variable_value.str` | string | Variable value. |
110+
| `variable_value.length` | string | Variable value length. |
111+
112+
## `command` *(FULL mode only)*
113+
114+
Supported events: `start`, `end`
115+
116+
| Field name | Field type | Description |
117+
| --- | --- | --- |
118+
| `status` | string | Command event status code. |
119+
| `connection_id` | string | Event connection ID. |
120+
| `command.str` | string | Command text. |
121+
| `command.length` | string | Command text length. |
122+
123+
## `query` *(FULL mode only)*
124+
125+
Supported events: `start`, `nested_start`, `status_end`, `nested_status_end`
126+
127+
| Field name | Field type | Description |
128+
| --- | --- | --- |
129+
| `status` | string | Query event status code. |
130+
| `connection_id` | string | Event connection ID. |
131+
| `sql_command_id` | string | SQL command string associated with the query event. The field name is retained as `sql_command_id` for compatibility. |
132+
| `query.str` | string | SQL query text. |
133+
| `query.length` | string | SQL query text length. |
134+
| `query_charset` | string | SQL query character set name. |
135+
136+
## `stored_program` *(FULL mode only)*
137+
138+
Supported events: `execute`
139+
140+
| Field name | Field type | Description |
141+
| --- | --- | --- |
142+
| `connection_id` | string | Event connection ID. |
143+
| `database.str` | string | Database where the stored program is defined. |
144+
| `database.length` | string | Database name length. |
145+
| `name.str` | string | Stored program name. |
146+
| `name.length` | string | Stored program name length. |
147+
148+
## `authentication` *(FULL mode only)*
149+
150+
Supported events: `flush`, `authid_create`, `credential_change`, `authid_rename`, `authid_drop`
151+
152+
| Field name | Field type | Description |
153+
| --- | --- | --- |
154+
| `status` | string | Authentication event status. |
155+
| `connection_id` | string | Event connection ID. |
156+
| `user.str` | string | User name. |
157+
| `user.length` | string | User name length. |
158+
| `host.str` | string | Host name. |
159+
| `host.length` | string | Host name length. |
160+
161+
## `message`
162+
163+
Supported events: `internal`, `user`
164+
REDUCED mode: all events
165+
166+
| Field name | Field type | Description |
167+
| --- | --- | --- |
168+
| `connection_id` | string | Event connection ID. |
169+
| `component.str` | string | Component name. |
170+
| `component.length` | string | Component name length. |
171+
| `producer.str` | string | Message producer name. |
172+
| `producer.length` | string | Message producer name length. |
173+
| `message.str` | string | Message text. |
174+
| `message.length` | string | Message text length. |
175+
176+
## `parse` *(FULL mode only)*
177+
178+
Supported events: `preparse`, `postparse`
179+
180+
| Field name | Field type | Description |
181+
| --- | --- | --- |
182+
| `connection_id` | string | Event connection ID. |
183+
| `flags` | string | Parse rewrite flags value. |
184+
| `query.str` | string | Original SQL query text. |
185+
| `query.length` | string | Original SQL query text length. |
186+
| `rewritten_query.str` | string | Rewritten SQL query text. |
187+
| `rewritten_query.length` | string | Rewritten SQL query text length. |
188+
189+
## Additional reading
190+
191+
* [Write audit_log_filter definitions](write-filter-definitions.md)
192+
* [Filter the Audit Log Filter logs](filter-audit-log-filter-files.md)
193+
* [Audit log filter functions, options, and variables](audit-log-filter-variables.md)
194+
* [Audit Log Filter restrictions](audit-log-filter-restrictions.md)
195+
* [Audit Log Filter file format overview](audit-log-filter-formats.md)
196+
* [Audit Log Filter overview](audit-log-filter-overview.md)
197+
* [Install the audit log filter](install-audit-log-filter.md)

‎docs/audit-log-filter-formats.md‎

Lines changed: 19 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,30 @@
11
# Audit Log Filter file format overview
22

3-
When an auditable event occurs, the component writes a record to the log file.
3+
On each auditable event, the component appends a record to the log. After startup, the first record describes the server and startup options; later records cover connections, disconnections, executed SQL, and more.
44

5-
After the component starts, the first record lists the description of the server and the options at startup. After the first record, the auditable events are connections, disconnections, SQL statements executed, and so on. Statements within stored procedures or triggers are not logged, only the top-level statements.
5+
Which server actions become audit records depends on [`audit_log_filter.event_mode`](audit-log-filter-variables.md#audit_log_filterevent_mode), not on the file format. See that variable for `REDUCED` versus `FULL` and for releases before it existed.
66

7-
If files are referenced by `LOAD_DATA`, the contents are not logged.
7+
If `LOAD_DATA` references files, the component does not log file contents.
88

9-
Set with the `audit_log_filter.format` system variable at startup. The available format types are the following;
9+
Set format with `audit_log_filter.format` at startup. Options:
1010

1111
| Format Type | Command | Description |
1212
|---|---|---|
13-
| [XML (new style)](audit-log-filter-new.md) | `audit_log_filter.format=NEW` | The default format |
14-
| [XML (old style)](audit-log-filter-old.md) | `audit_log_filter.format=OLD` | The original version of the XML format |
15-
| [JSON](audit-log-filter-json.md) | `audit_log_filter.format=JSON` | Files written as a JSON array |
16-
| [JSONL](audit-log-filter-json.md) | `audit_log_filter.format=JSONL` | Introduced in Percona Server for MySQL 8.4.9-9. Each event is a single compact JSON object on its own line inside a wrapping JSON array (the file is still valid JSON). Easy to process with `grep`, `jq`, `wc -l`, streaming pipelines, and log aggregation systems. |
13+
| [XML (new style)](audit-log-filter-new.md) | `audit_log_filter.format=NEW` | Default XML layout |
14+
| [XML (old style)](audit-log-filter-old.md) | `audit_log_filter.format=OLD` | Legacy XML layout |
15+
| [JSON](audit-log-filter-json.md) | `audit_log_filter.format=JSON` | One top-level JSON array of events |
16+
| [JSONL](audit-log-filter-json.md) | `audit_log_filter.format=JSONL` | Added in 8.4.9-9. One compact JSON object per line inside a wrapping array (see the JSON/JSONL topic). |
1717

18-
By default, the file contents in the new-style XML format are not compressed or encrypted.
18+
By default, new-style XML logs are neither compressed nor encrypted.
1919

20-
Changing the `audit_log_filter.format`, you should also change
21-
the `audit_log_filter.file` name. For example, changing the `audit_log_filter.format`
22-
to JSON or JSONL, change the `audit_log_filter.file` to `audit.json` or `audit.jsonl` respectively. If you don't change
23-
the `audit_log_filter.file` name, then all audit log filter files have the same
24-
base name and you won't be able to easily find when the format changed.
20+
When you change `audit_log_filter.format`, rename `audit_log_filter.file` as well—for example use `audit.json` or `audit.jsonl` for JSON or JSONL. Reusing one base name obscures format changes across rotated files.
2521

22+
## Additional reading
23+
24+
* [Audit Log Filter overview](audit-log-filter-overview.md)
25+
* [Audit Log Filter format - XML (new style)](audit-log-filter-new.md)
26+
* [Audit Log Filter format - XML (old style)](audit-log-filter-old.md)
27+
* [Audit Log Filter format - JSON and JSONL](audit-log-filter-json.md)
28+
* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) — `audit_log_filter.format`, `audit_log_filter.file`
29+
* [Reading Audit Log Filter files](reading-audit-log-filter-files.md)
30+
* [Audit Log Filter compression and encryption](audit-log-filter-compression-encryption.md)

0 commit comments

Comments
 (0)