Skip to content

Commit c2d52d3

Browse files
tisniktisnik
committed
LCORE-2661: Database description
1 parent fb0e2bf commit c2d52d3

1 file changed

Lines changed: 225 additions & 21 deletions

File tree

‎docs/design/automatic-conversation-expiration/automatic-conversation-expiration.md‎

Lines changed: 225 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -41,12 +41,25 @@
4141
* [Pros](#pros-2)
4242
* [Cons](#cons-2)
4343
* [Configuration options](#configuration-options)
44+
* [Conclusion](#conclusion)
45+
* [Audit log](#audit-log)
46+
* [Requirements for audit log](#requirements-for-audit-log)
47+
* [Security reviews](#security-reviews)
48+
* [Operational troubleshooting](#operational-troubleshooting)
49+
* [Accountability and governance](#accountability-and-governance)
50+
* [Correctness](#correctness)
4451
* [Performance impact](#performance-impact)
4552
* [Requirements](#requirements)
4653
* [Architecture](#architecture)
4754
* [Trigger mechanism](#trigger-mechanism)
4855
* [Existing storage / data model](#existing-storage--data-model)
56+
* [Table `user_conversation`](#table-user_conversation)
57+
* [Table `user_turn`](#table-user_turn)
58+
* [Table `cache`](#table-cache)
59+
* [Table `conversation_summaries`](#table-conversation_summaries)
60+
* [Table `conversations`](#table-conversations)
4961
* [Storage / data model changes](#storage--data-model-changes)
62+
* [Audit log table](#audit-log-table)
5063
* [Alternative designs considered](#alternative-designs-considered)
5164
* [Acceptance criteria](#acceptance-criteria)
5265
* [Configurable idle TTL enforcement (core)](#configurable-idle-ttl-enforcement-core)
@@ -111,7 +124,7 @@ Several use cases have been identified:
111124

112125
### Configurable idle TTL enforcement
113126

114-
- An admin sets an idle timeout (e.g., 30 minutes) for a tenant.
127+
- An administrator sets an idle timeout (e.g., 30 minutes) for a tenant.
115128
- When a conversation is inactive longer than the TTL, the system automatically deletes it from every configured persistence layer.
116129

117130
### Active session usability preserved
@@ -303,7 +316,73 @@ have to be persisted - it will be recreated after LCore restart.
303316
| database_reconnection_count | integer | Database reconnection count on startup. When database with conversations is not available on startup, the service tries to reconnect N times with specified delay. |
304317
| database_reconnection_delay | integer | Database reconnection delay specified in seconds. When database with conversations is not available on startup, the service tries to reconnect N times with specified delay. |
305318

306-
"2 days 5 hours"
319+
The `minimal_retention_time` is written in a human readable format, for example:
320+
321+
```
322+
2 days 5 hours
323+
600 minutes
324+
```
325+
326+
etc.
327+
328+
### Conclusion
329+
330+
In conclusion, the runner-based approach was selected because it is consistent
331+
with the system’s current background job architecture, can be implemented using
332+
the existing schema with limited or no disruptive changes, and provides
333+
predictable, policy-aligned deletion behavior. This minimizes integration risk
334+
and avoids shifting cleanup work into the critical request path, while still
335+
allowing performance safeguards through indexing and controlled execution.
336+
337+
## Audit log
338+
339+
### Requirements for audit log
340+
341+
Audit logging is needed for conversation deletion because deleting
342+
user-generated conversational data is a high-impact, policy-driven operation
343+
that must be transparent, traceable, and verifiable.
344+
345+
### Security reviews
346+
347+
First, audit logs provide evidence for compliance and security reviews. When
348+
deletion is triggered automatically (for example by an idle TTL/retention
349+
policy), the audit entry records that the system enforced the configured
350+
policy, including when the deletion happened, which conversation was affected,
351+
and the reason code (e.g., “idle TTL expired” vs “manual deletion”). This
352+
enables auditors to confirm that stale data is not retained longer than
353+
intended.
354+
355+
### Operational troubleshooting
356+
357+
Second, audit logs support operational troubleshooting. Conversation deletion
358+
may touch multiple persistence layers (primary storage, caches, indexes,
359+
derived artifacts). If a later inconsistency occurs—such as a conversation
360+
still appearing in a UI, search results, or a downstream index—audit records
361+
help identify the exact cleanup run, the deletion path taken, and the decision
362+
timing. This makes it far easier to diagnose whether the issue is due to missed
363+
propagation, temporary failures, or a race condition.
364+
365+
### Accountability and governance
366+
367+
Third, audit logs improve accountability and governance. Manual REST deletions
368+
should be attributable to a caller or request context, while automatic
369+
deletions should be attributable to the lifecycle/cleanup subsystem and its
370+
configuration state. Recording both types of events provides a complete
371+
narrative of “what happened” and “what triggered it,” which is important for
372+
incident response and for change control when retention settings are modified.
373+
374+
### Correctness
375+
376+
Fourth, audit logs help ensure correctness in edge cases. TTL-based cleanup can
377+
be affected by near-boundary timing, concurrent activity, retries, and partial
378+
failures. Having a durable audit trail makes it possible to confirm that the
379+
system applied idle semantics correctly (e.g., did not delete a conversation
380+
that received new activity just before TTL expiry) and that retries did not
381+
accidentally reprocess or resurrect data.
382+
383+
Overall, audit logging turns deletion from a silent background action into an
384+
observable, reviewable workflow—critical for both meeting retention
385+
expectations and maintaining confidence in the system’s behavior over time.
307386

308387
## Performance impact
309388

@@ -343,27 +422,61 @@ does not exceed 1% under representative production-like workloads.
343422

344423
## Requirements
345424

425+
Different part of source code need to be updated:
426+
427+
1. New runner need to be added
428+
1. Existing database schema need to be altered
429+
1. New table for audit log need to be added
430+
346431
# Architecture
347432

348433
## Trigger mechanism
349434

435+
The runner will trigger the conversation deletion subroutines in a timely
436+
manner, driven by a configurable `period` option. That period defines how often
437+
the runner wakes up to evaluate which conversations are eligible for cleanup
438+
(for example, based on idle time exceeding the configured TTL), and then
439+
initiates the appropriate deletion workflow.
440+
441+
When the configured period elapses, the runner executes its selection logic,
442+
processes eligible conversations (typically in controlled batches), and calls
443+
the underlying deletion subroutines for each candidate. After completing the
444+
current cycle, the runner waits until the next scheduled period before
445+
repeating the process. This ensures deletion is performed predictably according
446+
to configuration, while avoiding unnecessary database load from overly frequent
447+
scans.
448+
449+
A very similar mechanism is already implemented in the quota handler. The quota
450+
handler runs as a background component that periodically wakes up based on
451+
configuration, evaluates what work is due (for example, quota items or
452+
scheduled quota updates), and then executes the corresponding operations in a
453+
loop. This includes controlled batching to limit load, and it relies on the
454+
same overall runner lifecycle pattern used elsewhere in the system (start,
455+
periodic execution, and graceful shutdown).
456+
457+
Because the quota handler already solves the problem of “run scheduled work
458+
reliably without impacting request latency,” the conversation cleanup runner
459+
can reuse the same implementation approach: a configurable schedule/period, a
460+
loop-driven execution model, and consistent error handling and backoff
461+
behavior.
462+
350463
## Existing storage / data model
351464

352-
```sql
353-
CREATE TABLE user_conversation (
354-
id VARCHAR NOT NULL,
355-
user_id VARCHAR NOT NULL,
356-
last_used_model VARCHAR NOT NULL,
357-
last_used_provider VARCHAR NOT NULL,
358-
created_at DATETIME DEFAULT CURRENT_TIMESTAMP NOT NULL,
359-
last_message_at DATETIME DEFAULT CURRENT_TIMESTAMP NOT NULL,
360-
last_response_id VARCHAR,
361-
message_count INTEGER NOT NULL,
362-
topic_summary VARCHAR NOT NULL,
363-
PRIMARY KEY (id)
364-
);
365-
CREATE INDEX ix_user_conversation_user_id ON user_conversation (user_id);
366-
```
465+
The existing database schema affected by this feature request is described in
466+
this section. It covers the tables and columns that are currently used to store
467+
conversation records and track conversation metadata (such as tenant/account
468+
ownership and timestamps).
469+
470+
This includes any existing structures that the new conversation deletion runner
471+
will query against to determine which conversations have exceeded the
472+
configured idle TTL, as well as the structures that will be updated during
473+
cleanup (for example, status flags, deletion markers, or timestamp fields used
474+
to control “last used” semantics). It also highlights how the current schema
475+
supports (or limits) efficient cleanup operations, including which indexes
476+
exist today and which access patterns may require additional indexing to avoid
477+
performance regression.
478+
479+
### Table `user_conversation`
367480

368481
```sql
369482
CREATE TABLE user_conversation (
@@ -381,6 +494,10 @@ CREATE TABLE user_conversation (
381494
CREATE INDEX ix_user_conversation_user_id ON user_conversation (user_id);
382495
```
383496

497+
This table contains timestamp stored in a column named `last_message_at` that can be used by cleaner.
498+
499+
### Table `user_turn`
500+
384501
```sql
385502
CREATE TABLE user_turn (
386503
conversation_id VARCHAR NOT NULL,
@@ -396,6 +513,10 @@ CREATE TABLE user_turn (
396513
CREATE INDEX ix_user_turn_response_id ON user_turn (response_id);
397514
```
398515

516+
This table contains timestamp stored in a column named `completed_at` that can be used by cleaner.
517+
518+
### Table `cache`
519+
399520
```sql
400521
CREATE TABLE cache (
401522
user_id text NOT NULL,
@@ -418,6 +539,11 @@ CREATE INDEX timestamps
418539
;
419540
```
420541

542+
This table contains timestamp stored in a column named `completed_at` that can be used by cleaner.
543+
Additionally we can access this table via compound key `user_id`+`conversation_id`.
544+
545+
### Table `conversation_summaries`
546+
421547
```sql
422548
CREATE TABLE conversation_summaries (
423549
user_id text NOT NULL,
@@ -431,6 +557,11 @@ CREATE TABLE conversation_summaries (
431557
);
432558
```
433559

560+
This table contains timestamp stored in a column named `created_at` that can be used by cleaner.
561+
Additionally we can access this table via compound key `user_id`+`conversation_id`.
562+
563+
### Table `conversations`
564+
434565
```sql
435566
CREATE TABLE conversations (
436567
user_id text NOT NULL,
@@ -441,10 +572,82 @@ CREATE TABLE conversations (
441572
);
442573
```
443574

575+
This table contains timestamp stored in a column named `last_message_timestamp` that can be used by cleaner.
576+
We can access this table via compound key `user_id`+`conversation_id`.
577+
444578
## Storage / data model changes
445579

580+
The existing database schema is sufficient for implementing the requested
581+
behavior. The current tables already contain the key fields needed to determine
582+
conversation eligibility for deletion (such as tenant/account ownership and
583+
conversation activity timestamps), and they already support the queries
584+
required by the cleanup workflow without requiring disruptive structural
585+
changes.
586+
587+
In addition, the schema provides a place to record or infer cleanup state
588+
(e.g., whether a conversation is still active, pending cleanup, or already
589+
removed). As a result, the runner can select expired conversations efficiently
590+
and apply the deletion subroutines using the existing persistence structures.
591+
592+
Where needed for performance, the feature can rely on creating or adjusting
593+
database indexes to optimize the cleanup access patterns. This allows the
594+
system to avoid full scans and to keep the runtime cost predictable while
595+
implementing TTL/idle-based deletion using the schema that already exists.
596+
597+
598+
## Audit log table
599+
600+
The new table needs to be created (on the fly) for storing audit log information.
601+
Schema of this table can be following:
602+
603+
```sql
604+
CREATE TABLE audit_log_conversation_deletion (
605+
id serial NOT NULL,
606+
deletion_time DATETIME DEFAULT CURRENT_TIMESTAMP NOT NULL,
607+
user_id text NOT NULL,
608+
conversation_id text NOT NULL,
609+
deletion_source text NOT NULL,
610+
requested_by text NOT NULL,
611+
status text NOT NULL,
612+
error_code int,
613+
error_message text,
614+
PRIMARY KEY(user_id, conversation_id)
615+
);
616+
```
617+
618+
Deletion source can identify Who/what triggered the deletion:
619+
620+
1. automatic_cleaner
621+
1. manual_rest_call
622+
1. admin_action
623+
624+
446625
## Alternative designs considered
447626

627+
No alternate design was considered because the proposed solution aligns with
628+
the system’s existing architecture and operational model, and it directly meets
629+
the stated retention requirement with minimal disruption.
630+
631+
In particular, the platform already uses the runner pattern for periodic
632+
background work (e.g., the quota handler). Implementing conversation cleanup as
633+
another runner follows the same control flow, configuration approach, and
634+
lifecycle management, which reduces integration risk and avoids introducing a
635+
new scheduling/execution paradigm.
636+
637+
Additionally, the existing database schema is sufficient to implement idle
638+
TTL-based deletion without requiring major structural changes. This further
639+
lowers the cost and complexity of the implementation and avoids risky data
640+
migrations or re-modeling.
641+
642+
Finally, alternative designs (such as handling cleanup through request-path
643+
logic, external batch jobs, or implementing a different scheduling mechanism)
644+
would either increase runtime overhead on the critical request path or
645+
introduce operational complexity and harder-to-prove retention guarantees. The
646+
runner-based async execution model provides predictable timing, controllable
647+
load (via batching/period configuration), and clear observability—meeting the
648+
high-security customer expectations with the least change to the current
649+
system.
650+
448651
# Acceptance criteria
449652

450653
## Configurable idle TTL enforcement (core)
@@ -578,8 +781,9 @@ CREATE TABLE conversations (
578781

579782
TODO: Record significant changes after initial creation.
580783

581-
| Date | Change | Reason |
582-
|------------|---------------------|-----------------|
583-
| 2026-06-29 | Initial version | feature request |
584-
| 2026-07-01 | Problem description | initial spike |
784+
| Date | Change | Reason |
785+
|------------|-----------------------------|-----------------|
786+
| 2026-06-29 | Initial version | feature request |
787+
| 2026-07-01 | Problem description | initial spike |
788+
| 2026-07-06 | Database schema description | initial spike |
585789

0 commit comments

Comments
 (0)