Skip to content

[BUG] Hard delete of Table does not clean up profiler data from profiler_data_time_series — stale profiles resurface on re-creation #27041

@RajdeepKushwaha5

Description

@RajdeepKushwaha5

Affected module
Backend

Describe the bug
When a Table entity is permanently deleted (hard delete), its associated profiler data — table profiles, column profiles, and system profiles — remain in the profiler_data_time_series database table. If a table with the same fully qualified name is later re-created (e.g., re-ingested), the old profiling data resurfaces in the UI.

TableRepository writes to profiler_data_time_series using three extensions:

  • table.tableProfile
  • table.columnProfile
  • table.systemProfile

But it has no entitySpecificCleanup() or postDelete() override to delete these records on hard delete. The base EntityRepository.cleanup() does not include any generic time series cleanup either.

To Reproduce

  1. Create a database service and ingest a table
  2. Run the profiler to generate table and column profile data
  3. Permanently delete the table via API: DELETE /api/v1/tables/name/{fqn}?hardDelete=true&recursive=true
  4. Re-ingest the same table (same fully qualified name)
  5. Navigate to the Profiler tab for the table — old profile data from the deleted table appears

You can also verify directly in the database:

SELECT COUNT(*) FROM profiler_data_time_series 
WHERE entityFQNHash = '<hash_of_deleted_table_fqn>';
-- Returns > 0 even after hard delete

Expected behavior
Hard-deleting a table should remove all associated profiler data (table profiles, column profiles, system profiles) from profiler_data_time_series. Re-creating a table with the same name should start with a clean profiler history.

Suggested fix
Add an entitySpecificCleanup() override to TableRepository that deletes from profiler_data_time_series for all three profile extensions. Note: column profiles are stored with column-level FQNs (e.g., service.db.schema.table.column), so a prefix-based delete method may need to be added to EntityTimeSeriesDAO (currently only exact FQN match exists).

Version:

  • OpenMetadata version: 1.6.0-SNAPSHOT (main branch, commit 58a76d7)

Additional context
Similar to #27001 where PipelineRepository was missing time series cleanup on hard delete. This is part of a broader pattern where repositories that write to time series tables don't always clean up on entity deletion.


Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions