MSQ Compaction Not Working (Overlord Metadata Cache Syncing Forever)

[GWphua]

Affected Version

Master Branch
Between v36~v37: (HEAD at commit 8307d8a, PR#19030)

Encountered when trying out MSQ Compaction using MM-less ingestion.
The test datasource is ingested via Kafka supervisor.

Relevant Overlord settings:

## Required for Supervisor compaction in v36
    druid_supervisor_compaction_enabled: true
    druid_supervisor_compaction_engine: "msq"
    druid_manager_segments_useIncrementalCache: "always"

// Loadlist for MSQ extension + MM-less

Description

I have set up supervisor auto-compaction, using this and created a supervisor to compact my datasource.

However, the supervisor keeps showing 'RUNNING', with no tasks being created. Looking at the overlord logs, it seems like there's some problem metadata cache. The leader is trying to sync for an hour...

2026-03-09T09:38:12,533 INFO [qtp1741034833-54] org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Wait complete. Cache is now in state[LEADER_FIRST_SYNC_PENDING].
...
2026-03-09T09:44:12,325 INFO [CompactionScheduler-0] org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Wait complete. Cache is now in state[LEADER_FIRST_SYNC_PENDING].
... // Same logs till Overlord restart
2026-03-09T10:16:50,529 INFO [CompactionScheduler-0] org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Wait complete. Cache is now in state[FOLLOWER].
... // Pending logs after restart again
2026-03-09T10:36:58,682 INFO [CompactionScheduler-0] org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Wait complete. Cache is now in state[LEADER_FIRST_SYNC_PENDING]

I am testing this on a test cluster, with only 1 datasource of 6065 segments. This should not take an hour. I'm not sure if there's some kind of deadlocks preventing the sync.

Coordinator

INFO [org.apache.druid.metadata.SqlSegmentsMetadataManager-Exec--0] org.apache.druid.metadata.SqlSegmentsMetadataManager - Polled and found [6,065] segments in the database in [152]ms.

Overlord

WARN [qtp1741034833-73] org.apache.druid.metadata.segment.SqlSegmentMetadataTransactionFactory - Starting read-write transaction for datasource[test_ds]. Reads will be done directly from metadata store since cache is not synced yet.

[kfaraz]

Thanks for reporting the issue, @GWphua !
It seems that the docs are out-of-date. The property druid.supervisor.compaction.enabled and druid.supervisor.compaction.engine are not used anymore.
Instead, these fields are now available in the compaction dynamic config.
Set useSupervisors to true and engine to msq for your use case.

Fix for docs: #19113

[GWphua]

Hi @kfaraz

I have met an NPE bug while looking at the problem. Proposed fix is in #19118.

I have also tried setting the configs as you advised. However, the problem still persists. I think something might be wrong with the metadata cache sync. For a cluster of 6000 segments, this should not take a long time.

2026-03-10T03:38:59,523 WARN [qtp1741034833-58] org.apache.druid.metadata.segment.SqlSegmentMetadataTransactionFactory - Starting read-write transaction for datasource[pingmesh.inter_idc_rtt_projected]. Reads will be done directly from metadata store since cache is not synced yet.
2026-03-10T03:38:59,523 INFO [qtp1741034833-58] org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Waiting for cache to finish sync with metadata store.
2026-03-10T03:39:02,222 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Notified task runner to clean up [0] tasks with IDs[[]].
2026-03-10T03:39:02,230 INFO [TaskQueue-StorageSync] org.apache.druid.indexing.overlord.TaskQueue - Synced [0] tasks from storage (0 tasks added, 0 tasks removable, 0 tasks removed).
2026-03-10T03:39:02,323 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Notified task runner to clean up [0] tasks with IDs[[]].
2026-03-10T03:40:02,220 INFO [TaskQueue-StorageSync] org.apache.druid.indexing.overlord.TaskQueue - Synced [0] tasks from storage (0 tasks added, 0 tasks removable, 0 tasks removed).
2026-03-10T03:40:02,221 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Notified task runner to clean up [0] tasks with IDs[[]].
2026-03-10T03:41:00,024 WARN [qtp1741034833-72] org.apache.druid.metadata.segment.SqlSegmentMetadataTransactionFactory - Starting read-write transaction for datasource[pingmesh.inter_idc_rtt_projected]. Reads will be done directly from metadata store since cache is not synced yet.
2026-03-10T03:41:00,024 INFO [qtp1741034833-72] org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Waiting for cache to finish sync with metadata store.
... // 7 minutes of related logs
2026-03-10T03:48:59,523 INFO [qtp1741034833-58] org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Wait complete. Cache is now in state[LEADER_FIRST_SYNC_PENDING].
2026-03-10T03:48:59,524 INFO [qtp1741034833-58] org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache - Waiting for cache to finish sync with metadata store.

[kfaraz]

@GWphua , I have approved #19118 , thanks for the fix!

The cache taking too long to sync 6000 segments is indeed weird. It shouldn't take more than a couple of seconds.
This code has not been touched recently either and I have been using it in production for quite some time now.

Is the cache able to sync even once? How about the Coordinator, is that also taking time to sync?
You can set the property druid.manager.segments.useIncrementalCache: always on Coordinator too to see if the cache works fine there.

[GWphua]

@kfaraz

I found the fix to the problem.

It turns out that in my cluster configuration, the following config is only provided for the Coordinator. The Overlord did not have this config. Adding this config to the overlord will then allow me to see compaction tasks being run.

druid_coordinator_asOverlord_enabled: false

Failure to configure this config on the Overlord is harmless before #17935. The fact that the config is prefixed with druid_coordinator, and there is no mention in the documentations, made me believe that there is no need to consider applying this configuration on the Overlord.

After #17935, if users fail to add this config, isStandalone() will return false. Users will keep seeing Waiting for cache to finish sync with metadata store. in the logs, but the sync will never finish, because the Metadata Cache Sync will is not started at all.

if (isStandalone()) {
  segmentMetadataCache.start();
}

// How isStandalone is resolved.
private boolean isStandalone()
{
  return !coordinatorOverlordServiceConfig.isEnabled();
}

Documentation / upgrade notes will be helpful for users looking to upgrade / use this feature.

[kfaraz]

Oh, so you had to explicitly set druid.coordinator.asOverlord.enabled: false? But this config is already false by default.
Was the Overlord getting properties from some other source/template as well, which was probably setting this to true first?

Documentation / upgrade notes will be helpful for users looking to upgrade / use this feature.

Fair enough. I already have a PR up, let me include this there.

[GWphua]

I think it's because we are simply copying the configs from examples/conf/druid/auto/coordinator-overlord/runtime.properties

In here, druid.coordinator.asOverlord.enabled=true and hence the default is being overwritten.

Then, by specifying druid_coordinator_asOverlord_enabled: false in our Helm chart, we overwrite this again back to false and hence fixed the problem 🤣

[GWphua]

Documentation / upgrade notes will be helpful for users looking to upgrade / use this feature.

Fair enough. I already have a PR up, let me include this there.

Understand that #17935 is introduced a while back. If there's a need to propagate the docs back to earlier versions (if applicable), maybe a new PR is better to do backports (assuming we plan to)?

[kfaraz]

Understand that #17935 is introduced a while back. If there's a need to propagate the docs back to earlier versions (if applicable), maybe a new PR is better to do backports (assuming we plan to)?

Given that the issue was the config setup, I think we may just skip the docs.
The default behaviour of the config (false) is as expected and in that case, the Overlord cache would work fine.

Setting druid.coordinator.asOverlord.enabled=true on the Overlord when it is actually intended to run in a standalone mode is actually a misconfiguration anyway.

Let me know if you still feel that some docs would be helpful for this. Or if you have the changes in mind, please feel free to create a PR.

[GWphua]

I feel that some docs will still help, especially maybe a line saying that the druid.coordinator.asOverlord.enabled=true will should never be configured on the Overlord alone. (Or even incorporate such checks in the codebase? sounds complicated tho so ill skip on this)

This helps operators to be aware of this config. It took a while for me to find the cause, and I want to prevent similar cases from happening.