feat: Add pagination and search support to /workflow/latest-versions API

[akhilpathivada]

Resolves #781

Summary

• Add optional start, size, filterField and filterValue query parameters to GET /metadata/workflow/latest-versions for server-side pagination and field-level search
• Introduce PaginatedMetadataDAO interface as an opt-in capability for DAOs that support database-level pagination and filtering
• Implement native SQL pagination (LIMIT/OFFSET) and search (ILIKE) in PostgresMetadataDAO
• Add ALLOWED_FILTER_FIELDS whitelist in MetadataServiceImpl for input validation
• Add a partial index on meta_workflow_def to eliminate full table scans for pagination queries
• Service layer falls back to in-memory pagination for DAOs that don't implement the new interface; non-Postgres backends return unfiltered results and rely on UI client-side filtering

What changed

Layer	File	Change
DAO Interface	PaginatedMetadataDAO.java	New opt-in interface with searchWorkflowDefsLatestVersions(start, size) and overloaded searchWorkflowDefsLatestVersions(start, size, filterField, filterValue)
Service Interface	MetadataService.java	Added both pagination-only and pagination+filter method signatures
Service Impl	MetadataServiceImpl.java	ALLOWED_FILTER_FIELDS whitelist; delegates to PaginatedMetadataDAO if available, otherwise falls back to in-memory pagination (no server-side filtering for non-Postgres)
Postgres DAO	PostgresMetadataDAO.java	Implements both methods — pagination uses COUNT + SELECT ... LIMIT ? OFFSET ?; search uses name ILIKE ? for name field, json_data::jsonb->>'field' ILIKE ? for others
REST Controller	MetadataResource.java	Added optional start/size/filterField/filterValue params; changed return type from List<WorkflowDef> to SearchResult<WorkflowDef>
Unit Tests	MetadataServiceTest.java	Tests for DAO-delegated and in-memory fallback paths, filter with PaginatedDAO, filter with non-PaginatedDAO, empty filter, invalid filter field
Integration Tests	PostgresMetadataDAOTest.java	Tests for pagination boundaries, filter by name (case-insensitive), filter by JSON fields, no match, filter+pagination combined
REST Tests	MetadataResourceTest.java	Tests for various start/size/filterField/filterValue combinations and default behavior
DB Migration	V15__workflow_def_latest_version_index.sql	Partial index on meta_workflow_def(name) WHERE version = latest_version for optimized pagination queries

API change

GET /metadata/workflow/latest-versions?start={start}&size={size}&filterField={field}&filterValue={value}

Parameter	Required	Default	Description
start	No	0	Starting index (0-based)
size	No	100	Number of results per page
filterField	No	—	Workflow definition field to filter on (must be in ALLOWED_FILTER_FIELDS)
filterValue	No	—	Case-insensitive substring to match against the field

Allowed filter fields: name, description, ownerEmail, version, schemaVersion, timeoutPolicy, timeoutSeconds, restartable, workflowStatusListenerEnabled

Response (changed from List<WorkflowDef> to SearchResult<WorkflowDef>):

{
  "totalHits": 102,
  "results": [
    { "name": "workflow_a", "version": 3, ... },
    { "name": "workflow_b", "version": 1, ... }
  ]
}

When no parameters are provided, all latest versions are returned wrapped in SearchResult for backward compatibility.

Design decisions

1. Why an optional interface instead of modifying MetadataDAO?
Adding a new method to MetadataDAO would break all existing implementations (Redis, MySQL, etc.). The opt-in PaginatedMetadataDAO interface lets each persistence module adopt pagination at its own pace, with zero impact on others.

2. Why SearchResult instead of List?
The UI needs totalHits to render pagination controls (e.g., "Page 3 of 7"). This follows the existing pattern used by /workflow/search.

3. Why OFFSET/LIMIT over keyset pagination?
Workflow definitions typically number in the hundreds to low thousands. OFFSET/LIMIT is simpler, allows arbitrary page jumping, and performs well at this scale. Keyset pagination would add complexity (cursor management, no page jumping) with minimal benefit.

4. Why a partial index?
Without an index, Postgres performs a full table scan on every page request — even with LIMIT 15, it must read all rows to filter WHERE version = latest_version and sort by name. The partial index ON meta_workflow_def (name) WHERE version = latest_version pre-filters and pre-sorts only the latest version rows, reducing page queries from full table scans to direct index lookups. The index is created using CONCURRENTLY to avoid blocking writes on existing deployments.

6. Why json_data::jsonb cast?
The json_data column is text type, not jsonb. The ->> operator requires jsonb, so an explicit ::jsonb cast is needed. The name field has a dedicated DB column and is queried directly without cast.

7. Why no server-side filtering for non-Postgres backends?
Non-Postgres DAOs don't implement PaginatedMetadataDAO. Rather than loading all definitions into memory for in-memory filtering (expensive), the service returns unfiltered paginated results and the UI falls back to client-side filtering. This keeps the backend lightweight and maintains backward compatibility.

[akhilpathivada]

@v1r3n could you please help reviewing this?