Bug: Configuration list missing items due to unpaginated DynamoDB scan

[kaleko]

Summary

The UI's "View/Edit Configuration" page silently drops configs that land beyond the first page of the DynamoDB scan. With 230+ config versions in the table, only ~58 are returned to the UI. This issue likely impacts other areas of the UI that do the same scan, e.g. the upload document --> configuration version dropdown.

Reproduction

1. Have 100+config versions in ConfigurationTable
2. Upload a new config via idp-cli config-upload
3. Open the UI configuration page
4. New config does not appear in the list

Root Cause

ConfigurationManager.list_config_versions() in lib/idp_common_pkg/idp_common/config/configuration_manager.py (line 411) performs a single table.scan() without paginating through LastEvaluatedKey:

def list_config_versions(self) -> List[Dict[str, Any]]:
    response = self.table.scan(
        FilterExpression="begins_with(Configuration, :config_prefix)",
        ExpressionAttributeValues={":config_prefix": f"{CONFIG_TYPE_CONFIG}#"},
        ProjectionExpression="Configuration, IsActive, CreatedAt, UpdatedAt, Description, ..."
    )

    versions = []
    for item in response.get('Items', []):
        # ...
    return versions  # Only first page!

DynamoDB scan returns at most 1MB per call. With 230 items across 4 pages, any config not on page 1 is invisible to the UI.

Impact

• Configs uploaded via CLI or autotune agent may not appear in the UI
• Users cannot view/edit/activate these configs through the UI
• The configs still work when referenced by name in processing runs

Recommended Fix

Add pagination loop:

def list_config_versions(self) -> List[Dict[str, Any]]:
    versions = []
    scan_kwargs = {
        "FilterExpression": "begins_with(Configuration, :config_prefix)",
        "ExpressionAttributeValues": {":config_prefix": f"{CONFIG_TYPE_CONFIG}#"},
        "ProjectionExpression": "Configuration, IsActive, CreatedAt, UpdatedAt, Description, BdaProjectArn, BdaSyncStatus, BdaLastSyncedAt, Managed",
    }

    while True:
        response = self.table.scan(**scan_kwargs)
        for item in response.get('Items', []):
            config_key = item.get('Configuration', '')
            if "#" in config_key:
                _, version = config_key.split("#", 1)
                versions.append({
                    "versionName": version,
                    "isActive": item.get('IsActive'),
                    "createdAt": item.get('CreatedAt'),
                    "updatedAt": item.get('UpdatedAt'),
                    "description": item.get('Description', ""),
                    "bdaProjectArn": item.get('BdaProjectArn'),
                    "bdaSyncStatus": item.get('BdaSyncStatus'),
                    "bdaLastSyncedAt": item.get('BdaLastSyncedAt'),
                    "managed": item.get('Managed', False),
                })
        if 'LastEvaluatedKey' not in response:
            break
        scan_kwargs['ExclusiveStartKey'] = response['LastEvaluatedKey']

    return versions

File

lib/idp_common_pkg/idp_common/config/configuration_manager.py, line 411