Skip to content

Commit b312f03

Browse files
hari-kuriakoseclaudepre-commit-ci[bot]
authored
refactor: Add dynamic plugin loading for enterprise components (#1736)
* refactor: Add dynamic plugin loading for enterprise components ## What - Add dynamic plugin loading support to OSS codebase - Enable enterprise components to be loaded at runtime without modifying tracked files ## Why - Enterprise code was overwriting git-tracked OSS files causing dirty git state - Need clean separation between OSS and enterprise codebases - OSS should work independently without enterprise components ## How - `unstract_migrations.py`: Uses try/except ImportError to load from `pluggable_apps.migrations_ext` - `api_hub_usage_utils.py`: Uses try/except ImportError to load from `plugins.verticals_usage` - `utils.py`: Uses try/except ImportError to load from `pluggable_apps.manual_review_v2` and `plugins.workflow_manager.workflow_v2.rule_engine` - `backend.Dockerfile`: Conditional install of `requirements.txt` if present ## Can this PR break any existing features. If yes, please list possible items. If no, please explain why. - No. The changes add optional plugin loading that gracefully falls back to default behavior when plugins are not present. Existing OSS functionality is preserved. ## Database Migrations - None ## Env Config - None ## Relevant Docs - None ## Related Issues or PRs - None ## Dependencies Versions - None ## Notes on Testing - OSS build: Verify app starts and works without enterprise plugins - Enterprise build: Verify plugins are loaded and function correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor: Use get_plugin() for API Hub usage utilities 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Refactor random sampling logic in utils.py Removed redundant import of random and exception handling for manual_review_v2. Signed-off-by: Hari John Kuriakose <hari@zipstack.com> * fix: Add Traefik port labels and clean up service ignore list - Add explicit loadbalancer port labels for backend (8000) and frontend (3000) services in docker-compose to ensure proper Traefik routing - Rename spawned_services to ignored_services for clarity - Extend ignored_services list to include tool-classifier, tool-text_extractor, and worker-unified services that don't need environment setup 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Update frontend Docker config for nginx serving Update Traefik port label to 80 to match nginx and fix Dockerfile to use BUILD_CONTEXT_PATH for the runtime config script in both dev and prod stages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Use ARG instead of ENV for BUILD_CONTEXT_PATH in frontend Dockerfile Convert BUILD_CONTEXT_PATH from environment variable to build argument for proper Docker multi-stage build support. ARGs must be declared globally and re-declared in each stage that needs them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: typo in ignored services var name * fix: handle script execution via entrypoint --------- Signed-off-by: Hari John Kuriakose <hari@zipstack.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 435c6cf commit b312f03

7 files changed

Lines changed: 212 additions & 82 deletions

File tree

Lines changed: 62 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,66 @@
1+
"""Migration utilities for Unstract.
2+
3+
This module provides the UnstractMigration class which extends MigrationQuery.
4+
Additional migrations can be provided through pluggable_apps.migrations_ext
5+
which will be dynamically loaded if available.
6+
"""
7+
8+
import logging
9+
110
from migrating.v2.query import MigrationQuery
211

12+
logger = logging.getLogger(__name__)
13+
314

415
class UnstractMigration(MigrationQuery):
5-
pass
16+
"""Migration query class that supports extensible migrations.
17+
18+
Additional migrations can be loaded from pluggable_apps.migrations_ext
19+
if that module is available.
20+
"""
21+
22+
def get_public_schema_migrations(self) -> list[dict[str, str]]:
23+
"""Returns a list of dictionaries containing the schema migration details.
24+
25+
Returns:
26+
list: A list of dictionaries containing the schema migration details.
27+
"""
28+
core_migrations = super().get_public_schema_migrations()
29+
30+
try:
31+
from pluggable_apps.migrations_ext.migrations import (
32+
get_extended_public_schema_migrations,
33+
)
34+
35+
return core_migrations + get_extended_public_schema_migrations(self.v2_schema)
36+
except ImportError:
37+
pass
38+
39+
return core_migrations
40+
41+
def get_organization_migrations(
42+
self, schema: str, organization_id: str
43+
) -> list[dict[str, str]]:
44+
"""Returns a list of dictionaries containing the organization migration details.
45+
46+
Args:
47+
schema (str): The name of the schema for the organization.
48+
organization_id (str): The ID of the organization.
49+
50+
Returns:
51+
list: A list of dictionaries containing the organization migration details.
52+
"""
53+
core_migrations = super().get_organization_migrations(schema, organization_id)
54+
55+
try:
56+
from pluggable_apps.migrations_ext.migrations import (
57+
get_extended_organization_migrations,
58+
)
59+
60+
return core_migrations + get_extended_organization_migrations(
61+
self.v2_schema, schema, organization_id
62+
)
63+
except ImportError:
64+
pass
65+
66+
return core_migrations
Lines changed: 51 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,61 +1,79 @@
11
"""API Hub usage tracking utilities for workflow execution.
22
33
This module provides a clean interface for API Hub usage tracking in workflows.
4-
The OSS version provides default no-op implementations that can be overridden
5-
by enterprise plugins during the build process.
4+
Usage tracking functionality is loaded via get_plugin if available.
65
"""
76

87
import logging
98

9+
from plugins import get_plugin
10+
1011
logger = logging.getLogger(__name__)
1112

1213

1314
class APIHubUsageUtil:
14-
"""Utility class for handling API Hub usage tracking in workflows.
15-
16-
This is the OSS version that provides default no-op implementations.
17-
Enterprise builds will replace this file with enhanced functionality.
18-
"""
15+
"""Utility class for handling API Hub usage tracking in workflows."""
1916

2017
@staticmethod
2118
def track_api_hub_usage(
2219
workflow_execution_id: str,
2320
workflow_file_execution_id: str,
2421
organization_id: str | None = None,
2522
) -> bool:
26-
"""Track API hub usage if enterprise plugin is available.
27-
28-
OSS version: This is a no-op implementation that always returns False.
29-
Enterprise version: Will track actual usage in verticals.subscription_usage table.
23+
"""Track API hub usage if the verticals_usage plugin is available.
3024
3125
Args:
3226
workflow_execution_id: The workflow execution ID
3327
workflow_file_execution_id: The file execution ID
3428
organization_id: Optional organization ID
3529
3630
Returns:
37-
bool: False in OSS version (usage tracking not available).
38-
Enterprise version returns True if tracking succeeded.
31+
bool: True if tracking succeeded, False otherwise.
3932
"""
40-
# OSS version - no usage tracking available
41-
logger.debug("API hub usage tracking not available in OSS version")
42-
return False
33+
verticals_usage_plugin = get_plugin("verticals_usage")
34+
if not verticals_usage_plugin:
35+
return False
36+
37+
try:
38+
headers_cache = verticals_usage_plugin["headers_cache_class"]()
39+
usage_tracker = verticals_usage_plugin["service_class"]()
40+
41+
api_hub_headers = headers_cache.get_headers(workflow_execution_id)
42+
43+
if api_hub_headers:
44+
return usage_tracker.store_usage(
45+
file_execution_id=workflow_file_execution_id,
46+
api_hub_headers=api_hub_headers,
47+
organization_id=organization_id,
48+
)
49+
return False
50+
51+
except Exception as e:
52+
logger.error(
53+
f"Failed to track API hub usage for execution {workflow_execution_id}: {e}"
54+
)
55+
return False
4356

4457
@staticmethod
4558
def extract_api_hub_headers(request_headers: dict) -> dict | None:
4659
"""Extract API hub headers from request headers.
4760
48-
OSS version: Returns None (no API hub support).
49-
Enterprise version: Extracts and normalizes API hub headers.
50-
5161
Args:
5262
request_headers: Request headers dictionary
5363
5464
Returns:
55-
None in OSS version. Enterprise version returns normalized headers.
65+
Normalized API hub headers or None if not available.
5666
"""
57-
# OSS version - no API hub header extraction
58-
return None
67+
verticals_usage_plugin = get_plugin("verticals_usage")
68+
if not verticals_usage_plugin:
69+
return None
70+
71+
try:
72+
usage_tracker = verticals_usage_plugin["service_class"]()
73+
return usage_tracker.extract_api_hub_headers_from_request(request_headers)
74+
except Exception as e:
75+
logger.error(f"Error extracting API hub headers: {e}")
76+
return None
5977

6078
@staticmethod
6179
def cache_api_hub_headers(
@@ -65,16 +83,21 @@ def cache_api_hub_headers(
6583
) -> bool:
6684
"""Cache API hub headers for later usage tracking.
6785
68-
OSS version: No-op implementation.
69-
Enterprise version: Caches headers in Redis.
70-
7186
Args:
7287
execution_id: The execution ID to use as cache key
7388
headers: Headers to cache
7489
ttl_seconds: Time-to-live in seconds (default 2 hours)
7590
7691
Returns:
77-
False in OSS version. Enterprise version returns True if caching succeeded.
92+
True if caching succeeded, False otherwise.
7893
"""
79-
# OSS version - no caching available
80-
return False
94+
verticals_usage_plugin = get_plugin("verticals_usage")
95+
if not verticals_usage_plugin:
96+
return False
97+
98+
try:
99+
headers_cache = verticals_usage_plugin["headers_cache_class"]()
100+
return headers_cache.store_headers(execution_id, headers)
101+
except Exception as e:
102+
logger.error(f"Error caching API hub headers: {e}")
103+
return False

0 commit comments

Comments
 (0)