feat: Add HubSpot integration plugin hooks for user journey tracking (#1806)

* refactor: Add dynamic plugin loading for enterprise components

## What

- Add dynamic plugin loading support to OSS codebase
- Enable enterprise components to be loaded at runtime without modifying tracked files

## Why

- Enterprise code was overwriting git-tracked OSS files causing dirty git state
- Need clean separation between OSS and enterprise codebases
- OSS should work independently without enterprise components

## How

- `unstract_migrations.py`: Uses try/except ImportError to load from `pluggable_apps.migrations_ext`
- `api_hub_usage_utils.py`: Uses try/except ImportError to load from `plugins.verticals_usage`
- `utils.py`: Uses try/except ImportError to load from `pluggable_apps.manual_review_v2` and `plugins.workflow_manager.workflow_v2.rule_engine`
- `backend.Dockerfile`: Conditional install of `requirements.txt` if present

## Can this PR break any existing features. If yes, please list possible items. If no, please explain why.

- No. The changes add optional plugin loading that gracefully falls back to default behavior when plugins are not present. Existing OSS functionality is preserved.

## Database Migrations

- None

## Env Config

- None

## Relevant Docs

- None

## Related Issues or PRs

- None

## Dependencies Versions

- None

## Notes on Testing

- OSS build: Verify app starts and works without enterprise plugins
- Enterprise build: Verify plugins are loaded and function correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* refactor: Use get_plugin() for API Hub usage utilities

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Refactor random sampling logic in utils.py

Removed redundant import of random and exception handling for manual_review_v2.

Signed-off-by: Hari John Kuriakose <hari@zipstack.com>

* fix: Add Traefik port labels and clean up service ignore list

- Add explicit loadbalancer port labels for backend (8000) and frontend (3000)
  services in docker-compose to ensure proper Traefik routing
- Rename spawned_services to ignored_services for clarity
- Extend ignored_services list to include tool-classifier, tool-text_extractor,
  and worker-unified services that don't need environment setup

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Update frontend Docker config for nginx serving

Update Traefik port label to 80 to match nginx and fix Dockerfile to use
BUILD_CONTEXT_PATH for the runtime config script in both dev and prod stages.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Use ARG instead of ENV for BUILD_CONTEXT_PATH in frontend Dockerfile

Convert BUILD_CONTEXT_PATH from environment variable to build argument
for proper Docker multi-stage build support. ARGs must be declared
globally and re-declared in each stage that needs them.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: Add HubSpot integration plugin for contact event tracking

- Add new integrations plugin category under backend/plugins/integrations/
- Create HubSpot plugin with event-based contact updates
- Track user milestone events: project creation, document upload,
  prompt run, tool export, and API deployment
- Plugin validates is_first_for_org flag and first org member status
- Remove unused hubspot_signup_api() stub from authentication_service
- Update subscription_helper to use plugin pattern for form submissions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* [FIX] Fix HITL review screen showing "Never expires" despite TTL being set (#1785)

Fix two interacting bugs that prevented TTL from propagating to HITL
queue records:

1. WorkflowUtil.get_hitl_ttl_seconds was an OSS stub that always returned
   None. Now delegates to get_hitl_ttl_seconds_by_workflow via try/except
   import, falling back to None in OSS environments.

2. _push_to_queue_for_api_deployment never fetched TTL. Now mirrors the
   connector path by calling WorkflowUtil.get_hitl_ttl_seconds and passing
   ttl_seconds through to _create_queue_result and
   _enqueue_to_packet_or_regular_queue.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: Add dynamic plugin loading for enterprise components (#1736)

* refactor: Add dynamic plugin loading for enterprise components

- Add dynamic plugin loading support to OSS codebase
- Enable enterprise components to be loaded at runtime without modifying tracked files

- Enterprise code was overwriting git-tracked OSS files causing dirty git state
- Need clean separation between OSS and enterprise codebases
- OSS should work independently without enterprise components

- `unstract_migrations.py`: Uses try/except ImportError to load from `pluggable_apps.migrations_ext`
- `api_hub_usage_utils.py`: Uses try/except ImportError to load from `plugins.verticals_usage`
- `utils.py`: Uses try/except ImportError to load from `pluggable_apps.manual_review_v2` and `plugins.workflow_manager.workflow_v2.rule_engine`
- `backend.Dockerfile`: Conditional install of `requirements.txt` if present

- No. The changes add optional plugin loading that gracefully falls back to default behavior when plugins are not present. Existing OSS functionality is preserved.

- None

- None

- None

- None

- None

- OSS build: Verify app starts and works without enterprise plugins
- Enterprise build: Verify plugins are loaded and function correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* refactor: Use get_plugin() for API Hub usage utilities

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Refactor random sampling logic in utils.py

Removed redundant import of random and exception handling for manual_review_v2.

Signed-off-by: Hari John Kuriakose <hari@zipstack.com>

* fix: Add Traefik port labels and clean up service ignore list

- Add explicit loadbalancer port labels for backend (8000) and frontend (3000)
  services in docker-compose to ensure proper Traefik routing
- Rename spawned_services to ignored_services for clarity
- Extend ignored_services list to include tool-classifier, tool-text_extractor,
  and worker-unified services that don't need environment setup

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Update frontend Docker config for nginx serving

Update Traefik port label to 80 to match nginx and fix Dockerfile to use
BUILD_CONTEXT_PATH for the runtime config script in both dev and prod stages.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Use ARG instead of ENV for BUILD_CONTEXT_PATH in frontend Dockerfile

Convert BUILD_CONTEXT_PATH from environment variable to build argument
for proper Docker multi-stage build support. ARGs must be declared
globally and re-declared in each stage that needs them.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: typo in ignored services var name

* fix: handle script execution via entrypoint

---------

Signed-off-by: Hari John Kuriakose <hari@zipstack.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* feat: Add auth error code for forbidden emails (#1789)

* feat: add auth error code and frontend error display

* fix: run frontend dev server on port 80 and add signup handler

- Set PORT=80 env var in frontend Dockerfile development stage
- Change EXPOSE from 3000 to 80 to match production nginx
- Add handleSignup function and pass to LoginForm component

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: add ARG declaration to Dockerfile stages using BUILD_CONTEXT_PATH

SonarQube flagged that ARG must be declared in each Docker build stage
where it is used. Added the missing ARG BUILD_CONTEXT_PATH to both
development and builder stages.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

* [MISC] Improve dev experience by adding a compose debug override (#1765)

* [MISC] Improve Docker dev experience: separate debugpy, optimize memory, add V2 workers support

- Move debugpy to optional compose.debug.yaml for cleaner default dev setup
- Update compose.override.yaml with memory-optimized settings (1 worker, 2 threads)
- Add V2 workers configuration with build definitions
- Move V1 workers to optional workers-v1 profile
- Use modern uv run python -Xfrozen_modules=off pattern for services
- Updated README with compose.debug.yaml usage instructions

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [MISC] Add Docker Compose version requirement and scheduler comment

- Add Docker Compose 2.24.4+ requirement note for !override directive
- Add comment explaining why worker-log-history-scheduler-v2 has no watch

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [MISC] Fix debug ports table in README

- Fix port order: runner=5679, platform=5680, prompt=5681
- Remove x2text-service (not in compose.debug.yaml)
- Add V2 workers debug ports (5682-5688)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* [MISC] Simplify sample.compose.override.yaml

Remove db image override and V1 worker command overrides from
sample file. Users should reference compose.override.yaml for
actual dev setup.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* [FIX] Optimize queries made by worker and retry config of worker base client (#1798)

* Fix production queryset performance and retry amplification

Resolves 39.8s GET /internal/v1/file-execution/<uuid>/ latency by:
- Removing 7 debug COUNT(*) full table scans from get_queryset()
- Adding get_object() O(1) PK lookup override in ViewSet
- Adding @with_cache decorators to pipeline fetch methods
- Adding pipeline_data_key() to prevent cache key collisions
- Fixing urllib3 retry amplification: clear status_forcelist, let app-level retries handle status codes
- Use config defaults instead of hardcoded retry values

Root cause: Count queries on every request + retry storm (urllib3 × app-level).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address PR #1798 review comments

- Chain exception context in get_object() (Ruff B904)
- Fix cache key collision: get_workflow_definition() now uses
  CacheType.WORKFLOW_DEFINITION instead of CacheType.WORKFLOW to avoid
  type mismatch with get_workflow() sharing the same cache key
- Include check_active in get_pipeline_data() cache key to prevent
  active-status bypass when check_active=False is cached first
- Refactor status() and update_hash() to use self.get_object() instead
  of duplicating manual queryset + org filtering; add except APIException
  pass-through so NotFound propagates as 404 not 500

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* UN-2971 [FEAT] Pass selectedProduct to login/signup API for OAuth product scope (#1803)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Hari John Kuriakose <hari@zipstack.com>
Co-authored-by: Claude <noreply@anthropic.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: Scope HubSpot milestone count checks to current organization

PromptStudioOutputManager and DocumentManager lack
DefaultOrganizationManagerMixin, so .objects.count() was counting
across ALL organizations. Filter through the tool FK to CustomTool
(which is org-scoped) to get correct per-org counts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* [REFACTOR] Extract HubSpot notification logic into shared utility

Move all _notify_hubspot_* methods from views into a shared
utils/hubspot_notify.py module with a single notify_hubspot_event()
function, reducing duplication across prompt_studio views and
api_deployment_views.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Signed-off-by: Hari John Kuriakose <hari@zipstack.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: vishnuszipstack <117254672+vishnuszipstack@users.noreply.github.com>
Co-authored-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
Co-authored-by: vishnuszipstack <vishnu@zipstack.com>
Co-authored-by: Gayathri <142381512+gaya3-zipstack@users.noreply.github.com>

Author: 6 people

backend/account_v2/authentication_service.py

@@ -266,9 +266,6 @@ def get_invitations(self, organization_id: str) -> list[MemberInvitation]:
     def frictionless_onboarding(self, organization: Organization, user: User) -> None:
         raise MethodNotImplemented()
 
-    def hubspot_signup_api(self, request: Request) -> None:
-        raise MethodNotImplemented()
-
     def delete_invitation(self, organization_id: str, invitation_id: str) -> bool:
         raise MethodNotImplemented()
 

backend/api_v2/api_deployment_views.py

@@ -16,6 +16,7 @@
 from rest_framework.serializers import Serializer
 from tool_instance_v2.models import ToolInstance
 from utils.enums import CeleryTaskState
+from utils.hubspot_notify import notify_hubspot_event
 from utils.pagination import CustomPagination
 from workflow_manager.workflow_v2.dto import ExecutionResponse
 from workflow_manager.workflow_v2.models.execution import WorkflowExecution
@@ -293,6 +294,9 @@ def fetch_one(self, request: Request, pk: str | None = None) -> Response:
     def create(
         self, request: Request, *args: tuple[Any], **kwargs: dict[str, Any]
     ) -> Response:
+        # Check deployment count before create for HubSpot notification
+        deployment_count_before = APIDeployment.objects.count()
+
         serializer: Serializer = self.get_serializer(data=request.data)
         serializer.is_valid(raise_exception=True)
         self.perform_create(serializer)
@@ -301,6 +305,14 @@ def create(
             {"api_key": api_key.api_key, **serializer.data}
         )
 
+        # Notify HubSpot about first API deployment
+        notify_hubspot_event(
+            user=request.user,
+            event_name="API_DEPLOY",
+            is_first_for_org=deployment_count_before == 0,
+            action_label="API deployment",
+        )
+
         headers = self.get_success_headers(serializer.data)
         return Response(
             response_serializer.data,

backend/prompt_studio/prompt_studio_core_v2/views.py

@@ -22,6 +22,7 @@
 from rest_framework.versioning import URLPathVersioning
 from tool_instance_v2.models import ToolInstance
 from utils.file_storage.helpers.prompt_studio_file_helper import PromptStudioFileHelper
+from utils.hubspot_notify import notify_hubspot_event
 from utils.user_context import UserContext
 from utils.user_session import UserSessionUtils
 from workflow_manager.endpoint_v2.models import WorkflowEndpoint
@@ -57,6 +58,8 @@
     PromptStudioDocumentHelper,
 )
 from prompt_studio.prompt_studio_index_manager_v2.models import IndexManager
+from prompt_studio.prompt_studio_output_manager_v2.models import PromptStudioOutputManager
+from prompt_studio.prompt_studio_registry_v2.models import PromptStudioRegistry
 from prompt_studio.prompt_studio_registry_v2.prompt_studio_registry_helper import (
     PromptStudioRegistryHelper,
 )
@@ -119,6 +122,16 @@ def create(self, request: HttpRequest) -> Response:
         PromptStudioHelper.create_default_profile_manager(
             request.user, serializer.data["tool_id"]
         )
+
+        # Notify HubSpot if this is the first Prompt Studio project for the org
+        # (count == 1 means the one we just created is the first)
+        notify_hubspot_event(
+            user=request.user,
+            event_name="PROMPT_STUDIO_PROJECT_CREATE",
+            is_first_for_org=CustomTool.objects.count() == 1,
+            action_label="project creation",
+        )
+
         return Response(serializer.data, status=status.HTTP_201_CREATED)
 
     def perform_destroy(self, instance: CustomTool) -> None:
@@ -409,6 +422,14 @@ def fetch_response(self, request: HttpRequest, pk: Any = None) -> Response:
         if not run_id:
             # Generate a run_id
             run_id = CommonUtils.generate_uuid()
+
+        # Check output count before prompt run for HubSpot notification
+        # Filter through tool FK to scope by organization (PromptStudioOutputManager
+        # lacks DefaultOrganizationManagerMixin)
+        output_count_before = PromptStudioOutputManager.objects.filter(
+            tool_id__in=CustomTool.objects.values_list("tool_id", flat=True)
+        ).count()
+
         response: dict[str, Any] = PromptStudioHelper.prompt_responder(
             id=id,
             tool_id=tool_id,
@@ -418,6 +439,15 @@ def fetch_response(self, request: HttpRequest, pk: Any = None) -> Response:
             run_id=run_id,
             profile_manager_id=profile_manager,
         )
+
+        # Notify HubSpot about first prompt run
+        notify_hubspot_event(
+            user=request.user,
+            event_name="PROMPT_RUN",
+            is_first_for_org=output_count_before == 0,
+            action_label="prompt run",
+        )
+
         return Response(response, status=status.HTTP_200_OK)
 
     @action(detail=True, methods=["post"])
@@ -574,6 +604,13 @@ def upload_for_ide(self, request: HttpRequest, pk: Any = None) -> Response:
         uploaded_files: Any = serializer.validated_data.get("file")
         file_converter_plugin = get_plugin("file_converter")
 
+        # Check document count before upload for HubSpot notification
+        # Filter through tool FK to scope by organization (DocumentManager
+        # lacks DefaultOrganizationManagerMixin)
+        doc_count_before = DocumentManager.objects.filter(
+            tool__in=CustomTool.objects.all()
+        ).count()
+
         documents = []
         for uploaded_file in uploaded_files:
             # Store file
@@ -624,6 +661,15 @@ def upload_for_ide(self, request: HttpRequest, pk: Any = None) -> Response:
                 "tool": document.tool.tool_id,
             }
             documents.append(doc)
+
+        # Notify HubSpot about first document upload
+        notify_hubspot_event(
+            user=request.user,
+            event_name="DOCUMENT_UPLOAD",
+            is_first_for_org=doc_count_before == 0,
+            action_label="document upload",
+        )
+
         return Response({"data": documents})
 
     @action(detail=True, methods=["delete"])
@@ -675,13 +721,24 @@ def export_tool(self, request: Request, pk: Any = None) -> Response:
         user_ids = set(serializer.validated_data.get("user_id"))
         force_export = serializer.validated_data.get("force_export")
 
+        # Check registry count before export for HubSpot notification
+        registry_count_before = PromptStudioRegistry.objects.count()
+
         PromptStudioRegistryHelper.update_or_create_psr_tool(
             custom_tool=custom_tool,
             shared_with_org=is_shared_with_org,
             user_ids=user_ids,
             force_export=force_export,
         )
 
+        # Notify HubSpot about first tool export
+        notify_hubspot_event(
+            user=request.user,
+            event_name="TOOL_EXPORT",
+            is_first_for_org=registry_count_before == 0,
+            action_label="tool export",
+        )
+
         return Response(
             {"message": "Custom tool exported sucessfully."},
             status=status.HTTP_200_OK,

backend/utils/hubspot_notify.py

@@ -0,0 +1,37 @@
+import logging
+
+from plugins import get_plugin
+
+logger = logging.getLogger(__name__)
+
+
+def notify_hubspot_event(
+    user,
+    event_name: str,
+    is_first_for_org: bool,
+    action_label: str,
+) -> None:
+    """Send a HubSpot event notification if the plugin is available.
+
+    Args:
+        user: The user performing the action.
+        event_name: The HubSpotEvent attribute name (e.g. "PROMPT_RUN").
+        is_first_for_org: Whether this is the first such action for the org.
+        action_label: Human-readable label for logging (e.g. "prompt run").
+    """
+    hubspot_plugin = get_plugin("hubspot")
+    if not hubspot_plugin:
+        return
+
+    try:
+        from plugins.integrations.hubspot import HubSpotEvent
+
+        event = getattr(HubSpotEvent, event_name)
+        service = hubspot_plugin["service_class"]()
+        service.update_contact(
+            user=user,
+            events=[event],
+            is_first_for_org=is_first_for_org,
+        )
+    except Exception as e:
+        logger.warning(f"Failed to notify HubSpot for {action_label}: {e}")