feat(guardrails): PII redaction via Presidio sidecar (native VIN, per-rule language) (#5174)

* fix(logs): run PII redaction over HTTP and fix Presidio provisioning

- resolve the guardrails venv via candidate paths and fail fast instead of
  silently falling back to system python3 (the misleading "Presidio not
  installed" that broke redaction and the guardrails block in deployed runtimes)
- install the en_core_web_lg spaCy model in setup.sh and app.Dockerfile
- route log redaction through an internal /api/guardrails/mask-batch endpoint
  so Presidio always runs in the app container, including async executions that
  persist inside the trigger.dev runtime

* fix(guardrails): chunk + time-bound internal PII mask requests

- chunk maskPIIBatchViaHttp by count (2000) and bytes (256KB) so large
  executions split across requests and never hit the contract's 100k cap
- add AbortSignal.timeout(45s) per request so a slow/unreachable app container
  aborts and the caller scrubs, instead of hanging the trigger.dev job
- catch maskPIIBatch failures in the route: log and return a structured 500
  (broken venv fails loudly server-side; caller still scrubs, no leak)
- add mask-client tests (order across chunks, count split, non-2xx, empty)

* fix(guardrails): mint internal token per mask request

A single token (5min TTL) could expire mid-batch when a large execution
fans out into many sequential chunk requests; mint one per request instead.

* feat(guardrails): run PII via Presidio sidecars + TS recognizer registry

- replace the per-call python3 subprocess (cold spaCy load every call) with
  two long-lived Presidio sidecars (analyzer + anonymizer) reached over HTTP;
  the app image no longer carries Python/Presidio/venv
- add PRESIDIO_ANALYZER_URL / PRESIDIO_ANONYMIZER_URL
- move VIN out of Python into a TS recognizer (check-digit validated) behind a
  CUSTOM_RECOGNIZERS registry so new custom detectors are one entry; masking is
  handled uniformly by the anonymizer
- drive the guardrails block's PII type picker from the shared pii-entities
  catalog (adds VIN, fixes drift) so block + Data Retention never diverge
- delete validate_pii.py, requirements.txt, setup.sh and the Dockerfile venv step

* fix(guardrails): bound-parallelize mask batch; refresh stale comments

- maskPIIBatch runs per-string sidecar calls with bounded concurrency (8) via
  mapWithConcurrency, so a chunk of many small leaves finishes within the 45s
  request timeout instead of aborting and scrubbing; order + fail-on-error kept
- drop stale comments referencing the deleted Python venv / 30s subprocess timeout

* refactor(guardrails): single Presidio image, native VIN, per-rule redaction language

- collapse the analyzer/anonymizer URLs into one PRESIDIO_URL (combined image
  serves /analyze + /anonymize)
- remove the TS VIN recognizer (vin.ts, recognizers.ts) — VIN is now native +
  multi-language in the image; validate_pii is a thin analyze→anonymize client
- trim KR_RRN/TH_TNIN from the catalog (no Korean/Thai model in the image)
- add per-rule redaction language: PII_LANGUAGES catalog drives the contract enum,
  the Data Retention rule modal, and the guardrails block dropdown; resolver +
  logger thread it through to maskPIIBatch (default en), so non-English entity
  rules (e.g. ES_NIF) actually fire instead of silently no-op'ing under en

* fix(guardrails): correct sidecar port (5001) + README for combined image

The combined Presidio image (docker/pii.Dockerfile) serves /analyze + /anonymize
on a single port 5001 with native VIN + multi-language recognizers. Fix the
PRESIDIO_URL default (was 5002) and rewrite the README, which still described two
stock containers and a TS VIN recognizer.

* fix(guardrails): coerce stored redaction language in the resolver

The persist-path resolver accepted any stored language string, so a stale/invalid
code (e.g. a dropped locale) would reach Presidio and scrub the log even though the
admin UI shows English. Coerce against the supported set via a shared
coercePiiLanguage helper (now reused by the data-retention route too), falling back
to en for unknown values.

* fix(guardrails): rename PRESIDIO_URL env var to PII_URL

Match the infra taskdef, which sets PII_URL on the app container for the
combined Presidio sidecar.

Author: TheodoreSpeaks

apps/sim/app/api/guardrails/mask-batch/route.test.ts

@@ -0,0 +1,64 @@
+/**
+ * @vitest-environment node
+ */
+import { createMockRequest } from '@sim/testing'
+import { beforeEach, describe, expect, it, vi } from 'vitest'
+
+const { mockCheckInternalAuth, mockMaskPIIBatch } = vi.hoisted(() => ({
+  mockCheckInternalAuth: vi.fn(),
+  mockMaskPIIBatch: vi.fn(),
+}))
+
+vi.mock('@/lib/auth/hybrid', () => ({
+  checkInternalAuth: mockCheckInternalAuth,
+}))
+
+vi.mock('@/lib/guardrails/validate_pii', () => ({
+  maskPIIBatch: mockMaskPIIBatch,
+}))
+
+import { POST } from '@/app/api/guardrails/mask-batch/route'
+
+describe('POST /api/guardrails/mask-batch', () => {
+  beforeEach(() => {
+    vi.clearAllMocks()
+    mockCheckInternalAuth.mockResolvedValue({ success: true })
+    mockMaskPIIBatch.mockImplementation(async (texts: string[]) => texts.map((t) => `M(${t})`))
+  })
+
+  it('returns 401 without internal auth', async () => {
+    mockCheckInternalAuth.mockResolvedValue({
+      success: false,
+      error: 'Internal authentication required',
+    })
+
+    const res = await POST(
+      createMockRequest('POST', { texts: ['a@b.com'], entityTypes: ['EMAIL_ADDRESS'] })
+    )
+
+    expect(res.status).toBe(401)
+    expect(mockMaskPIIBatch).not.toHaveBeenCalled()
+  })
+
+  it('masks the batch in-process and preserves order', async () => {
+    const res = await POST(
+      createMockRequest('POST', {
+        texts: ['a@b.com', 'hello'],
+        entityTypes: ['EMAIL_ADDRESS'],
+        language: 'en',
+      })
+    )
+
+    expect(res.status).toBe(200)
+    const json = await res.json()
+    expect(json.masked).toEqual(['M(a@b.com)', 'M(hello)'])
+    expect(mockMaskPIIBatch).toHaveBeenCalledWith(['a@b.com', 'hello'], ['EMAIL_ADDRESS'], 'en')
+  })
+
+  it('rejects an invalid body with 400', async () => {
+    const res = await POST(createMockRequest('POST', { texts: 'not-an-array', entityTypes: [] }))
+
+    expect(res.status).toBe(400)
+    expect(mockMaskPIIBatch).not.toHaveBeenCalled()
+  })
+})

apps/sim/app/api/guardrails/mask-batch/route.ts

@@ -0,0 +1,45 @@
+import { createLogger } from '@sim/logger'
+import { getErrorMessage } from '@sim/utils/errors'
+import { type NextRequest, NextResponse } from 'next/server'
+import { guardrailsMaskBatchContract } from '@/lib/api/contracts'
+import { parseRequest } from '@/lib/api/server'
+import { checkInternalAuth } from '@/lib/auth/hybrid'
+import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
+import { maskPIIBatch } from '@/lib/guardrails/validate_pii'
+
+const logger = createLogger('GuardrailsMaskBatchAPI')
+
+/**
+ * Internal batch PII masking. The log-redaction persist path runs in both the
+ * Next.js server and the trigger.dev runtime, but the Presidio sidecars live only
+ * in the app task — so redaction calls this endpoint server-to-server (internal
+ * JWT) to keep Presidio centralized here.
+ */
+export const POST = withRouteHandler(async (request: NextRequest) => {
+  const auth = await checkInternalAuth(request, { requireWorkflowId: false })
+  if (!auth.success) {
+    return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
+  }
+
+  const parsed = await parseRequest(guardrailsMaskBatchContract, request, {})
+  if (!parsed.success) return parsed.response
+
+  const { texts, entityTypes, language } = parsed.data.body
+
+  try {
+    const masked = await maskPIIBatch(texts, entityTypes, language)
+    logger.info('Masked PII batch', { count: texts.length })
+    return NextResponse.json({ masked })
+  } catch (error) {
+    // An unreachable/misconfigured Presidio sidecar makes maskPIIBatch throw; fail
+    // loudly here (the caller scrubs to REDACTION_FAILED, so PII is never leaked).
+    logger.error('PII batch masking failed', {
+      error: getErrorMessage(error),
+      count: texts.length,
+    })
+    return NextResponse.json(
+      { error: getErrorMessage(error, 'PII masking failed') },
+      { status: 500 }
+    )
+  }
+})

apps/sim/app/api/organizations/[id]/data-retention/route.ts

@@ -16,6 +16,7 @@ import { isOrganizationOnEnterprisePlan } from '@/lib/billing/core/subscription'
 import { isBillingEnabled } from '@/lib/core/config/env-flags'
 import { isFeatureEnabled } from '@/lib/core/config/feature-flags'
 import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
+import { coercePiiLanguage } from '@/lib/guardrails/pii-entities'
 
 const logger = createLogger('DataRetentionAPI')
 
@@ -35,7 +36,14 @@ function normalizeConfigured(
     logRetentionHours: settings?.logRetentionHours ?? null,
     softDeleteRetentionHours: settings?.softDeleteRetentionHours ?? null,
     taskCleanupHours: settings?.taskCleanupHours ?? null,
-    piiRedaction: settings?.piiRedaction?.rules ? { rules: settings.piiRedaction.rules } : null,
+    piiRedaction: settings?.piiRedaction?.rules
+      ? {
+          rules: settings.piiRedaction.rules.map((rule) => ({
+            ...rule,
+            language: coercePiiLanguage(rule.language),
+          })),
+        }
+      : null,
   }
 }
 

apps/sim/blocks/blocks/guardrails.ts

@@ -1,4 +1,5 @@
 import { ShieldCheckIcon } from '@/components/icons'
+import { PII_ENTITY_GROUPS, PII_LANGUAGES } from '@/lib/guardrails/pii-entities'
 import type { BlockConfig } from '@/blocks/types'
 import {
   getModelOptions,
@@ -170,65 +171,15 @@ Return ONLY the regex pattern - no explanations, no quotes, no forward slashes,
       title: 'PII Types to Detect',
       type: 'grouped-checkbox-list',
       maxHeight: 400,
-      options: [
-        // Common PII types
-        { label: 'Person name', id: 'PERSON', group: 'Common' },
-        { label: 'Email address', id: 'EMAIL_ADDRESS', group: 'Common' },
-        { label: 'Phone number', id: 'PHONE_NUMBER', group: 'Common' },
-        { label: 'Location', id: 'LOCATION', group: 'Common' },
-        { label: 'Date or time', id: 'DATE_TIME', group: 'Common' },
-        { label: 'IP address', id: 'IP_ADDRESS', group: 'Common' },
-        { label: 'URL', id: 'URL', group: 'Common' },
-        { label: 'Credit card number', id: 'CREDIT_CARD', group: 'Common' },
-        { label: 'International bank account number (IBAN)', id: 'IBAN_CODE', group: 'Common' },
-        { label: 'Cryptocurrency wallet address', id: 'CRYPTO', group: 'Common' },
-        { label: 'Medical license number', id: 'MEDICAL_LICENSE', group: 'Common' },
-        { label: 'Nationality / religion / political group', id: 'NRP', group: 'Common' },
-
-        // USA
-        { label: 'US bank account number', id: 'US_BANK_NUMBER', group: 'USA' },
-        { label: 'US driver license number', id: 'US_DRIVER_LICENSE', group: 'USA' },
-        {
-          label: 'US individual taxpayer identification number (ITIN)',
-          id: 'US_ITIN',
-          group: 'USA',
-        },
-        { label: 'US passport number', id: 'US_PASSPORT', group: 'USA' },
-        { label: 'US Social Security number', id: 'US_SSN', group: 'USA' },
-
-        // UK
-        { label: 'UK National Insurance number', id: 'UK_NINO', group: 'UK' },
-        { label: 'UK NHS number', id: 'UK_NHS', group: 'UK' },
-
-        // Spain
-        { label: 'Spanish NIF number', id: 'ES_NIF', group: 'Spain' },
-        { label: 'Spanish NIE number', id: 'ES_NIE', group: 'Spain' },
-
-        // Italy
-        { label: 'Italian fiscal code', id: 'IT_FISCAL_CODE', group: 'Italy' },
-        { label: 'Italian driver license', id: 'IT_DRIVER_LICENSE', group: 'Italy' },
-        { label: 'Italian identity card', id: 'IT_IDENTITY_CARD', group: 'Italy' },
-        { label: 'Italian passport', id: 'IT_PASSPORT', group: 'Italy' },
-
-        // Poland
-        { label: 'Polish PESEL', id: 'PL_PESEL', group: 'Poland' },
-
-        // Singapore
-        { label: 'Singapore NRIC/FIN', id: 'SG_NRIC_FIN', group: 'Singapore' },
-
-        // Australia
-        { label: 'Australian business number (ABN)', id: 'AU_ABN', group: 'Australia' },
-        { label: 'Australian company number (ACN)', id: 'AU_ACN', group: 'Australia' },
-        { label: 'Australian tax file number (TFN)', id: 'AU_TFN', group: 'Australia' },
-        { label: 'Australian Medicare number', id: 'AU_MEDICARE', group: 'Australia' },
-
-        // India
-        { label: 'Indian Aadhaar', id: 'IN_AADHAAR', group: 'India' },
-        { label: 'Indian PAN', id: 'IN_PAN', group: 'India' },
-        { label: 'Indian vehicle registration', id: 'IN_VEHICLE_REGISTRATION', group: 'India' },
-        { label: 'Indian voter number', id: 'IN_VOTER', group: 'India' },
-        { label: 'Indian passport', id: 'IN_PASSPORT', group: 'India' },
-      ],
+      // Driven by the shared catalog (includes VIN and custom recognizers) so the
+      // block and the Data Retention settings never drift.
+      options: PII_ENTITY_GROUPS.flatMap((group) =>
+        group.entities.map((entity) => ({
+          label: entity.label,
+          id: entity.value,
+          group: group.label,
+        }))
+      ),
       condition: {
         field: 'validationType',
         value: ['pii'],
@@ -255,13 +206,7 @@ Return ONLY the regex pattern - no explanations, no quotes, no forward slashes,
       id: 'piiLanguage',
       title: 'Language',
       type: 'dropdown',
-      options: [
-        { label: 'English', id: 'en' },
-        { label: 'Spanish', id: 'es' },
-        { label: 'Italian', id: 'it' },
-        { label: 'Polish', id: 'pl' },
-        { label: 'Finnish', id: 'fi' },
-      ],
+      options: PII_LANGUAGES.map((language) => ({ label: language.label, id: language.value })),
       defaultValue: 'en',
       condition: {
         field: 'validationType',

apps/sim/ee/data-retention/components/data-retention-settings.tsx

@@ -21,7 +21,13 @@ import {
 } from '@/components/emcn'
 import { useSession } from '@/lib/auth/auth-client'
 import { isBillingEnabled } from '@/lib/core/config/env-flags'
-import { PII_ENTITY_GROUPS, SUPPORTED_PII_ENTITIES } from '@/lib/guardrails/pii-entities'
+import {
+  DEFAULT_PII_LANGUAGE,
+  PII_ENTITY_GROUPS,
+  PII_LANGUAGES,
+  type PIILanguage,
+  SUPPORTED_PII_ENTITIES,
+} from '@/lib/guardrails/pii-entities'
 import { getUserRole } from '@/lib/workspaces/organization/utils'
 import { SettingsSection } from '@/app/workspace/[workspaceId]/settings/components/settings-section/settings-section'
 import { InfoNote } from '@/ee/components/info-note'
@@ -59,6 +65,7 @@ interface RuleDraft {
   id: string
   entityTypes: string[]
   workspaceId: string | null
+  language: PIILanguage
 }
 
 function hoursToDisplayDays(hours: number | null): string {
@@ -75,6 +82,7 @@ function normalizeRule(rule: RuleDraft): string {
   return JSON.stringify({
     entityTypes: [...rule.entityTypes].sort(),
     workspaceId: rule.workspaceId,
+    language: rule.language,
   })
 }
 
@@ -227,6 +235,18 @@ function RuleModal({
             onChange={(entityTypes) => onChange({ ...draft, entityTypes })}
           />
         </ChipModalField>
+        <ChipModalField
+          type='custom'
+          title='Language'
+          hint='Detection runs with this language’s recognizers — match it to your log content.'
+        >
+          <ChipSelect
+            value={draft.language}
+            onChange={(language) => onChange({ ...draft, language: language as PIILanguage })}
+            options={PII_LANGUAGES.map((l) => ({ value: l.value, label: l.label }))}
+            align='start'
+          />
+        </ChipModalField>
       </ChipModalBody>
       <ChipModalFooter
         onCancel={onClose}
@@ -291,6 +311,7 @@ export function DataRetentionSettings() {
         id: r.id,
         entityTypes: r.entityTypes,
         workspaceId: r.workspaceId,
+        language: r.language ?? DEFAULT_PII_LANGUAGE,
       }))
     )
     hydratedOrgRef.current = orgId
@@ -327,6 +348,7 @@ export function DataRetentionSettings() {
             id: r.id,
             entityTypes: r.entityTypes,
             workspaceId: r.workspaceId,
+            language: r.language,
           })),
         },
       },
@@ -335,7 +357,12 @@ export function DataRetentionSettings() {
   }
 
   function openEditDefault() {
-    const rule: RuleDraft = defaultRule ?? { id: generateId(), entityTypes: [], workspaceId: null }
+    const rule: RuleDraft = defaultRule ?? {
+      id: generateId(),
+      entityTypes: [],
+      workspaceId: null,
+      language: DEFAULT_PII_LANGUAGE,
+    }
     setModalIsNew(defaultRule === null)
     setModalOriginal(rule)
     setModalDraft({ ...rule })
@@ -344,7 +371,12 @@ export function DataRetentionSettings() {
   function openAddOverride() {
     const workspaceId = freeWorkspaces[0]?.value
     if (!workspaceId) return
-    const blank: RuleDraft = { id: generateId(), entityTypes: [], workspaceId }
+    const blank: RuleDraft = {
+      id: generateId(),
+      entityTypes: [],
+      workspaceId,
+      language: DEFAULT_PII_LANGUAGE,
+    }
     setModalIsNew(true)
     setModalOriginal(blank)
     setModalDraft(blank)

apps/sim/lib/api/contracts/hotspots.ts

@@ -45,6 +45,34 @@ export const guardrailsValidateContract = defineRouteContract({
   },
 })
 
+const guardrailsMaskBatchBodySchema = z.object({
+  texts: z.array(z.string()).max(100_000),
+  entityTypes: z.array(z.string().min(1, 'Entity type cannot be empty')).max(200),
+  language: z.string().min(1).max(20).optional(),
+})
+
+const guardrailsMaskBatchResponseSchema = z.object({
+  masked: z.array(z.string()),
+})
+
+/**
+ * Internal batch PII masking. Called server-to-server (internal JWT) from the
+ * log-redaction persist path so Presidio always runs in the app container,
+ * including for async executions that persist inside the trigger.dev runtime.
+ */
+export const guardrailsMaskBatchContract = defineRouteContract({
+  method: 'POST',
+  path: '/api/guardrails/mask-batch',
+  body: guardrailsMaskBatchBodySchema,
+  response: {
+    mode: 'json',
+    schema: guardrailsMaskBatchResponseSchema,
+  },
+})
+
+export type GuardrailsMaskBatchBody = z.input<typeof guardrailsMaskBatchBodySchema>
+export type GuardrailsMaskBatchResult = z.output<typeof guardrailsMaskBatchResponseSchema>
+
 const chatMessageSchema = z.object({
   role: z.enum(['user', 'assistant', 'system']),
   content: z.string(),

apps/sim/lib/api/contracts/primitives.ts

@@ -1,4 +1,5 @@
 import { z } from 'zod'
+import { PII_LANGUAGE_CODES } from '@/lib/guardrails/pii-entities'
 
 export const unknownRecordSchema = z.record(z.string(), z.unknown())
 
@@ -93,6 +94,8 @@ export const piiRedactionRuleSchema = z.object({
   entityTypes: z.array(z.string().min(1, 'Entity type cannot be empty')).max(100),
   /** null = all workspaces; otherwise the single targeted workspace. */
   workspaceId: z.string().min(1).nullable(),
+  /** Language whose Presidio recognizers apply; defaults to English. */
+  language: z.enum(PII_LANGUAGE_CODES).optional(),
 })
 
 export type PiiRedactionRule = z.output<typeof piiRedactionRuleSchema>