Address CK-DOCS-ASKS-HYBRID findings and Tim's diagram feedback

kemister85 · kemister85 · commit 139cd39158e5 · 2026-05-25T09:22:08.000+10:00
Diagrams:
- overview-fig-2: add LB/proxy, HTTPS/HTTP labels, fix token flow
  direction, split data layer into labeled nodes, fix label positioning
- complete-guide-fig-1: reduce to single replica (+N), add HTTPS labels,
  add read/write labels to data layer connections
- complete-guide-fig-2: remove bare-metal option, rename to
  "Container orchestrator"
- database-setup-fig-1: remove bare-metal, clarify native DB path

Critical fixes (from CK-DOCS-ASKS-HYBRID):
- G6: Fix WEBSEARCH_HEADERS from broken JSON to colon-CSV format
- G1: Add ai:conversations:webSearch to JWT permissions table

High-severity:
- D2: Add Postgres TLS known-issue warning (managed PG + missing SSL)
- B4: Add Bedrock API-key billing trap note
- E1: Document I/O-bound workload shape for HPA guidance
- E2: Add HA primitives section (PDB, anti-affinity, topology spread)
- E3: Add Azure (AKS) and GCP (GKE) cluster bring-up pointers

Medium/Low:
- D3: MySQL role-inherited grants note for Cloud SQL
- E7: License key shared across replicas statement
- G7: host.docker.internal loopback-bind warning
- Network requirements: license.container.tiny.cloud firewall whitelist
- MCP page: web search prominence TIP
diff --git a/modules/ROOT/images/tinymceai-on-premises/complete-guide-fig-1.mmd b/modules/ROOT/images/tinymceai-on-premises/complete-guide-fig-1.mmd
@@ -1,17 +1,13 @@
 flowchart TB
     Browser["Browser<br>TinyMCE editor + tinymceai plugin"]
     TokenEP["Your token endpoint<br>signs HS256 JWTs"]
-    Browser -->|"fetch JWT"| TokenEP
+    Browser -->|"HTTPS"| TokenEP
     Browser -->|"HTTPS + Bearer JWT"| LB
 
-    subgraph App["Application layer (stateless, N replicas)"]
+    subgraph App["Application layer (stateless, +N replicas)"]
         LB["Reverse proxy / Load balancer<br>nginx · ALB · K8s Ingress<br>TLS termination · SSE pass-through"]
-        AIN["ai-service replica N"]
-        AI2["ai-service replica 2"]
-        AI1["ai-service replica 1"]
-        LB --> AIN
-        LB --> AI2
-        LB --> AI1
+        AI1["ai-service"]
+        LB -->|"HTTP"| AI1
     end
 
     subgraph Data["Shared data layer"]
@@ -20,7 +16,9 @@ flowchart TB
         Storage[("File storage<br>S3 · Azure Blob · filesystem")]
     end
 
-    AI1 --> Data
+    AI1 <-->|"read/write"| DB
+    AI1 <-->|"read/write"| Cache
+    AI1 <-->|"read/write"| Storage
 
     AI1 -->|"HTTPS"| LLM["LLM provider<br>OpenAI · Anthropic · Google ·<br>Azure · Bedrock · Vertex ·<br>self-hosted"]
 
diff --git a/modules/ROOT/images/tinymceai-on-premises/complete-guide-fig-1.svg b/modules/ROOT/images/tinymceai-on-premises/complete-guide-fig-1.svg
diff --git a/modules/ROOT/images/tinymceai-on-premises/complete-guide-fig-2.mmd b/modules/ROOT/images/tinymceai-on-premises/complete-guide-fig-2.mmd
@@ -1,16 +1,14 @@
 flowchart TD
     Start([New deployment]) --> Q1{Evaluating or<br>going to production?}
     Q1 -->|Evaluating locally| Compose[Docker Compose<br>all services on one host<br>Getting started guide]
-    Q1 -->|Production| Q2{Orchestrator?}
+    Q1 -->|Production| Q2{Container orchestrator?}
     Q2 -->|Kubernetes| K8s[Kubernetes deployment<br>Production guide]
     Q2 -->|AWS ECS / Fargate| ECS[ECS task definition<br>Production guide]
     Q2 -->|Docker / Podman on VMs| VMs[Docker or Podman compose<br>Database guide]
-    Q2 -->|Bare metal / no containers| Bare[Native install for<br>data layer; container<br>for AI service<br>Database guide]
     Compose --> DB{Database?}
     K8s --> DB
     ECS --> DB
     VMs --> DB
-    Bare --> DB
     DB -->|Managed cloud DB| Managed[RDS · Cloud SQL ·<br>Azure Database]
     DB -->|Self-managed| Self[Containers or native install]
     Managed --> Done([Continue with<br>LLM providers guide])
diff --git a/modules/ROOT/images/tinymceai-on-premises/complete-guide-fig-2.svg b/modules/ROOT/images/tinymceai-on-premises/complete-guide-fig-2.svg
diff --git a/modules/ROOT/images/tinymceai-on-premises/database-setup-fig-1.mmd b/modules/ROOT/images/tinymceai-on-premises/database-setup-fig-1.mmd
@@ -5,7 +5,7 @@ flowchart TD
     Q2 -->|Cloud / managed services| Managed[AWS RDS · Cloud SQL ·<br>Azure Database<br>+ ElastiCache · Memorystore ·<br>Azure Cache for Redis]
     Q2 -->|Self-managed| Q3{Container runtime<br>available?}
     Q3 -->|Docker or Podman| Containers[Containers on the same<br>network or pod as ai-service]
-    Q3 -->|None - bare metal or VM| Native[Native install<br>brew · apt · yum · dnf<br>service runs on host]
+    Q3 -->|Native on host| Native[Native install<br>brew · apt · yum · dnf<br>AI service connects via<br>host.docker.internal]
     Compose --> Verify([Verify: nc -zv host port<br>then start ai-service])
     Managed --> Verify
     Containers --> Verify
diff --git a/modules/ROOT/images/tinymceai-on-premises/database-setup-fig-1.svg b/modules/ROOT/images/tinymceai-on-premises/database-setup-fig-1.svg
diff --git a/modules/ROOT/images/tinymceai-on-premises/overview-fig-2.mmd b/modules/ROOT/images/tinymceai-on-premises/overview-fig-2.mmd
@@ -6,18 +6,25 @@ flowchart LR
     end
 
     subgraph Service["Application layer"]
+        LB["Load balancer /\nreverse proxy\n(TLS termination)"]
         AI["AI service\n(container)"]
     end
 
     subgraph Data["Data layer"]
-        DB[("Database\n+ Redis\n+ Storage")]
+        DB[("SQL database\n(conversations, configs)")]
+        Redis[("Redis\n(cache, coordination)")]
+        Storage[("File storage\n(uploads, documents)")]
     end
 
     LLM["LLM provider"]
 
-    Token -->|"1. signed JWT"| App
-    App -->|"2. prompt + JWT"| AI
-    AI -->|"5. SSE stream"| App
-    AI -->|"3. forward"| LLM
+    App -->|"1. request JWT"| Token
+    Token -->|"JWT"| App
+    App -->|"2. HTTPS"| LB
+    LB -->|"HTTP"| AI
+    AI -->|"3. HTTPS"| LLM
     LLM -->|"4. stream"| AI
-    AI --- DB
+    AI -->|"5. SSE response"| App
+    AI <-->|"read/write"| DB
+    AI <-->|"read/write"| Redis
+    AI <-->|"read/write"| Storage
diff --git a/modules/ROOT/images/tinymceai-on-premises/overview-fig-2.svg b/modules/ROOT/images/tinymceai-on-premises/overview-fig-2.svg
diff --git a/modules/ROOT/pages/tinymceai-on-premises-database.adoc b/modules/ROOT/pages/tinymceai-on-premises-database.adoc
@@ -127,6 +127,8 @@ GRANT ALL PRIVILEGES ON ai_service.* TO 'ai_service'@'%';
 [NOTE]
 --
 Some versions of the AI service image report false-positive "Not enough permissions to access database" errors even with `ALL PRIVILEGES`. If this occurs, grant the privileges globally rather than per-database, or use the MySQL `root` user for development.
+
+On Cloud SQL MySQL, grant privileges to the service user **directly** — not via a role (e.g. `cloudsqlsuperuser`). The startup grant check runs `SHOW GRANTS FOR user` and does not resolve role-inherited grants.
 --
 
 === PostgreSQL
@@ -157,7 +159,10 @@ GRANT ALL ON SCHEMA "cs-on-premises" TO ai_service;
 ----
 ====
 
-
+[WARNING]
+--
+*Managed PostgreSQL TLS issue:* Amazon RDS, Cloud SQL, and Azure Database for PostgreSQL default to requiring TLS (`rds.force_ssl=1` / `require_secure_transport=ON`). When TLS is enforced and the AI service has not been configured with `DATABASE_SSL_CA`, the connection is rejected — but the error message surfaces as a generic "permissions" error, not a TLS error. If the service fails to start with a permissions-related database error after grants have been verified, check whether TLS is the underlying cause.
+--
 
 == Database setup
 
diff --git a/modules/ROOT/pages/tinymceai-on-premises-jwt.adoc b/modules/ROOT/pages/tinymceai-on-premises-jwt.adoc
@@ -95,10 +95,11 @@ This is the canonical permission list for the AI service.
 [cols=",",options="header",]
 |===
 |Permission |Grants
-|`ai:conversations:*` |All conversation operations: create, list, send message, delete
+|`ai:conversations:*` |All conversation operations: create, list, send message, delete, and web search
 |`ai:conversations:create` |Create new conversations
 |`ai:conversations:read` |List and read existing conversations
 |`ai:conversations:delete` |Delete conversations
+|`ai:conversations:webSearch` |Enable the web search toggle in conversations. Without this permission, `GET /v1/models/1` reports `capabilities.webSearch.allowed: false` even when `WEBSEARCH_ENABLED=true` and `capabilities.webSearch: true` is set on the model.
 |`ai:models:agent` |Access the built-in agent model (model ID `agent-1`)
 |`ai:models:<provider>:<model-id>` |Access a specific custom model configured through the `MODELS` env var
 |`ai:actions:system:*` |All built-in quick actions (rewrite, summarize, expand, translate, change tone, and related operations)
diff --git a/modules/ROOT/pages/tinymceai-on-premises-mcp.adoc b/modules/ROOT/pages/tinymceai-on-premises-mcp.adoc
@@ -5,6 +5,8 @@
 
 The AI service extends model capabilities through two integration points: the https://modelcontextprotocol.io/[Model Context Protocol] (MCP) for tool calling, and pluggable web endpoints for page fetching and search. Both features operate within AI conversations only.
 
+TIP: Web search and scraping allow the AI to reference live internet content during conversations. For most deployments, enabling at least web search significantly improves response quality. These features are configured entirely through environment variables on the AI service container — no additional infrastructure is required beyond a search endpoint.
+
 
 
 [[mcp-integration]]
@@ -42,6 +44,8 @@ Set the `MCP_SERVERS` environment variable to a JSON object. Each key is a serve
 
 TIP: On Linux Docker, add `extra_hosts: ["host.docker.internal:host-gateway"]` to the AI service compose entry to reach MCP servers running on the host machine.
 
+WARNING: `host-gateway` resolves to the Docker bridge IP, not `127.0.0.1`. Host services bound to `127.0.0.1` only (common in Python MCP SDK examples and CLI-style servers) are unreachable from inside the container (`ECONNREFUSED`). Bind MCP servers to `0.0.0.0` or run them as sibling containers on the same Docker network.
+
 [[mcp-shared-token]]
 === Shared-token authentication limitation
 
@@ -153,10 +157,12 @@ The AI service can forward web page fetches and search queries to external endpo
 -e WEBSEARCH_ENABLED='true' \
 -e WEBSEARCH_ENDPOINT='http://host.docker.internal:4001/search' \
 -e WEBSEARCH_REQUEST_TIMEOUT='10000' \
--e WEBSEARCH_HEADERS='{"Authorization":"Bearer search-api-key"}'
+-e WEBSEARCH_HEADERS='Authorization: Bearer search-api-key, X-Source: tinymce'
 ----
 
-NOTE: A model must include `capabilities.webSearch: true` in its xref:tinymceai-on-premises-providers.adoc#models-required[`MODELS` entry] for the web search toggle to appear in the editor.
+IMPORTANT: `WEBSEARCH_HEADERS` uses **colon-CSV format** (`Header-Name: value, Another: value`), not JSON. Passing a JSON object produces `ERR_INVALID_HTTP_TOKEN` and silently prevents all outbound search requests.
+
+NOTE: A model must include `capabilities.webSearch: true` in its xref:tinymceai-on-premises-providers.adoc#models-required[`MODELS` entry] for the web search toggle to appear in the editor. Additionally, the JWT must include the `ai:conversations:webSearch` permission — see xref:tinymceai-on-premises-jwt.adoc#permissions-reference[JWT permissions].
 
 === Web scraping endpoint contract
 
diff --git a/modules/ROOT/pages/tinymceai-on-premises-production.adoc b/modules/ROOT/pages/tinymceai-on-premises-production.adoc
@@ -286,7 +286,19 @@ spec:
 
 TIP: For PostgreSQL, change `DATABASE_DRIVER` to `"postgres"`, update `DATABASE_HOST` to the PostgreSQL endpoint, and add `DATABASE_SCHEMA` set to `"public"` (or ensure the `cs-on-premises` schema exists). See xref:tinymceai-on-premises-database.adoc#postgresql-schema-prerequisite[PostgreSQL schema prerequisite].
 
-NOTE: `terminationGracePeriodSeconds` is set to 300 to match the maximum SSE stream duration. On SIGTERM, the service finishes in-flight SSE streams before shutting down. Set this value equal to or greater than the longest expected AI response time. For multi-zone clusters, add `topologySpreadConstraints` to the pod spec to spread replicas across availability zones. Add a `PodDisruptionBudget` (`minAvailable: 1` or a percentage at scale) to prevent all replicas being evicted simultaneously during node maintenance. These resource values are evaluation defaults; adjust for production workload.
+NOTE: `terminationGracePeriodSeconds` is set to 300 to match the maximum SSE stream duration. On SIGTERM, the service finishes in-flight SSE streams before shutting down. Set this value equal to or greater than the longest expected AI response time. These resource values are evaluation defaults; adjust for production workload.
+
+IMPORTANT: The workload is *network-I/O bound* (waiting on LLM provider streams), not CPU-bound. CPU-based HPA scaling is a safe starting point but may not trigger under high concurrency. For production, consider supplementing with custom metrics (concurrent SSE streams, request queue depth) through KEDA or Prometheus Adapter.
+
+=== High-availability primitives
+
+Any multi-replica deployment should include the following Kubernetes primitives — configure them in line with the platform team's standards:
+
+* *`PodDisruptionBudget`*: prevents all replicas being evicted simultaneously during node maintenance (`minAvailable: 1` or a percentage at scale).
+* *`podAntiAffinity`*: distributes replicas across nodes to avoid single-node failures.
+* *`topologySpreadConstraints`*: spreads replicas across availability zones for zone-level resilience.
+
+The canonical Deployment example above does not include these — they are site-specific. Without them, the scheduler may bin-pack all replicas onto a single node in a single availability zone.
 
 === Step 4: Service
 
@@ -451,7 +463,48 @@ The AI service does not use platform-native credential chains. AWS IRSA, EC2 ins
 |Registry pull credentials |Secrets Manager {plus} ECR pull-through cache, or a private repository mirroring `registry.containers.tiny.cloud`
 |===
 
+=== Azure (AKS)
+
+Deploy on Azure Kubernetes Service with Azure Database for PostgreSQL Flexible Server, Azure Cache for Redis, and Azure Blob Storage. Service-shape considerations:
+
+* `STORAGE_DRIVER=azure` is the recommended storage backend on Azure.
+* PostgreSQL Flexible Server defaults to enforcing TLS — see the xref:tinymceai-on-premises-database.adoc#_postgresql[known TLS error surface].
+* The AI service does not use Workload Identity or managed identity for LLM credentials — inline Azure OpenAI credentials in `PROVIDERS` (see xref:tinymceai-on-premises-providers.adoc#_azure_openai[Azure OpenAI]).
+
+For AKS cluster setup, node pools, and networking, refer to the https://learn.microsoft.com/en-us/azure/aks/[Azure Kubernetes Service documentation].
+
+=== GCP (GKE)
+
+Deploy on Google Kubernetes Engine (Standard mode — not Autopilot) with Cloud SQL for PostgreSQL, Memorystore for Redis, and GCS for file storage. Service-shape considerations:
+
+* No native GCS driver exists. Use GCS via S3-interop (`STORAGE_DRIVER=s3` with HMAC credentials). The bucket must have `uniform_bucket_level_access` disabled (the SDK sends `x-amz-acl` headers).
+* The AI service does not use GKE Workload Identity or Application Default Credentials — inline Vertex SA credentials in `PROVIDERS` (see xref:tinymceai-on-premises-providers.adoc#_google_vertex_ai[Google Vertex AI]).
+* Cloud SQL for PostgreSQL defaults to enforcing TLS — see the xref:tinymceai-on-premises-database.adoc#_postgresql[known TLS error surface].
+
+For GKE cluster creation and node pool configuration, refer to the https://cloud.google.com/kubernetes-engine/docs[Google Kubernetes Engine documentation].
+
+== Network requirements
+
+The AI service requires outbound HTTPS access to the following endpoints. Configure firewall rules and security groups to allow this traffic:
+
+[cols="1,2,2",options="header"]
+|===
+|Endpoint |Purpose |Required
+
+|LLM provider APIs (e.g. `api.openai.com`, `bedrock-runtime.*.amazonaws.com`)
+|Forward AI requests to the configured providers
+|Yes
+
+|`license.container.tiny.cloud`
+|License key validation (phone-home check on startup)
+|Yes — service will not start if this is blocked
+
+|`registry.containers.tiny.cloud`
+|Pull container image updates
+|Yes (at deploy time)
+|===
 
+TIP: If the deployment environment blocks all outbound traffic by default, add these to the egress allowlist before starting the service. The license check occurs on every container start.
 
 == Security hardening
 
diff --git a/modules/ROOT/pages/tinymceai-on-premises-providers.adoc b/modules/ROOT/pages/tinymceai-on-premises-providers.adoc
@@ -419,6 +419,8 @@ IMPORTANT: The AI service does *not* use the AWS SDK default credential chain. `
 
 The `sessionToken` field is optional but required for STS-issued short-lived credentials. Plan a rotation procedure when using temporary credentials.
 
+NOTE: Bedrock console-issued API keys have a separate billing entitlement check that can return `INVALID_PAYMENT_INSTRUMENT` on accounts where the IAM-credential path works fine. Prefer the `credentials` block (IAM user with `accessKeyId` / `secretAccessKey`) for production.
+
 *Prerequisites checklist:*
 
 . *Enable model access.* Bedrock console -> *Model access* -> Manage model access. Each model must be approved per-region.
diff --git a/modules/ROOT/pages/tinymceai-on-premises-reference.adoc b/modules/ROOT/pages/tinymceai-on-premises-reference.adoc
@@ -27,7 +27,7 @@ Alphabetized. Required-ness is marked relative to a minimum working deployment.
 |`LANGFUSE_DEBUG` |No |- |Verbose Langfuse logging.
 |`LANGFUSE_PUBLIC_KEY` |If using Langfuse |- |Langfuse public key.
 |`LANGFUSE_SECRET_KEY` |If using Langfuse |- |Langfuse secret key.
-|`LICENSE_KEY` |Yes |- |AI service license key (long string from Tiny).
+|`LICENSE_KEY` |Yes |- |AI service license key (long string from Tiny). The same key is shared across all replicas of a deployment — no per-pod licensing is required.
 |`LLM_TELEMETRY_ENABLED` |No |`false` |Primary OpenTelemetry switch.
 |`LLM_TIMEOUT_MS` |No |180000 |Per-request large language model (LLM) timeout in ms. Raise for large self-hosted models.
 |`MCP_SERVERS` |No |- |JSON object; Model Context Protocol (MCP) server configuration. See xref:tinymceai-on-premises-mcp.adoc#mcp-integration[MCP integration].
