[QA-G16] OpenSearch Migration — Limited (Non-Admin) OpenSearch User (TC-053–TC-058)

[fabrizzio-dotCMS]

Parent Epic

#35476

Goal

Prove dotCMS can run the OpenSearch migration against an OS 3.x cluster reached through a
restricted, non-admin user (the managed-cloud setup validated in spike #35922) — not just an
open, security-disabled cluster. Covers provisioning the user, pointing dotCMS at it, and confirming
index operations work through the limited role.

Glossary (quick)

• ES (old engine): Elasticsearch — http://localhost:9200, plain HTTP, no auth.
• OS (new engine): OpenSearch 3.x — https://localhost:9201, TLS + basic auth.
• Migration phase: 0 = ES only, 1 = dual-write (search ES), 2 = dual-write (search OS).
• .os tag: new-engine index names end in .os (e.g. cluster_dotcms….working_….os).

Setup — limited-user stack + the provisioning script

Stack: docker/docker-compose-examples/os-migration/docker-compose.limited-user.yml (OS 3.x with
the security plugin ON, provisioned with the non-admin dotcms-es-user). The
docker/docker-compose-examples/single-node-os-migration/ variant runs OS 1.x + OS 3.x, provisioned
the same way.

docker compose -f docker/docker-compose-examples/os-migration/docker-compose.limited-user.yml up -d

Users on OS 3.x (port 9201, HTTPS):

User	Password	Role
admin	Dev!Search3-Kx9mP-2026	cluster admin
dotcms-es-user	Dev!dotcms-EsUser-2026	dotcms-role (limited)

Provisioning script opensearch.py — runs once via the opensearch-provision service; creates
per customer an internal user <customer>-es-user, action groups, a role <customer>-role, and the
role mapping. Run it manually:

# Re-run the bundled provisioner against the running stack (idempotent):
docker compose -f docker/docker-compose-examples/os-migration/docker-compose.limited-user.yml run --rm opensearch-provision

# Or standalone against any reachable cluster:
./opensearch.py --admin-user admin --admin-pass 'Dev!Search3-Kx9mP-2026' \
  --password 'Dev!dotcms-EsUser-2026' --customer dotcms --host localhost --port 9201

The dotcms-role grants:

• Index (pattern cluster_<customer>*): indices_all, indices_monitor
• Cluster: cluster:monitor/health, cluster:monitor/state, cluster:monitor/nodes/stats,
  indices:data/write/bulk, indices:data/read/scroll, indices:data/read/scroll/clear, and
  cluster:monitor/main (GET / version probe — the spike Spike: Validate dotCMS OpenSearch client requests against a non-admin OS 3.x user #35922 fix; without it dotCMS misreads
  OS as unreachable and silently falls back to ES)
• All-indices: indices:monitor/stats, indices:monitor/settings/get, indices:admin/aliases/get

⚠️ Cluster-id gotcha: the role only grants index permissions on names matching cluster_<customer>*.
dotCMS names indexes cluster_<DOT_DOTCMS_CLUSTER_ID>.*, so DOT_DOTCMS_CLUSTER_ID must start with
the customer name (e.g. dotcms-os-migration) or the limited user gets 403 on its own indexes.

dotCMS config:

DOT_ES_ENDPOINTS=http://<host>:9200
DOT_OS_ENDPOINTS=https://<host>:9201
DOT_OS_AUTH_TYPE=BASIC
DOT_OS_AUTH_BASIC_USER=dotcms-es-user
DOT_OS_AUTH_BASIC_PASSWORD=Dev!dotcms-EsUser-2026
DOT_OS_TLS_TRUST_SELF_SIGNED=true
DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE=1
DOT_DOTCMS_CLUSTER_ID=dotcms-os-migration   # MUST start with the customer name

Test Cases

TC-053 — Provision the limited user and verify the role/permissions (Risk: Medium)

1. Launch the limited-user stack; let opensearch-provision finish.
2. List users: curl -sk https://localhost:9201/_plugins/_security/api/internalusers?pretty -u admin:'Dev!Search3-Kx9mP-2026' → dotcms-es-user present.
3. Inspect role: curl -sk https://localhost:9201/_plugins/_security/api/roles/dotcms-role?pretty -u admin:'Dev!Search3-Kx9mP-2026'.
4. curl -sk https://localhost:9201/ -u dotcms-es-user:'Dev!dotcms-EsUser-2026' → HTTP 200.

• Expected: dotcms-es-user + dotcms-role exist with the listed permissions (incl. cluster:monitor/main); GET / as the limited user returns 200.

TC-054 — dotCMS starts against OS as the limited user; migration NOT disabled (Risk: High)

1. Stack up; dotCMS configured as in Setup (Phase 1, cluster id starts with dotcms).
2. Start dotCMS; watch the log; confirm the phase (Admin → System → Index, or log).

• Expected: dotCMS starts and runs normally; migration NOT auto-disabled (no "OpenSearch cluster is not reachable" / phase-reset lines); log shows OS version check passed: 3.x; both ES and .os indexes listed → Phase 1.

TC-055 — Dual-write works through the limited role (Phase 1) (Risk: High)

1. (TC-054 passed.) Create and publish a content item; note its identifier.
2. As the limited user, confirm it in OS (index ends in .os): curl -sk "https://localhost:9201/<cluster_dotcms….working_….os>/_doc/<id>" -u dotcms-es-user:'Dev!dotcms-EsUser-2026'.

• Expected: document present in the OS working index (200/found); no 403 in the dotCMS log for the OS write.

TC-056 — Cluster-id mismatch → limited user gets 403 on its own indexes (gotcha) (Risk: Medium)

1. Configure dotCMS with a cluster id NOT starting with the customer name, e.g. DOT_DOTCMS_CLUSTER_ID=acme-test. Start in Phase 1.
2. Publish content; watch the log and try an OS write/read as the limited user.

• Expected: OS ops on cluster_acme-test.* fail with 403 (outside cluster_dotcms*); dotCMS logs the OS write failure (fire-and-forget in Phase 1 — ES unaffected). Fix: cluster id must start with dotcms (or re-provision with matching --customer) → 403 gone.

TC-057 — Read/search works through the limited user (Risk: Medium)

1. Content dual-written (TC-055). As the limited user: curl -sk "https://localhost:9201/<…os>/_search?q=*:*" -u dotcms-es-user:'Dev!dotcms-EsUser-2026'.
2. (Phase 2) run a dotCMS content search and confirm results.

• Expected: search returns results (no 403); scroll-based reads succeed.

TC-058 — (Regression) Without cluster:monitor/main, dotCMS misreads OS as unreachable (Risk: Medium)

1. Remove cluster:monitor/main from dotcms-role (security API, or re-provision without it).
2. Confirm GET / as the limited user now returns 403.
3. Restart dotCMS in Phase 1 against this user; watch the log.

• Expected: version probe fails → migration auto-disabled (the "OpenSearch cluster is not reachable" + phase-reset lines), dotCMS falls back to ES. Restoring cluster:monitor/main (re-run the provisioner) → TC-054 passes again.

Acceptance Criteria

• TC-053: dotcms-es-user/dotcms-role provisioned with expected perms (incl. cluster:monitor/main); GET / as limited user = 200.
• TC-054: dotCMS starts as the limited user; migration stays ON (Phase 1); no halt lines.
• TC-055: published content lands in the OS index via the limited role; no 403.
• TC-056: mismatched cluster id → 403 on own indexes; matching prefix fixes it.
• TC-057: search/read through the limited user succeeds.
• TC-058: removing cluster:monitor/main reproduces the unreachable→halt failure; restoring it recovers.

Notes

• Validates the managed-cloud limited-user setup from spike Spike: Validate dotCMS OpenSearch client requests against a non-admin OS 3.x user #35922; the cluster:monitor/main grant is the key fix that keeps the OS version probe (client.info()) working for a non-admin user.
• ES stays plain HTTP / no auth; only OS 3.x is HTTPS + authenticated in this stack.

[rjvelazco]

❌ QA FAILED — TC-056 (Cluster-id mismatch → 403 on own indexes)

Risk: Medium
Phase: 1
Result: ❌ Fail: the mismatch does not degrade gracefully; index creation fails outright instead of the OS write failing fire-and-forget with ES unaffected.

What the case expects

Per TC-056, with a cluster id that does not match the role's index pattern, OS ops on the mismatched index should fail with 403, dotCMS should log the OS write failure (fire-and-forget in Phase 1), and ES should be unaffected — dotCMS keeps running normally.

Setup used to induce the mismatch

• Non-admin user with the index permission scoped to cluster_dotcms*.
• DOT_DOTCMS_CLUSTER_ID set to a custom value that does not start with dotcms, so index names fall outside cluster_dotcms* (as the case requires).
• Started dotCMS in Phase 1 and attempted the index operation.

Permission:

Observed behavior

Instead of a fire-and-forget 403 with ES unaffected, index creation failed and the index never ran. The log shows:

INFO  util.SecurityLogger - class com.dotcms.rest.exception.mapper.DotBadRequestExceptionMapper : Failed to create index:

video-part-1.mov

video-part-2.mov

[rjvelazco]

Remaining cases

Aside from TC-056 (reported separately above), all other cases in this group pass. Ran against the limited-user stack (docker-compose.limited-user.yml, OS 3.x with the security plugin ON, non-admin dotcms-es-user / dotcms-role), Phase 1, cluster id starting with dotcms.

Test Case	Scenario	Result
TC-053	Provision limited user + verify role/permissions	✅ Pass
TC-054	dotCMS starts as limited user; migration NOT disabled	✅ Pass
TC-055	Dual-write works through the limited role (Phase 1)	✅ Pass
TC-056	Cluster-id mismatch → 403 on own indexes	❌ Fail (see comment above)
TC-057	Read/search works through the limited user	✅ Pass
TC-058	Regression: without cluster:monitor/main, OS misread as unreachable	✅ Pass

TC-053 — dotcms-es-user and dotcms-role provisioned with the expected permissions (including cluster:monitor/main); GET / as the limited user returned 200.

video-1-unauthorized.mov

video-1-non-admin.mov

TC-054 — dotCMS started normally as the limited user; migration stayed ON (Phase 1), no halt / phase-reset lines; log showed OS version check passed: 3.x; both ES and .os indexes listed.

TC-055 — Published content landed in the OS working index (.os) via the limited role; no 403 on the OS write.

TC-057 — Search/read through the limited user succeeded (no 403); scroll-based reads worked.

video.mov

TC-058 — Removing cluster:monitor/main reproduced the failure: GET / returned 403, the version probe failed, and dotCMS auto-disabled the migration and fell back to ES. Restoring the grant (re-running the provisioner) recovered — TC-054 passed again.

video-2.mov

Acceptance Criteria

• TC-053: user/role provisioned with expected perms; GET / = 200.
• TC-054: dotCMS starts as limited user; migration stays ON; no halt lines.
• TC-055: published content lands in OS index via limited role; no 403.
• TC-056: mismatched cluster id → 403 on own indexes. ← failing (see comment above)
• TC-057: search/read through the limited user succeeds.
• TC-058: removing cluster:monitor/main reproduces the halt; restoring it recovers.

cc @fabrizzio-dotCMS