Replace Direct Elasticsearch Indexing With Publisher Changeset Upload

[chuang-CE]

Replace Direct Elasticsearch Indexing With Publisher Changeset Upload

Background

Ruby Registry currently publishes data and directly initiates Elasticsearch indexing workflows.

As part of the Registry transition strategy, indexing responsibilities are being moved out of Ruby Registry and into Publisher. Ruby Registry will continue generating graph artifacts, but Publisher will become responsible for orchestrating downstream processing.

Required Changes

After a successful publish, Ruby Registry should:

1. Generate a changeset ZIP file containing:

Changeset ZIP structure expected.

changeset.zip
|
+- upserts/
|  +- graphs/
|  |  +- <ctid1>.json
|  |
|  +- metadata/
|  |  +- <ctid1>.json
|  |
|  +- resources/
|     +- <ctid1>.json
|
+- deletes/
|  +- graphs/
|  |  +- <ctid1>.json
|  |
|  +- metadata/
|  |  +- <ctid1>.json
|  |
|  +- resources/
|     +- <ctid1>.json

Example deletes file content for deletes/graphs/.json:

[
  "deletes/graphs/<ctid1>.json",
  "deletes/graphs/<ctid2>.json"
]

Follow similar structure for metadata and resources.

Delete artifacts should mirror the upsert structure and identify resources that should be removed from the target storage location.

Example:

deletes/
  graphs/
    ce-123456789abcdef.json

2. Use the envelope community specified during publish.

3. Upload the ZIP file to the new Publisher endpoint.

Publisher Endpoint

Request

POST /changesets
Content-Type: multipart/form-data

Form Fields

Field	Type	Required
community	string	Yes
changeset	file	Yes

Example

curl -X POST \
  https://publisher.test.credentialengine.org/changesets \
  -F "community=public" \
  -F "changeset=@changeset.zip"

Storage Convention

Publisher will store the uploaded ZIP using:

{community}-publisher/changesets/{timestamp}.zip

Example:

public-publisher/changesets/20260603184522123.zip

The caller should not construct or manage the storage path. Publisher will generate the timestamped filename.

Responsibilities Removed From Ruby Registry

Ruby Registry should no longer:

• Submit Elasticsearch indexing workflows.
• Call Argo workflows related to indexing.
• Manage Elasticsearch index refresh operations.
• Manage graph/resource/metadata synchronization after publish.

Those responsibilities will be handled by Publisher and downstream workflow orchestration.

Expected Publisher Behavior

After receiving the changeset ZIP, Publisher will:

1. Store the ZIP in:

{community}-publisher/changesets/{timestamp}.zip

2. Trigger the CEOPS copy-changeset workflow.

3. Handle downstream indexing orchestration through existing workflow events and sensors.

Ruby Registry does not need to be aware of these implementation details.

Acceptance Criteria

• changeset ZIP is generated after publish.
• ZIP contains the required upserts and deletes structure.
• ZIP is uploaded to Publisher using the new endpoint.
• The envelope community is supplied as the community parameter.
• Existing direct Elasticsearch indexing workflow calls are removed.
• Successful publish behavior remains unchanged.
• Publisher becomes the sole orchestrator for post-publish synchronization and indexing.

---

Start Date: 06/03/2026

Deadline: 06/03/2026

Estimated Cost (hours): 2

Actual Cost (hours):