Skip to content

feat: expand team-export surface to remaining team models#3712

Merged
SmittieC merged 17 commits into
mainfrom
cs/export_apis_followup
Jun 30, 2026
Merged

feat: expand team-export surface to remaining team models#3712
SmittieC merged 17 commits into
mainfrom
cs/export_apis_followup

Conversation

@SmittieC

Copy link
Copy Markdown
Contributor

Product Description

no change — internal team-export tooling.

Technical Description

Follow-up to the export_apis branch: expands the team-export manifest from a handful of core models to the remaining team-owned surface (chats/messages, traces, evaluations, human annotations, analyses, notifications, cost tracking, custom actions, documents/files, events, generic-tagging).

Points worth a reviewer's attention:

  • Manifest ordering is load-bearing. Entries are topologically ordered so every FK target is imported before its referrers. Generic-FK models (customtaggeditem, usercomment, score) are placed last because their content_object can point at many of the preceding models; usage_records follow pricing_rules.
  • Generic FKs are now handled end-to-end: the serializer emits the content type as an app_label.model label (stable across servers) instead of the source ContentType pk, and the importer resolves the type by natural key and translates the object id through the FK store. Both sides read the GFK column names off the field itself (generic_fk_fields) rather than hardcoding. Coverage is guarded by test_generic_fk_target_models_are_synced.
  • Multi-path team scoping. TEAM_PATH_REGISTRY values may now be a list of ORM paths (OR'd together, distinct() applied) for models reachable via more than one relation — e.g. EventAction has no team FK and is scoped through either of its two trigger one-to-ones.
  • EXTRA_FILTERS adds post-team-scoping row exclusions; used to drop assistant-attached CustomActionOperation rows (assistants aren't synced) so the model's check constraint holds on import.
  • New serializer field types for columns the default DRF mapping can't round-trip: pgvector columns → list of floats, FileField/ImageField → stored relative path (not /media/ URL, which Django's safe-join rejects on assign), and django_pydantic_field schema fields → plain JSON.
  • Through-M2M exclusions: Collection.files / DocumentSource.files are excluded from serialization (their CollectionFile through rows carry the link) since .set() is rejected for explicit through models.
  • annotationqueueaggregate matches an existing target row (recreated as a side effect of importing annotations) rather than creating a second, which the queue one-to-one forbids.

export.yml (the generated OpenAPI schema) is regenerated to match.

Migrations

N/A — no migrations.

Docs and Changelog

  • This PR requires docs/changelog update

SmittieC and others added 7 commits June 30, 2026 07:49
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add 38 models to the export manifest (custom actions, collections, files, embeddings, channels, events, notifications, chat messages, traces, evaluations, human annotations, analysis) with their team-scoping, secret, exclude, and extra-filter registries; serialize file/vector/pydantic fields; and import them via the existing FK/m2m machinery plus an annotation-queue-aggregate matcher.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds tags, comments, scores, pricing rules and usage records to the export
surface. Generic-FK content types are serialized as app_label.model labels and
resolved through the translation store on import; every possible target model is
itself synced (pinned by test_generic_fk_target_models_are_synced).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
codescene-delta-analysis[bot]

This comment was marked as outdated.

codescene-delta-analysis[bot]

This comment was marked as outdated.

The post_save signal sets last_activity_at to timezone.now() (import time)
rather than the original message created_at. After importing all chat messages
for a batch, backfill each session's last_activity_at with the max HUMAN
message created_at via a subquery scoped to the target team.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Base automatically changed from cs/export_apis to main June 30, 2026 07:57
codescene-delta-analysis[bot]

This comment was marked as outdated.

codescene-delta-analysis[bot]

This comment was marked as outdated.

@coderabbitai

coderabbitai Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR extends the team data sync export/import system with generic foreign key support end-to-end. The export serializer gains custom field classes for file paths, pgvector values, and pydantic schemas, plus a factory that emits content-type labels (app_label.model) instead of raw PKs for generic FK fields. The sync manifest is expanded with new model entries, multi-path team scoping, and a generic_fk_fields() introspection helper. The importer resolves generic FK content types and object IDs during row import, adds queue-aggregate deduplication, and backfills ExperimentSession.last_activity_at after chat message import. New test factories cover all newly supported models, and tests validate each new behavior including a full manifest round-trip smoke test.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 62.22% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: expanding the team-export surface to remaining team models.
Description check ✅ Passed The description matches the template and covers product, technical details, migrations, and changelog notes.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
apps/utils/factories/evaluations.py (1)

199-199: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Drop the redundant computed_at factory value.
EvaluationRunAggregate.computed_at is auto_now_add=True, so factory.LazyFunction(timezone.now) is overwritten on create and only adds noise. If a deterministic timestamp is needed for tests, set it after creation with update() or a post-generation hook.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/utils/factories/evaluations.py` at line 199, Remove the redundant
computed_at assignment from the evaluation factory so EvaluationRunAggregate
relies on its auto_now_add behavior instead of
factory.LazyFunction(timezone.now). Update the factory definition in the
evaluations factory class that builds EvaluationRunAggregate instances, and if
tests need a fixed timestamp, apply it after creation with an update call or a
post-generation hook rather than setting it during factory construction.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@apps/teams/export/importer.py`:
- Around line 277-281: The generic-FK handling in importer logic is silently
accepting missing translations by assigning None from
FKTranslationStore.get_target(), which can leave unresolved relations behind.
Update the gfk_pairs loop in the importer method that builds field_values so it
fails fast the same way regular FK resolution does: detect when get_target()
returns no translated target for the type_label and fk field, and raise the
unresolved-FK error instead of storing None. Keep the fix localized to the
generic relation translation path so ContentType lookup and valid target
assignment still behave the same.

In `@apps/teams/export/manifest.py`:
- Around line 81-82: The export manifest ordering in Manifest needs to be
adjusted so evaluation_datasets is processed before evaluation_messages. Update
the ManifestEntry sequence in manifest.py to place the evaluation_datasets entry
ahead of the evaluation_messages entry, since EvaluationMessage depends on
evaluationdataset__team and must not be imported before its related dataset
exists.

---

Nitpick comments:
In `@apps/utils/factories/evaluations.py`:
- Line 199: Remove the redundant computed_at assignment from the evaluation
factory so EvaluationRunAggregate relies on its auto_now_add behavior instead of
factory.LazyFunction(timezone.now). Update the factory definition in the
evaluations factory class that builds EvaluationRunAggregate instances, and if
tests need a fixed timestamp, apply it after creation with an update call or a
post-generation hook rather than setting it during factory construction.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 4b433f6d-f188-4e0c-b93b-e66fa8a50eed

📥 Commits

Reviewing files that changed from the base of the PR and between 5ae36f1 and 38fd0cd.

📒 Files selected for processing (19)
  • api-schemas/export.yml
  • apps/api/export/serializers.py
  • apps/api/export/tests/test_serializers.py
  • apps/api/export/tests/test_views.py
  • apps/teams/export/importer.py
  • apps/teams/export/manifest.py
  • apps/teams/export/tests/test_importer.py
  • apps/teams/export/tests/test_manifest.py
  • apps/teams/management/commands/sync_team.py
  • apps/utils/factories/analysis.py
  • apps/utils/factories/annotations.py
  • apps/utils/factories/custom_actions.py
  • apps/utils/factories/documents.py
  • apps/utils/factories/evaluations.py
  • apps/utils/factories/experiment.py
  • apps/utils/factories/files.py
  • apps/utils/factories/human_annotations.py
  • apps/utils/factories/notifications.py
  • open-chat-studio.cs-export_apis

Comment thread apps/teams/export/importer.py Outdated
Comment thread apps/teams/export/manifest.py
Mirror regular-FK resolution: raise UnresolvedForeignKey when a generic
FK's target was never imported instead of silently nulling the relation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
codescene-delta-analysis[bot]

This comment was marked as outdated.

SmittieC and others added 4 commits June 30, 2026 12:43
A mode-160000 gitlink named after the repo's own working directory was
committed by mistake (there is no .gitmodules), which breaks fresh clones
and any git submodule operation. Drop it from the index.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Fail loudly (UnresolvedForeignKey) when an m2m member of a synced model
  has no translation, instead of silently dropping the link -- matching
  scalar FK resolution. Members of deliberately-unsynced models are still
  left out.
- Mute model signals for the whole import (mute_signals) so the ChatMessage
  post_save handler no longer stamps each session's first/last_activity_at
  to the import time; the source values are preserved instead. Removes the
  now-unneeded activity backfill.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_VectorField subclassed serializers.Field with no schema hint, so
drf-spectacular rendered embedding columns as bare strings. Annotate it as
a list of floats so the generated schema matches the JSON actually served.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Picks up the pgvector array types and the 400 (previously a stale 409)
response on the new secret-resource list endpoints.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
codescene-delta-analysis[bot]

This comment was marked as outdated.

@SmittieC SmittieC marked this pull request as ready for review June 30, 2026 11:08
@SmittieC SmittieC requested a review from snopoke June 30, 2026 11:08

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gates Failed
Enforce advisory code health rules (1 file with Complex Method, Complex Conditional)

Our agent can fix these. Install it.

Gates Passed
3 Quality Gates Passed

Reason for failure
Enforce advisory code health rules Violations Code Health Impact
test_importer.py 2 advisory rules 10.00 → 9.39 Suppress

See analysis details in CodeScene

Quality Gate Profile: Clean Code Collective
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.

@snopoke snopoke left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

return value.model_dump(mode="json") if isinstance(value, pydantic.BaseModel) else value


class _RelativeFileField(serializers.FileField):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this work when using S3 storage?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

@claude

claude Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

@SmittieC SmittieC merged commit c7d860c into main Jun 30, 2026
30 of 31 checks passed
@SmittieC SmittieC deleted the cs/export_apis_followup branch June 30, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants