Background
When multiple reviewers annotate the same session their tags may differ. Currently there is no mechanism to:
- Identify which sessions have conflicting annotations
- Designate one annotation as the authoritative / gold-label answer
- Drive concordance metrics and annotation queue metrics from that gold label
The agreed approach is not to create a third "gold label" annotation, but to mark one of the existing annotations as authoritative. Reviewers would each submit their own annotation independently, then sit together to align, and the correct annotation would be edited — with one of them flagged as authoritative.
Proposed change
Add an is_authoritative boolean flag to the annotation/review model (default False).
Rules
- At most one annotation per session can be
is_authoritative = True
- Setting a different annotation as authoritative on the same session should clear the flag on the previous one
- Reviewers (or admins) can toggle the flag from the annotation UI
Metric behaviour
- Concordance metrics should be calculated using only authoritative annotations where one exists; fall back to all annotations for sessions with no authoritative annotation yet
- Annotation queue metrics (completion, coverage, etc.) should count a session as "resolved" only when it has an authoritative annotation (for sessions with >1 reviewer)
Workaround (until this is implemented)
For projects with multiple reviewers, the interim process is:
- Each reviewer independently annotates every session
- Reviewers meet and agree on the correct tags
- Both reviewers edit their annotations to match — ensuring no conflicts remain in the current metrics
Out of scope
- A dedicated "gold label" third annotation type (explicitly rejected)
- Automatic conflict detection / flagging (could be a follow-up)
Background
When multiple reviewers annotate the same session their tags may differ. Currently there is no mechanism to:
The agreed approach is not to create a third "gold label" annotation, but to mark one of the existing annotations as authoritative. Reviewers would each submit their own annotation independently, then sit together to align, and the correct annotation would be edited — with one of them flagged as authoritative.
Proposed change
Add an
is_authoritativeboolean flag to the annotation/review model (defaultFalse).Rules
is_authoritative = TrueMetric behaviour
Workaround (until this is implemented)
For projects with multiple reviewers, the interim process is:
Out of scope