Skip to content

Human annotation: add 'authoritative' flag to resolve multi-reviewer conflicts #3342

@ocs-agent

Description

@ocs-agent

Background

When multiple reviewers annotate the same session their tags may differ. Currently there is no mechanism to:

  1. Identify which sessions have conflicting annotations
  2. Designate one annotation as the authoritative / gold-label answer
  3. Drive concordance metrics and annotation queue metrics from that gold label

The agreed approach is not to create a third "gold label" annotation, but to mark one of the existing annotations as authoritative. Reviewers would each submit their own annotation independently, then sit together to align, and the correct annotation would be edited — with one of them flagged as authoritative.

Proposed change

Add an is_authoritative boolean flag to the annotation/review model (default False).

Rules

  • At most one annotation per session can be is_authoritative = True
  • Setting a different annotation as authoritative on the same session should clear the flag on the previous one
  • Reviewers (or admins) can toggle the flag from the annotation UI

Metric behaviour

  • Concordance metrics should be calculated using only authoritative annotations where one exists; fall back to all annotations for sessions with no authoritative annotation yet
  • Annotation queue metrics (completion, coverage, etc.) should count a session as "resolved" only when it has an authoritative annotation (for sessions with >1 reviewer)

Workaround (until this is implemented)

For projects with multiple reviewers, the interim process is:

  1. Each reviewer independently annotates every session
  2. Reviewers meet and agree on the correct tags
  3. Both reviewers edit their annotations to match — ensuring no conflicts remain in the current metrics

Out of scope

  • A dedicated "gold label" third annotation type (explicitly rejected)
  • Automatic conflict detection / flagging (could be a follow-up)

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

Status
✅ Done

Relationships

None yet

Development

No branches or pull requests

Issue actions