Calibration Session

Revision of Annotation Guidelines and Scheme

Task: Discuss any questions or disagreements with current annotation guideline and scheme

Discuss representative examples

1. 
2. 
3. 
4. 
5. 
6. 
7. 
8. 
9. 
10. 
11. 
12. 

Annotate calibration set

1. Task: Annotate calibration set
   • Pull: Latest main (you can discard your changes, I already appended all labels)
   • Open: beyond_hate/data_processing/hateful_meme_reannotation.ipynb
   • Annotate: After setting your name annotate the 33 images
   • Share annotations: Send the calibration annotations beyond_hate/data/labels_calibration.jsonl
2. Explore results
3. Discuss

Next steps: Re-annotate

• Open: beyond_hate/data_processing/hateful_meme_reannotation.ipynb
• Setup: Set annotator_name and task = "reannotate"
• Annotate the 566 images
• Share annotations: Send the calibration annotations beyond_hate/data/labels_reannotation.jsonl