Skip to content

Latest commit

 

History

History
513 lines (364 loc) · 16.8 KB

File metadata and controls

513 lines (364 loc) · 16.8 KB

🚀 VideoAnnotator v1.4.2 JOSS Submission Roadmap

Release Overview

This roadmap targets v1.4.2 as the JOSS (Journal of Open Source Software) submission release. It focuses on JOSS requirements, reproducibility assets, and reviewer-friendly onboarding.

Target Release: Q1 2026 Current Status: PLANNING (v1.4.1 released Dec 18, 2025) Main Goal: JOSS paper submission + acceptance, backed by a tagged v1.4.2 release Duration: 8-10 weeks (focused scope)

Note: VideoAnnotator already has public releases; this roadmap is about making the JOSS submission and review as low-friction as possible.


👥 Authorship (JOSS Submission)

Author order for the JOSS paper:

  1. Caspar Addyman (ILCHR, Stellenbosch)
  2. Jeremiah Ishaya (ILCHR, Stellenbosch)
  3. Irene Uwerikowe (ILCHR, Stellenbosch)
  4. Daniel Stamate (Computing, Goldsmiths, University of London)
  5. Jamie Lachman (DISP, University of Oxford)
  6. Mark Tomlinson (ILCHR, Stellenbosch)

✅ JOSS Submission Checklist (Must-Have)

These items map directly to JOSS submission/review expectations and should be treated as release gates:

  • Open repository: public Git repository with issues and PRs enabled
  • OSI license: license file present and referenced in docs
  • Installation: clear install path for reviewers (CPU-only, local), plus Docker as optional reproducibility
  • Automated tests: documented commands and a “reviewer smoke test” path
  • Documentation: README + quick start + troubleshooting sufficient for first-time users
  • Paper artifacts: paper/paper.md and paper/paper.bib in the repository
  • Paper format: short JOSS format, with proper citations and statement of need
  • Archival DOI: tag v1.4.2, archive release (e.g., Zenodo), and reference the DOI consistently (paper + CITATION + README)
  • Disclosure: conflicts of interest and funding statements in the paper
  • AI/provenance: brief statement describing AI assistance boundaries and confirming authors’ responsibility/maintainership

Prerequisites - ALL COMPLETED in v1.3.0:

  • ✅ Persistent storage with retention policies
  • ✅ Job cancellation and concurrency control
  • ✅ Schema-based config validation
  • ✅ Secure-by-default configuration
  • ✅ Standardized error envelope
  • ✅ Modern videoannotator package namespace
  • ✅ Test suite at 94.4% passing (720/763)
  • ✅ Real test fixtures infrastructure
  • ✅ ffmpeg across all platforms

🎯 Core Principles - JOSS FOCUSED

This release has a tight scope focused on JOSS requirements ONLY:

  • JOSS Paper - Complete manuscript meeting all JOSS requirements
  • Research Reproducibility - Example datasets with reproducible workflows
  • Publication Documentation - Method descriptions, validation, benchmarks
  • Community Onboarding - Quick start, tutorials, clear installation
  • PyPI Release - pip install videoannotator for easy adoption
  • Minor Polish - Only v1.3.0 deferred items (16 hours total)

Explicitly OUT OF SCOPE (moved to v1.5.0):

  • ❌ Advanced features (quality assessment, batch optimization, pipeline comparison)
  • ❌ Enhanced progress indicators and notifications
  • ❌ Alternative export formats (FiftyOne, Label Studio, custom CSV)
  • ❌ Structured logging and log analysis tools
  • ❌ Interactive config wizard and templates
  • ❌ Model auto-download and setup wizard
  • ❌ Resource monitoring and advanced health metrics
  • ❌ All advanced ML features, plugins, enterprise features

📄 JOSS Paper Requirements

Essential Components for Publication

1. Statement of Need

  • Problem Definition: Why video annotation for research is hard
  • Existing Tools: Comparison with ELAN, BORIS, Anvil, commercial solutions
  • Our Contribution: Modular pipelines, standardized outputs, reproducibility
  • Target Audience: Developmental psychologists, interaction researchers, behavioral scientists

2. Implementation & Architecture

  • Pipeline Architecture: Modular design, registry system, standard outputs
  • Supported Models: Person tracking, face analysis, audio diarization, emotion, gaze
  • Output Formats: COCO, WebVTT, RTTM - rationale and conversion tools
  • Extensibility: How researchers can adapt pipelines for their needs

3. Example Usage & Workflows

  • Parent Child Interaction: Multi-person tracking + speech diarization
  • Clinical Assessment: Face analysis + emotion recognition over time
  • Infant Attention: Gaze tracking + object detection coordination
  • Group Dynamics: Social interaction patterns across multiple individuals
  • Reproducible Tutorial: Step-by-step walkthrough with provided data

4. Quality & Validation

  • Benchmark Results: Accuracy on standard datasets (if available)
  • Performance Metrics: Speed benchmarks (CPU/GPU, various video lengths)
  • Comparison Study: Side-by-side with manual coding or other tools
  • Limitations: Known issues, edge cases, when NOT to use VideoAnnotator

5. Community & Sustainability

  • Documentation: Comprehensive user and developer guides
  • Installation: Multiple paths (pip, Docker, conda)
  • Support: GitHub issues, discussion forum, contribution guidelines
  • Roadmap: Future development plans and feature requests

📋 v1.4.2 Deliverables - JOSS FOCUSED

Phase 1: Quick Polish (Week 1)

Complete the minimal deferred items from v1.3.0 (16 hours total):

1.1 Queue Position Display

  • Add computed queue_position property for PENDING jobs
  • Include in API responses
  • Add tests
  • Effort: 2 hours

Acceptance criteria:

  • queue_position is 1-based and only set for pending jobs
  • Ordering matches the worker dequeue order (FIFO by created_at)
  • Verified by: uv run pytest -q tests/api/test_api_server.py::TestJobEndpoints::test_queue_position_pending_jobs

1.2 Deterministic Test Fixtures

  • Synthetic video generator with known properties
  • Mock OpenCV capture for unit tests
  • Eliminate flaky tests
  • Effort: 8 hours

Acceptance criteria:

  • Tests stop relying on invalid “fake video bytes” and use deterministic, OpenCV-readable fixtures
  • Default fixture avoids codec edge cases (prefer AVI/MJPG for broad OpenCV support)
  • Verified by at least one targeted smoke run (e.g., uv run pytest -q tests/pipelines/test_scene_detection.py::TestSceneDetectionPipeline::test_pipeline_initialization)

1.3 Documentation Touch-ups

  • Enhance contributor workflow docs
  • Expand troubleshooting based on feedback
  • Effort: 6 hours

Acceptance criteria:

  • Reviewer-friendly commands use uv consistently (install + run + tests)
  • At least one “smoke test” flow documented (CPU-only, local)

Checkpoint: v1.3.0 polish complete (16 hours)


Phase 2: Research Examples (Weeks 2-3)

2.1 Four Research Workflow Examples

Complete, reproducible research examples (JOSS requirement):

  • Example 1: Classroom Interaction - Multi-person tracking + speech diarization
  • Example 2: Clinical Assessment - Face analysis + emotion recognition
  • Example 3: Infant Attention - Gaze tracking coordination example
  • Example 4: Group Dynamics - Multi-person interaction patterns

Each example includes:

  • Jupyter notebook with narrative
  • Sample video (5-10 minutes, properly licensed)
  • Ground truth annotations (where applicable)
  • Expected outputs
  • README with clear instructions

Deliverables: examples/research_workflows/ with 4 complete examples

Effort: 40 hours (10 hours per example)

2.2 Reproducibility Assets

Minimal reproducibility requirements for JOSS:

  • Docker Images - CPU and GPU images with pinned dependencies

    • videoannotator:1.4.2-cpu (lightweight)
    • videoannotator:1.4.2-gpu (CUDA 12.1)
    • Published to Docker Hub
  • Benchmark Data - Basic performance metrics

    • Processing time vs video length (CPU/GPU)
    • Memory usage profiles
    • Hardware specifications used
  • Validation Scripts - Compare outputs to ground truth

    • Basic accuracy calculation where applicable
    • Simple visualization of results

Deliverables:

  • docker/ with Dockerfiles
  • benchmarks/results.md with performance data
  • validation/ with comparison scripts

Effort: 24 hours (8 hours Docker, 8 hours benchmarks, 8 hours validation)

Checkpoint: Research examples complete (64 hours total)

Phase 3: JOSS Paper & Academic Documentation (Weeks 4-5)

3.1 JOSS Paper Manuscript

Complete JOSS paper (JOSS requirement):

  • Abstract & Introduction - Problem statement and contribution
  • Implementation Section - Architecture and pipeline descriptions
  • Example Usage - Reference to the 4 research examples
  • Quality & Performance - Benchmark results and validation
  • Acknowledgments & References - Citations and funding

Deliverables: paper/paper.md (JOSS format) + paper/paper.bib

Effort: 40 hours

3.2 Method Documentation (for JOSS)

Scientific documentation for each major pipeline:

  • Pipeline Methods - Brief descriptions of:

    • Person tracking (YOLO-based detection + tracking)
    • Face analysis (detection + recognition pipelines)
    • Audio diarization (speaker segmentation)
    • Speech recognition (Whisper-based transcription)
  • Model Citations - Proper attribution for all models used

  • Limitations - Known issues and edge cases

  • Comparison Table - VideoAnnotator vs alternatives (ELAN, BORIS, OpenPose)

Deliverables:

  • docs/methods/ with pipeline documentation
  • CITATION.cff updated with complete citation info
  • docs/comparison.md with ecosystem positioning

Effort: 24 hours

Checkpoint: JOSS paper and academic docs complete (64 hours)

Phase 4: User Documentation (Week 6)

Essential documentation for community onboarding (JOSS requirement):

4.1 Quick Start Guide

  • 5-minute quick start (docs/quickstart.md)
    • Installation on Linux/macOS/Windows
    • Test video processing
    • View results
    • Next steps

Effort: 8 hours

4.2 Tutorial Series (3 tutorials minimum)

  • Tutorial 1: Single pipeline (person tracking)
  • Tutorial 2: Multi-pipeline workflow (person + audio)
  • Tutorial 3: Configuration and customization

Effort: 16 hours (5-6 hours per tutorial)

4.3 Contributing Guide Enhancement

  • Update CONTRIBUTING.md with:
    • Development setup walkthrough
    • Adding a new pipeline (basic example)
    • Writing tests
    • Submitting PRs

Effort: 8 hours

Checkpoint: Core user documentation complete (32 hours)


Phase 5: PyPI Release & Testing (Weeks 7-8)

5.1 PyPI Package Preparation

  • Package Setup - Prepare for PyPI
    • Update pyproject.toml for PyPI
    • Build wheels for major platforms
    • Test installation from test.pypi.org
    • Publish to PyPI: pip install videoannotator

Effort: 16 hours

5.2 Platform Testing

  • Test installation on:
    • Ubuntu 22.04, 24.04
    • macOS (Intel and Apple Silicon)
    • Windows 11 (WSL2)
    • Docker (both CPU and GPU images)

Effort: 16 hours

5.3 Final Testing & QA

  • Run all examples on fresh installs
  • Verify benchmark reproducibility
  • Test documentation accuracy
  • Fix any critical issues found

Effort: 16 hours

Checkpoint: PyPI release tested (48 hours)


Phase 6: Community Launch (Weeks 9-10)

6.1 JOSS Submission

  • Finalize Paper - Review and polish manuscript
  • Submit to JOSS - Follow JOSS submission process
  • Address Reviewer Feedback - Respond to initial review

Effort: 24 hours

6.2 Release Preparation

  • Release Notes - Comprehensive changelog since v1.3.0
  • Migration Guide - v1.4.1 → v1.4.2 changes
  • GitHub Release - Tag v1.4.2 and create release
  • Zenodo Archive - Create DOI for paper

Effort: 16 hours

6.3 Community Setup (Minimal)

  • README Polish - Ensure GitHub README is excellent
  • GitHub Discussions - Enable and create initial topics
  • Issue Templates - Ensure issue templates are present and current
  • Demo Video - 5-minute overview screencast

Effort: 16 hours

Checkpoint: v1.4.2 released, JOSS submitted (56 hours)


✅ Success Criteria - JOSS FOCUSED

Must-Have for Release (JOSS Requirements)

  • JOSS paper submitted and under review
  • 4 complete research workflow examples (reproducible)
  • PyPI package published (pip install videoannotator)
  • Docker images published (CPU + GPU)
  • Method documentation for major pipelines
  • Benchmark data published
  • Quick start + 3 tutorials minimum
  • Enhanced contributing guide
  • Multi-platform testing passed
  • All examples verified on fresh installs

Quality Targets (Minimal)

  • Example reproducibility: 100% success rate
  • Documentation clarity: Clear for JOSS reviewers
  • API stability: No breaking changes from v1.3.0
  • Installation: Works on Linux, macOS, Windows
  • Test coverage: Maintain ≥80% from v1.3.0

JOSS Acceptance

  • Paper meets all JOSS submission requirements
  • Responds to reviewer feedback within 2 weeks
  • Paper accepted for publication

🚫 Explicitly DEFERRED to v1.5.0

All "nice-to-have" features are moved to v1.5.0 to keep v1.4.2 focused on JOSS:

Usability Enhancements → v1.5.0

  • Model auto-download with progress bars
  • Setup wizard and first-run configuration
  • Real-time progress indicators and ETA
  • Resource usage monitoring
  • Job notifications (email, webhook, desktop)
  • Interactive config wizard
  • Config templates library
  • Structured logging (JSON format)
  • Log analysis tools

Export & Integration → v1.5.0

  • FiftyOne export integration
  • Label Studio import/export
  • Custom CSV templates
  • Advanced export formats

Performance & Quality → v1.5.0

  • Quality assessment pipeline
  • Batch processing optimization
  • Smart job scheduling
  • Pipeline comparison tools
  • Parameter optimization framework

Advanced Features → v1.5.0+

  • Active learning system
  • Multi-modal correlation analysis
  • Plugin system architecture
  • Real-time streaming
  • GraphQL API
  • Enterprise features (SSO, RBAC, multi-tenancy)
  • Advanced analytics dashboard
  • Cloud provider integration

📅 Streamlined Timeline - 10 Weeks

Weeks 1: Quick Polish (16 hours)

  • Queue position display
  • Deterministic test fixtures
  • Documentation touch-ups

Weeks 2-3: Research Examples (64 hours)

  • 4 complete research workflow examples
  • Docker images (CPU/GPU)
  • Benchmark data and validation scripts

Weeks 4-5: JOSS Paper & Academic Docs (64 hours)

  • Complete JOSS paper manuscript
  • Method documentation for pipelines
  • Comparison with alternatives
  • Citations and references

Week 6: User Documentation (32 hours)

  • Quick start guide
  • 3 progressive tutorials
  • Enhanced contributing guide

Weeks 7-8: PyPI & Testing (48 hours)

  • PyPI package preparation and publication
  • Multi-platform testing
  • Example verification on fresh installs
  • Critical bug fixes

Weeks 9-10: Launch (56 hours)

  • JOSS paper submission
  • Release preparation (notes, migration guide)
  • Minimal community setup (README, discussions, demo video)
  • v1.4.2 release and announcement

Total Effort: ~280 hours (8-10 weeks with focused effort)


📊 Effort Summary

Phase Duration Effort (hours) Key Deliverables
Phase 1: Quick Polish Week 1 16 v1.3.0 deferred items complete
Phase 2: Research Examples Weeks 2-3 64 4 examples + Docker + benchmarks
Phase 3: JOSS Paper Weeks 4-5 64 Paper manuscript + methods docs
Phase 4: User Docs Week 6 32 Quick start + 3 tutorials
Phase 5: PyPI & Testing Weeks 7-8 48 PyPI release + testing
Phase 6: Launch Weeks 9-10 56 JOSS submission + release
TOTAL 10 weeks 280 hours JOSS-ready release

🎯 Summary: Tight JOSS Focus

v1.4.2 is intentionally streamlined to focus on JOSS acceptance:

Core Goals:

  1. ✅ Complete, reproducible research examples
  2. ✅ JOSS paper manuscript and submission
  3. ✅ Essential documentation (methods, tutorials, quick start)
  4. ✅ PyPI release for easy installation
  5. ✅ Multi-platform testing

Deferred to v1.5.0:

  • All usability enhancements (progress bars, wizards, notifications)
  • All advanced features (quality assessment, batch optimization)
  • All alternative integrations (FiftyOne, Label Studio)
  • All advanced tooling (log analysis, parameter optimization)

This tight scope ensures we can:

  • Complete v1.4.2 in 10 weeks (~280 hours)
  • Get JOSS paper submitted and accepted
  • Make the JOSS review experience reviewer-friendly
  • Build momentum for v1.5.0 feature enhancements

Last Updated: December 18, 2025 Target Release: Q1 2026 Duration: 8-10 weeks Status: Planning Phase - v1.4.2 JOSS Focused