Skip to content

Add model response output to GitHub job summary and support multiple model presets#150

Merged
kirklandsign merged 28 commits into
mainfrom
report-text-output
Jan 15, 2026
Merged

Add model response output to GitHub job summary and support multiple model presets#150
kirklandsign merged 28 commits into
mainfrom
report-text-output

Conversation

@kirklandsign
Copy link
Copy Markdown
Contributor

@kirklandsign kirklandsign commented Jan 15, 2026

Summary

  • Add model response extraction from instrumentation tests to GitHub job summary
  • Support configurable model presets (stories, llama, qwen3, custom) with proper filename handling
  • Pass model filenames to instrumentation tests via Gradle arguments for preset-aware testing

Changes

GitHub Workflow

  • Extract LLM conversation output from logcat and display in job summary
  • Add debug output showing which model preset and files are being used

Gradle Configuration

  • Keep original model filenames when pushing to device (instead of renaming to model.pte)
  • Pass modelFile and tokenizerFile to instrumentation tests via runner arguments
  • Add skipModelDownload flag for local testing with existing files

Instrumentation Tests

  • SanityCheck and UIWorkflowTest now read model filenames from instrumentation arguments
  • Default to stories preset (stories110M.pte, tokenizer.model) for Android Studio compatibility
  • Log model responses to logcat with LLAMA_RESPONSE tag for CI extraction

Documentation

  • Add README section documenting model preset usage and available options

Test plan

  • Run ./gradlew connectedCheck -PmodelPreset=stories locally
  • Run ./gradlew connectedCheck -PmodelPreset=llama and verify different model is used
  • Trigger CI workflow and verify model response appears in job summary
  • Run tests from Android Studio play button (should use default stories preset)

- UIWorkflowTest now writes conversation to /data/local/tmp/llama/response.txt
- Workflow pulls response.txt after tests complete
- New step appends model response to job summary
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jan 15, 2026
- Update LlamaDemo README to document new text-only report mode, usage steps, and expected behavior.
- Adjust app/build.gradle.kts to configure dependencies and flags for text output, disabling image/audio generation paths where appropriate.
@kirklandsign kirklandsign changed the title Extract model response and add to GitHub job summary Add model response output to GitHub job summary and support multiple model presets Jan 15, 2026
Move the test execution logic from inline heredoc in the workflow
to a separate run-ci-tests.sh script for better maintainability
and to avoid shell parsing issues with the android-emulator-runner.
- Remove run-instrumentation-tests.sh as users can run tests directly
  with ./gradlew connectedCheck -PmodelPreset=stories
- Consolidate README testing sections into single streamlined section
@kirklandsign kirklandsign merged commit 5b9154f into main Jan 15, 2026
2 of 3 checks passed
@kirklandsign kirklandsign deleted the report-text-output branch January 15, 2026 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants