Ml driver poc#254
Closed
ti3x wants to merge 2 commits into
Closed
Conversation
Lands the eval-pipeline ml_driver runner inside Firefox so eval suites can
invoke ML APIs through an in-tree, CI-tested entry point instead of
re-injecting JavaScript into the browser via executeAsyncScript every run.
Source mapping (eval-pipeline ml_driver/src/ml_driver/firefox/scripts -> MC):
runner.js -> src/firefox_inference/runner.js
(12-line executeAsyncScript shim) +
src/firefox_inference/Runner.sys.mjs
(handleMessage / run / serializeError)
imports.js -> folded into per-module
ChromeUtils.defineESModuleGetters
ml_engine.js -> src/firefox_inference/MlEngineRunner.sys.mjs
(createMlEngine, runMlEngine, destroyMlEngine,
getExistingEngine, plus serializeEntry and
serializeInferenceResponse - referenced but
never defined upstream)
translations.js -> src/firefox_inference/TranslationsRunner.sys.mjs
(createTranslationsSession,
runTranslationsSession,
destroyTranslationsSession,
normalizeLanguagePair, createDeferred)
page_extractor.js -> src/firefox_inference/PageExtractorRunner.sys.mjs
(getPageText, getPageInfo, getReaderModeContent,
getSelectionText, runHeadlessExtractor)
Module wiring (already committed in the parent change):
- src/firefox_inference/moz.build registers the four .sys.mjs modules
under EXTRA_JS_MODULES.firefox_inference, exposing them as
resource://gre/modules/firefox_inference/<Name>.sys.mjs.
- toolkit/components/ml/jar.mn ships runner.js at
chrome://global/content/ml/firefox_inference/runner.js.
- toolkit/components/ml/moz.build adds src/firefox_inference to DIRS and
registers the new browser.toml under BROWSER_CHROME_MANIFESTS.
New mochitest suite (tests/browser/browser_ml_driver/):
- browser.toml: pulls parent head.js + shared-head.js as support files.
- head.js: loadSubScript the parent ML browser-test head so setup(),
EngineProcess, remoteClients, etc. are available.
- browser_ml_driver.js: 6 add_task blocks (19 subtests) that all route
through `run(done, args)` via a `dispatch(...)` Promise wrapper.
Covers happy-path lifecycle (create -> run -> destroy moz-echo),
unknown-message error envelope, unknown/blank engineId error envelope,
and translations missing-source/target validation. Internal helpers
(serializeEntry/serializeError/normalizeLanguagePair/...) are verified
implicitly through the observable envelope contract.
Result: ./mach test toolkit/components/ml/tests/browser/browser_ml_driver
prints `Passed: 19 / Failed: 0 / Unexpected results: 0`.
TODO / deferred (not in this change):
- chat.js, memories.js, search_browsing_history.js from ml_driver are
NOT ported. They depend on moz-src:///browser/components/aiwindow/
(Smart Window), which is browser/, not toolkit/, and is still
actively changing. Defer until aiwindow stabilizes or a parallel
runner shell lands under browser/.
- PageExtractorRunner is wired but its actor (PageExtractorParent) is
not in upstream toolkit yet; lazy import means the module loads,
invocation throws clearly until the actor lands.
- Per-handler happy-path mochitests for translations sessions and
PageExtractor handlers; engine lifecycle is the only end-to-end
success path covered today.
- eval-pipeline FirefoxDriver (ml_driver/src/ml_driver/firefox/driver.py)
still bundles its own runner.js via script_utils.get_runner_script()
and ships it over Marionette per session. Switching it to load
chrome://global/content/ml/firefox_inference/runner.js directly is a
follow-up patch in eval-pipeline, gated on this landing upstream.
Differential Revision: https://phabricator.services.mozilla.com/D299465
Continues d051a08 (toolkit port) by landing the six handlers that were deferred because they depend on browser/components/aiwindow/ — Smart Window code, which sits in browser/ and cannot be reached from toolkit. A parallel runner shell under aiwindow/scripts/ now owns these handlers and falls through to the toolkit Runner.sys.mjs for the other twelve, so ml_driver gets one chrome:// URL that responds to all ~18 message names with an identical wire envelope. Files: - scripts/runner.js — 19-line executeAsyncScript shim, mirror of the toolkit shim. - scripts/AiwindowRunner.sys.mjs — dispatcher. 8-case switch over the smart-window handlers; default branch forwards to toolkit Runner.handleMessage. Exports run + serializeError matching the toolkit envelope contract bit-for-bit. - scripts/ChatRunner.sys.mjs — setRealTimeInfo, initConversation, submitTurn (5-round tool loop, DDG search fallback, runWithGenerator streaming). Lazy imports of ChatConversation, Utils, Tools, PageExtractorParent. - scripts/MemoriesRunner.sys.mjs — setMemories, getAllMemories, getRelevantMemories, clearMemories. Normalization defaults preserved (Hobbies & Leisure, Research / Learn, score 5). - scripts/SearchBrowsingHistoryRunner.sys.mjs — thin wrapper over the aiwindow searchBrowsingHistory model export. - scripts/moz.build, scripts/jar.mn — register MOZ_SRC_FILES, the chrome://browser/content/aiwindow/scripts/runner.js URL, and the mochitest manifest. - aiwindow/moz.build — one-line addition of "scripts" to DIRS, the only edit outside the new directory. Behavior change worth flagging: chat/conversation state moves from the chrome window (chromeWindow.__realTimeInfo, __conversationState) to module-level singletons. Per-process scope instead of per-window; ml_driver drives one window per process so eval-pipeline sees no difference, but the contract is documented. Tests: scripts/tests/browser/browser_aiwindow_runner.js covers the dispatcher envelope shape for set_real_time_info, init_conversation, the memories CRUD roundtrip, search_browsing_history, the toolkit fallthrough error envelope (unknown handler), and submit_turn input validation. Heavier submit_turn end-to-end with the aiwindow mock OpenAI fixture is a follow-up. Out of scope: ml_driver eval-pipeline migration to load this URL directly (gated on this landing, same staging pattern as d051a08); toolkit Runner.sys.mjs registry refactor (rejected as inverted layering — toolkit shouldn't acquire an extension point for a single browser-layer consumer). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
No Taskcluster jobs started for this pull requestThe |
Contributor
|
(Automated Close) Please do not file pull requests here, see https://firefox-source-docs.mozilla.org/contributing/how_to_submit_a_patch.html |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.