feat(frontend): add HuggingFace media output rendering in result panel#5675
Open
juliethecao wants to merge 57 commits into
Open
feat(frontend): add HuggingFace media output rendering in result panel#5675juliethecao wants to merge 57 commits into
juliethecao wants to merge 57 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #5675 +/- ##
============================================
+ Coverage 53.32% 53.59% +0.26%
- Complexity 2670 2827 +157
============================================
Files 1098 1108 +10
Lines 42532 43361 +829
Branches 4575 4730 +155
============================================
+ Hits 22682 23239 +557
- Misses 18520 18762 +242
- Partials 1330 1360 +30
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
✅ No material benchmark regressions detected🟢 0 better · 🔴 0 worse · ⚪ 15 noise (<±5%) · 0 without baseline
Baseline detailsLatest main
Raw CSVconfig_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,498.75,200,128000,401,0.245,24819.47,33998.30,33998.30
1,100,10,64,20,2454.86,2000,1280000,815,0.497,120420.79,143264.75,143264.75
2,1000,10,64,20,21442.79,20000,12800000,933,0.569,1072045.95,1119081.48,1119081.48 |
juliethecao
force-pushed
the
hf/08-result-media
branch
from
1d05e38 to
339130b
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends Texera’s HuggingFace operator experience by (1) detecting/rendering media-like outputs in the workflow result UI and (2) adding/expanding HuggingFace property-editor support (task/model selection plus image/audio inputs). It also includes backend HuggingFace codegen additions and a small schema/output-column handling adjustment.
Changes:
- Add frontend media-type detection helpers (+ unit tests) and use them in the result table and row-detail modal.
- Add HuggingFace Formly field components (model selector, image upload, audio upload) and task preview/visibility logic in the property editor.
- Add backend HuggingFace task codegen objects (image/audio/media/QA) and adjust output-schema/result-column handling.
Reviewed changes
Copilot reviewed 33 out of 35 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| frontend/src/app/workspace/service/workflow-graph/model/mock-workflow-data.ts | Adds a mock HuggingFace operator predicate for frontend tests/mocks. |
| frontend/src/app/workspace/service/operator-metadata/mock-operator-metadata.data.ts | Adds a mock HuggingFace operator schema and registers it in the mock schema list. |
| frontend/src/app/workspace/component/result-panel/result-table-frame/result-table-frame.component.ts | Adds is*Cell helpers delegating to media-type util. |
| frontend/src/app/workspace/component/result-panel/result-table-frame/result-table-frame.component.spec.ts | Adds tests around media URL detection for result cells and util consistency. |
| frontend/src/app/workspace/component/result-panel/result-table-frame/result-table-frame.component.html | Updates table cell rendering to show media placeholders/icons based on detected type. |
| frontend/src/app/workspace/component/result-panel/result-panel-model.component.scss | Adds modal/row-detail styling for media display and copy actions. |
| frontend/src/app/workspace/component/result-panel/result-panel-modal.component.ts | Builds per-field row entries with media metadata + copy-to-clipboard + media proxy URL resolution. |
| frontend/src/app/workspace/component/result-panel/result-panel-modal.component.html | Renders video/audio/image inline in the modal with copy fallback for text. |
| frontend/src/app/workspace/component/property-editor/operator-property-edit-frame/operator-property-edit-frame.component.ts | Adds HuggingFace task preview samples, task-based field visibility/validators, and Formly mapping adjustments. |
| frontend/src/app/workspace/component/property-editor/operator-property-edit-frame/operator-property-edit-frame.component.spec.ts | Adds tests for HuggingFace task preview behavior (known/unknown/empty/non-HF). |
| frontend/src/app/workspace/component/property-editor/operator-property-edit-frame/operator-property-edit-frame.component.scss | Adds styling for the HuggingFace task preview card. |
| frontend/src/app/workspace/component/property-editor/operator-property-edit-frame/operator-property-edit-frame.component.html | Renders the HuggingFace task preview card above the Formly form. |
| frontend/src/app/workspace/component/hugging-face/hugging-face.component.ts | Introduces a HuggingFace model/task selector Formly field with caching, pagination, and search. |
| frontend/src/app/workspace/component/hugging-face/hugging-face.component.spec.ts | Adds basic unit tests for exported constants/cache invalidation. |
| frontend/src/app/workspace/component/hugging-face/hugging-face.component.scss | Adds styles for the HuggingFace model select UI. |
| frontend/src/app/workspace/component/hugging-face/hugging-face.component.html | Adds the HuggingFace task dropdown, search, model list, and pagination UI. |
| frontend/src/app/workspace/component/hugging-face-image-upload/hugging-face-image-upload.component.ts | Adds client-side image compression + data-URL storage for HF image inputs. |
| frontend/src/app/workspace/component/hugging-face-image-upload/hugging-face-image-upload.component.spec.ts | Adds unit tests for image upload component state/error paths. |
| frontend/src/app/workspace/component/hugging-face-image-upload/hugging-face-image-upload.component.scss | Styles the image upload and preview UI. |
| frontend/src/app/workspace/component/hugging-face-image-upload/hugging-face-image-upload.component.html | Adds template for selecting/previewing/clearing uploaded images. |
| frontend/src/app/workspace/component/hugging-face-audio-upload/hugging-face-audio-upload.component.ts | Adds backend-stream upload + preview for HF audio inputs. |
| frontend/src/app/workspace/component/hugging-face-audio-upload/hugging-face-audio-upload.component.spec.ts | Adds a minimal spec scaffolding for the audio upload component. |
| frontend/src/app/workspace/component/hugging-face-audio-upload/hugging-face-audio-upload.component.scss | Styles the audio upload and preview UI. |
| frontend/src/app/workspace/component/hugging-face-audio-upload/hugging-face-audio-upload.component.html | Adds template for selecting/uploading/previewing/clearing audio. |
| frontend/src/app/common/util/media-type.util.ts | Adds isImageUrl / isAudioUrl / isVideoUrl helpers. |
| frontend/src/app/common/util/media-type.util.spec.ts | Adds unit tests covering detection across prefixes/extensions/query strings/CDN URLs. |
| frontend/src/app/common/formly/formly-config.ts | Registers new HuggingFace Formly field types. |
| frontend/src/app/app.module.ts | Adds HuggingFace components to the app module declarations. |
| common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceInferenceOpDesc.scala | Adjusts result-column fallback logic in codegen and output schema. |
| common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/QaRankingCodegen.scala | Adds codegen for QA/zero-shot/similarity/ranking task family. |
| common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/MediaGenCodegen.scala | Adds codegen for prompt-driven media generation tasks (text-to-image/video). |
| common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/ImageTaskCodegen.scala | Adds codegen for image task family (image-only and image+prompt). |
| common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/AudioTaskCodegen.scala | Adds codegen for audio task family (ASR/audio-classification/TTS). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…nent Register the huggingface formly field type and declare HuggingFaceComponent in AppModule. Provides a task dropdown, paginated model list with client-side search, and per-task field state preservation when switching tasks.
The rxjs/no-implicit-any-catch ESLint rule requires explicit type annotations on error callbacks in .subscribe() calls. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…mponent Satisfies the rxjs-angular/prefer-takeuntil lint rule. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…mponent Fix setInterval/setTimeout leaks by tracking timer IDs and clearing them in ngOnDestroy. Remove takeUntil(destroy$) from shared module-level task fetch to prevent cache poisoning when the initiating component is destroyed. Remove unused TASK_TAG_MAP and TASK_NAMES exports. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…FaceComponent inFlightByTag was written but never read, so multiple component instances mounting for the same uncached task would each fire a full backend request. Add an in-flight guard that polls for the existing request's completion, matching the pattern already used for task fetches. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… in HuggingFaceComponent
Read X-Texera-Truncated header via { observe: "response" } to detect
when the backend's model list is incomplete. Show a notice prompting
users to search. When truncated, search queries are sent to the backend
search endpoint with debounce; otherwise local filtering is used.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…r lint rule Add back destroy$ and takeUntil(this.destroy$) on all subscribe calls. For the shared task fetch, add finalize() to reset tasksFetchSubscription when takeUntil fires before next/error, preventing the stale-guard bug the reviewer originally flagged. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…anup Mirror the defensive pattern from the tasks fetch so that when takeUntil(destroy$) cancels mid-flight, inFlightByTag is cleared and subsequent component instances can re-fetch instead of polling forever. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Elliot Lin <36275109+ELin2025@users.noreply.github.com>
- Fix model polling indefinite loop by detecting cancelled in-flight fetch - Fix CSS class mismatch: hf-tasks-error → hf-error to match SCSS - Remove unused imports in spec file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Catch HTTP errors inside switchMap so search stream survives failures - Use empty string instead of null when clearing model selection Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add mockHuggingFaceSchema and mockHuggingFacePredicate to test infrastructure. Add 7 spec tests covering huggingFaceTaskPreview for known tasks (text, image, audio, video), unknown tasks (fallback), empty tasks, and non-HF operators.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ew sample Fix inverted fallback for systemPrompt/maxNewTokens/temperature: these fields now correctly hide when no task is selected, matching the behavior of all other HuggingFace fields. Add missing image-text-to-text entry to huggingFaceTaskPreviewSamples so it no longer falls through to the generic text fallback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add sample-video.mp4 and sample-audio.wav for task preview cards - Check hasOperator() before getOperator() in isHuggingFaceOperator() Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…lassification Prefer self.CANDIDATE_LABELS over prompt column for candidate labels, with fallback to prompt_value for backward compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cover hide expressions for imageInput, audioInput, promptColumn, systemPrompt, contextColumn, and candidateLabels across task types. Also test requiredPromptColumn, requiredImageInput, and requiredAudioInput validators pass/fail for relevant tasks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Automated Reviewer SuggestionsBased on the
|
Plugs the 9-task image family into the dispatcher pattern established
in PR 2:
image-only image-classification, object-detection,
image-segmentation, image-to-text
image + prompt visual-question-answering, document-question-answering,
zero-shot-image-classification, image-text-to-text,
image-to-image
- ImageTaskCodegen supplies payload + parse Python for all 9 tasks
- TaskCodegen trait gains a `tasks: Set[String]` default method so a
single codegen can register under multiple task strings; the
dispatcher map in HuggingFaceInferenceOpDesc is built from
registeredCodegens.tasks.flatMap(...)
- CodegenContext extended with imageInput + inputImageColumn
(EncodableString)
- HuggingFaceInferenceOpDesc gains 2 new @JsonProperty fields and
registers ImageTaskCodegen
PythonCodegenBase grows to host the shared image infrastructure:
- image_only_tasks / image_prompt_tasks / image_tasks tuples and
image_headers in process_table
- per-row image bytes resolution from upload (self._read_image_input)
or input column (self._read_binary_value + self._compress_image_bytes)
- use_raw_binary_body / raw_binary_headers state threaded through
_post_with_fallback (signature extended)
- _post_with_fallback adds the image-text-to-text chat-completions
branch and the model-author vision branch
- _call_provider adds branches for zai-org's custom API, Replicate
predictions + polling, Fal-ai, Wavespeed submit+poll, and image
embedding in OpenAI-compatible / unknown-provider fallbacks
- image-content-type response handling returns data:image URLs
- image helpers added: _read_image_input, _compress_image_bytes,
_image_input_as_base64, _read_binary_value, _looks_like_html,
_html_to_image_bytes, _extract_json_arg, _url_to_data_url
User-input strings continue to flow through pyb"..." + EncodableString
so they reach Python as self.decode_python_template('<base64>') rather
than raw literals. PythonCodeRawInvalidTextSpec still passes
(117/117 descriptors py_compile cleanly).
Frontend integration adds only the HF lines (no agent / dataset
noise from the source branch):
- HuggingFaceImageUploadComponent declared in app.module.ts
- huggingface-image-upload formly type registered in formly-config.ts
- Image upload component .ts/.html/.scss cherry-picked from huggingFace
- HuggingFace.png + sample-image.png assets
PR 3 of a stacked 9-PR series. Stacks on hf/02-operator-textgen.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…d media proxy Introduces a new Jersey REST resource exposing endpoints used by the upcoming HuggingFace operator UI: - GET /api/huggingface/models — browse / search models per task - GET /api/huggingface/tasks — list HF pipeline tags with hosted inference - POST /api/huggingface/upload-audio — upload audio for HF audio tasks - GET /api/huggingface/audio-preview — stream uploaded audio (path-validated) - GET /api/huggingface/media-proxy — proxy remote media URLs to bypass CORS This is the first PR in a stacked series landing the HF operator end-to-end. No operator code yet; this resource is independently useful and lets the frontend integrate with HF before the operator class lands.
Addresses xuang7's review on PR apache#5124 — both endpoints previously buffered the full payload into a heap-resident byte[] with no upper bound, leaving the JVM open to OOM on a hostile or buggy upstream response (/media-proxy) or out-of-band write into the audio temp dir (/audio-preview). - /media-proxy: switch from Unirest.asBytes() to asObject(Function<RawResponse, T>), streaming the upstream body in 8 KiB chunks with a running byte counter. Aborts with 413 if the declared Content-Length exceeds the cap (pre-check) or if the body crosses the cap mid-read (defends against missing/lying Content-Length). New MAX_MEDIA_PROXY_BYTES = 50 MiB, sized for HF inference media (text-to-image ~5 MiB, text-to-video ~30 MiB) with headroom. - /audio-preview: add Files.size() defense-in-depth check before readAllBytes. /upload-audio already enforces MAX_AUDIO_BYTES on ingest; this catches the case where a bug or out-of-band write puts an oversized file in the temp dir. Adds a spec covering the audio-preview cap using a sparse-file fixture so the test stays fast (87/87 spec passes). The media-proxy cap path is exercised via the existing input-validation suite plus the new streamMediaWithCap helper - a follow-up can add a fake-RawResponse unit test if reviewers want explicit coverage of the chunked-read cap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per review on apache#5124 (xuang7, Ma77Ball): mark the resource with @RolesAllowed(Array("REGULAR", "ADMIN")) to document that all five endpoints require an authenticated user. The annotation isn't enforced yet — that's coming with the auth-enforcement PR @Yicong-Huang and @Ma77Ball are working on — but adding it now means no follow-up change is needed when enforcement lands, and it matches the convention used by UserConfigResource / AdminSettingsResource. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… panel Add isImageUrl/isAudioUrl/isVideoUrl detection helpers and wire them into the result table and row detail modal so image, audio, and video outputs from HuggingFace tasks render inline instead of as raw URLs. Gate backend string truncation on output mode so HF data URLs are never cut off.
…media-type spec Vitest does not support Jasmine-style toBeTrue() and toBeFalse() matchers.
… headers and body Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Julie Cao <116243642+juliethecao@users.noreply.github.com>
…sult panel row mapping
…y undefined, and use ogv for video extension
juliethecao
force-pushed
the
hf/08-result-media
branch
from
b83d948 to
1eeba34
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Render image, audio, and video outputs from HuggingFace tasks inline in the workflow result panel instead of displaying raw data URLs as text.
New file —
media-type.util.ts:isImageUrl— detectsdata:image/data URLs and common image extensions (.png,.jpg,.jpeg,.gif,.webp)isAudioUrl— detectsdata:audio/data URLs and common audio extensions (.mp3,.wav,.ogg,.m4a,.flac)isVideoUrl— detectsdata:video/data URLs, common video extensions (.mp4,.webm,.ogg), and thefal.mediaCDN host used by fal.ai text-to-video outputsChanges to
result-table-frame.component.{ts,html}:isImageCell/isAudioCell/isVideoCellmethods that delegate to the detection helpers<img>,<audio controls>,<video controls>elements render inline in the row detail modalChanges to
result-panel-modal.component.{ts,html}:rowEntrieswith per-field media metadata on modal openChanges to
result-panel-model.component.scss:New file —
media-type.util.spec.ts:fal.mediaCDN URL, cross-type rejection, and empty/plain-string inputsAny related issues, documentation, discussions?
How was this PR tested?
Unit tests in
media-type.util.spec.ts(19 cases) andresult-table-frame.component.spec.ts. Run withng test.Was this PR authored or co-authored using generative AI tooling?
Co-authored with Claude Sonnet 4.6 in compliance with ASF guidelines