feat: add DiffSynth backend support#1260
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces support for the DiffSynth backend, enabling built-in image and video generation. It includes localized configuration strings, a new parameters definition file, and logic to automatically select DiffSynth for image models while disabling distributed inference for it. The review feedback highlights a robustness issue in checkIsImageModel when handling string-based categories or comma-separated tasks. Additionally, the reviewer suggests extending the distributed inference exclusion logic to also cover the voxBox backend for consistency across the UI and form submission.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| const checkIsImageModel = (model: any, source?: string) => { | ||
| const task = model?.task; | ||
| const categories = model?.categories || []; | ||
| if ( | ||
| model?.image_only || | ||
| categories.includes(modelTaskMap.image) || | ||
| categories.includes(modelTaskMap.textToImage) | ||
| ) { | ||
| return true; | ||
| } | ||
| if (source === modelSourceMap.huggingface_value) { | ||
| return task === HuggingFaceTaskMap[modelTaskMap.textToImage]; | ||
| } | ||
| if (source === modelSourceMap.modelscope_value) { | ||
| return task === ModelscopeTaskMap[modelTaskMap.textToImage]; | ||
| } | ||
| return task === modelTaskMap.textToImage || task === modelTaskMap.image; | ||
| }; |
There was a problem hiding this comment.
The checkIsImageModel function has potential robustness issues:
categoriescan be a string (e.g.,'image') or an array. If it is a string, calling.includes()performs a substring check, which can lead to false positives (e.g.,'image_only'would match'image').taskcan be a comma-separated string of tasks (especially for ModelScope models, e.g.,'text-to-image,image-to-image'). An exact equality checktask === ModelscopeTaskMap[...]will fail when multiple tasks are present.
We should split task by , and check if the resulting array contains the target task, and handle categories safely by checking if it is an array or string.
const checkIsImageModel = (model: any, source?: string) => {
const task = model?.task;
const categories = model?.categories || [];
const hasImageCategory = Array.isArray(categories)
? categories.includes(modelTaskMap.image) || categories.includes(modelTaskMap.textToImage)
: categories === modelTaskMap.image || categories === modelTaskMap.textToImage;
if (
model?.image_only ||
hasImageCategory
) {
return true;
}
const tasks = typeof task === 'string' ? task.split(',').map((t: string) => t.trim()) : [];
if (source === modelSourceMap.huggingface_value) {
return tasks.includes(HuggingFaceTaskMap[modelTaskMap.textToImage]);
}
if (source === modelSourceMap.modelscope_value) {
return tasks.includes(ModelscopeTaskMap[modelTaskMap.textToImage]);
}
return (
tasks.includes(modelTaskMap.textToImage) ||
tasks.includes(modelTaskMap.image)
);
};| {currentBackendOptions?.isBuiltIn && | ||
| backend !== backendOptionsMap.diffSynth && ( |
There was a problem hiding this comment.
Like diffSynth, the voxBox backend also does not support distributed inference across workers (and is forced to false on submit). We should exclude both diffSynth and voxBox from showing this option in the UI to keep it consistent.
| {currentBackendOptions?.isBuiltIn && | |
| backend !== backendOptionsMap.diffSynth && ( | |
| {currentBackendOptions?.isBuiltIn && | |
| ![backendOptionsMap.diffSynth, backendOptionsMap.voxBox].includes(backend) && ( |
| const updateDistributedInferenceConfig = (backend: string) => { | ||
| if (backend === backendOptionsMap.diffSynth) { | ||
| return { | ||
| distributed_inference_across_workers: false | ||
| }; | ||
| } | ||
| return {}; | ||
| }; |
There was a problem hiding this comment.
Unify the distributed inference configuration helper to also disable it for voxBox, as it is not supported by voxBox either.
| const updateDistributedInferenceConfig = (backend: string) => { | |
| if (backend === backendOptionsMap.diffSynth) { | |
| return { | |
| distributed_inference_across_workers: false | |
| }; | |
| } | |
| return {}; | |
| }; | |
| const updateDistributedInferenceConfig = (backend: string) => { | |
| if ([backendOptionsMap.diffSynth, backendOptionsMap.voxBox].includes(backend)) { | |
| return { | |
| distributed_inference_across_workers: false | |
| }; | |
| } | |
| return {}; | |
| }; |
| ...(data.backend === backendOptionsMap.diffSynth | ||
| ? { distributed_inference_across_workers: false } | ||
| : {}), |
There was a problem hiding this comment.
Ensure distributed_inference_across_workers is forced to false on submit for both diffSynth and voxBox backends.
| ...(data.backend === backendOptionsMap.diffSynth | |
| ? { distributed_inference_across_workers: false } | |
| : {}), | |
| ...([backendOptionsMap.diffSynth, backendOptionsMap.voxBox].includes(data.backend) | |
| ? { distributed_inference_across_workers: false } | |
| : {}), |
No description provided.