Skip to content

feat: add DiffSynth backend support#1260

Open
mysuochao2-boop wants to merge 1 commit into
gpustack:mainfrom
sudoprivacy:main
Open

feat: add DiffSynth backend support#1260
mysuochao2-boop wants to merge 1 commit into
gpustack:mainfrom
sudoprivacy:main

Conversation

@mysuochao2-boop

Copy link
Copy Markdown

No description provided.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for the DiffSynth backend, enabling built-in image and video generation. It includes localized configuration strings, a new parameters definition file, and logic to automatically select DiffSynth for image models while disabling distributed inference for it. The review feedback highlights a robustness issue in checkIsImageModel when handling string-based categories or comma-separated tasks. Additionally, the reviewer suggests extending the distributed inference exclusion logic to also cover the voxBox backend for consistency across the UI and form submission.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +23 to +40
const checkIsImageModel = (model: any, source?: string) => {
const task = model?.task;
const categories = model?.categories || [];
if (
model?.image_only ||
categories.includes(modelTaskMap.image) ||
categories.includes(modelTaskMap.textToImage)
) {
return true;
}
if (source === modelSourceMap.huggingface_value) {
return task === HuggingFaceTaskMap[modelTaskMap.textToImage];
}
if (source === modelSourceMap.modelscope_value) {
return task === ModelscopeTaskMap[modelTaskMap.textToImage];
}
return task === modelTaskMap.textToImage || task === modelTaskMap.image;
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The checkIsImageModel function has potential robustness issues:

  1. categories can be a string (e.g., 'image') or an array. If it is a string, calling .includes() performs a substring check, which can lead to false positives (e.g., 'image_only' would match 'image').
  2. task can be a comma-separated string of tasks (especially for ModelScope models, e.g., 'text-to-image,image-to-image'). An exact equality check task === ModelscopeTaskMap[...] will fail when multiple tasks are present.

We should split task by , and check if the resulting array contains the target task, and handle categories safely by checking if it is an array or string.

  const checkIsImageModel = (model: any, source?: string) => {
    const task = model?.task;
    const categories = model?.categories || [];
    const hasImageCategory = Array.isArray(categories)
      ? categories.includes(modelTaskMap.image) || categories.includes(modelTaskMap.textToImage)
      : categories === modelTaskMap.image || categories === modelTaskMap.textToImage;

    if (
      model?.image_only ||
      hasImageCategory
    ) {
      return true;
    }

    const tasks = typeof task === 'string' ? task.split(',').map((t: string) => t.trim()) : [];

    if (source === modelSourceMap.huggingface_value) {
      return tasks.includes(HuggingFaceTaskMap[modelTaskMap.textToImage]);
    }
    if (source === modelSourceMap.modelscope_value) {
      return tasks.includes(ModelscopeTaskMap[modelTaskMap.textToImage]);
    }
    return (
      tasks.includes(modelTaskMap.textToImage) ||
      tasks.includes(modelTaskMap.image)
    );
  };

Comment on lines +110 to +111
{currentBackendOptions?.isBuiltIn &&
backend !== backendOptionsMap.diffSynth && (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Like diffSynth, the voxBox backend also does not support distributed inference across workers (and is forced to false on submit). We should exclude both diffSynth and voxBox from showing this option in the UI to keep it consistent.

Suggested change
{currentBackendOptions?.isBuiltIn &&
backend !== backendOptionsMap.diffSynth && (
{currentBackendOptions?.isBuiltIn &&
![backendOptionsMap.diffSynth, backendOptionsMap.voxBox].includes(backend) && (

Comment on lines +204 to +211
const updateDistributedInferenceConfig = (backend: string) => {
if (backend === backendOptionsMap.diffSynth) {
return {
distributed_inference_across_workers: false
};
}
return {};
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Unify the distributed inference configuration helper to also disable it for voxBox, as it is not supported by voxBox either.

Suggested change
const updateDistributedInferenceConfig = (backend: string) => {
if (backend === backendOptionsMap.diffSynth) {
return {
distributed_inference_across_workers: false
};
}
return {};
};
const updateDistributedInferenceConfig = (backend: string) => {
if ([backendOptionsMap.diffSynth, backendOptionsMap.voxBox].includes(backend)) {
return {
distributed_inference_across_workers: false
};
}
return {};
};

Comment on lines +243 to +245
...(data.backend === backendOptionsMap.diffSynth
? { distributed_inference_across_workers: false }
: {}),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Ensure distributed_inference_across_workers is forced to false on submit for both diffSynth and voxBox backends.

Suggested change
...(data.backend === backendOptionsMap.diffSynth
? { distributed_inference_across_workers: false }
: {}),
...([backendOptionsMap.diffSynth, backendOptionsMap.voxBox].includes(data.backend)
? { distributed_inference_across_workers: false }
: {}),

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants