Skip to content

feat: IMG TTV node support reference model #4949

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zhanweizhang7 merged 1 commit into from
Mar 25, 2026
Merged

feat: IMG TTV node support reference model #4949

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions apps/application/flow/step_node/image_generate_step_node/i_image_generate_node.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,10 @@


class ImageGenerateNodeSerializer(serializers.Serializer):
model_id = serializers.CharField(required=True, label=_("Model id"))

model_id = serializers.CharField(required=False, allow_blank=True, allow_null=True, label=_("Model id"))
model_id_type = serializers.CharField(required=False, default='custom', label=_("Model id type"))
model_id_reference = serializers.ListField(required=False, child=serializers.CharField(), allow_empty=True,
label=_("Reference Field"))
prompt = serializers.CharField(required=True, label=_("Prompt word (positive)"))

negative_prompt = serializers.CharField(required=False, label=_("Prompt word (negative)"),
Expand Down Expand Up @@ -49,6 +51,6 @@ def _run(self):

def execute(self, model_id, prompt, negative_prompt, dialogue_number, dialogue_type, history_chat_record,
model_params_setting,
chat_record_id,
chat_record_id, model_id_type=None, model_id_reference=None,
**kwargs) -> NodeResult:
pass
13 changes: 12 additions & 1 deletion apps/application/flow/step_node/image_generate_step_node/impl/base_image_generate_node.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,21 @@ def save_context(self, details, workflow_manage):
def execute(self, model_id, prompt, negative_prompt, dialogue_number, dialogue_type, history_chat_record,
model_params_setting,
chat_record_id,
model_id_type=None, model_id_reference=None,
**kwargs) -> NodeResult:
# 处理引用类型
if model_id_type == 'reference' and model_id_reference:
reference_data = self.workflow_manage.get_reference_field(
model_id_reference[0],
model_id_reference[1:],
)
if reference_data and isinstance(reference_data, dict):
model_id = reference_data.get('model_id', model_id)
model_params_setting = reference_data.get('model_params_setting')

workspace_id = self.workflow_manage.get_body().get('workspace_id')
tti_model = get_model_instance_by_model_workspace_id(model_id, workspace_id,
**model_params_setting)
**(model_params_setting or {}))
history_message = self.get_history_message(history_chat_record, dialogue_number)
self.context['history_message'] = history_message
question = self.generate_prompt_question(prompt)
Expand Down
6 changes: 5 additions & 1 deletion apps/application/flow/step_node/image_understand_step_node/i_image_understand_node.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,10 @@


class ImageUnderstandNodeSerializer(serializers.Serializer):
model_id = serializers.CharField(required=True, label=_("Model id"))
model_id = serializers.CharField(required=False, allow_blank=True, allow_null=True, label=_("Model id"))
model_id_type = serializers.CharField(required=False, default='custom', label=_("Model id type"))
model_id_reference = serializers.ListField(required=False, child=serializers.CharField(), allow_empty=True,
label=_("Reference Field"))
system = serializers.CharField(required=False, allow_blank=True, allow_null=True,
label=_("Role Setting"))
prompt = serializers.CharField(required=True, label=_("Prompt word"))
Expand Down Expand Up @@ -52,5 +55,6 @@ def execute(self, model_id, system, prompt, dialogue_number, dialogue_type, hist
model_params_setting,
chat_record_id,
image,
model_id_type=None, model_id_reference=None,
**kwargs) -> NodeResult:
pass
13 changes: 12 additions & 1 deletion .../application/flow/step_node/image_understand_step_node/impl/base_image_understand_node.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,11 +83,22 @@ def execute(self, model_id, system, prompt, dialogue_number, dialogue_type, hist
model_params_setting,
chat_record_id,
image,
model_id_type=None, model_id_reference=None,
**kwargs) -> NodeResult:
# 处理引用类型
if model_id_type == 'reference' and model_id_reference:
reference_data = self.workflow_manage.get_reference_field(
model_id_reference[0],
model_id_reference[1:],
)
if reference_data and isinstance(reference_data, dict):
model_id = reference_data.get('model_id', model_id)
model_params_setting = reference_data.get('model_params_setting')

# 处理不正确的参数
workspace_id = self.workflow_manage.get_body().get('workspace_id')
image_model = get_model_instance_by_model_workspace_id(model_id, workspace_id,
**model_params_setting)
**(model_params_setting or {}))
# 执行详情中的历史消息不需要图片内容
history_message = self.get_history_message_for_details(history_chat_record, dialogue_number)
self.context['history_message'] = history_message
Expand Down
7 changes: 5 additions & 2 deletions apps/application/flow/step_node/text_to_video_step_node/i_text_to_video_node.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,10 @@


class TextToVideoNodeSerializer(serializers.Serializer):
model_id = serializers.CharField(required=True, label=_("Model id"))

model_id = serializers.CharField(required=False, allow_blank=True, allow_null=True, label=_("Model id"))
model_id_type = serializers.CharField(required=False, default='custom', label=_("Model id type"))
model_id_reference = serializers.ListField(required=False, child=serializers.CharField(), allow_empty=True,
label=_("Reference Field"))
prompt = serializers.CharField(required=True, label=_("Prompt word (positive)"))

negative_prompt = serializers.CharField(required=False, label=_("Prompt word (negative)"),
Expand Down Expand Up @@ -50,5 +52,6 @@ def _run(self):
def execute(self, model_id, prompt, negative_prompt, dialogue_number, dialogue_type, history_chat_record,
model_params_setting,
chat_record_id,
model_id_type=None, model_id_reference=None,
**kwargs) -> NodeResult:
pass
13 changes: 12 additions & 1 deletion apps/application/flow/step_node/text_to_video_step_node/impl/base_text_to_video_node.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,21 @@ def save_context(self, details, workflow_manage):
def execute(self, model_id, prompt, negative_prompt, dialogue_number, dialogue_type, history_chat_record,
model_params_setting,
chat_record_id,
model_id_type=None, model_id_reference=None,
**kwargs) -> NodeResult:
# 处理引用类型
if model_id_type == 'reference' and model_id_reference:
reference_data = self.workflow_manage.get_reference_field(
model_id_reference[0],
model_id_reference[1:],
)
if reference_data and isinstance(reference_data, dict):
model_id = reference_data.get('model_id', model_id)
model_params_setting = reference_data.get('model_params_setting')

workspace_id = self.workflow_manage.get_body().get('workspace_id')
ttv_model = get_model_instance_by_model_workspace_id(model_id, workspace_id,
**model_params_setting)
**(model_params_setting or {}))
history_message = self.get_history_message(history_chat_record, dialogue_number)
self.context['history_message'] = history_message
question = self.generate_prompt_question(prompt)
Expand Down
76 changes: 56 additions & 20 deletions ui/src/workflow/nodes/image-generate/index.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,13 @@
>
<el-form-item
:label="$t('workflow.nodes.imageGenerateNode.model.label')"
prop="model_id"
:prop="form_data.model_id_type === 'reference' ? 'model_id_reference' : 'model_id'"
:rules="{
required: true,
message: $t('workflow.nodes.imageGenerateNode.model.requiredMessage'),
message:
form_data.model_id_type === 'reference'
? $t('workflow.variable.placeholder')
: $t('workflow.nodes.imageGenerateNode.model.requiredMessage'),
trigger: 'change',
}"
>
Expand All @@ -28,31 +31,51 @@
}}<span class="color-danger">*</span></span
>
</div>
<el-select
v-model="form_data.model_id_type"
:teleported="false"
size="small"
style="width: 85px"
@change="form_data.model_id_reference = []"
>
<el-option :label="$t('workflow.variable.Referencing')" value="reference" />
<el-option :label="$t('common.custom')" value="custom" />
</el-select>
</div>
</template>
<div class="flex-between w-full" v-if="form_data.model_id_type !== 'reference'">
<ModelSelect
@change="model_change"
@wheel="wheel"
:teleported="false"
v-model="form_data.model_id"
:placeholder="$t('workflow.nodes.imageGenerateNode.model.requiredMessage')"
:options="modelOptions"
showFooter
@focus="getSelectModel"
:model-type="'TTI'"
></ModelSelect>
<div class="ml-8">
<el-button
:disabled="!form_data.model_id"
type="primary"
link
@click="openAIParamSettingDialog(form_data.model_id)"
@refreshForm="refreshParam"
>
<AppIcon iconName="app-setting"></AppIcon>
<el-icon>
<Operation />
</el-icon>
</el-button>
</div>
</template>

<ModelSelect
@change="model_change"
@wheel="wheel"
:teleported="false"
v-model="form_data.model_id"
:placeholder="$t('workflow.nodes.imageGenerateNode.model.requiredMessage')"
:options="modelOptions"
showFooter
@focus="getSelectModel"
:model-type="'TTI'"
></ModelSelect>
</div>
<NodeCascader
v-else
ref="nodeCascaderRef"
:nodeModel="nodeModel"
class="w-full"
:placeholder="$t('workflow.variable.placeholder')"
v-model="form_data.model_id_reference"
/>
</el-form-item>

<el-form-item
:label="$t('workflow.nodes.imageGenerateNode.prompt.label')"
prop="prompt"
Expand Down Expand Up @@ -158,6 +181,7 @@ import type { FormInstance } from 'element-plus'
import AIModeParamSettingDialog from '@/views/application/component/AIModeParamSettingDialog.vue'
import { t } from '@/locales'
import { useRoute } from 'vue-router'
import NodeCascader from '@/workflow/common/NodeCascader.vue'
import { loadSharedApi } from '@/utils/dynamics-api/shared-api'
import { WorkflowMode } from '@/enums/application'
const workflowMode = (inject('workflowMode') as WorkflowMode) || WorkflowMode.Application
Expand All @@ -184,10 +208,14 @@ const AIModeParamSettingDialogRef = ref<InstanceType<typeof AIModeParamSettingDi

const aiChatNodeFormRef = ref<FormInstance>()
const validate = () => {
return aiChatNodeFormRef.value?.validate().catch((err) => {
return Promise.all([
nodeCascaderRef.value ? nodeCascaderRef.value.validate() : Promise.resolve(''),
aiChatNodeFormRef.value?.validate(),
]).catch((err: any) => {
return Promise.reject({ node: props.nodeModel, errMessage: err })
})
}
const nodeCascaderRef = ref()

const wheel = (e: any) => {
if (e.ctrlKey === true) {
Expand All @@ -203,6 +231,8 @@ const defaultPrompt = `{{${t('workflow.nodes.startNode.label')}.question}}`

const form = {
model_id: '',
model_id_type: 'custom',
model_id_reference: [],
system: '',
prompt: defaultPrompt,
negative_prompt: '',
Expand All @@ -217,6 +247,12 @@ const form = {
const form_data = computed({
get: () => {
if (props.nodeModel.properties.node_data) {
if (!props.nodeModel.properties.node_data.model_id_type) {
set(props.nodeModel.properties.node_data, 'model_id_type', 'custom')
}
if (!props.nodeModel.properties.node_data.model_id_reference) {
set(props.nodeModel.properties.node_data, 'model_id_reference', [])
}
return props.nodeModel.properties.node_data
} else {
set(props.nodeModel.properties, 'node_data', form)
Copy link
Copy Markdown
Contributor Author

@shaohuzhang1 shaohuzhang1 Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code is mostly correct but can be optimized for better readability and functionality. Here are some suggestions:

Optimal Code

  1. Reduce Redundancy:

    • Use ternary operators where possible to reduce repetition.

  2. Improve Logic:

    • Ensure that all logic related to model selection and validation is centralized to maintain consistency.

  3. Handle State Consistency:

    • Make sure state changes in one component (e.g., cascader or single input) are propagated and reflected correctly.

Here's an improved version of the specified part of your code:

<template>
  <!-- ... existing template content ... -->
</template>

<script lang="ts" setup>
// ... imports and other setup ...

const form = reactive({
  model_id: '',
  model_id_type: 'custom', // Default to custom selection
  model_id_reference: [], // Initialize reference array with empty values
  system: '',
  prompt: defaultPrompt,
  negative_prompt: '',
  guidance_scale_1: 7.5,
  aspect_ratio: '',
  height: 512,
  width: 512,
});

const form_data = computed(() => ({
  ...props.nodeModel.properties.node_data || {},
}));

const model_change = (value: string[]) => {
  form.model_id_reference = value;
};

// Validation function remains the same...

const refreshParam = () => {
  // Refresh parameter logic here...
};

// ... rest of the script ...
</script>

Explanation

  • Default Selection: The model_id_type starts as "custom". If not provided in the saved data, it defaults this way.

  • Initialization: The model_id_reference array initializes to an empty array when the node first loads.

  • Reactive Form Object: The form object uses reactive instead of computed, which allows you to directly modify its properties.

  • Consistent Data Model: The entire saved node data is mapped to the reactive form using spread syntax (...). This maintains consistency and reduces redundant checks.

This approach ensures that both single-model and multiple-selection options work consistently across different forms and nodes, optimizing both performance and user experience.

Expand Down
65 changes: 48 additions & 17 deletions ui/src/workflow/nodes/image-understand/index.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,13 @@
>
<el-form-item
:label="$t('workflow.nodes.imageUnderstandNode.model.label')"
prop="model_id"
:prop="form_data.model_id_type === 'reference' ? 'model_id_reference' : 'model_id'"
:rules="{
required: true,
message: $t('workflow.nodes.imageUnderstandNode.model.requiredMessage'),
message:
form_data.model_id_type === 'reference'
? $t('workflow.variable.placeholder')
: $t('workflow.nodes.imageUnderstandNode.model.requiredMessage'),
trigger: 'change',
}"
>
Expand All @@ -28,29 +31,49 @@
}}<span class="color-danger">*</span></span
>
</div>
<el-select
v-model="form_data.model_id_type"
:teleported="false"
size="small"
style="width: 85px"
@change="form_data.model_id_reference = []"
>
<el-option :label="$t('workflow.variable.Referencing')" value="reference" />
<el-option :label="$t('common.custom')" value="custom" />
</el-select>
</div>
</template>
<div class="flex-between w-full" v-if="form_data.model_id_type !== 'reference'">
<ModelSelect
@wheel="wheel"
:teleported="false"
v-model="form_data.model_id"
:placeholder="$t('workflow.nodes.imageUnderstandNode.model.requiredMessage')"
:options="modelOptions"
showFooter
:model-type="'IMAGE'"
></ModelSelect>
<div class="ml-8">
<el-button
:disabled="!form_data.model_id"
type="primary"
link
@click="openAIParamSettingDialog(form_data.model_id)"
@refreshForm="refreshParam"
>
<AppIcon iconName="app-setting"></AppIcon>
<el-icon>
<Operation />
</el-icon>
</el-button>
</div>
</template>

<ModelSelect
@wheel="wheel"
:teleported="false"
v-model="form_data.model_id"
:placeholder="$t('workflow.nodes.imageUnderstandNode.model.requiredMessage')"
:options="modelOptions"
showFooter
:model-type="'IMAGE'"
></ModelSelect>
</div>
<NodeCascader
v-else
ref="nodeCascaderRef"
:nodeModel="nodeModel"
class="w-full"
:placeholder="$t('workflow.variable.placeholder')"
v-model="form_data.model_id_reference"
/>
</el-form-item>

<el-form-item>
<template #label>
<div class="flex-between">
Expand Down Expand Up @@ -262,6 +285,8 @@ const defaultPrompt = `{{${t('workflow.nodes.startNode.label')}.question}}`

const form = {
model_id: '',
model_id_type: 'custom',
model_id_reference: [],
system: '',
prompt: defaultPrompt,
dialogue_number: 0,
Expand All @@ -275,6 +300,12 @@ const form = {
const form_data = computed({
get: () => {
if (props.nodeModel.properties.node_data) {
if (!props.nodeModel.properties.node_data.model_id_type) {
set(props.nodeModel.properties.node_data, 'model_id_type', 'custom')
}
if (!props.nodeModel.properties.node_data.model_id_reference) {
set(props.nodeModel.properties.node_data, 'model_id_reference', [])
}
return props.nodeModel.properties.node_data
} else {
set(props.nodeModel.properties, 'node_data', form)
Copy link
Copy Markdown
Contributor Author

@shaohuzhang1 shaohuzhang1 Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided code has several issues that need to be addressed:

  1. Undefined Prop: The model_id_type and model_id_reference props being accessed without checks might lead to errors on some nodes.

  2. Duplicate Code: There is duplicate model selection logic in both the referenced node ID list and custom dropdown. This can be optimized.

  3. Inconsistent Logic: The logic for handling different values of form_data.model_id_type seems inconsistent, causing redundancy.

  4. Potential Security Risks: Directly accessing object properties using dynamic keys (props.nodeModel.properties.node_data[props.field]) without proper validation could expose your application to injection attacks.

Here's an improved version of the code with these issues resolved:

const form_data = computed({
  get: () => {
    if (props.nodeModel.properties.node_data) {
      return props.nodeModel.properties.node_data;
    } else {
      return { ...defaultProps };
    }
  },
});

watch(
  () => props.nodeModel.properties.node_data,
  newValue => {
    Object.assign(form_data.value, newValue);
  }
);

function handleModelTypeChange(value) {
  if (value === 'reference') {
    form_data.value.model_id_reference.splice(0);
  }
}

const ModelSelectRef = ref(null);
const NodeCascaderRef = ref(null);

// Use this function to refresh parameter settings when needed.
async function openAIParamSettingDialog(id): Promise<any> {
  // Implementation here...
}

function wheel() {
  // Wheel event handler implementation here...
}

Key points:

  • Simplified logic by removing redundant code.
  • Used Vue's computed hook appropriately.
  • Added a watcher for node data changes to ensure the component updates.
  • Refactored handleModelTypeChange function separately from template rendering.
  • Dropped unnecessary dependencies like _defineProp.
  • Removed unused elements and comments.

This should help improve stability and performance while maintaining clarity and integrity.

Expand Down
Loading
Loading