how to load a finetuned model especially during validation phase

[micklexqg]

As the above, I have finetuned the model and want to validate it, but the given demo which is train_dreambooth_sd3.py still uses "pipeline = StableDiffusion3Pipeline.from_pretrained( args.pretrained_model_name_or_path, transformer=transformer, text_encoder=text_encoder_one, text_encoder_2=text_encoder_two, text_encoder_3=text_encoder_three, ) " .

I wonder why it still load from args.pretrained_model_name_or_path as it has saved the finetuned model in the save_path which is "os.path.join(args.output_dir, f"checkpoint-{global_step}")".

so, how to how to load the finetuned model during validation phase?

Another confusion, what is the difference between " StableDiffusion3Pipeline.from_pretrained() " and "SD3Transformer2DModel.from_pretrained" as the following:

[DN6]

cc: @sayakpaul @linoytsaban

[sayakpaul]

During validation, we send a pipeline:

diffusers/examples/dreambooth/train_dreambooth_sd3.py

Line 162 in 1c50a5f

pipeline,

The pipeline there is first initialized like this:

diffusers/examples/dreambooth/train_dreambooth_sd3.py

Lines 1724 to 1734 in 1c50a5f

	pipeline = StableDiffusion3Pipeline.from_pretrained(
	args.pretrained_model_name_or_path,
	vae=vae,
	text_encoder=accelerator.unwrap_model(text_encoder_one),
	text_encoder_2=accelerator.unwrap_model(text_encoder_two),
	text_encoder_3=accelerator.unwrap_model(text_encoder_three),
	transformer=accelerator.unwrap_model(transformer),
	revision=args.revision,
	variant=args.variant,
	torch_dtype=weight_dtype,
	)

This is to ensure that the models being requested for fine-tuning are present in pipeline. We're using args.pretrained_model_name_or_path, with from_pretrained() to load the other components of the pipeline such as tokenizers and the scheduler.

[micklexqg]

During validation, we send a pipeline:

diffusers/examples/dreambooth/train_dreambooth_sd3.py

Line 162 in 1c50a5f
pipeline,

The pipeline there is first initialized like this:

diffusers/examples/dreambooth/train_dreambooth_sd3.py

Lines 1724 to 1734 in 1c50a5f
pipeline = StableDiffusion3Pipeline.from_pretrained(
args.pretrained_model_name_or_path,
vae=vae,
text_encoder=accelerator.unwrap_model(text_encoder_one),
text_encoder_2=accelerator.unwrap_model(text_encoder_two),
text_encoder_3=accelerator.unwrap_model(text_encoder_three),
transformer=accelerator.unwrap_model(transformer),
revision=args.revision,
variant=args.variant,
torch_dtype=weight_dtype,
)

This is to ensure that the models being requested for fine-tuning are present in pipeline. We're using args.pretrained_model_name_or_path, with from_pretrained() to load the other components of the pipeline such as tokenizers and the scheduler.

Thanks, I understand it, from_pretrained() with args.pretrained_model_name_or_path is to load the other components and the finetuned model can be directly loaded with corresponding argument.
for example,

pipeline = StableDiffusion3Pipeline.from_pretrained(
args.pretrained_model_name_or_path,
vae=vae,
text_encoder=accelerator.unwrap_model(text_encoder_one),
text_encoder_2=accelerator.unwrap_model(text_encoder_two),
text_encoder_3=accelerator.unwrap_model(text_encoder_three),
transformer=accelerator.unwrap_model(transformer),
revision=args.revision,
variant=args.variant,
torch_dtype=weight_dtype,
)
in the above code, vae maybe the non fine-tuned components which is "other components" stated above, text_encoder and transformer maybe the fine-tuned components and can be loaded with trained model argument that maybe fine-tuned" accelerator.unwrap_model(text_encoder_one)" and "accelerator.unwrap_model(transformer)" here?
I do not know if is it right?
Another problem, if I load the pretrained components with the given pipline that maybe StableDiffusion3Pipeline here, but one component is modified (such as the input channel is changed) and I saved components in a new folder. In this case, should I create a new pipline as the modified component is not appropriate with the previous pipline?

[sayakpaul]

accelerator.unwrap_model is a utility that we use to ensure the corresponding model isn't wrapped with any distributed utilities. So, it's safe.

In this case, should I create a new pipline as the modified component is not appropriate with the previous pipline?

You can just save the component and use the pipeline like so:

finetuned_component = AutoModel.from_pretrained("your-finetuned-directory")
pipe = DiffusionPipeline(pretrained_ckpt_id, transformer=finetuned_component)

[micklexqg]

accelerator.unwrap_model is a utility that we use to ensure the corresponding model isn't wrapped with any distributed utilities. So, it's safe.

In this case, should I create a new pipline as the modified component is not appropriate with the previous pipline?

You can just save the component and use the pipeline like so:

finetuned_component = AutoModel.from_pretrained("your-finetuned-directory")
pipe = DiffusionPipeline(pretrained_ckpt_id, transformer=finetuned_component)

Thanks, as some core processings need to be modified, I copied the previous pipeline which maybe DiffusionPipline here and renamed it, but i can not be imported. I also try to import it manually with the relative path, but still failed, so how to import a newly custom pipeline？

[sayakpaul]

Do you have a minimal example of how you're doing it?