Add Anima regional control support by Sen-sou · Pull Request #2524 · Acly/krita-ai-diffusion

Sen-sou · 2026-06-12T18:17:16Z

Added Anima regional control support. Regional control requires an optional (https://huggingface.co/Sen-sou/Anima-LLLite-Regional-Controlnet) model to work best. Since Anima-LLLite models are already supported in krita, therefore added a new controlnet type 'regional'. It doesnt require explicit control image, it automatically combines the current regions to form a control image. Regions can still work without the control model but the region effect is relatively weaker.

Edit: Both 'regional' control net and region attention should be able to work independently. But currently region attention works independently but 'regional' controlnet needs regions to work.

Acly · 2026-06-13T14:55:16Z

I don't think this is a good way to add it - Regions are already the mechanism to define regional prompts, adding another way that is somehow related but needs to be setup manually on top is not intuitive (and also a hassle).

I will look at the attention couple PR in tooling nodes and hopefully get around to testing it soon. Don't have access to my GPU at home atm so might take a bit. Definitely want to try and get that in and see how it performs.

If the control lllite is needed on top, it should be implemented as part of using regions, without any further setup or knowledge required on user side. But it will increase code complexity quite a bit, and without being able to actually assign prompts to colors in the control image it seems a bit random. I'm not really sure what the control model actually learned here...? Would be interested what your validation looked like.

Sen-sou · 2026-06-13T16:50:42Z

The controlnet model originally started as an experiment for training a regional control model, thinking that the prompts would associate with their respective colored region in the control image, but that didnt happen, so it ended up with just characters which follow those colored region without adhering to specific character prompts. That was the regional controlnet.
And you are already aware how weak attention masking is for Anima model, so i thought pairing the two together would cancel out each others drawbacks (attention couple masks text tokens and regional controlnet masks the image tokens), and it does remarkably well. Together they produce strong regional adherence compared to their individual results.
But to use it in krita, would require both to be active at the same time. Attention couple/masking was can be easily merged into whats already there, as for having the lllite patch run along with the regions would make the implementation complicated, so i just added it as a regional controlnet type.

Acly · 2026-06-21T03:48:05Z

Attention masking is always somewhat "weak" when it conflicts with what the model thinks it needs to do to keep the image consistent. But IMO it's better than to force regions too hard and sacrifice composition. Slight prompt drift is much easier to fix in follow-up steps (eg. "Refine Region") than bad overall composition.

I'm not entirely convinced on the controlnet - for relatively simple setups that are already decent without it, it improves adherence, but also seems to fit content to the mask shape on a structural level somewhat (not necessarily desired). And it changes the image style completely.
For challenging setups where Anima already struggles it doesn't help that much, and composition is noticeable worse.

Apart from complexity I'm concerned that the small scale of the CN made it learn to fit the dataset and not just the general task. I'm not even entirely sure what the task was supposed to be, since the CN doesn't interact with prompts - it seems to go in the direction of segmentation CN.

Sen-sou · 2026-06-21T11:38:15Z

Those are fair concerns.

The regional controlnet model's original goal is to learn true prompt to region assignment, but it didnt end up behaving that way. From my testing, it behaved closer to a spatial/layout conditioning model that encourages subjects to occupy the specified regions. Thats why i found it worked best when paired with attention masking (which i tested in comfyui). The control model influences where the content appears and attention masking influences which prompt affected the region. Both approaches have limitation, but together they produced better adherence in my testing.
Regarding changing image style, maybe i didn't test or notice it much, but it might be possible, since the dataset was only 580 images. The project was largely experimental and it showed promising results and a lot of people seems to have really positive reviews on this. I also have plans to increase the dataset to 4000+ images for the next version, so that might fix the image style issue. As for the content structure fitting to mask shape, lower strength can help reduce that (but in my opinion, to able to define your own shape for the content gives much more control to the user).

Here are some example and tests from other users:
https://x.com/ushifuto/status/2067186826981634372
https://x.com/sndiws184/status/2066718730999263500
https://x.com/kiyoshi_shin/status/2066339654731960403
https://x.com/kurayamimousou/status/2066211793899557218
https://x.com/uni_nori_/status/2066164887236284542
https://x.com/uni_nori_/status/2066049172168642784
https://x.com/higana_yogana/status/2065654661261246785

As for integration, i understand your concern about adding it as separate regional workflows. But from my testing, and you know too, that attention masking by itself doesn't perform well at all, and it literally doesnt work with complex regions. For that reason, i view this controlnet model as a complementary requirement for regions to properly work in krita.

Implementation-wise i will leave it to your discretion.

Acly · 2026-06-22T01:05:25Z

Defining shape isn't something the region feature ever did or is supposed to do though. Usually it's combined with a sketch or depth image for structural guidance.

It sounds like the region CN fills a similar purpose, and is quite close the "Segmentation" control net. I would suggest to just expose it as Anima's segmentation control layer for now, since it doesn't have one currently.

If you want to generate a segmentation guidance image from the defined regions, this could be added as a button next to the "Generate control image" button.

If I got it correctly, the CN expect regions as plain color blobs in the image. Current _regional_control_image implementation just uses the layer content, which often will not be plain color, ie. when editing an existing image. Should probably change that to just take coverage/alpha from the layer, and fill it with distinct plain color.

Sen-sou · 2026-06-22T04:53:05Z

That does sound better. I'll take up your suggestions.

Edit: I think all the changes can be kept, and 'regional' just need to be changed to 'segmentation'

…ol mode support for Anima

Acly · 2026-06-28T02:33:37Z

My idea was to add a new button next to the existing generate/pre-processor button, which generates a segmentation image from current regions. It should work the same as the pre-processors, ie. create an explicit new layer with the colored regions that becomes the source for the control layer. Also should be available to all models, not a special Anima thing.

The behavior as it is in this PR isn't intuitive at all, Anima has special behavior that just ignores any configured layer and generates something else behind the scenes which users never see and somehow depends on regions when there is no obvious connection.

Also the function that generations the segmentation image needs to draw plain colors, NOT the region layer content.

Sen-sou · 2026-06-28T07:14:09Z

My idea was to add a new button next to the existing generate/pre-processor button, which generates a segmentation image from current regions. It should work the same as the pre-processors, ie. create an explicit new layer with the colored regions that becomes the source for the control layer. Also should be available to all models, not a special Anima thing.

The behavior as it is in this PR isn't intuitive at all, Anima has special behavior that just ignores any configured layer and generates something else behind the scenes which users never see and somehow depends on regions when there is no obvious connection.

Also the function that generations the segmentation image needs to draw plain colors, NOT the region layer content.

totally forgot about that. Made the changes, please check

Acly

Looks good, just some small stuff and the linting error

Acly · 2026-06-28T11:21:21Z

+        return image
+
    def generate(self):
+        if not self.can_generate:


any particular reason this was added?

safeguard for unintended calls

what unintended calls? if there is no way to hit this code please remove it.

yea sorry about that, i didnt check there were only two calls for this, and they would be hidden. Removed it.

Acly · 2026-06-28T11:22:29Z

+            and layer.compute_bounds().area > 0
+            and Bounds.intersection(bounds, layer.bounds).area > 0


Suggested change

and layer.compute_bounds().area > 0

and Bounds.intersection(bounds, layer.bounds).area > 0

and Bounds.intersection(bounds, layer.compute_bounds()).area > 0

Acly · 2026-06-28T11:24:38Z

+        if self.mode is not ControlMode.segmentation:
+            return
+


Suggested change

if self.mode is not ControlMode.segmentation:

return

assert self.mode is ControlMode.segmentation

changed this, but assertion can be disabled right?

technically yes, but they are not disabled in Krita (and I'm not sure what your point is?)

Acly

LGTM, Lint check still needs to be fixed, but I probably won't be able to merge until weekend anyway

Add Anima regional control support

da8f931

Sen-sou mentioned this pull request Jun 12, 2026

Regions not working with Anima models #2480

Open

Fixed regional controlnet only usage

a702110

Removed regional control mode and integrated it as segmentation contr…

8e98cd4

…ol mode support for Anima

Sen-sou added 2 commits June 28, 2026 12:29

Add explicit region segmentation control layer generation

050d178

Fill transparent segmentation controls with white

9fa5755

Acly reviewed Jun 28, 2026

View reviewed changes

Use computed region bounds for segmentation control

dc7dbc4

Acly approved these changes Jun 30, 2026

View reviewed changes

Sen-sou added 2 commits June 30, 2026 18:20

cleanup

9b5e085

lint

f51f547

		and layer.compute_bounds().area > 0
		and Bounds.intersection(bounds, layer.bounds).area > 0

	and layer.compute_bounds().area > 0
	and Bounds.intersection(bounds, layer.bounds).area > 0
	and Bounds.intersection(bounds, layer.compute_bounds()).area > 0

	if self.mode is not ControlMode.segmentation:
	return
	assert self.mode is ControlMode.segmentation

Uh oh!

Conversation

Sen-sou commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Acly commented Jun 13, 2026

Uh oh!

Sen-sou commented Jun 13, 2026

Uh oh!

Acly commented Jun 21, 2026

Uh oh!

Sen-sou commented Jun 21, 2026

Uh oh!

Acly commented Jun 22, 2026

Uh oh!

Sen-sou commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Acly commented Jun 28, 2026

Uh oh!

Sen-sou commented Jun 28, 2026

Uh oh!

Acly left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Acly left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sen-sou commented Jun 12, 2026 •

edited

Loading

Sen-sou commented Jun 22, 2026 •

edited

Loading