Could I add support for grouping-based methods for multi-task methods? I.e., methods that cluster tasks based on gradient conflict to update one group at a time. These methods are quite good at reducing negative transfer. Example from ICLR 2025 is "Selective Task Group Updates for Multi-Task Optimization": https://openreview.net/forum?id=EdNSQHaaMR.
To support this, I would make changes to the repo to add a new abstraction class and minor changes to the training loop. This obviously wouldn't impact any existing optimizers or their results.
This is an emerging class of methods that I think would be helpful to implement. I can make a fully functional implementation and test it locally. Let me know what you think.
Could I add support for grouping-based methods for multi-task methods? I.e., methods that cluster tasks based on gradient conflict to update one group at a time. These methods are quite good at reducing negative transfer. Example from ICLR 2025 is "Selective Task Group Updates for Multi-Task Optimization": https://openreview.net/forum?id=EdNSQHaaMR.
To support this, I would make changes to the repo to add a new abstraction class and minor changes to the training loop. This obviously wouldn't impact any existing optimizers or their results.
This is an emerging class of methods that I think would be helpful to implement. I can make a fully functional implementation and test it locally. Let me know what you think.