add interruption handling by Samoed · Pull Request #169 · deeppavlov/AutoIntent

Samoed · 2025-03-23T22:50:56Z

No description provided.

voorhs

круто!

Samoed · 2025-03-24T14:22:16Z

Я не понимаю из-за чего тесты ломаются совсем. Локально по отдельности они работают

voorhs · 2025-03-28T08:24:50Z

у нас еще есть такая тема что во время оптимизации есть два способа сохранять обученные модули: либо в ОЗУ либо на диск. по идее сохранение на ОЗУ не совместимо с resuming, поэтому надо выдавать соответствующие ворнинги

и в целом мне кажется надо еще протестировать как работает Pipeline.dump и Pipeline.load после resuming

voorhs · 2025-03-28T08:26:05Z

наверное еще стоит сделать проверку полного конфига на то что мы резюмим ран с тем же конфигом

voorhs · 2025-04-02T10:43:20Z

У нас трудновосстановимая промежуточная инфа (numpy массивы с предсказанными вероятностями и имя лучшего подобранного эмбедера) сохраняется в виде артефактов (context.optimization_info.artifacts).

Поэтому, чтобы реализовать полноценный interruption handling, я предлагаю реализовать метод Context.optimization_info.load(), который будет загружать эти артефакты из папки проекта. Если этих артефактов нет, то просто бросать ошибку и говорить "продолжить с прежнего момента невозможно, начните заново с настройками dump_modules=True если хотите вновь резьюмить ран`.

Гипотетическая сложность в том, что надо еще реализовать дампинг артефактов в память во время оптимизации :) у нас сохраняются только Trials и то в конце рана и в модифицированном виде, а вот для артефактов нет пайплайна для дампинга/лоадинга

voorhs

если тесты проходят то получается все ок? или ты еще не все тесты написал

мне надо будет подробнее потом посмотреть

Samoed · 2025-04-09T09:56:29Z

Вроде все, но не уверен

* try to fix * fix typing errors * bug fix * Update autointent/nodes/_node_optimizer.py Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> --------- Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

* try to fix * dump context constantly and fix serialization issues * add exclude option to dumper * fix codestyle and typing errors * try to fix file exists error * fix no fixture found error

* full tuning (#165) * Added code for full tuning * work on review * renaming * fix ruff * mypy test * ignote mypy * Feat/bert scorer config refactoring (#168) * refactor configs * add proper configs to BERTScorer * fix typing * fix tokenizer's parameters * fix transformers and accelerate issue * Update optimizer_config.schema.json * bug fix * update callback test * fix tests --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * delete validate_task * report_to * batches * Fix/docs building for bert scorer (#171) * fix * fix codestyle --------- Co-authored-by: Алексеев Илья <44509110+voorhs@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * bert-scorer ending (#172) * batches * tests check * fix * return to torch * fix for tests * Fix/bert scorer (#174) * fix str and float issue and shrinken search space * update `inference node config` overriding logic * fix typing * fix codestyle * fix multilabel issue * attempt to fix `inference node config` bugs * another attempt --------- Co-authored-by: Алексеев Илья <44509110+voorhs@users.noreply.github.com> * Feat/code carbon each node (#175) * feat: update codecarbon * feat: update codecarbon * feat: added codecarbon * Update optimizer_config.schema.json * fix: fixed import mypy * fix: codecarbon package * fix: only float\integer log * fix: codecarbon package * fix: mypy * fix: test * fix: delete emissions * fix: test --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * standartize pyproject & speedup tests (#176) * speedup tests * fix pyproject * Update optimizer_config.schema.json * move optional dependencies * fixes * add xdist * fix ci * download data from hub in doc * add caching * add doc cache --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: voorhs <ilya_alekseev_2016@list.ru> * add proper `omit` definition for tests coverage report (#179) * add proper `omit` definition * Update optimizer_config.schema.json * exclude tmp from coverage report --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * add node validators (#177) * add node validators * add comments * Update optimizer_config.schema.json * rename bert model * lint * fixes * fix test --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: voorhs <ilya_alekseev_2016@list.ru> * dumper saving (#180) * added main code for saving models * Update optimizer_config.schema.json * checker fixes * Revert "checker fixes" This reverts commit 6e32eb9. * Revert "added main code for saving models" This reverts commit 5637fb8. * drat main code for new dumper * ruf fix * comments * added code for test dumper * Check dumper (#182) * Feat/code carbon each node (#175) * feat: update codecarbon * feat: update codecarbon * feat: added codecarbon * Update optimizer_config.schema.json * fix: fixed import mypy * fix: codecarbon package * fix: only float\integer log * fix: codecarbon package * fix: mypy * fix: test * fix: delete emissions * fix: test --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * standartize pyproject & speedup tests (#176) * speedup tests * fix pyproject * Update optimizer_config.schema.json * move optional dependencies * fixes * add xdist * fix ci * download data from hub in doc * add caching * add doc cache --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: voorhs <ilya_alekseev_2016@list.ru> * add proper `omit` definition for tests coverage report (#179) * add proper `omit` definition * Update optimizer_config.schema.json * exclude tmp from coverage report --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * add node validators (#177) * add node validators * add comments * Update optimizer_config.schema.json * rename bert model * lint * fixes * fix test --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: voorhs <ilya_alekseev_2016@list.ru> * update makefile * update bert test * mypy workaround * attempt to fix windows permission error * workaround --------- Co-authored-by: Darinochka <39233990+Darinochka@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Алексеев Илья <44509110+voorhs@users.noreply.github.com> Co-authored-by: Darinochka <39233990+Darinochka@users.noreply.github.com> Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> * Update embedder prompt (#183) * Add trust remote code (#185) * lint * fix trust remote code * Update optimizer_config.schema.json * update fix trust remote code * fix test cllback --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Remove autointent org from docs (#186) * lint * update paths * feat: added crossencoder (#181) * feat: added crossencoder * refactor * feat: added arg similarity * Update optimizer_config.schema.json * feat: added tests * feat: added errors * fix: scoring test * fix: description vectors error * fix: description vectors error * fix: lint * fix: test * add node validators (#177) * add node validators * add comments * Update optimizer_config.schema.json * rename bert model * lint * fixes * fix test --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: voorhs <ilya_alekseev_2016@list.ru> * fix: unit tests * feat: added test for description * feat: delete encoder_type from the class args * feat: update assets * feat: update assets * fix: fixed test * Update optimizer_config.schema.json --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> Co-authored-by: voorhs <ilya_alekseev_2016@list.ru> * Add few shot (#187) * init few shot * Update optimizer_config.schema.json * apply few shot to all * Update optimizer_config.schema.json * fix test * lint --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * update numpy typing (#188) * Lora scorer (#170) * added lora scorer * fix ruff * Update __init__.py * updated after mr #165 * Update pyproject.toml * fixed requested changes * fixed ruff failing * fixed remarks * Update optimizer_config.schema.json * added test * ruff fix * convert labels to float * Update autointent/modules/scoring/_lora/lora.py Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> * Update autointent/modules/scoring/_lora/lora.py Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> * change model_config name, added trust_remote_code * Update lora.py * inherited lora from bert * fix ruff * fix search space * Update lora.py * Update lora.py * added dump check * Update test_lora.py * Update test_lora.py * added docstring * fix ruff * Update test_lora.py * Update test_lora.py --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> * PTuningScorer (#178) * Initial commit of PTuningScorer module * Added peft (>=0.10.0, <0.15.0) in dependencies * Implement fit/predict PTuningScorer * Added PTuningScorer in __init__ file * Update optimizer_config.schema.json * Minor fixs * PGH00 * Refactor clear_cache in fit method * Refactor typing ignore + remove unnecessary * Fix fit method status check * Added test for PTuningScorer * Fix mypy typing * Update and fix peft version dependencies * Fix mypy typing * Added test in multiclass.yaml, multilabel.yaml * Update docs strings * Fix mypy typing * Added trust_remote_code * make proper rst reference * Added test for dump lod * feat: added crossencoder (#181) * feat: added crossencoder * refactor * feat: added arg similarity * Update optimizer_config.schema.json * feat: added tests * feat: added errors * fix: scoring test * fix: description vectors error * fix: description vectors error * fix: lint * fix: test * add node validators (#177) * add node validators * add comments * Update optimizer_config.schema.json * rename bert model * lint * fixes * fix test --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: voorhs <ilya_alekseev_2016@list.ru> * fix: unit tests * feat: added test for description * feat: delete encoder_type from the class args * feat: update assets * feat: update assets * fix: fixed test * Update optimizer_config.schema.json --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> Co-authored-by: voorhs <ilya_alekseev_2016@list.ru> * Added fixed seed to test reproduction * Pull LoraScorer and Bert Refactor * Refactor PTuningScorer * Refactor test for ptuning * Fix typing * Fix multilabel multiclass tests * Fix typing --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: voorhs <ilya_alekseev_2016@list.ru> Co-authored-by: Darinochka <39233990+Darinochka@users.noreply.github.com> Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> * Rerank scorer: опция для выбора источника для расчета вектора вероятностей (#115) * Enable rerank scorer to use crossencoder scores for the probability vector * add cross encoder scores range options * upd test --------- Co-authored-by: voorhs <ilya_alekseev_2016@list.ru> * feat: add DISABLE_EMISSIONS_TRACKING (#191) * feat: add DISABLE_EMISSIONS_TRACKING * try to fix docs error * Update optimizer_config.schema.json * another attempt * Update optimizer_config.schema.json * i give up for now * Update optimizer_config.schema.json --------- Co-authored-by: voorhs <ilya_alekseev_2016@list.ru> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fix issue (#194) * Refactor/embedding caching (#195) * implement new hashing strategy * fix codestyle * Update optimizer_config.schema.json * minor bug fix * fix typing error * refactor similarity calculation * Update optimizer_config.schema.json * upd callback test * solve 429 error --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * forgot something --------- Co-authored-by: Сергей Малышев <68858104+SeBorgey@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Darinochka <39233990+Darinochka@users.noreply.github.com> Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> Co-authored-by: VALERIA RUBANOVA <76725077+riapush@users.noreply.github.com> Co-authored-by: nikiduki <72929274+nikiduki@users.noreply.github.com> Co-authored-by: Dmitryv-2024 <dmitry.v.zhelobanov@yandex.ru>

add interruption handling

2e23004

Samoed requested a review from voorhs March 23, 2025 22:50

fix test

f082b94

voorhs reviewed Mar 24, 2025

View reviewed changes

Comment thread autointent/nodes/_node_optimizer.py Outdated

Comment thread tests/assets/configs/optuna.yaml

Comment thread autointent/nodes/_node_optimizer.py Outdated

Samoed added 8 commits March 24, 2025 13:55

fix test

907e224

update

c848a27

fix test

465571b

lint

e6fef34

remove step

5d7659d

use patch instead of monkeypatch

160d7b4

Merge branch 'dev' into add_interruption_handling

f957390

add n_jobs as param

400cc46

change n_jobs to -1

9b22e17

Samoed mentioned this pull request Mar 26, 2025

Fix/bert scorer #174

Merged

Samoed added 2 commits March 26, 2025 15:24

try fix

37ea18a

remove old study

2237ddd

voorhs reviewed Mar 28, 2025

View reviewed changes

Comment thread autointent/nodes/_node_optimizer.py

Samoed and others added 3 commits March 31, 2025 14:04

add logging warning

764a7bf

Update optimizer_config.schema.json

9a999e8

lint

0bdd785

Samoed force-pushed the add_interruption_handling branch from bc1f31e to 0bdd785 Compare March 31, 2025 12:22

Samoed added 3 commits April 8, 2025 23:24

try dumping

96822fd

lint

e13ab64

np encoder

52f14f0

voorhs reviewed Apr 9, 2025

View reviewed changes

voorhs reviewed Apr 21, 2025

View reviewed changes

Comment thread autointent/_pipeline/_pipeline.py Outdated

Samoed commented Apr 21, 2025

View reviewed changes

Comment thread pyproject.toml Outdated

update warning trigger

b57c002

Samoed force-pushed the add_interruption_handling branch from b607d12 to b57c002 Compare April 21, 2025 11:18

voorhs and others added 4 commits May 3, 2025 17:40

Fix/n trials issue (#196)

b11f845

* try to fix * fix typing errors * bug fix * Update autointent/nodes/_node_optimizer.py Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> --------- Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

Fix/context not dumped error (#197)

d4249aa

* try to fix * dump context constantly and fix serialization issues * add exclude option to dumper * fix codestyle and typing errors * try to fix file exists error * fix no fixture found error

minor commit to refresh branch

f876811

voorhs merged commit 8580f19 into dev May 3, 2025
24 of 26 checks passed

voorhs deleted the add_interruption_handling branch May 3, 2025 16:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add interruption handling#169

add interruption handling#169
voorhs merged 24 commits intodevfrom
add_interruption_handling

Samoed commented Mar 23, 2025

Uh oh!

voorhs left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Samoed commented Mar 24, 2025 •

edited

Loading

Uh oh!

Uh oh!

voorhs commented Mar 28, 2025

Uh oh!

voorhs commented Mar 28, 2025

Uh oh!

voorhs commented Apr 2, 2025

Uh oh!

voorhs left a comment

Uh oh!

Samoed commented Apr 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Samoed commented Mar 23, 2025

Uh oh!

voorhs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Samoed commented Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

voorhs commented Mar 28, 2025

Uh oh!

voorhs commented Mar 28, 2025

Uh oh!

voorhs commented Apr 2, 2025

Uh oh!

voorhs left a comment

Choose a reason for hiding this comment

Uh oh!

Samoed commented Apr 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Samoed commented Mar 24, 2025 •

edited

Loading