Skip to content

Fix status_message with missing on_events#2788

Merged
r4victor merged 1 commit intomasterfrom
pr_retry_on_events_status_message
Jun 11, 2025
Merged

Fix status_message with missing on_events#2788
r4victor merged 1 commit intomasterfrom
pr_retry_on_events_status_message

Conversation

@r4victor
Copy link
Copy Markdown
Collaborator

_status_message() didn't respect profile parameters and retry defaults. It could also lead to run processing being stuck and the following server errors if on_events was not set:

[10:48:02] DEBUG    dstack._internal.server.background.tasks.proce
                    ss_runs:95 run(248015)james-2-w2ur93:         
                    processing run                                
           ERROR    apscheduler.executors.default:195 Job         
                    "process_runs (trigger: interval[0:00:02],    
                    next run at: 2025-06-11 10:48:05 +05)" raised 
                    an exception                                  
                    Traceback (most recent call last):            
                      File                                        
                    "/Users/r4victor/Projects/dstack/dstack/.venv/
                    lib/python3.12/site-packages/apscheduler/execu
                    tors/base.py", line 181, in run_coroutine_job 
                        retval = await job.func(*job.args,        
                    **job.kwargs)                                 
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    ^^^^^^                                        
                      File                                        
                    "/Users/r4victor/Projects/dstack/dstack/src/ds
                    tack/_internal/server/background/tasks/process
                    _runs.py", line 49, in process_runs           
                        await asyncio.gather(*tasks)              
                      File                                        
                    "/Users/r4victor/Projects/dstack/dstack/src/ds
                    tack/_internal/server/background/tasks/process
                    _runs.py", line 88, in _process_next_run      
                        await _process_run(session=session,       
                    run_model=run_model)                          
                      File                                        
                    "/Users/r4victor/Projects/dstack/dstack/src/ds
                    tack/_internal/server/background/tasks/process
                    _runs.py", line 113, in _process_run          
                        await _process_active_run(session,        
                    run_model)                                    
                      File                                        
                    "/Users/r4victor/Projects/dstack/dstack/src/ds
                    tack/_internal/server/background/tasks/process
                    _runs.py", line 217, in _process_active_run   
                        run = run_model_to_run(run_model)         
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^         
                      File                                        
                    "/Users/r4victor/Projects/dstack/dstack/src/ds
                    tack/_internal/server/services/runs.py", line 
                    698, in run_model_to_run                      
                        run = Run(                                
                              ^^^^                                
                      File                                        
                    "/Users/r4victor/Projects/dstack/dstack/.venv/
                    lib/python3.12/site-packages/pydantic_duality/
                    __init__.py", line 246, in __new__            
                        return cls.__request__(*args, **kwargs)   
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   
                      File "pydantic/main.py", line 347, in       
                    pydantic.main.BaseModel.__init__              
                    pydantic.error_wrappers.ValidationError: 1    
                    validation error for RunRequest               
                    __root__                                      
                      argument of type 'NoneType' is not iterable 
                    (type=type_error)

It was due to run_spec.configuration.retry.on_events that contains original on_events submitted by the user and not resolved on_events with defaults set (jobs[0].job_spec.retry.on_events)

@r4victor r4victor merged commit 6cdae26 into master Jun 11, 2025
25 checks passed
@r4victor r4victor deleted the pr_retry_on_events_status_message branch June 11, 2025 06:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant