dpdisp submit implicitly deletes remote working directory via undocumented clean=True default in Submission.run_submission

[qchempku2017]

The current implementation of Submission.run_submission()
(link)

introduces a clean parameter that defaults to True.

When enabled, this triggers machine.context.clean() after job completion, which for SSHContext results in removing the entire remote working directory (i.e., everything under the submission-specific remote root) via recursive deletion (rmtree):

(clean)
(rmtree)

When using the CLI entrypoint:

dpdisp submit submission.json

the clean parameter is not exposed or configurable, neither via JSON schema nor via command-line options:
(submit.py)

As a result, dpdisp submit always executes with clean=True, leading to implicit deletion of all remote job artifacts after completion.

Impact

This behavior has several serious implications:

• Loss of traceability and auditability
  Users cannot inspect intermediate or full results on the remote HPC environment after job completion.
• Silent destructive default
  The cleanup is destructive (rm -rf) and occurs without explicit user consent.
• Agent / automation incompatibility
  For automated workflows (e.g., LLM/agent-based pipelines), this implicit side-effect is particularly problematic, as it is:
  not inferable from the CLI interface,
  not declared in configuration,
  and not documented.

Documentation Gap

The following are currently not documented in Submission.run_submission():

• The existence of the clean parameter
• Its default value (True)
• Its destructive effect on remote directories
• The fact that CLI submission always enables it

This significantly increases the risk of unintended data loss.

[njzjz-bot]

This is mostly the expected workflow in dpdispatcher: files that should be preserved after a submission finishes need to be listed explicitly in backward_files or backward_common_files. Files that are not listed there are not guaranteed to remain available on the remote side after the job is completed and synchronized back.

That said, I agree the issue points out a real UX/documentation problem. dpdisp submit submission.json currently calls run_submission(..., clean=True), while the CLI/config path does not expose an obvious way to disable cleanup. Since SSHContext.clean() removes the submission remote root, this behavior should be documented clearly. It may also be reasonable to add a --no-clean CLI option, or a clean field in the submission config, so users can opt out when they intentionally want to inspect the full remote working directory.

So I would not treat this as a bug in the backward-file mechanism itself, but we should improve the docs and probably the CLI ergonomics around cleanup.

— OpenClaw 2026.4.22 (model: gpt-5.5)