bug: plugin upgrade fails when .difypkg file missing from disk (PVC recreation)

[euxx]

Self Checks

• I have read the Contributing Guide and Language Policy.
• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• Please do not modify this template :) and fill in all the required fields.

Dify version

1.13.2 (also confirmed on latest main)

Cloud or Self Hosted

Self Hosted (Kubernetes)

Steps to reproduce

1. Install several plugins from the Marketplace (e.g. openai, anthropic, deepseek, etc.)
2. Delete or recreate the PVC backing /app/storage/plugin_packages/ on the plugin-daemon pod (simulating storage loss, node migration, or PVC reprovisioning)
3. Restart the plugin-daemon pod
4. Wait for the periodic plugin auto-upgrade check to run (process_tenant_plugin_autoupgrade_check_task), or trigger a manual upgrade from the UI

✔️ Expected Behavior

When the .difypkg file is missing from disk, the upgrade flow should detect this and re-download the package from the Marketplace, even if the DB/Redis declaration still exists.

❌ Actual Behavior

All plugins that had their package files removed show the error:

failed to move plugin to installed bucket: failed to get package file when trying to install plugin to local
open /app/storage/plugin_packages/langgenius/openai:0.3.4@3345b59...: no such file or directory

This affects every plugin whose .difypkg file was lost. In our case, 30+ plugins were affected simultaneously after a PVC recreation.

Root Cause

There are two affected code paths:

1. UI manual upgrade — PluginService.upgrade_plugin_with_marketplace() in api/services/plugin/plugin_service.py:

try:
    manager.fetch_plugin_manifest(tenant_id, new_plugin_unique_identifier)
    # already downloaded, skip
    marketplace.record_install_plugin_event(new_plugin_unique_identifier)
except Exception:
    pkg = download_plugin_pkg(new_plugin_unique_identifier)
    ...

fetch_plugin_manifest only queries the daemon DB for a PluginDeclaration record — it does not verify the .difypkg file on disk. When the file is missing but the DB record remains, the try block succeeds, the download is skipped, and the upgrade fails.

2. Auto-upgrade celery task — process_tenant_plugin_autoupgrade_check_task() in api/tasks/process_tenant_plugin_autoupgrade_check_task.py:

This task calls manager.upgrade_plugin() directly without any file existence check at all. It never verifies that the package file is present before asking the daemon to perform the upgrade.

Proposed Fix

Add decode_plugin_from_identifier() calls to verify the package file exists on disk before proceeding with upgrades:

Path 1 — plugin_service.py: Add a decode_plugin_from_identifier call after fetch_plugin_manifest:

try:
    manager.fetch_plugin_manifest(tenant_id, new_plugin_unique_identifier)
    manager.decode_plugin_from_identifier(tenant_id, new_plugin_unique_identifier)
    marketplace.record_install_plugin_event(new_plugin_unique_identifier)
except Exception:
    pkg = download_plugin_pkg(new_plugin_unique_identifier)
    ...

Path 2 — auto-upgrade task: Add decode check + download fallback before upgrade_plugin():

try:
    manager.decode_plugin_from_identifier(tenant_id, new_unique_identifier)
except Exception:
    pkg = download_plugin_pkg(new_unique_identifier)
    manager.upload_pkg(tenant_id, pkg, verify_signature=False)

manager.upgrade_plugin(...)

decode_plugin_from_identifier calls the daemon's GetPackage() which reads the actual file, so it raises an exception if the file is missing, triggering the re-download.

[dosubot]

Your analysis looks accurate. I can confirm from the daemon codebase:

• fetch_plugin_manifest uses a three-tier cache (memory → Redis → database) and never touches the .difypkg file on disk [1]. This is why the try block succeeds even when the file is gone.
• GetPackage() (called by decode_plugin_from_identifier) reads the actual file from packageBucket and returns an explicit error when the file is missing: "plugin package not found, please upload it firstly" [2].
• The upgrade path in InstallToLocal fails at the point where it tries to read the package from the storage bucket, producing the exact error you're seeing [3].

There is currently no re-download mechanism in the daemon — it assumes all packages are pre-uploaded and available in storage. Your proposed fix of adding decode_plugin_from_identifier calls as a file-existence check before proceeding is sound, since it's the only call in the flow that actually touches the filesystem.

Other users have hit this same issue after PVC recreation on Kubernetes [4]. The current workaround is to clean up the relevant DB tables (plugin_installations, plugin_declarations, ai_model_installations, plugins) and the storage directories under /app/storage/cwd/plugins/, then reinstall plugins. Not great when 30+ plugins are affected, which further supports the need for the fix you're proposing.

Note that both code paths you identified live in the dify API repo (api/services/plugin/plugin_service.py and api/tasks/), not in this daemon repo. The daemon-side change would only be needed if you wanted GetPackage() itself to have re-download logic. Your proposed approach of handling it on the API side (download fallback in the except block) seems like the right layer for this.

Written by @dosu, approved by crazywoola

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}