Skip to content

PluginDaemonInternalServerError: no available node, plugin runtime not found #670

Closed as not planned
#677
Closed as not planned
PluginDaemonInternalServerError: no available node, plugin runtime not found#670
#677
Labels
bugSomething isn't workingplugin-daemon
@NiuBlibing

Description

@NiuBlibing
NiuBlibing
opened on Apr 2, 2026

Self Checks

To make sure we get to you in time, please check the following :)

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • "Please do not modify this template :) and fill in all the required fields."

Versions

  1. dify-plugin-daemon Version 0.5.1-local
  2. dify-api Version 1.11.1

Describe the bug
After S3 crash, plugin daemon step into error "PluginDaemonInternalServerError: no available node, plugin runtime not found" and the health check still return ok.

To Reproduce
Steps to reproduce the behavior:

  1. s3 error like network down or read/write error.
  2. plugin errors

Expected behavior

  1. plugin daemon can auto recover from s3 error.
  2. health check return failed when stuck in error.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
The error log:

2026/04/01 09:57:28 factory.go:28: �[31m[ERROR]PluginDaemonInternalServerError: no available node, plugin runtime not found
goroutine 50928022 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:26 +0x5e
github.com/langgenius/dify-plugin-daemon/internal/types/exception.InternalServerError({0x2b22e20, 0xc001212a80})
	/app/internal/types/exception/factory.go:27 +0x27
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).redirectPluginInvokeByPluginIdentifier(0xc0006b6e60, 0xc000662300, {0xc0003896c0?, 0xcdbed4?}, {0x2b22e20, 0x4058ad0})
	/app/internal/server/middleware.go:132 +0x105
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).RedirectPluginInvoke.func3(0xc000662300)
	/app/internal/server/middleware.go:106 +0x8a
github.com/gin-gonic/gin.(*Context).Next(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).FetchPluginInstallation.func2(0xc000662300)
	/app/internal/server/middleware.go:78 +0x235
github.com/gin-gonic/gin.(*Context).Next(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.CollectActiveDispatchRequests.func1(0x27b49ae?)
	/app/internal/server/controllers/health_check.go:28 +0x25
github.com/gin-gonic/gin.(*Context).Next(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginGroup.CheckingKey.func1(0xc000662300)
	/app/internal/server/middleware.go:26 +0x95
github.com/gin-gonic/gin.(*Context).Next(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).server.CollectActiveRequests.func3(0x0?)
	/app/internal/server/controllers/health_check.go:20 +0x25
github.com/gin-gonic/gin.(*Context).Next(...)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/recovery.go:102 +0x6f
github.com/gin-gonic/gin.(*Context).Next(...)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/logger.go:249 +0xe5
github.com/gin-gonic/gin.(*Context).Next(...)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc00065c000, 0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/gin.go:633 +0x94b
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc00065c000, {0x2b52cb0, 0xc00022d0e0}, 0xc0002c0140)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/gin.go:589 +0x1ad
net/http.serverHandler.ServeHTTP({0x2b45268?}, {0x2b52cb0?, 0xc00022d0e0?}, 0x6?)
	/usr/local/go/src/net/http/server.go:3340 +0x8e
net/http.(*conn).serve(0xc000a48360, {0x2b57378, 0xc00066d080})
	/usr/local/go/src/net/http/server.go:2109 +0x665
created by net/http.(*Server).Serve in goroutine 147
	/usr/local/go/src/net/http/server.go:3493 +0x485
�[0m
[GIN] 2026/04/01 - 09:57:28 | 404 |    2.722258ms |   11.121.160.78 | POST     "/plugin/7679c3f3-bd95-4762-b720-303bd75b8b0e/dispatch/model/schema"
[GIN] 2026/04/01 - 09:57:29 | 200 |      67.499µs |    10.236.89.77 | GET      "/health/check"
[GIN] 2026/04/01 - 09:57:29 | 200 |   10.734684ms |   11.121.160.78 | GET      "/plugin/7679c3f3-bd95-4762-b720-303bd75b8b0e/management/models?page=1&page_size=256"
2026/04/01 09:57:29 factory.go:28: �[31m[ERROR]PluginDaemonInternalServerError: no available node, plugin runtime not found
goroutine 50928024 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:26 +0x5e
github.com/langgenius/dify-plugin-daemon/internal/types/exception.InternalServerError({0x2b22e20, 0xc001212c00})
	/app/internal/types/exception/factory.go:27 +0x27
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).redirectPluginInvokeByPluginIdentifier(0xc0006b6e60, 0xc000662300, {0xc000389b20?, 0xcdbed4?}, {0x2b22e20, 0x4058ad0})
	/app/internal/server/middleware.go:132 +0x105
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).RedirectPluginInvoke.func3(0xc000662300)
	/app/internal/server/middleware.go:106 +0x8a
github.com/gin-gonic/gin.(*Context).Next(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).FetchPluginInstallation.func2(0xc000662300)
	/app/internal/server/middleware.go:78 +0x235
github.com/gin-gonic/gin.(*Context).Next(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.CollectActiveDispatchRequests.func1(0x27b49ae?)
	/app/internal/server/controllers/health_check.go:28 +0x25
github.com/gin-gonic/gin.(*Context).Next(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginGroup.CheckingKey.func1(0xc000662300)
	/app/internal/server/middleware.go:26 +0x95
github.com/gin-gonic/gin.(*Context).Next(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).server.CollectActiveRequests.func3(0x0?)
	/app/internal/server/controllers/health_check.go:20 +0x25
github.com/gin-gonic/gin.(*Context).Next(...)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/recovery.go:102 +0x6f
github.com/gin-gonic/gin.(*Context).Next(...)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/logger.go:249 +0xe5
github.com/gin-gonic/gin.(*Context).Next(...)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc00065c000, 0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/gin.go:633 +0x94b
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc00065c000, {0x2b52cb0, 0xc00022d3b0}, 0xc0002c0280)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/gin.go:589 +0x1ad
net/http.serverHandler.ServeHTTP({0x2b45268?}, {0x2b52cb0?, 0xc00022d3b0?}, 0x6?)
	/usr/local/go/src/net/http/server.go:3340 +0x8e
net/http.(*conn).serve(0xc000a48510, {0x2b57378, 0xc00066d080})
	/usr/local/go/src/net/http/server.go:2109 +0x665
created by net/http.(*Server).Serve in goroutine 147
	/usr/local/go/src/net/http/server.go:3493 +0x485
�[0m
[GIN] 2026/04/01 - 09:57:29 | 404 |    2.681256ms |   11.121.160.78 | POST     "/plugin/7679c3f3-bd95-4762-b720-303bd75b8b0e/dispatch/model/schema"
[GIN] 2026/04/01 - 09:57:29 | 200 |   12.799318ms |   11.121.160.78 | GET      "/plugin/7679c3f3-bd95-4762-b720-303bd75b8b0e/management/models?page=1&page_size=256"
[GIN] 2026/04/01 - 09:57:29 | 200 |   10.669684ms |   11.121.160.78 | GET      "/plugin/7679c3f3-bd95-4762-b720-303bd75b8b0e/management/models?page=1&page_size=256"
2026/04/01 09:57:29 factory.go:28: �[31m[ERROR]PluginDaemonInternalServerError: no available node, plugin runtime not found
goroutine 50927991 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:26 +0x5e
github.com/langgenius/dify-plugin-daemon/internal/types/exception.InternalServerError({0x2b22e20, 0xc000d7c0c0})
	/app/internal/types/exception/factory.go:27 +0x27
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).redirectPluginInvokeByPluginIdentifier(0xc0006b6e60, 0xc000662900, {0xc000b3e0e0?, 0xcdbed4?}, {0x2b22e20, 0x4058ad0})
	/app/internal/server/middleware.go:132 +0x105
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).RedirectPluginInvoke.func3(0xc000662900)
	/app/internal/server/middleware.go:106 +0x8a
github.com/gin-gonic/gin.(*Context).Next(0xc000662900)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).FetchPluginInstallation.func2(0xc000662900)
	/app/internal/server/middleware.go:78 +0x235
github.com/gin-gonic/gin.(*Context).Next(0xc000662900)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.CollectActiveDispatchRequests.func1(0x27b49ae?)
	/app/internal/server/controllers/health_check.go:28 +0x25
github.com/gin-gonic/gin.(*Context).Next(0xc000662900)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginGroup.CheckingKey.func1(0xc000662900)
	/app/internal/server/middleware.go:26 +0x95
github.com/gin-gonic/gin.(*Context).Next(0xc000662900)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).server.CollectActiveRequests.func3(0x26001e?)
	/app/internal/server/controllers/health_check.go:20 +0x25
github.com/gin-gonic/gin.(*Context).Next(...)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000662900)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/recovery.go:102 +0x6f
github.com/gin-gonic/gin.(*Context).Next(...)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000662900)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/logger.go:249 +0xe5
github.com/gin-gonic/gin.(*Context).Next(...)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc00065c000, 0xc000662900)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/gin.go:633 +0x94b
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc00065c000, {0x2b52cb0, 0xc000e6e000}, 0xc0002e0000)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/gin.go:589 +0x1ad
net/http.serverHandler.ServeHTTP({0x2b45268?}, {0x2b52cb0?, 0xc000e6e000?}, 0x6?)
	/usr/local/go/src/net/http/server.go:3340 +0x8e
net/http.(*conn).serve(0xc0008aa090, {0x2b57378, 0xc00066d080})
	/usr/local/go/src/net/http/server.go:2109 +0x665
created by net/http.(*Server).Serve in goroutine 147
	/usr/local/go/src/net/http/server.go:3493 +0x485
�[0m
[GIN] 2026/04/01 - 09:57:29 | 404 |    2.773432ms |   11.121.160.78 | POST     "/plugin/7679c3f3-bd95-4762-b720-303bd75b8b0e/dispatch/model/schema"
2026/04/01 09:57:29 factory.go:28: �[31m[ERROR]PluginDaemonInternalServerError: no available node, plugin runtime not found
goroutine 50928066 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:26 +0x5e
github.com/langgenius/dify-plugin-daemon/internal/types/exception.InternalServerError({0x2b22e20, 0xc001242270})
	/app/internal/types/exception/factory.go:27 +0x27
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).redirectPluginInvokeByPluginIdentifier(0xc0006b6e60, 0xc000662300, {0xc0003896c0?, 0xcdbed4?}, {0x2b22e20, 0x4058ad0})
	/app/internal/server/middleware.go:132 +0x105
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).RedirectPluginInvoke.func3(0xc000662300)
	/app/internal/server/middleware.go:106 +0x8a
github.com/gin-gonic/gin.(*Context).Next(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).FetchPluginInstallation.func2(0xc000662300)
	/app/internal/server/middleware.go:78 +0x235
github.com/gin-gonic/gin.(*Context).Next(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.CollectActiveDispatchRequests.func1(0x27b49ae?)
	/app/internal/server/controllers/health_check.go:28 +0x25
github.com/gin-gonic/gin.(*Context).Next(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginGroup.CheckingKey.func1(0xc000662300)
	/app/internal/server/middleware.go:26 +0x95
github.com/gin-gonic/gin.(*Context).Next(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
github.com/langgenius/dify-plugin-daemon/internal/server.(*App).server.CollectActiveRequests.func3(0x1?)
	/app/internal/server/controllers/health_check.go:20 +0x25
github.com/gin-gonic/gin.(*Context).Next(...)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/recovery.go:102 +0x6f
github.com/gin-gonic/gin.(*Context).Next(...)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/logger.go:249 +0xe5
github.com/gin-gonic/gin.(*Context).Next(...)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc00065c000, 0xc000662300)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/gin.go:633 +0x94b
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc00065c000, {0x2b52cb0, 0xc00022cf00}, 0xc0002c0280)
	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/gin.go:589 +0x1ad
net/http.serverHandler.ServeHTTP({0x2b45268?}, {0x2b52cb0?, 0xc00022cf00?}, 0x6?)
	/usr/local/go/src/net/http/server.go:3340 +0x8e
net/http.(*conn).serve(0xc000a48120, {0x2b57378, 0xc00066d080})
	/usr/local/go/src/net/http/server.go:2109 +0x665
created by net/http.(*Server).Serve in goroutine 147
	/usr/local/go/src/net/http/server.go:3493 +0x485
�[0m
[GIN] 2026/04/01 - 09:57:29 | 404 |     2.73677ms |   11.121.160.78 | POST     "/plugin/7679c3f3-bd95-4762-b720-303bd75b8b0e/dispatch/model/schema"```

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingplugin-daemon

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions