-
Notifications
You must be signed in to change notification settings - Fork 874
feat: critic agent #3320
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
KERVIN-FARMER
wants to merge
27
commits into
langgenius:main
Choose a base branch
from
KERVIN-FARMER:feat/Critic-agent
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
feat: critic agent #3320
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
ede8dbf
feat: critic agent
KERVIN-FARMER 8652f51
Update agent-strategies/self_refine_agent/strategies/self_refine.py
KERVIN-FARMER 6fa7a51
Fix: Type error deleted
KERVIN-FARMER 5818ba6
Fix: Phase 3 completed
KERVIN-FARMER bb5a7bc
Update agent-strategies/self_refine_agent/strategies/self_refine.py
KERVIN-FARMER 3bcf383
Update agent-strategies/self_refine_agent/strategies/self_refine.py
KERVIN-FARMER 244c8c0
Update agent-strategies/self_refine_agent/strategies/self_refine.py
KERVIN-FARMER 1ad1367
Update agent-strategies/self_refine_agent/strategies/self_refine.py
KERVIN-FARMER 8bb3842
fix: bugs solved
KERVIN-FARMER 7c665b0
Fix: Minor fixes in grammar and syntax
KERVIN-FARMER e16161d
Update agent-strategies/self_refine_agent/strategies/self_refine.py
KERVIN-FARMER 1693147
Update agent-strategies/self_refine_agent/strategies/self_refine.py
KERVIN-FARMER 850d3c8
Update agent-strategies/self_refine_agent/strategies/self_refine.py
KERVIN-FARMER 4b6ddad
Update agent-strategies/self_refine_agent/strategies/self_refine.py
KERVIN-FARMER d36de97
Fix: Minor grammar mistakes
KERVIN-FARMER 12bccfb
Merge branch 'main' into feat/Critic-agent
KERVIN-FARMER 0041318
Merge branch 'main' into feat/Critic-agent
KERVIN-FARMER d7cadcd
Apply suggestion from @crazywoola
crazywoola 72ae2b0
Apply suggestions from code review
crazywoola 4394e3a
Apply suggestions from code review
crazywoola 8ffd156
Apply suggestion from @crazywoola
crazywoola 1906437
Merge branch 'main' into feat/Critic-agent
KERVIN-FARMER 61b8401
Update README.md
KERVIN-FARMER 59049b7
Merge branch 'main' into feat/Critic-agent
KERVIN-FARMER 1abb10b
Merge branch 'main' into feat/Critic-agent
KERVIN-FARMER 262a125
Merge branch 'main' into feat/Critic-agent
KERVIN-FARMER 3e09faf
Merge branch 'main' into feat/Critic-agent
crazywoola File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| INSTALL_METHOD=remote | ||
| REMOTE_INSTALL_HOST=debug-plugin.dify.dev | ||
| REMOTE_INSTALL_PORT=5003 | ||
| REMOTE_INSTALL_KEY=your-plugin-key-here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| # Self-Refine Agent Strategy | ||
|
|
||
| A Dify agent strategy plugin that improves output quality through iterative self-critique and refinement. | ||
|
|
||
| ## What It Does | ||
|
|
||
| Self-Refine Agent Strategy implements the [Self-Refine](https://arxiv.org/abs/2303.17651) (NeurIPS 2023) algorithm, enabling agents to automatically evaluate and improve their own outputs through multiple refinement cycles. | ||
|
|
||
| The strategy works by having the LLM act as its own critic: | ||
| 1. **Execute** - Agent generates an initial response | ||
| 2. **Evaluate** - LLM judges the output quality (0-100 score) | ||
| 3. **Critique** - LLM identifies specific issues and improvement opportunities | ||
| 4. **Refine** - Critique is injected into context for the next iteration | ||
| 5. **Repeat** - Process continues until quality threshold is met or max refinements reached | ||
|
|
||
| This approach is particularly effective for tasks requiring high-quality outputs such as writing, analysis, code generation, and complex reasoning. | ||
|
|
||
| ## Key Features | ||
|
|
||
| - **Automatic Quality Control** - LLM evaluates its own outputs without human intervention | ||
| - **Iterative Improvement** - Each refinement cycle builds on previous critiques | ||
| - **Transparent Process** - All intermediate steps (execution, evaluation, critique) are visible | ||
| - **Configurable** - Control maximum iterations and refinement cycles | ||
|
|
||
| ## Usage | ||
|
|
||
| ### Parameters | ||
|
|
||
| | Parameter | Type | Default | Description | | ||
| |-----------|------|---------|-------------| | ||
| | `model` | model-selector | - | LLM model to use | | ||
| | `tools` | array[tools] | - | Available tools for the agent | | ||
| | `instruction` | string | - | System instructions for the agent | | ||
| | `query` | string | - | User query to process | | ||
| | `context` | array[object] | - | Optional context information | | ||
| | `maximum_iterations` | number | 5 | Max iterations per agent execution | | ||
| | `max_refinements` | number | 2 | Max refinement cycles (0-5) | | ||
|
|
||
| ### Configuration Example | ||
|
|
||
| When configuring the agent in Dify: | ||
|
|
||
| 1. Select **Self-Refine Agent** as your strategy | ||
| 2. Choose your preferred LLM model | ||
| 3. Set `max_refinements` to control quality vs. speed tradeoff: | ||
| - `0` - No refinement (standard agent behavior) | ||
| - `1-2` - Balanced (recommended for most cases) | ||
| - `3-5` - High quality (best for critical outputs) | ||
|
|
||
| ### Local Development | ||
|
|
||
| To test the plugin locally: | ||
|
|
||
| 1. Create a `.env` file: | ||
| ```bash | ||
| INSTALL_METHOD=remote | ||
| DIFY_PLUGIN_DAEMON_URL=http://localhost:5003 | ||
| ``` | ||
|
|
||
| 2. Ensure Dify is running locally (via Docker Compose) | ||
|
|
||
| 3. Start the plugin: | ||
| ```bash | ||
| python main.py | ||
| ``` | ||
|
|
||
| ## How It Works | ||
|
|
||
| The refinement loop follows this pattern: | ||
|
|
||
| ``` | ||
| Initial Query → Agent Execution → Output | ||
| ↓ | ||
| Evaluation (score) | ||
| ↓ | ||
| [Score ≥ 80?] → Yes → Return Final Output | ||
| ↓ No | ||
| Generate Critique | ||
| ↓ | ||
| Inject Critique into Context | ||
| ↓ | ||
| Re-execute Agent → Output | ||
| ↓ | ||
| [Max refinements reached?] → Yes → Return Best Output | ||
| ↓ No | ||
| Repeat Cycle | ||
| ``` | ||
|
|
||
| The agent automatically stops refining when: | ||
| - Output quality score reaches 80 or above | ||
| - Maximum refinement cycles are reached | ||
| - No significant improvements are detected | ||
|
|
||
| ## Reference | ||
|
|
||
| Based on the paper: | ||
|
|
||
| Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., ... & Clark, P. (2023). | ||
| **Self-Refine: Iterative Refinement with Self-Feedback.** | ||
| NeurIPS 2023. [arXiv:2303.17651](https://arxiv.org/abs/2303.17651) | ||
|
|
||
| ## License | ||
|
|
||
| MIT License |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| import sys | ||
|
|
||
| from dify_plugin import DifyPluginEnv, Plugin | ||
|
|
||
| plugin = Plugin(DifyPluginEnv(MAX_REQUEST_TIMEOUT=240, MAX_INVOCATION_TIMEOUT=900)) | ||
|
|
||
| if __name__ == "__main__": | ||
| plugin.run() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| version: 0.0.1 | ||
| type: plugin | ||
| author: "KERVIN-FARMER" | ||
| name: "self_refine_agent" | ||
| label: | ||
| en_US: "Self-Refine Agent Strategy" | ||
| zh_Hans: "自我优化 Agent 策略" | ||
| created_at: "2025-06-18T00:00:00.000000000Z" | ||
| icon: icon.svg | ||
| description: | ||
| en_US: "Agent strategy with automatic self-critique and iterative refinement. Based on Self-Refine (NeurIPS 2023)." | ||
| zh_Hans: "支持自动自我批判和迭代优化的 Agent 策略。基于 Self-Refine (NeurIPS 2023)。" | ||
| tags: | ||
| - "agent" | ||
| resource: | ||
| memory: 1048576 | ||
| permission: | ||
| tool: | ||
| enabled: true | ||
| model: | ||
| enabled: true | ||
| llm: true | ||
| plugins: | ||
| agent_strategies: | ||
| - "provider/self_refine.yaml" | ||
| meta: | ||
| version: 0.0.1 | ||
| minimum_dify_version: "1.7.0" | ||
| arch: | ||
| - "amd64" | ||
| - "arm64" | ||
| runner: | ||
| language: "python" | ||
| version: "3.12" | ||
| entrypoint: "main" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| """ | ||
| Prompt templates for Self-Refine Agent Strategy | ||
|
|
||
| Based on Self-Refine: Iterative Refinement with Self-Feedback (NeurIPS 2023) | ||
| https://arxiv.org/abs/2303.17651 | ||
| """ | ||
|
|
||
| # System prompt for initial execution | ||
| EXECUTION_SYSTEM_PROMPT = """You are a helpful AI assistant. Follow the user's instructions carefully and provide detailed, accurate responses. | ||
|
|
||
| {{instruction}} | ||
|
|
||
| Available tools: | ||
| {{tools}} | ||
|
|
||
| Use the tools when necessary to gather information and complete the task. Provide thorough and well-reasoned answers.""" | ||
|
|
||
| # System prompt for execution with refinement context | ||
| REFINEMENT_EXECUTION_PROMPT = """You are a helpful AI assistant. Follow the user's instructions carefully and provide detailed, accurate responses. | ||
|
|
||
| {{instruction}} | ||
|
|
||
| Available tools: | ||
| {{tools}} | ||
|
|
||
| Use the tools when necessary to gather information and complete the task. | ||
|
|
||
| [SELF-REFINEMENT CONTEXT] | ||
| Your previous attempt had the following issues: | ||
| {{critique}} | ||
|
|
||
| Please address these issues in this attempt and provide an improved response.""" | ||
|
|
||
| # Evaluation prompt for assessing output quality | ||
| EVALUATION_PROMPT = """You are an expert evaluator. Assess the quality of the following agent output for the given query. | ||
|
|
||
| Query: {{query}} | ||
|
|
||
| Agent Output: | ||
| {{output}} | ||
|
|
||
| Evaluate the output based on these criteria: | ||
| 1. Accuracy: Is the information correct and factual? | ||
| 2. Completeness: Does it fully address the query? | ||
| 3. Clarity: Is it clear and well-structured? | ||
| 4. Relevance: Does it stay focused on the query? | ||
|
|
||
| Respond in the following JSON format: | ||
| { | ||
| "is_satisfactory": true/false, | ||
| "issues": "Specific description of issues if not satisfactory, or empty string if satisfactory", | ||
| "score": 0-100 | ||
| } | ||
|
|
||
| Only set is_satisfactory to false if there are significant issues that require correction.""" | ||
|
|
||
| # Fallback critique when evaluation fails | ||
| FALLBACK_CRITIQUE = "Unable to evaluate output quality. Please review and improve the response for accuracy, completeness, and clarity." | ||
|
|
||
| SELF_REFINE_TEMPLATES = { | ||
| "execution_system": EXECUTION_SYSTEM_PROMPT, | ||
| "refinement_execution": REFINEMENT_EXECUTION_PROMPT, | ||
| "evaluation": EVALUATION_PROMPT, | ||
| "fallback_critique": FALLBACK_CRITIQUE | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| from dify_plugin.interfaces.agent import AgentProvider | ||
|
|
||
|
|
||
| class SelfRefineAgentProvider(AgentProvider): | ||
| pass |
15 changes: 15 additions & 0 deletions
15
agent-strategies/self_refine_agent/provider/self_refine.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| identity: | ||
| author: KERVIN-FARMER | ||
| name: self_refine | ||
| label: | ||
| en_US: Self-Refine Agent | ||
| zh_Hans: 自我优化 Agent | ||
| description: | ||
| en_US: Self-refining agent with automatic critique and iterative improvement | ||
| zh_Hans: 支持自动批判和迭代改进的自我优化 Agent | ||
| icon: icon.svg | ||
| strategies: | ||
| - strategies/self_refine.yaml | ||
| extra: | ||
| python: | ||
| source: provider/self_refine.py |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| [project] | ||
| name = "self_refine_agent" | ||
| version = "0.0.1" | ||
| description = "Self-Refine Agent Strategy for Dify - Automatic critique and iterative refinement" | ||
| readme = "README.md" | ||
| requires-python = ">=3.12" | ||
|
|
||
| dependencies = [ | ||
| "dify_plugin>=0.9.0", | ||
| ] | ||
|
|
||
| [dependency-groups] | ||
| dev = [] |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.