-
Notifications
You must be signed in to change notification settings - Fork 21
Add advisory support #54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
a618b14
26f6d3d
ef5f389
7276a2d
250186f
f1ba721
9ec009c
35e41ee
94cd6f3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -29,20 +29,26 @@ taskflow: | |
| - seclab_taskflows.personalities.web_application_security_expert | ||
| model: code_analysis | ||
| user_prompt: | | ||
| The issue is in repo {{ result.repo }} with id {{ result.issue_id }}. The component is under the directory | ||
| The issue is in repo {{ result.repo }} with id {{ result.issue_id }}. The component is under the directory | ||
| {{ result.location }} with component_id {{ result.component_id }}. The notes of the component is: | ||
|
|
||
| {{ result.component_notes }} | ||
|
|
||
| You should use this to understand the intended purpose of the component and take it into account when | ||
| You should use this to understand the intended purpose of the component and take it into account when | ||
| you audit the issue. | ||
|
|
||
| The type of the issue is {{ result.issue_type }} and here is the notes of the issue: | ||
|
|
||
| {{ result.issue_notes }} | ||
|
|
||
| ## Known Security Advisories for this Repository | ||
|
|
||
| Fetch the security advisories for {{ globals.repo }} from memcache (stored under the key 'security_advisories_{{ globals.repo }}'). If the value in the memcache is null, clearly state so. Otherwise, state how many advisories were found. | ||
| Review these advisories and consider them when identifying security risks. If you identify code that is an actual vulnerability with similar pattern to a known advisory, highlight that connection. | ||
|
|
||
| {% include 'seclab_taskflows.prompts.audit.audit_issue' %} | ||
|
Comment on lines
+46
to
49
|
||
| toolboxes: | ||
| - seclab_taskflows.toolboxes.repo_context | ||
| - seclab_taskflows.toolboxes.local_file_viewer | ||
|
|
||
| - seclab_taskflow_agent.toolboxes.memcache | ||
| - seclab_taskflows.toolboxes.ghsa | ||
|
Kwstubbs marked this conversation as resolved.
Outdated
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -36,8 +36,13 @@ taskflow: | |
| Fetch the entry points and web entry points of the component, then the user actions of this component. | ||
| Based on the entry points, web entry points, components, user actions and README.md and if available, SECURITY.md in the {{ globals.repo }}, | ||
| can you tell me what type of application this repo is and what kind of security boundary it has. | ||
| Based on this, determine whether the component is likely to have security problems. | ||
|
|
||
| Based on this, determine whether the component is likely to have security problems. | ||
|
|
||
| ## Known Security Advisories for this Repository | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe pull this out and turn it into a reusable prompt and include it in this and the other taskflow? Use this version is better because it instructs the LLM to skip the advisory analysis if advisory isn't found.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sure, makes sense. Let me know if it's what you were thinking. |
||
|
|
||
| Fetch the security advisories for {{ globals.repo }} from memcache (stored under the key 'security_advisories_{{ globals.repo }}'). If the value in the memcache is null, clearly state so and skip advisory analysis. Otherwise, state how many advisories were found. | ||
| Review these advisories and consider them when identifying security risks. If you identify code that is similar to a known advisory pattern, highlight that connection. | ||
|
|
||
| Identify the most likely security problems in the component. Your task is not to carry out a full audit, but to | ||
| identify the main risk in the component so that further analysis can be carried out. | ||
| Do not be too specific about an issue, but rather craft your report based on the general functionality and type of | ||
|
|
@@ -50,7 +55,7 @@ taskflow: | |
| - Is this component likely to take untrusted user input? For example, remote web requests or IPC, RPC calls? | ||
| - What is the intended purpose of this component and its functionality? Does it allow high privileged actions? | ||
| Is it intended to provide such functionalities for all users? Or is there complex access control logic involved? | ||
| - The component itself may also have its own `README.md` (or a subdirectory of it may have a `README.md`). Take | ||
| - The component itself may also have its own `README.md` (or a subdirectory of it may have a `README.md`). Take | ||
| a look at those files to help understand the functionality of the component. | ||
|
|
||
| For example, an Admin UI/dashboard may be susceptible to client side Javascript vulnerabilities such as XSS, CSRF. | ||
|
|
@@ -60,7 +65,7 @@ taskflow: | |
| a web frontend may allow users to access their own content and admins to access all content, but users should not | ||
| be able to access another users' content in general. | ||
|
|
||
| We're looking for more concrete and serious security issues that affects system integrity or | ||
| We're looking for more concrete and serious security issues that affects system integrity or | ||
| lead to information leak, so please do not include issues like brute force, Dos, log injection etc. | ||
|
|
||
| Also do not include issues that require the system to be already compromised, such as issues that rely on malicious | ||
|
|
@@ -72,9 +77,9 @@ taskflow: | |
| Your task is to identify risk rather than properly audit and find security issues. Do not look too much into | ||
| the implementation or scrutinize the security measures such as access control and sanitizers at this stage. | ||
| Instead, report more general risks that are associated with the type of component | ||
| that you are looking at. | ||
| that you are looking at. | ||
|
|
||
| It is not your task to audit the security measures, but rather just to identify the risks and suggest some issues | ||
| It is not your task to audit the security measures, but rather just to identify the risks and suggest some issues | ||
| that is worth auditing. | ||
|
|
||
| Reflect on your notes and check that the attack scenario meets the above requirements. Exclude low severity issues or | ||
|
|
@@ -84,4 +89,6 @@ taskflow: | |
| If you think the issues satisfy the criteria, store a component issue entry for each type of issue identified. | ||
| toolboxes: | ||
| - seclab_taskflows.toolboxes.repo_context | ||
| - seclab_taskflows.toolboxes.local_file_viewer | ||
| - seclab_taskflows.toolboxes.local_file_viewer | ||
| - seclab_taskflow_agent.toolboxes.memcache | ||
| - seclab_taskflows.toolboxes.ghsa | ||
|
Kwstubbs marked this conversation as resolved.
Outdated
|
||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,34 @@ | ||||||
| # SPDX-FileCopyrightText: GitHub, Inc. | ||||||
| # SPDX-License-Identifier: MIT | ||||||
|
|
||||||
| seclab-taskflow-agent: | ||||||
| filetype: taskflow | ||||||
| version: "1.0" | ||||||
|
|
||||||
| model_config: seclab_taskflows.configs.model_config | ||||||
|
|
||||||
| globals: | ||||||
| repo: | ||||||
|
|
||||||
| # Example taskflow to fetch and review security advisories for a repository | ||||||
| taskflow: | ||||||
| - task: | ||||||
| must_complete: true | ||||||
| exclude_from_context: false | ||||||
| agents: | ||||||
| - seclab_taskflow_agent.personalities.assistant | ||||||
| model: general_tasks | ||||||
| user_prompt: | | ||||||
| Fetch all GitHub Security Advisories (GHSAs) for the repo {{ globals.repo }}. | ||||||
|
|
||||||
| After fetching, store the list of advisories in memcache under the key 'security_advisories_{{ globals.repo }}'. | ||||||
|
|
||||||
| Provide a summary of: | ||||||
| 1. How many advisories were found | ||||||
| 2. The severity levels of the advisories | ||||||
| 3. Key recommendations for addressing them | ||||||
|
||||||
| toolboxes: | ||||||
| - seclab_taskflows.toolboxes.ghsa | ||||||
| - seclab_taskflow_agent.toolboxes.memcache | ||||||
| - seclab_taskflows.toolboxes.local_file_viewer | ||||||
| - seclab_taskflows.toolboxes.gh_file_viewer | ||||||
|
Comment on lines
+35
to
+36
|
||||||
| - seclab_taskflows.toolboxes.local_file_viewer | |
| - seclab_taskflows.toolboxes.gh_file_viewer |
Uh oh!
There was an error while loading. Please reload this page.