You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The introduction of AI agents is being considered to address the challenges faced by many workplaces, such as the aging of the population, lack of human resources, and delays in decision-making. In order to improve the functionality of AI agents, we have developed and provided a benchmark suite to evaluate AI agents by extending the evaluation method of web operations to field operations.
@@ -9,148 +13,160 @@ FieldWorkArena is a groundbreaking benchmark suite for evaluating AI agents. By
- 2025-06-30: The **Retail** dataset has been released on Hugging Face. If you would like to obtain it, please apply [here](https://en-portal.research.global.fujitsu.com/fieldworkarena/).
14
-
- 2025-06-30: The **Warehouse** dataset has been released on Hugging Face. If you would like to obtain it, please apply [here](https://en-portal.research.global.fujitsu.com/fieldworkarena/).
15
-
- 2025-02-27: The **Factory** dataset has been released on Hugging Face.
16
-
17
-
18
-
## Getting Started
19
-
The current reporting functionality of FieldWorkArena utilizes
20
-
[Browsergym](https://github.com/ServiceNow/BrowserGym) and [WorkArena](https://github.com/ServiceNow/WorkArena). Therefore, it is necessary to use ServiceNow instance in this implementation. \
21
-
In the future, the implementation may change in line with modification to the action space and task definitions.
22
-
23
-
### Create ServiceNow Instance
24
-
1. Go to https://developer.servicenow.com/ and create an account.
25
-
2. Click on `Request an instance` and select the `Washington` release (initializing the instance will take a few minutes),
26
-
If you can't select release, once you request an instance for default release, do `Release instance `and click `Request an instance` again.
27
-
3. Once the instance is ready, you should see your instance URL and credentials. If not, click _Return to the Developer Portal_, then navigate to _Manage instance password_ and click _Reset instance password_.
28
-
4. You should now see your URL and credentials. Based on this information, set the following environment variables:
29
-
*`SNOW_INSTANCE_URL`: The URL of your ServiceNow developer instance
30
-
*`SNOW_INSTANCE_UNAME`: The username, should be "admin"
31
-
*`SNOW_INSTANCE_PWD`: The password, make sure you place the value in quotes "" and be mindful of [escaping special shell characters](https://onlinelinuxtools.com/escape-shell-characters). Running `echo $SNOW_INSTANCE_PWD` should print the correct password.
32
-
5. Log into your instance via a browser using the admin credentials. Close any popup that appears on the main screen (e.g., agreeing to analytics).
33
-
34
-
**Warning:** Feel free to look around the platform, but please make sure you revert any changes (e.g., changes to list views, pinning some menus, etc.) as these changes will be persistent and affect the benchmarking process.
35
-
36
-
### Install FieldWorkArena and Initialize your instance
Finally, run this command in a terminal to upload the benchmark data to your ServiceNow instance:
49
-
```
50
-
workarena-install
18
+
src/
19
+
└─ fieldworkarena/
20
+
├─ run_scenario.py # run agents and start assessment
21
+
├─ agent/
22
+
├─ client.py # CLI client to start assessment
23
+
├─ metrics/ # Utils for metrics of FWA
24
+
└─ fwa_green_agent.py # A2A GreenAgent server
25
+
└─ core/
26
+
├─ green_executor.py # base A2A green agent executor
27
+
├─ models.py # pydantic models for green agent IO
28
+
├─ purple_client.py # A2A client tool to communicate with PurpleAgent
29
+
└─ client_utils.py # A2A messaging helpers
30
+
31
+
scenarios/
32
+
└─ fwa/ # implementation of the FWA
33
+
├─ purple_agent/ # put your Agent to solve FWA task
34
+
├─ all_task_ids.toml # config of which task should be input in the scenario
35
+
└─ scenario.toml # config for evaluation in the FWA environment
36
+
37
+
benchmark/
38
+
├─ tasks/ # Task detailed file
39
+
└─ all_task_ids.toml # Task Definition file
51
40
```
52
41
53
-
### Download dataset
42
+
## Prerequisites
43
+
44
+
### Request Access to Hugging Face Dataset
45
+
46
+
This project requires access to the FieldWorkArena dataset hosted on Hugging Face. To request access:
47
+
54
48
1. Go to https://en-documents.research.global.fujitsu.com/fieldworkarena/ .
55
49
2. Click link on `Evaluation dataset` and apply from Forms page,
56
50
3. Confirm the download URL in email sent from FieldWorkArena. (It may take a few business days.)
57
-
4. Unzip downloaded file. The files should be organized in the following directory structure:
58
-
```
59
-
FieldWorkArena \
60
-
├── ...\
61
-
├── data\
62
-
│ ├── document \
63
-
│ ├── image\
64
-
│ └── movie\
65
-
└── ...
66
-
```
67
-
## Use Sample Agent
51
+
- If you do not receive a response within one week, please reapply using the Form from step 2.
52
+
4. Wait for approval from the dataset maintainers
53
+
5. Once approved, generate an access token:
54
+
- Go to your Hugging Face Settings → Access Tokens
55
+
- Create a new token with `read` permissions
56
+
- Copy the token and use it in your `scenario.toml` configuration file
68
57
69
-
### OpenAI API setting (for demo agent)
70
-
set environment variable
71
-
*`OPENAI_API_KEY=You OpenAI API key`
58
+
**Note:** You must have an approved access token before running the benchmark tasks. Please note that access permission handling procedures may be subject to change.
72
59
73
-
### Demo
74
-
**To run the demo, Xserver environment is required.**
75
-
76
-
In these demos, the tasks is to search for incidents in the image according to the query and to report any incidents found.
Run the following script, the results will be saved in the `results` directory.
86
-
#### Linux
87
-
All tasks
67
+
2. Set environment variables
88
68
```
89
-
bash run_tasks.sh all
69
+
cp sample.env .env
90
70
```
91
71
92
-
Each tasks
93
-
```
94
-
# for factory
95
-
bash run_tasks.sh factory
72
+
3. Edit your scenario scenarios/fwa/scenario.toml [How to edit](#scenariotoml)
96
73
97
-
# for warehouse
98
-
bash run_tasks.sh warehouse
74
+
4. Edit task Configuration if needed benchmark/all_task_ids.toml [How to edit](#all_task_idstoml)
99
75
100
-
# for retail
101
-
bash run_tasks.sh retail
76
+
## Quick Start (Running Locally)
77
+
```
78
+
uv sync
79
+
uv run fwa-run scenarios/fwa/scenario.toml
102
80
```
81
+
This command will:
82
+
- Start the agent servers, which include GreenAgent and PurpleAgent, using the commands specified in scenario.toml
83
+
- Construct an `assessment_request` message containing the participant's role-endpoint mapping and the assessment config
84
+
- Send the `assessment_request` to the green agent and print streamed responses
85
+
86
+
**Note:** Use `--show-logs` to see agent outputs during the assessment, and `--serve-only` to start agents without running the assessment.
87
+
88
+
To run this example manually, start the agent servers in separate terminals, and then in another terminal run the A2A client on the scenario.toml file to initiate the assessment.
103
89
104
-
#### Windows
105
-
All tasks
90
+
## Running with Docker
106
91
```
107
-
.\run_all_tasks.bat all
92
+
docker build -t fwa_green_agent .
93
+
docker run -p 9009:9009 fwa_green_agent
108
94
```
109
95
110
-
Each tasks
111
-
```
112
-
# for factory
113
-
.\run_tasks.bat factory
96
+
**Note:** This only starts the GreenAgent server and does not execute the assessment using test_agent.
114
97
115
-
# for warehouse
116
-
.\run_tasks.bat warehouse
98
+
## Scenario Configuration
117
99
118
-
# for retail
119
-
.\run_tasks.bat retail
120
-
```
100
+
### all_task_ids.toml
101
+
102
+
The `all_task_ids.toml` file defines which tasks should be executed in your scenario. It contains four categories:
103
+
104
+
-**`factory`**: Factory tasks (predefined, do not modify)
105
+
-**`warehouse`**: Warehouse tasks (predefined, do not modify)
106
+
-**`retail`**: Retail tasks (predefined, do not modify)
107
+
-**`custom`**: Custom task selection (modify this to pick specific tasks)
121
108
122
-
## Test Your Agent
123
-
### Edit Agent
124
-
Agent is defined in 'demo/agent.py'.
125
-
For testing your agent, you should mainly modify 'get_action()' method.
109
+
#### How to Use
126
110
127
-
<spanstyle="color: red; ">**Attention!!**</span>
111
+
1.**Run all predefined tasks**: In `scenario.toml`, set `target = "all"` to execute all tasks from `factory`, `warehouse`, and `retail` categories (excludes `custom`).
128
112
129
-
**After you added your own agents and scenarios, Plase call `pip install .` .**
113
+
2.**Run specific category**: Set `target = "factory"`, `target = "warehouse"`, or `target = "retail"` to run tasks from a single category.
130
114
131
-
### Submit Your Result
132
-
Compress the `results` directory and reply it to the email address with the download URL of the evaluation data .
115
+
3.**Run custom task selection**:
116
+
- Set `target = "custom"` in `scenario.toml`
117
+
- Copy task IDs from `factory`, `warehouse`, or `retail` categories and paste them into the `custom` array
118
+
- For development use only
119
+
120
+
Example:
121
+
```toml
122
+
custom = [
123
+
"fieldworkarena.1.1.0001",
124
+
"fieldworkarena.2.1.0005",
125
+
"fieldworkarena.3.1.0010"
126
+
]
127
+
```
133
128
129
+
**Note**: Do not modify the `factory`, `warehouse`, or `retail` categories. Use `custom` for custom task selections only.
134
130
135
-
## Inquiries and Support
131
+
### scenario.toml
132
+
133
+
The `scenario.toml` file configures the evaluation environment, including agent endpoints and assessment settings.
-`role`: Role identifier for the agent (must be "agent")
161
+
-`endpoint`: URL where the Purple Agent server will be accessible
162
+
-`cmd`: Command to start the Purple Agent server
163
+
- You can define multiple participants by adding more `[[participants]]` sections
144
164
145
-
## Acknowledment
146
-
This implementation was created with reference to the source code for WorkArena, developed by ServiceNow Research.
147
-
- github: https://github.com/ServiceNow/WorkArena
148
-
- arxiv:
149
-
*[WorkArena](https://arxiv.org/pdf/2403.07718)
150
-
*[WorkArena++](https://arxiv.org/pdf/2407.05291)
165
+
**`[config]`**: Assessment configuration
166
+
-`target`: Target category to run (`"factory"`, `"warehouse"`, `"retail"`, `"custom"`, or `"all"`)
167
+
-`token`: your FWA-authenticated Hugging Face access token
151
168
152
-
## Trouble shooting
153
-
When the browser launched and the proxy auth dialog blocks the startup, please install chrome extension "Proxy Helper". After that, fill PAC URL and your account/password.
This project uses `pytest` for testing. For detailed information about running tests, environment variable configuration, and security best practices, please see [tests/README.md](tests/README.md).
0 commit comments