Skip to content

Commit d3bc984

Browse files
aliyuldashevaliyuldashev
authored
Merge pull request #124 from RedTeamSubnet/rt-474-release-webflow-vpn-detection-challenge
Rt 474 release webflow vpn detection challenge
2 parents 9d4a08e + d558a33 commit d3bc984

13 files changed

Lines changed: 254 additions & 36 deletions

File tree

‎.gitmodules‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[submodule "src/challenges/historical_fingerprinter"]
22
path = src/challenges/historical_fingerprinter
33
url = https://github.com/RedTeamSubnet/historical-fingerprinter-challenge.git
4-
[submodule "src/challenges/dev_fingerprinter"]
5-
path = src/challenges/dev_fingerprinter
6-
url = https://github.com/RedTeamSubnet/dev-fingerprinter-challenge.git
4+
[submodule "src/challenges/flowradar"]
5+
path = src/challenges/flowradar
6+
url = https://github.com/RedTeamSubnet/flowradar-challenge.git

‎docs/challenges/.nav.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,9 @@ nav:
22
- README.md
33
- Active Challenges:
44
- historical_fingerprinter: ./historical_fingerprinter
5-
- dev_fingerprinter: ./dev_fingerprinter
5+
- flowradar vpn detection: ./flowradar
66
- Inactive Challenges:
7+
- dev_fingerprinter: ./dev_fingerprinter
78
- ada_detection: ./ada_detection
89
- humanize_behaviour: ./humanize_behaviour
910
- ab_sniffer: ./ab_sniffer

‎docs/challenges/flowradar/README.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
---
2+
title: FlowRadar VPN Detection Challenge
3+
---
4+
5+
# FlowRadar VPN Detection Challenge
6+
7+
## Overview
8+
9+
FlowRadar is a VPN detection challenge. Miners receive network flow features as JSON and must return a single boolean indicating VPN usage.
10+
11+
## What You Build
12+
13+
You submit a single `submission.py` file that implements a VPN detector. The system sends one network flow at a time to your API and expects:
14+
15+
```json
16+
{
17+
"is_vpn": true
18+
}
19+
```
20+
21+
## Data
22+
23+
- **Training data**: ~51k rows extracted from open-source MIT datasets (collected in 2022)
24+
- **Testing data**: internal production dataset collected with modern top VPN providers
25+
26+
Training data is for local experimentation. Testing data is used for production scoring.
27+
28+
## Challenge Flow (High Level)
29+
30+
1. Your submission is packaged into a container with a `/vpn_detector` endpoint.
31+
2. The scoring system iterates through flow rows and posts each flow as JSON.
32+
3. Your response is compared against ground-truth labels.
33+
4. Score is computed using F1.
34+
35+
## Challenge Versions
36+
37+
- [v1 (Active)](./v1.md)
38+
39+
## Resources
40+
41+
- [Building a Submission Commit](../../miner/workflow/3.build-and-publish.md)
42+
- [Dashboard](../../miner/concepts/dashboard.md)
43+
44+
## References
45+
46+
- RedTeam Subnet: <https://www.theredteam.io>
47+
- FastAPI: <https://fastapi.tiangolo.com>

‎docs/challenges/flowradar/testing_manuals.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
---
2+
title: Testing Manual
3+
---
4+
5+
# FlowRadar VPN Detection Testing Manual
6+
7+
This manual provides instructions for testing the FlowRadar VPN detection challenge using Docker and Docker Compose.
8+
9+
## Overview
10+
11+
- Tests are designed to evaluate the effectiveness of your VPN detection implementation.
12+
- Uses Docker for easy submission and testing
13+
14+
## Quick Start Guide
15+
16+
### Prerequisites
17+
18+
- Docker
19+
- Docker Compose
20+
21+
### Step 0: Clone the Repository [skip if you cloned already]
22+
23+
```bash
24+
git clone git@github.com:RedTeamSubnet/flowradar-challenge.git flowradar
25+
cd flowradar
26+
```
27+
28+
### Step 1: Provide Your `submission.py` Script
29+
30+
- Paste your `submission.py` script into **src/flr_challenge/challenge/flowradar/src** folder.
31+
32+
### Step 2: Update Configuration Files
33+
34+
```bash
35+
cp .env.example .env
36+
cp ./templates/compose/compose.override.dev.yml ./compose.override.yml
37+
```
38+
39+
### Step 3: Setting up environmental variables
40+
41+
- Change `FLR_CHALLENGE_API_KEY` with your own preferred API key. You can use any string as API key, but make sure to update it in your detection scripts as well.
42+
43+
### Step 4: Start the Challenge Server
44+
45+
```bash
46+
docker compose up -d challenge-api
47+
```
48+
49+
### Step 5: Test Your Detector Script
50+
51+
- Visit <https://localhost:10001/docs>
52+
- Authenticate using provided authentication API that you put into the `.env` with `FLR_CHALLENGE_API_KEY` variable.
53+
![alt text](https://github.com/RedTeamSubnet/historical-fingerprinter-challenge/blob/08e3deea03d551a5d97b9f93c41b7b31a5c2ee01/docs/images/image.png?raw=true)
54+
- Test your detection files by running the `/score` endpoint
55+
56+
## Important Notes
57+
58+
- The server runs on port 10001 by default
59+
- Make sure port 10001 is available on your system
60+
- All interactions are logged for analysis. Miners can check logs by running `docker compose logs -f`
61+
- All commands must be executed from challenge's root directory.
62+
- If you want score justification, you can check all results with `results` endpoint after running the `score` endpoint.
63+
64+
## Troubleshooting
65+
66+
If you encounter issues:
67+
68+
1. Check if Docker is running
69+
2. Verify port 10001 is not in use
70+
3. Check Docker logs using `docker compose logs`
71+
4. Ensure you have proper permissions to run Docker commands
72+
5. If you got lower or higher score than expected, you can check the justification of your score by running `results` endpoint after running the `score` endpoint. You can also check the logs for more details on how your detection script performed against the test cases.

‎docs/challenges/flowradar/v1.md‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
---
2+
title: FlowRadar v1
3+
---
4+
5+
# FlowRadar v1
6+
7+
## Overview
8+
9+
FlowRadar v1 evaluates a miner's ability to detect VPN usage using network flow features. Each request contains a single flow represented as JSON. Your model must return a boolean `is_vpn` value.
10+
11+
For general challenge context, data description, and links, see the [FlowRadar README](./README.md).
12+
13+
---
14+
15+
## Input/Output Format
16+
17+
### Request
18+
19+
```json
20+
{
21+
"products": {
22+
"fwd_num_pkts": 120,
23+
"bwd_num_pkts": 98,
24+
"fwd_sum_pkt_len": 31400,
25+
"bwd_sum_pkt_len": 38100,
26+
"flow_duration": 2500
27+
...
28+
}
29+
}
30+
```
31+
32+
### Response
33+
34+
```json
35+
{
36+
"is_vpn": true,
37+
"request_id": "<server-generated>"
38+
}
39+
```
40+
41+
Notes:
42+
43+
- The `products` object is a single CSV row serialized as JSON.
44+
- Fields and counts can vary by dataset version.
45+
46+
---
47+
48+
## Scoring
49+
50+
Scoring uses **F1 score** based on true positives, false positives, true negatives, and false negatives.
51+
52+
- Typical scores in early iterations may be 0.7-0.8, but **FlowRadar v1 requires >= 0.9** to be eligible for emissions.
53+
- F1 is computed across the full scoring dataset.
54+
55+
The dashboard also surfaces a confusion matrix (TP, FP, TN, FN) as JSON for debugging and evaluation.
56+
See: [Dashboard](../../miner/concepts/dashboard.md)
57+
58+
---
59+
60+
## Performance Constraints
61+
62+
- **Per-request time limit**: 0.5 seconds
63+
- Scoring stops early if too many requests fail or time out
64+
65+
---
66+
67+
## Submission Requirements
68+
69+
- **Single file only**: `submission.py`
70+
- **Line limit**: 1000 lines max
71+
- **No external libraries**: only Python built-ins
72+
- Must expose a VPN detection function that returns a boolean
73+
74+
---
75+
76+
## Acceptance Criteria
77+
78+
!!! important "Limits and Acceptance Criteria"
79+
- **F1 score >= 0.9** for eligibility
80+
- **Request time <= 0.5 seconds**
81+
- **One file only**: `submission.py`
82+
- **Maximum 1000 lines**
83+
- **No additional libraries** (Python built-ins only)
84+
85+
---
86+
87+
## Example Logic (Simplified)
88+
89+
```python
90+
def detect_vpn(features: dict) -> bool:
91+
# Example heuristic only
92+
fwd = features.get("fwd_sum_pkt_len", 0)
93+
bwd = features.get("bwd_sum_pkt_len", 0)
94+
if fwd > 0 and (bwd / fwd) > 1.2:
95+
return True
96+
return False
97+
```
98+
99+
This illustrates the expected shape and return type. Production submissions should use more robust logic to reach the required F1 threshold.

‎docs/challenges/historical_fingerprinter/testing_manuals.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,13 +46,13 @@ cp ./templates/compose/compose.override.dev.yml ./compose.override.yml
4646
docker compose up -d challenge-api
4747
```
4848

49-
### Step 4: Test Your Bot
49+
### Step 5: Test Your Detection Scripts
5050

5151
- Visit <https://localhost:10001/docs>
5252
- Authenticate using provided authentication API that you put into the `.env` with `HFP_CHALLENGE_API_KEY` variable.
5353
![alt text](https://github.com/RedTeamSubnet/historical-fingerprinter-challenge/blob/08e3deea03d551a5d97b9f93c41b7b31a5c2ee01/docs/images/image.png?raw=true)
5454
- Test your detection files by running the `/score` endpoint
55-
- if you see the warning log(like `Please visit endpoint <URI> to complete human verification for the task.`) then open given uri in log in your browser to complete human verification in your side.
55+
- Check your score and justification by running the `/results` endpoint
5656

5757
## Important Notes
5858

‎requirements.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,5 @@ substrate-interface>=1.7.11,<1.8
66
python-dotenv>=1.0.1,<2.0.0
77
pydantic[email,timezone]>=2.0.3,<3.0.0
88
pydantic-settings>=2.8.1,<3.0.0
9-
dev_fingerprinter @ git+https://github.com/RedTeamSubnet/dev-fingerprinter-challenge.git@v2.0.4
10-
historical_fingerprinter @ git+https://github.com/RedTeamSubnet/historical-fingerprinter-challenge.git@v1.1.5
9+
historical_fingerprinter @ git+https://github.com/RedTeamSubnet/historical-fingerprinter-challenge.git@v1.1.6
10+
flr_challenge @ git+https://github.com/RedTeamSubnet/flowradar-challenge.git@v1.0.0

‎requirements/requirements.docs.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
pylint>=3.0.4,<5.0.0
22
click>=8.0.0,<8.2.2
33
mkdocs-material>=9.5.50,<10.0.0
4-
mkdocs-awesome-nav>=3.0.0,<4.0.0
5-
mkdocstrings[python]>=0.24.3,<2.0.0
4+
mkdocs-awesome-nav>=3.3.0,<4.0.0
5+
mkdocstrings[python]>=1.0.4,<2.0.0
66
mkdocs-render-swagger-plugin>=0.1.2,<1.0.0
77
mike>=2.1.3,<3.0.0

‎src/challenges/dev_fingerprinter‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎src/challenges/flowradar‎

Submodule flowradar added at 745e024

0 commit comments

Comments
 (0)