Skip to content

Commit cbc074d

Browse files
author
sw33tLie
committed
docs
1 parent 019a9f6 commit cbc074d

35 files changed

Lines changed: 2069 additions & 1 deletion

‎.github/workflows/docker.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,14 @@ on:
99
- 'README.md'
1010
- '*.md'
1111
- 'website/**'
12+
- 'docs/**'
1213
pull_request:
1314
branches: [ "main", "refactor/v2" ]
1415
paths-ignore:
1516
- 'README.md'
1617
- '*.md'
1718
- 'website/**'
19+
- 'docs/**'
1820

1921
jobs:
2022

‎.github/workflows/docs.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: Deploy Docs
2+
3+
on:
4+
push:
5+
branches: [main]
6+
paths:
7+
- 'docs/**'
8+
9+
permissions:
10+
contents: read
11+
pages: write
12+
id-token: write
13+
14+
concurrency:
15+
group: pages
16+
cancel-in-progress: true
17+
18+
jobs:
19+
deploy:
20+
runs-on: ubuntu-latest
21+
environment:
22+
name: github-pages
23+
url: ${{ steps.deployment.outputs.page_url }}
24+
steps:
25+
- uses: actions/checkout@v4
26+
27+
- name: Install mdBook
28+
run: |
29+
curl -sSL https://github.com/rust-lang/mdBook/releases/latest/download/mdbook-v0.4.44-x86_64-unknown-linux-gnu.tar.gz | tar -xz
30+
chmod +x mdbook
31+
32+
- name: Build docs
33+
run: ./mdbook build docs/
34+
35+
- name: Upload Pages artifact
36+
uses: actions/upload-pages-artifact@v3
37+
with:
38+
path: docs/book
39+
40+
- name: Deploy to GitHub Pages
41+
id: deployment
42+
uses: actions/deploy-pages@v4

‎.gitignore‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
/bbscope
22
/bbscope.*
33
*.sqlite
4-
website/.env
4+
website/.env
5+
docs/book/

‎README.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@
2424

2525
Visit [bbscope.com](https://bbscope.com/) to explore an hourly-updated list of public scopes from all supported platforms, stats, and more!
2626

27+
**[Read the full documentation](https://sw33tlie.github.io/bbscope/)** — installation, CLI reference, database guide, web deployment, library API, and more.
28+
2729
---
2830

2931
## ✨ Features
@@ -364,6 +366,29 @@ bbscope poll immunefi
364366

365367
---
366368

369+
## 📚 Documentation
370+
371+
Full documentation is available at **[sw33tlie.github.io/bbscope](https://sw33tlie.github.io/bbscope/)**.
372+
373+
Covers:
374+
- **Getting Started** — installation, configuration, quick start
375+
- **CLI Reference** — all commands, flags, output formatting
376+
- **Database** — setup, schema, querying, SQL examples
377+
- **Web Interface** — self-hosting, Docker deployment, REST API
378+
- **Platforms** — auth setup for each platform
379+
- **AI Normalization** — configuration, custom endpoints, cost
380+
- **Library Usage** — using `pkg/polling`, `pkg/platforms`, `pkg/storage`, `pkg/targets` in your own Go projects
381+
- **Architecture** — project structure, data flow, design decisions
382+
383+
To build the docs locally:
384+
385+
```bash
386+
cd docs && mdbook serve
387+
# Opens at http://localhost:3000
388+
```
389+
390+
---
391+
367392
## 🌐 bbscope.com (Self-Hosting)
368393

369394
The `website/` folder contains the code powering [bbscope.com](https://bbscope.com/) — a web interface and API for browsing aggregated scopes from all supported platforms.

‎docs/book.toml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
[book]
2+
title = "bbscope Documentation"
3+
authors = ["sw33tLie"]
4+
language = "en"
5+
src = "src"
6+
7+
[output.html]
8+
default-theme = "navy"
9+
preferred-dark-theme = "navy"
10+
git-repository-url = "https://github.com/sw33tLie/bbscope"
11+
edit-url-template = "https://github.com/sw33tLie/bbscope/edit/main/docs/{path}"

‎docs/src/SUMMARY.md‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# Summary
2+
3+
[Introduction](./introduction.md)
4+
5+
# Getting Started
6+
7+
- [Installation](./getting-started/installation.md)
8+
- [Configuration](./getting-started/configuration.md)
9+
- [Quick Start](./getting-started/quickstart.md)
10+
11+
# CLI Usage
12+
13+
- [Polling Scopes](./cli/polling.md)
14+
- [Database Commands](./cli/database.md)
15+
- [Extracting Targets](./cli/targets.md)
16+
- [Output Formatting](./cli/output.md)
17+
18+
# Database
19+
20+
- [Setup](./database/setup.md)
21+
- [Schema & Change Tracking](./database/schema.md)
22+
- [Querying & Searching](./database/querying.md)
23+
24+
# Web Interface
25+
26+
- [Self-Hosting](./web/self-hosting.md)
27+
- [Docker Deployment](./web/docker.md)
28+
- [REST API](./web/api.md)
29+
- [Background Poller](./web/poller.md)
30+
31+
# Platforms
32+
33+
- [HackerOne](./platforms/hackerone.md)
34+
- [Bugcrowd](./platforms/bugcrowd.md)
35+
- [Intigriti](./platforms/intigriti.md)
36+
- [YesWeHack](./platforms/yeswehack.md)
37+
- [Immunefi](./platforms/immunefi.md)
38+
39+
# AI Normalization
40+
41+
- [Overview](./ai/overview.md)
42+
- [Configuration](./ai/configuration.md)
43+
44+
# Using as a Library
45+
46+
- [Overview](./library/overview.md)
47+
- [Polling Package](./library/polling.md)
48+
- [Platforms Package](./library/platforms.md)
49+
- [Storage Package](./library/storage.md)
50+
- [Targets Package](./library/targets.md)
51+
52+
# Advanced
53+
54+
- [Burp Suite Integration](./advanced/burp.md)
55+
- [Architecture](./advanced/architecture.md)

‎docs/src/advanced/architecture.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Architecture
2+
3+
## Project structure
4+
5+
```
6+
bbscope/
7+
├── cmd/ # CLI commands (Cobra)
8+
│ ├── root.go # Root command, global flags
9+
│ ├── poll.go # bbscope poll
10+
│ ├── poll_*.go # Platform-specific poll subcommands
11+
│ ├── db.go # bbscope db
12+
│ ├── db_*.go # DB subcommands (stats, changes, find, etc.)
13+
│ ├── serve.go # bbscope serve
14+
│ └── dev.go # bbscope dev
15+
├── pkg/ # Library packages (importable)
16+
│ ├── platforms/ # Platform interface + implementations
17+
│ │ ├── platform.go # PlatformPoller interface
18+
│ │ ├── hackerone/
19+
│ │ ├── bugcrowd/
20+
│ │ ├── intigriti/
21+
│ │ ├── yeswehack/
22+
│ │ ├── immunefi/
23+
│ │ └── dev/
24+
│ ├── polling/ # Shared polling orchestrator
25+
│ │ └── polling.go
26+
│ ├── storage/ # PostgreSQL persistence
27+
│ │ ├── storage.go # DB operations
28+
│ │ ├── types.go # Entry, Change, UpsertEntry, etc.
29+
│ │ ├── normalize.go # Target normalization
30+
│ │ ├── transform.go # Aggressive transforms
31+
│ │ └── extra.go # Additional queries
32+
│ ├── scope/ # Core types and category normalization
33+
│ ├── targets/ # Target extraction (wildcards, domains, etc.)
34+
│ ├── ai/ # AI normalization
35+
│ ├── whttp/ # HTTP client wrapper (retryablehttp)
36+
│ └── otp/ # TOTP generation
37+
├── website/ # Web server
38+
│ ├── pkg/core/ # Server core (routes, handlers, poller)
39+
│ ├── static/ # CSS, JS, images
40+
│ ├── docker-compose.yml
41+
│ └── Dockerfile
42+
├── internal/
43+
│ └── utils/ # Logging setup
44+
└── docs/ # This documentation (mdBook)
45+
```
46+
47+
## Data flow
48+
49+
### CLI polling (`bbscope poll --db`)
50+
51+
```
52+
Platform API
53+
↓ ListProgramHandles()
54+
↓ FetchProgramScope() × N (concurrent workers)
55+
↓
56+
pkg/polling.PollPlatform()
57+
↓ AI normalize (optional, with DB cache)
58+
↓ BuildEntries()
59+
↓ UpsertProgramEntries() → changes
60+
↓ LogChanges()
61+
↓ OnProgramDone callback → printChanges()
62+
↓
63+
↓ SyncPlatformPrograms() → removed program changes
64+
↓
65+
stdout (change output)
66+
```
67+
68+
### Web server (`bbscope serve`)
69+
70+
```
71+
Background goroutine (every N hours)
72+
↓ buildPollers() from env vars
73+
↓ For each platform (concurrent):
74+
│ polling.PollPlatform()
75+
↓ invalidateProgramsCache()
76+
77+
HTTP requests
78+
↓ /api/v1/* → query DB → JSON response (cached)
79+
↓ /programs, /program/* → query DB → HTML (gomponents)
80+
```
81+
82+
## Key design decisions
83+
84+
- **Platform interface**: All platforms implement `PlatformPoller`. Adding a new platform means implementing 4 methods.
85+
- **Shared orchestrator**: `pkg/polling` contains the polling logic used by both CLI and web server, avoiding duplication.
86+
- **Change detection in DB**: `UpsertProgramEntries` does an atomic compare-and-update, returning only what changed.
87+
- **Safety checks**: Multiple layers prevent accidental data loss (scope wipe detection, platform-level 0-program check).
88+
- **AI caching**: Normalized targets are stored in `targets_ai_enhanced` so only new/changed targets hit the API.
89+
- **Category unification**: Platform-specific category names are mapped to a unified set in `pkg/scope`.

‎docs/src/advanced/burp.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Burp Suite Integration
2+
3+
bbscope includes a Burp Suite script that checks whether HTTP request targets are in scope by querying the bbscope REST API.
4+
5+
## Setup
6+
7+
1. Start the bbscope web server (or use a hosted instance).
8+
2. In Burp Suite, load the script from `website/find-scope-burp-action.java`.
9+
3. Configure the script to point to your bbscope instance URL.
10+
11+
## How it works
12+
13+
The script intercepts HTTP requests in Burp and checks each hostname against the bbscope API's wildcard/domain list. If the target matches an in-scope entry, it's flagged accordingly.
14+
15+
This is useful for passive scope validation during testing — you can see at a glance whether a target you're interacting with is actually in scope for a bug bounty program.
16+
17+
## Requirements
18+
19+
- A running bbscope web server with the REST API accessible
20+
- Programs polled and stored in the database

‎docs/src/ai/configuration.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# AI Normalization — Configuration
2+
3+
## Config file
4+
5+
```yaml
6+
ai:
7+
api_key: "sk-..."
8+
model: "gpt-4.1-mini" # default
9+
# provider: "openai" # default
10+
# endpoint: "" # custom OpenAI-compatible endpoint
11+
# max_batch: 0 # items per API call (0 = auto)
12+
# max_concurrency: 0 # parallel API calls (0 = auto)
13+
# proxy: "" # HTTP proxy for AI API calls
14+
```
15+
16+
## Environment variable fallback
17+
18+
If `ai.api_key` is not set in the config, bbscope falls back to the `OPENAI_API_KEY` environment variable.
19+
20+
For the web server, use:
21+
22+
```
23+
OPENAI_API_KEY=sk-...
24+
OPENAI_MODEL=gpt-4.1-mini
25+
```
26+
27+
## Usage
28+
29+
### CLI
30+
31+
```bash
32+
bbscope poll --db --ai
33+
```
34+
35+
The `--ai` flag only works with `--db`. Without a database, there's nowhere to store the normalized results.
36+
37+
### Web server
38+
39+
If `OPENAI_API_KEY` is set, AI normalization is automatically enabled for the background poller. Check the `/debug` page to confirm.
40+
41+
## Custom endpoints
42+
43+
For self-hosted or alternative OpenAI-compatible APIs (e.g., Ollama, vLLM, Azure OpenAI), set the endpoint:
44+
45+
```yaml
46+
ai:
47+
api_key: "your-key"
48+
endpoint: "http://localhost:11434/v1"
49+
model: "llama3"
50+
```
51+
52+
## Cost considerations
53+
54+
- Only **new or changed** targets are sent to the API. Previously normalized targets are loaded from the database cache.
55+
- The default model (`gpt-4.1-mini`) is cost-effective for this use case.
56+
- Batch size and concurrency can be tuned with `max_batch` and `max_concurrency` if needed.

‎docs/src/ai/overview.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# AI Normalization — Overview
2+
3+
Bug bounty platforms often have messy scope entries. Targets might look like:
4+
5+
- `*.example.com (main website)` instead of `*.example.com`
6+
- `https://app.example.com/api/v2 - REST API` instead of `https://app.example.com/api/v2`
7+
- `All subdomains of example.com` instead of `*.example.com`
8+
9+
AI normalization uses an LLM (OpenAI-compatible API) to clean these up automatically.
10+
11+
## What it does
12+
13+
1. **Cleans up entries** — strips descriptions, comments, and formatting artifacts from target strings.
14+
2. **Handles wildcards** — converts "All subdomains of X" to `*.X`.
15+
3. **Classifies scope intent** — determines if a messy entry is a wildcard, URL, domain, etc.
16+
4. **Normalizes categories** — maps platform-specific category names to unified ones.
17+
5. **Caches results** — stores AI outputs in the `targets_ai_enhanced` database table. Only new/changed targets are sent to the API on subsequent polls, avoiding redundant API calls and costs.
18+
19+
## How it looks
20+
21+
In the web UI, the program detail page has an "AI / Raw" toggle to switch between views.
22+
23+
In the CLI with `--db`, changes show the mapping:
24+
25+
```
26+
🆕 h1 https://hackerone.com/program *.example.com (main site) -> *.example.com
27+
```
28+
29+
## Requirements
30+
31+
- A database (`--db` flag)
32+
- An OpenAI-compatible API key
33+
34+
See [Configuration](./configuration.md) for setup.

0 commit comments

Comments
 (0)