Skip to content

Commit defa23e

Browse files
0x46616c6bOpenCode
andcommitted
📝 Add documentation and configuration for policy server
- Add POLICY_LISTEN_ADDR to README configuration section - Document rate limiting feature with Postfix integration example - Add policy/rate limiting metrics documentation - Expose policy server port in docker-compose.yml - Update copilot-instructions.md with new architecture overview Co-Authored-By: OpenCode <noreply@opencode.ai>
1 parent bb35443 commit defa23e

5 files changed

Lines changed: 161 additions & 103 deletions

File tree

.github/copilot-instructions.md

Lines changed: 64 additions & 102 deletions
Original file line numberDiff line numberDiff line change
@@ -2,141 +2,103 @@
22

33
## Project Overview
44

5-
This is a Postfix socketmap adapter for the [Userli](https://github.com/systemli/userli) email user management system. It provides TCP-based lookup services for Postfix to query virtual aliases, domains, mailboxes, and sender login maps from Userli's REST API.
5+
Postfix adapter for [Userli](https://github.com/systemli/userli) email management. Provides two TCP servers:
66

7-
## Architecture
8-
9-
### Core Components
10-
11-
- **Socketmap Server** (`server.go`): TCP server implementing Postfix's socketmap protocol on port 10001
12-
- **Userli Client** (`userli.go`): REST API client for querying Userli backend
13-
- **Metrics Server** (`prometheus.go`): Prometheus metrics endpoint on port 10002
14-
- **Adapter Logic** (`adapter.go`): Request routing and response formatting for four map types (alias, domain, mailbox, senders)
15-
16-
### Request Flow
17-
18-
1. Postfix sends socketmap query via TCP (format: `<netstring>MAP_NAME <SP> KEY`)
19-
2. Socketmap server parses request and routes to appropriate handler in adapter
20-
3. Adapter queries Userli REST API (`/api/postfix/{map_type}?query={key}`)
21-
4. Response converted back to socketmap netstring format
22-
5. Metrics updated for observability
23-
24-
## Development Workflow
7+
- **Lookup Server** (`:10001`): Lookups for aliases, domains, mailboxes, senders
8+
- **Policy Server** (`:10003`): Rate limiting via Postfix SMTP Access Policy Delegation
259

26-
### Local Setup
10+
## Architecture
2711

28-
```bash
29-
# Copy environment template
30-
cp .env.dist .env
12+
```
13+
┌─────────┐ ┌──────────────────┐ ┌────────────┐
14+
│ Postfix │────▶│ tcpserver.go │────▶│ Userli API │
15+
└─────────┘ │ (shared infra) │ └────────────┘
16+
├──────────────────┤
17+
│ lookup.go │ ← ConnectionHandler interface
18+
│ policy.go │ ← ConnectionHandler interface
19+
└──────────────────┘
20+
```
3121

32-
# Edit .env and set USERLI_TOKEN (required)
33-
# Token can be created in Userli: Settings -> Api Tokens
22+
### Key Pattern: ConnectionHandler Interface
3423

35-
# Start full stack (adapter + postfix + userli + mariadb + mailcatcher)
36-
docker-compose up
24+
Both servers implement `ConnectionHandler` from `tcpserver.go`:
3725

38-
# Adapter runs on :10001 (socketmap) and :10002 (metrics)
26+
```go
27+
type ConnectionHandler interface {
28+
HandleConnection(ctx context.Context, conn net.Conn)
29+
}
3930
```
4031

41-
### Testing Postfix Integration
32+
`StartTCPServer()` provides shared infrastructure: connection pooling (semaphore), graceful shutdown, TCP keep-alive, metrics hooks.
4233

43-
```bash
44-
# Test socketmap queries directly
45-
echo -e "10:alias test" | nc localhost 10001
34+
### File Structure
4635

47-
# Test via Postfix container
48-
docker-compose exec postfix postmap -q "user@example.org" socketmap:inet:adapter:10001:alias
49-
docker-compose exec postfix postmap -q "example.org" socketmap:inet:adapter:10001:domain
50-
docker-compose exec postfix postmap -q "user@example.org" socketmap:inet:adapter:10001:mailbox
51-
docker-compose exec postfix postmap -q "user@example.org" socketmap:inet:adapter:10001:senders
52-
53-
# View caught test emails
54-
open http://localhost:1080 # Mailcatcher web UI
55-
```
36+
| File | Purpose |
37+
| --------------- | -------------------------------------------------------------------- |
38+
| `tcpserver.go` | Shared TCP server with connection pooling, graceful shutdown |
39+
| `lookup.go` | Socketmap protocol + `LookupServer` (implements `ConnectionHandler`) |
40+
| `policy.go` | Policy protocol + `PolicyServer` + rate limit logic |
41+
| `ratelimit.go` | Sliding window rate limiter (in-memory, per-sender) |
42+
| `userli.go` | HTTP client for Userli API with Bearer auth |
43+
| `prometheus.go` | Metrics server + all metric definitions |
44+
| `config.go` | Environment variable configuration |
5645

57-
### Building & Testing
46+
## Development
5847

5948
```bash
60-
# Run tests with coverage
61-
go test ./...
49+
cp .env.dist .env # Set USERLI_TOKEN
50+
docker-compose up # Full stack: adapter + postfix + userli + mariadb + mailcatcher
6251

63-
# Build binary
64-
go build -o userli-postfix-adapter
52+
# Test lookup (via socketmap protocol)
53+
docker-compose exec postfix postmap -q "example.org" socketmap:inet:adapter:10001:domain
6554

66-
# Build Docker image
67-
docker build -t systemli/userli-postfix-adapter .
55+
# Test policy (sends raw policy request)
56+
echo -e "request=smtpd_access_policy\nprotocol_state=END-OF-MESSAGE\nsender=test@example.org\n\n" | nc localhost 10003
6857
```
6958

7059
## Code Conventions
7160

72-
### Configuration Pattern
61+
### Context Propagation
7362

74-
- Use environment variables exclusively (no config files)
75-
- **Required**: `USERLI_TOKEN` - application will fatal if missing
76-
- Defaults defined in `config.go:NewConfig()`:
77-
- `USERLI_BASE_URL`: `http://localhost:8000`
78-
- `SOCKETMAP_LISTEN_ADDR`: `:10001`
79-
- `METRICS_LISTEN_ADDR`: `:10002`
80-
- `LOG_LEVEL`: `info`
81-
- `LOG_FORMAT`: `text` (or `json`)
63+
- Never store `context.Context` in structs - pass through function parameters
64+
- Use parent context for timeouts: `ctx, cancel := context.WithTimeout(parentCtx, 5*time.Second)`
8265

8366
### Error Handling
8467

85-
- Use `logrus` for structured logging: `log.WithField("key", value).Error()`
86-
- Socketmap protocol requires specific error responses:
87-
- `"TEMP "` - temporary failure (HTTP errors, network issues)
88-
- `"PERM "` - permanent failure (404 not found)
89-
- `"NOTFOUND "` - valid query but no result
90-
- `"OK <value>"` - successful lookup
91-
- Fatal errors only for startup issues (missing token, port bind failures)
92-
- Network/API errors are logged but return TEMP to Postfix for retry
68+
- **Socketmap responses**: `OK <data>`, `NOTFOUND`, `TEMP <msg>`, `PERM <msg>`
69+
- **Policy responses**: `action=DUNNO\n\n` (allow) or `action=REJECT <msg>\n\n`
70+
- **Fail-open**: API errors return DUNNO/allow, never block mail on failures
9371

94-
### Adapter Response Pattern
72+
### Metrics (prometheus.go)
9573

96-
The adapter in `adapter.go` follows this flow:
74+
- No PII in labels - aggregate counters only, no email addresses
75+
- Metrics defined as package-level vars, registered in `StartMetricsServer()`
9776

98-
```go
99-
// 1. Parse socketmap request (map name and key)
100-
// 2. Query Userli API: GET /api/postfix/{mapName}?query={key}
101-
// 3. Parse JSON response structure: {"exists": bool, "result": string}
102-
// 4. Return formatted response: "OK result" or "NOTFOUND "
103-
```
77+
### Testing
10478

105-
### Socketmap Protocol Implementation
79+
- Mocks generated via mockery (`.mockery.yml`) - regenerate with `mockery`
80+
- Use `context.Background()` in tests for handlers
10681

107-
- Request format: `<length>:<data>,` (netstring format)
108-
- Data format: `<mapName> <key>`
109-
- Responses must end with space and newline per Postfix spec
110-
- See `server.go:handleConnection()` for full protocol details
82+
## Protocols
11183

112-
### Testing with Mocks
84+
### Socketmap (RFC-like netstring)
11385

114-
- Mock interfaces generated with `mockery` (see `mock_UserliService.go`)
115-
- Test files follow `*_test.go` naming convention
116-
- Use table-driven tests for multiple scenarios
117-
118-
## Key Files
86+
```
87+
Request: <len>:<mapname> <key>, e.g., "18:domain example.org,"
88+
Response: <len>:<status> <data>, e.g., "4:OK 1,"
89+
```
11990

120-
- `main.go` - Entry point, initializes config and starts servers
121-
- `server.go` - TCP server and socketmap protocol implementation
122-
- `adapter.go` - Request routing and Userli API interaction
123-
- `userli.go` - HTTP client with Bearer token authentication
124-
- `config.go` - Environment-based configuration
125-
- `prometheus.go` - Metrics instrumentation
126-
- `docker-compose.yml` - Full test environment with Postfix, Userli, MariaDB, and Mailcatcher
91+
### Policy Delegation (Postfix SMTPD)
12792

128-
## External Dependencies
93+
```
94+
Request: name=value\n pairs, empty line terminates
95+
Response: action=ACTION\n\n
96+
```
12997

130-
- **Userli API**: REST endpoints at `/api/postfix/{alias,domain,mailbox,senders}?query={key}`
131-
- Returns JSON: `{"exists": true/false, "result": "value"}`
132-
- Requires Bearer token authentication
133-
- **Postfix Configuration**: Uses `socketmap:inet:adapter:10001:{mapName}` in virtual\_\*\_maps directives
134-
- **Prometheus**: Scrapes metrics from `:10002/metrics`
98+
Only process at `protocol_state=END-OF-MESSAGE` for accurate counting.
13599

136100
## Common Pitfalls
137101

138-
- Forgetting to set `USERLI_TOKEN` in `.env` causes immediate startup failure
139-
- Map names in Postfix config must exactly match: `alias`, `domain`, `mailbox`, `senders`
140-
- Netstring format is strict: must include length prefix and comma suffix
141-
- Empty API responses (exists=false) should return "NOTFOUND ", not an error
142-
- All socketmap responses must end with space + newline for Postfix compatibility
102+
- `USERLI_TOKEN` is required - app exits immediately if missing
103+
- Rate limiter cleanup runs every 5 minutes in background goroutine
104+
- Map names must match exactly: `alias`, `domain`, `mailbox`, `senders`

README.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ The adapter is configured via environment variables:
1313
- `USERLI_BASE_URL`: The base URL of the userli API.
1414
- `POSTFIX_RECIPIENT_DELIMITER`: The recipient delimiter used in Postfix (e.g., `+`). Default: empty.
1515
- `SOCKETMAP_LISTEN_ADDR`: The address to listen on for socketmap requests. Default: `:10001`.
16+
- `POLICY_LISTEN_ADDR`: The address to listen on for policy requests (rate limiting). Default: `:10003`.
1617
- `METRICS_LISTEN_ADDR`: The address to listen on for metrics. Default: `:10002`.
1718

1819
In Postfix, you can configure the adapter using the socketmap protocol like this:
@@ -24,6 +25,28 @@ virtual_mailbox_maps = socketmap:inet:localhost:10001:mailbox
2425
smtpd_sender_login_maps = socketmap:inet:localhost:10001:senders
2526
```
2627

28+
### Rate Limiting (Policy Server)
29+
30+
The adapter also provides a Postfix SMTP Access Policy Delegation server for rate limiting outgoing mail.
31+
It queries the Userli API for per-user quotas and enforces sending limits.
32+
33+
Configure in Postfix `main.cf`:
34+
35+
```text
36+
smtpd_end_of_data_restrictions = check_policy_service inet:localhost:10003
37+
```
38+
39+
The Userli API endpoint `/api/postfix/smtp_quota/{email}` returns:
40+
41+
```json
42+
{
43+
"per_hour": 100,
44+
"per_day": 1000
45+
}
46+
```
47+
48+
Where `0` means unlimited. If the API is unreachable, messages are allowed (fail-open).
49+
2750
## Docker
2851

2952
You can run the adapter using Docker.
@@ -141,6 +164,15 @@ The adapter exposes Prometheus metrics on `/metrics` (port 10002) and provides h
141164

142165
- `userli_postfix_adapter_health_check_status` - Health check status (1=healthy, 0=unhealthy)
143166

167+
**Policy/Rate Limiting Metrics:**
168+
169+
- `userli_postfix_adapter_policy_active_connections` - Active policy connections gauge
170+
- `userli_postfix_adapter_policy_requests_total` - Total policy request counter
171+
- `userli_postfix_adapter_policy_request_duration_seconds` - Policy request duration histogram
172+
- `userli_postfix_adapter_quota_exceeded_total` - Total messages rejected due to quota
173+
- `userli_postfix_adapter_quota_checks_total` - Total quota checks performed
174+
- `userli_postfix_adapter_tracked_senders` - Number of senders tracked by rate limiter
175+
144176
All metrics include relevant labels (handler, status, endpoint, etc.).
145177

146178
### Health Endpoints

docker-compose.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ services:
77
ports:
88
- "10001:10001"
99
- "10002:10002"
10+
- "10003:10003"
1011
env_file:
1112
- .env
1213
networks:
@@ -22,6 +23,7 @@ services:
2223
POSTFIX_virtual_mailbox_domains: "socketmap:inet:adapter:10001:domain"
2324
POSTFIX_virtual_mailbox_maps: "socketmap:inet:adapter:10001:mailbox"
2425
POSTFIX_smtpd_sender_login_maps: "socketmap:inet:adapter:10001:senders"
26+
POSTFIX_smtpd_end_of_data_restrictions: "check_policy_service inet:adapter:10003"
2527
networks:
2628
- userli
2729

userli.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -224,7 +224,7 @@ func (u *Userli) GetQuota(ctx context.Context, email string) (*Quota, error) {
224224
return nil, err
225225
}
226226

227-
resp, err := u.call(ctx, fmt.Sprintf("%s/api/postfix/quota/%s", u.baseURL, sanitizedEmail))
227+
resp, err := u.call(ctx, fmt.Sprintf("%s/api/postfix/smtp_quota/%s", u.baseURL, sanitizedEmail))
228228
if err != nil {
229229
return nil, err
230230
}

userli_test.go

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,68 @@ func (s *UserliTestSuite) TestGetSenders() {
227227
})
228228
}
229229

230+
func (s *UserliTestSuite) TestGetQuota() {
231+
s.Run("success", func() {
232+
gock.New("http://localhost:8000").
233+
Get("/api/postfix/smtp_quota/user@example.com").
234+
MatchHeader("Authorization", "Bearer insecure").
235+
MatchHeader("Accept", "application/json").
236+
MatchHeader("Content-Type", "application/json").
237+
MatchHeader("User-Agent", "userli-postfix-adapter").
238+
Reply(200).
239+
JSON(map[string]int{"per_hour": 50, "per_day": 500})
240+
241+
quota, err := s.userli.GetQuota(context.Background(), "user@example.com")
242+
s.NoError(err)
243+
s.NotNil(quota)
244+
s.Equal(50, quota.PerHour)
245+
s.Equal(500, quota.PerDay)
246+
s.True(gock.IsDone())
247+
})
248+
249+
s.Run("no email", func() {
250+
quota, err := s.userli.GetQuota(context.Background(), "user")
251+
s.Error(err)
252+
s.Nil(quota)
253+
})
254+
255+
s.Run("server error", func() {
256+
gock.New("http://localhost:8000").
257+
Get("/api/postfix/smtp_quota/user@example.com").
258+
MatchHeader("Authorization", "Bearer insecure").
259+
MatchHeader("Accept", "application/json").
260+
MatchHeader("Content-Type", "application/json").
261+
MatchHeader("User-Agent", "userli-postfix-adapter").
262+
Reply(500).
263+
JSON(map[string]string{"error": "internal server error"})
264+
265+
// call() does not check HTTP status codes, so the response body
266+
// is decoded into a Quota struct with zero values (no matching keys).
267+
quota, err := s.userli.GetQuota(context.Background(), "user@example.com")
268+
s.NoError(err)
269+
s.NotNil(quota)
270+
s.Equal(0, quota.PerHour)
271+
s.Equal(0, quota.PerDay)
272+
s.True(gock.IsDone())
273+
})
274+
275+
s.Run("invalid json", func() {
276+
gock.New("http://localhost:8000").
277+
Get("/api/postfix/smtp_quota/user@example.com").
278+
MatchHeader("Authorization", "Bearer insecure").
279+
MatchHeader("Accept", "application/json").
280+
MatchHeader("Content-Type", "application/json").
281+
MatchHeader("User-Agent", "userli-postfix-adapter").
282+
Reply(200).
283+
BodyString("not valid json")
284+
285+
quota, err := s.userli.GetQuota(context.Background(), "user@example.com")
286+
s.Error(err)
287+
s.Nil(quota)
288+
s.True(gock.IsDone())
289+
})
290+
}
291+
230292
func (s *UserliTestSuite) TestWithClient() {
231293
s.Run("sets custom client", func() {
232294
customClient := &http.Client{}

0 commit comments

Comments
 (0)