Skip to content

Commit cbbae13

Browse files
authored
feat(dev): introduce AGENTS.md (#23858)
* feat(dev): introduce AGENTS.md * redesign * simplify and fix * remove extraneous example from string formatting section * add CLAUDE.md * spell checker fix
1 parent c101e3c commit cbbae13

3 files changed

Lines changed: 365 additions & 0 deletions

File tree

.github/actions/spelling/allow.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -557,3 +557,4 @@ zstandard
557557
ZTE
558558
Zync
559559
sighup
560+
CLAUDE

AGENTS.md

Lines changed: 363 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,363 @@
1+
# Quick Reference for Vector Development
2+
3+
This guide provides quick commands and coding conventions for Vector development. It's designed to help both AI assistants and human
4+
contributors get started quickly.
5+
6+
**For comprehensive information, see [CONTRIBUTING.md](CONTRIBUTING.md) and [docs/DEVELOPING.md](docs/DEVELOPING.md).**
7+
8+
## Project Summary
9+
10+
Vector is a high-performance, end-to-end observability data pipeline written in Rust. It collects, transforms, and routes logs, metrics, and
11+
traces from various sources to any destination. Vector is designed to be reliable, fast, and vendor-neutral, enabling dramatic cost
12+
reduction and improved data quality for observability infrastructure.
13+
14+
## Project Structure
15+
16+
### Core Directories
17+
18+
- `/src/` - Main Rust source code
19+
- `sources/` - Data ingestion components
20+
- `transforms/` - Data processing and routing components
21+
- `sinks/` - Data output destinations
22+
- `config/` - Configuration system and validation
23+
- `topology/` - Component graph management
24+
- `api/` - GraphQL API for management and monitoring
25+
- `cli.rs` - Command-line interface
26+
27+
- `/lib/` - Modular library crates
28+
- `vector-lib/` - Unified library re-exporting core Vector components
29+
- `vector-core/` - Core event system and abstractions
30+
- `vector-config/` - Configuration framework with schema generation
31+
- `vector-buffers/` - Buffering and backpressure management
32+
- `codecs/` - Data encoding/decoding (JSON, Avro, Protobuf)
33+
- `enrichment/` - Data enrichment (GeoIP, custom tables)
34+
- `file-source/` - File watching and reading
35+
- `prometheus-parser/` - Prometheus metrics parsing
36+
37+
- `/config/` - Configuration examples and templates
38+
- `/distribution/` - Packaging and deployment configs
39+
- `docker/` - Docker images (Alpine, Debian, Distroless)
40+
- `kubernetes/` - Kubernetes manifests
41+
- `systemd/` - SystemD service files
42+
- `debian/`, `rpm/` - Linux package configurations
43+
44+
- `/scripts/` - Build, test, and deployment automation
45+
- `/docs/` - Developer documentation
46+
- `/tests/` - Integration and E2E tests
47+
48+
## Two Different Workflows
49+
50+
### Rust Development (Most Common)
51+
52+
If you're working on Vector's Rust codebase (sources, sinks, transforms, core functionality):
53+
54+
**Format your code:**
55+
56+
```bash
57+
make fmt
58+
```
59+
60+
**Check formatting:**
61+
62+
```bash
63+
make check-fmt
64+
```
65+
66+
**Run Clippy (linter):**
67+
68+
```bash
69+
make check-clippy
70+
```
71+
72+
**Auto-fix Clippy issues:**
73+
74+
```bash
75+
make clippy-fix
76+
```
77+
78+
**Run unit tests:**
79+
80+
```bash
81+
make test
82+
# or
83+
cargo nextest run --workspace --no-default-features --features "${FEATURES}"
84+
```
85+
86+
**Run integration tests:**
87+
88+
```bash
89+
# See available integration tests:
90+
# cargo vdev int show
91+
./scripts/run-integration-test.sh <integration-name>
92+
```
93+
94+
See [Integration Tests](#integration-tests) section below for more details.
95+
96+
**Before committing (recommended checks):**
97+
98+
```bash
99+
make fmt # Format code
100+
make check-fmt # Verify formatting
101+
make check-clippy # Run Clippy linter
102+
make check-component-docs # Check component documentation
103+
./scripts/check_changelog_fragments.sh # Verify changelog
104+
```
105+
106+
### Website/Docs Development (Separate Process)
107+
108+
If you're working on vector.dev website or documentation content:
109+
110+
**Prerequisites:**
111+
112+
- Hugo static site generator
113+
- CUE CLI tool
114+
- Node.js and Yarn
115+
- htmltest
116+
117+
**Run the site locally:**
118+
119+
```bash
120+
cd website
121+
make serve
122+
# Navigate to http://localhost:1313
123+
```
124+
125+
**Build website:**
126+
127+
```bash
128+
cd website
129+
make cue-build
130+
```
131+
132+
**Note:** Website changes use Hugo, CUE, Tailwind CSS, and TypeScript. See [website/README.md](website/README.md) for details.
133+
134+
## Rust Coding Conventions
135+
136+
### Import Statements (`use`)
137+
138+
All `use` statements must be at the **top of the file/module** or at the top of `mod tests`.
139+
This is for consistency.
140+
141+
**Correct:**
142+
143+
```rust
144+
use std::time::Duration;
145+
use governor::clock;
146+
use crate::config::TransformConfig;
147+
148+
fn my_function() {
149+
// function code
150+
}
151+
```
152+
153+
**Incorrect:**
154+
155+
```rust
156+
fn my_function() {
157+
use std::time::Duration; // WRONG; Do not insert `use` inside functions
158+
// function code
159+
}
160+
```
161+
162+
**Organization:**
163+
164+
- Group imports: `std` → external crates → internal (`crate::`)
165+
- Use `rustfmt` to automatically organize them: `make fmt`
166+
167+
### Logging Style
168+
169+
Always use the [Tracing crate](https://tracing.rs/tracing/)'s key/value style:
170+
171+
**Correct:**
172+
173+
```rust
174+
warn!(message = "Failed to merge value.", %error);
175+
info!(message = "Processing batch.", batch_size, internal_log_rate_secs = 1);
176+
```
177+
178+
**Incorrect:**
179+
180+
```rust
181+
warn!("Failed to merge value: {}.", err); // Don't do this
182+
```
183+
184+
**Rules:**
185+
186+
- Events should be capitalized and end with a period
187+
- Use `error` (not `e` or `err`) for error values
188+
- Prefer Display over Debug: `%error` not `?error`
189+
- Key/value pairs provide structured logging
190+
191+
### String Formatting
192+
193+
Prefer inline variable syntax in format strings (Rust 1.58+).
194+
195+
**Correct:**
196+
197+
```rust
198+
format!("Error: {err}");
199+
println!("Processing {count} items");
200+
```
201+
202+
**Incorrect:**
203+
204+
```rust
205+
format!("Error: {}", err); // Unnecessary positional argument
206+
println!("Processing {} items", count);
207+
```
208+
209+
**Why:** Inline syntax is more readable and reduces mistakes with argument ordering.
210+
211+
### Panics
212+
213+
Code in Vector should **NOT** panic under normal circumstances.
214+
215+
- Panics are only acceptable when assumptions about internal state are violated (indicating a bug)
216+
- All potential panics **MUST** be documented in function documentation
217+
- Prefer `Result<T, E>` and proper error handling
218+
219+
### Feature Flags
220+
221+
New components (sources, sinks, transforms) must be behind feature flags:
222+
223+
```bash
224+
# Build only specific component for faster iteration
225+
cargo test --lib --no-default-features --features sinks-console sinks::console
226+
```
227+
228+
See `features` section in `Cargo.toml` for examples.
229+
230+
## Common Patterns
231+
232+
### Development Tools
233+
234+
Vector uses `cargo vdev` for most development tasks. This is a custom CLI tool that wraps common operations:
235+
236+
```bash
237+
cargo vdev check rust # Clippy
238+
cargo vdev check fmt # Formatting check
239+
cargo vdev check events # Event instrumentation check
240+
cargo vdev check licenses # License compliance
241+
cargo vdev test # Unit tests
242+
cargo vdev int test <name> # Integration tests
243+
cargo vdev fmt # Format code
244+
```
245+
246+
### Pre-Push Hook (Optional but Recommended)
247+
248+
Create `.git/hooks/pre-push` with:
249+
250+
```bash
251+
#!/bin/sh
252+
set -e
253+
254+
echo "Format code"
255+
make fmt
256+
257+
echo "Running pre-push checks..."
258+
make check-licenses
259+
make check-fmt
260+
make check-clippy
261+
make check-component-docs
262+
263+
./scripts/check_changelog_fragments.sh
264+
```
265+
266+
Then: `chmod +x .git/hooks/pre-push`
267+
268+
### Container Development
269+
270+
Vector supports development in Docker/Podman containers:
271+
272+
```bash
273+
ENVIRONMENT=true make <target>
274+
# Example: ENVIRONMENT=true make test
275+
```
276+
277+
## Architecture Notes
278+
279+
### Component Development
280+
281+
- **Sources**: Ingest data from external systems
282+
- **Transforms**: Modify, filter, or enrich event data
283+
- **Sinks**: Send data to external systems
284+
285+
Component docs are auto-generated from code annotations. Run `make check-component-docs` after changes.
286+
287+
### Integration Tests
288+
289+
Integration tests verify Vector works with real external services. Require Docker or Podman.
290+
291+
**Run integration tests:**
292+
293+
```bash
294+
# List available tests
295+
cargo vdev int show
296+
297+
# Run specific test (example: aws)
298+
cargo vdev int start aws # need to initiate dev environment first
299+
cargo vdev int test aws
300+
```
301+
302+
See [docs/DEVELOPING.md](docs/DEVELOPING.md#integration-tests) for adding new integration tests.
303+
304+
### Key Files
305+
306+
- `Makefile` - Common build/test/check targets
307+
- `vdev/` - Custom development CLI tool
308+
- `src/` - Rust source code
309+
- `website/` - Hugo-based documentation site
310+
- `tests/` - Integration and behavior tests
311+
312+
## Common Issues
313+
314+
### Formatting Fails
315+
316+
Run `make fmt` before committing. Formatting must be exact.
317+
318+
### Clippy Errors
319+
320+
Run `make clippy-fix` to auto-fix many issues. Manual fixes may be required.
321+
322+
### Component Docs Out of Sync
323+
324+
Component documentation is generated from code. Run:
325+
326+
```bash
327+
make check-component-docs
328+
```
329+
330+
### License Check Fails
331+
332+
After adding/updating dependencies:
333+
334+
```bash
335+
cargo install dd-rust-license-tool --locked
336+
make build-licenses
337+
```
338+
339+
## Reference Documentation
340+
341+
These documents provide context that AI agents and developers need when working on Vector code.
342+
343+
### Essential for Code Changes
344+
345+
- **[STYLE.md](STYLE.md)** - Code style rules (formatting, const strings, code organization)
346+
- **[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)** - System architecture (sources, transforms, sinks, topology)
347+
- **[docs/DEVELOPING.md](docs/DEVELOPING.md)** - Development workflow and testing
348+
349+
### Component Development
350+
351+
- **[docs/specs/component.md](docs/specs/component.md)** - Component specification (naming, configuration, health checks)
352+
- **[docs/specs/instrumentation.md](docs/specs/instrumentation.md)** - Instrumentation requirements (event/metric naming)
353+
- **[src/internal_events](src/internal_events)** - Internal event examples for telemetry
354+
355+
### Adding Documentation
356+
357+
- **[docs/DOCUMENTING.md](docs/DOCUMENTING.md)** - How to document code changes
358+
- **[changelog.d/README.md](changelog.d/README.md)** - Adding changelog entries
359+
360+
### Full Guides
361+
362+
- **[CONTRIBUTING.md](CONTRIBUTING.md)** - Complete contributing guide
363+
- **[website/README.md](website/README.md)** - Website development only (separate from Rust code)

CLAUDE.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
@AGENTS.md

0 commit comments

Comments
 (0)